Upon arrival at trailer this morning, pam and flux computers in the trailer were in the state that they were put into on 23 December when an error appears to have caused a failure in numerous parts of the data system.
pam, which is used for s-plus realtime analysis was operating, but flux had become unresponsive. The /usr/local mount on pam did not contain any updated netcdf files from the isfs sites, so realtime analysis was not working. It was advised to restart both pam and flux, starting with pam, and then flux.
This was done, and pam restarted fine, but when remounting, the files did not appear. Then flux was restarted (using the power button due to unresponsiveness) which came back and I was able to start up cockpit and observe that data was flowing properly.
However, when flux was restarted, pam crashed, and became unresponsive. It is apparent that pam mounts flux, and when flux dissapears, pam does not work. The result was a need to restart using the power button. This operation resulted in pam not remounting flux on startup, and therefore does not have access to netcdf data or s-plus.
For future situations where flux is down, it is advised to shut down pam using the computer controls, then reboot flux, and then restart pam, so that it proceeds in mounting, as well as allows flux to reboot.
pam will need to be addressed in the near future, as it is not particularly useful at this time.
Restarting flux has allowed Tom Horst to have access to base computers again, and netcdfs have resumed coming into the computer. Realtime analysis can be done on flux for now, and cockpit is working just fine. It is reported that Tom can no longer access flux. This will be addressed in the morning of the 28th.