The Sabbath has ended.
At around 11am, chenning noticed that the QC plots had not updated since 9:00am. We confirmed that data was still coming into the local drives int he DSMs, and then we confirmed that data was coming into ustar. So we thought that there may be an outage or something at the NCAR servers. We waited a little while and then the QC plots for the past few hours all updated at once. The QC plots are updated properly now.
The disk space on t0t was about 97% filled today. There was around 2 Gb of space left, and the Nagios reported that t0t populated a average of around 5Gb a day (between 4 and 6 depending). Gary said in a note that he sent to Chris, that Rick subsequently forwarded to me that we could delete bulky hotfilm data files of the t0t drive so long as we made sure they were backed up on the ustar drive.
t0t had bulky hotfilm data files from 9/12 through today (9/17). We made sure they were backed up on ustar, then deleted all the hotfilm data form 09/12 and 09/13. This freed about 12 Gb of space on the t0t drive, which we think is plenty to last until Gary's glorious return to our working ranks, and can perform his dark magic rituals to solve the problem, or he may just delete more redundant files.
1 Comment
Gary Granger
Thanks for cleaning hotfilm data off t0t, nice work. Last night I created some more space by removing up until the last couple days, by running the following on ustar:
As for ustar, I was able to free up disk space by removing older hotfilm data and all of the staging data, by running the following on mercury as user isfs:
For the record, these data are all old enough that by now I think they have been copied to campaign storage from /scr/isfs. So there are copies of M2HATS ISFS and hotfilm data in these places: