I've just noticed that /scr/isfs isn't being mounted on barolo. I've submitted a Jira help request. In the meantime, I expect rsyncs (and my plot scripts) to fail.
Update, middle of next day:
Ted is working on this. Apparently, 2 disks in the RAID shared by /scr/isfs and /scr/raf died. The first attempt to rebuild from the other disks also died after going 50% through. Another rebuild is now in process (takes 5hrs) and is at 89% as of now. If it fails, we will have to rebuild all of /scr/isfs from scratch. So...we're crossing our fingers, but in any case should consider replacing this RAID drive.
In the meantime, CHEESEHEAD data are not available mirrored in Boulder.
Just an update: the recovery effort finished, with 11Tb of /scr/isfs recovered and 4.7Tb of corrupt data – about 1/3. These files currently are being rsync'ed to a new RAID disk now mounted as /scr/isfs. We do not yet know which 1/3 is gone.
Powered by a free Atlassian Confluence Community License granted to University Corporation for Atmospheric Research (UCAR). Evaluate Confluence today.