Meeting on December 4th: http://www.fin.ucar.edu/it/mms/v2/fl-1001-chat.html
Read only Jan 20, 2020
End of life Oct 1, 2021
Deadlines won't be extended. Oracle is not manufacturing the tape drives we need.
87PB on HPSS. Only time to migrate 50% of data before EOL.
You can run gladequota to look at your CS quotas (cheyenne).
https://www2.cisl.ucar.edu/user-support/allocations
https://www2.cisl.ucar.edu/user-support/allocations/ncar-allocations
On cheyenne, run 'id' to list your groups (projects). (Can also look at sam.ucar.edu).
If you know what paths your files exist:
hsi ls -lRU /path >& ~/hsi.ls.path.txt |
If you don't know where your files are. Go to support.ucar.edu. Submit ticket and ask for an HPSS inventory.
You can move files from the HPSS to the HPSS for free (i.e. hsi mkdir & hsi mv cost basically nothing).
Recommendation - create directories for migration plan:
/home/user/project/campaign
/home/user/project/delete
/home/user/project/local_storage
Then you can use recursive commands to delete, get, etc.
Go to data-access node.
cd /glade/campaign/ral/
cd path/to/cs/dir
hsi cget -RA /home/user/project/campaign
-R Recursive
-A Tape Ordering (makes a huge difference - gets everything from one tape before moving to the next tape)
(This is also why it is useful to move everything into a single directory, so HSI can figure out the most efficient way to get data from the tapes).
(RUN hsi cget in tmux or screen or something so that it doesn't die on you).
https://www.hamvocke.com/blog/a-quick-and-easy-guide-to-tmux/
https://thoughtbot.com/blog/a-tmux-crash-course
Advantage - globus resumes, and provides tracking.
Disadvantage - need to stage data locally via hsi, then globus to push to CS.
If you are putting data somewhere that is not a globus endpoint:
scp, rsync, bbcp (requires install on both ends - uses multiple streams, so faster)
confirm that file sizes are as expected. (ls)
Remove data you don't need and data that you have transfered to CampaignStore
Verify first! Data removed from HPSS is not recoverable!
Only get files once - getting them multiple times is a waste of valuable tape time.
Record commands into log files so you can figure out what you did.
Use verbose mode when available
Putting data onto HPSS will slow your migration (concurrent transfer limit of 5 files on HPSS).
Dave Hart told me that he thought projects at NCAR could purchase additional data storage from CISL at the rate of $45/TB/year, but that he was not sure. Can you confirm that this data storage is available and let me know if the rate is correct. Can you also tell me anything more about this storage? i.e. what will the interface and medium be for the storage?
Send ticket to support.ucar.edu.