Transferring Data and Other Tips

• To allow others in the group to read a project space on zeus, use: setfacl (from Timothy Brown). 

• The data transfer nodes (DTN) on zeus have now been opened for access from bluefire. To transfer large amounts of data from jet to bluefire, use bbcp:

• bbcp -P 1 -w 16M -s 64 <file> Jonathan.Vigh@dtn-zeus.rdhpcs.noaa.gov:/scratch1/portfolios/BMC/dtc-hwrf/diagnostics_climatology/input_data/ECMWF/
  Other notes from Chris Harrop:

When moving data from Zeus to Bluefire, you need to use the -z bbcp option to reverse the direction of the authentication.  Otherwise the transfer will fail.

Also, due to the nature of bbcp and AIX vs Linux, the optimal bbcp window size and streams may be different depending on the direction you are going.  If you are going from Bluefire to Zeus, it looks like the more streams, the better.  If you are going the other direction, that doesn't appear to be the case, and using only 4 streams may be best. In any case, it may take some experimentation to find the optimal options for each direction.

Note: so from zeus to bluefire, one might use: -s 4 -w 16M   or -s4 -w 2M. So to get data from zeus back to NCAR, from bluefire, one would do something like:

    bbcp -P 1 -s 4 -w 16M -z Jonathan.Vigh@dtn-zeus.rdhpcs.noaa.gov:/scratch1/portfolios/BMC/dtc-hwrf/diagnostics_climatology/processed_data/basin_HWRF/hwrf-baseline-basinscale_final_Weiugo.Wang/bhwrf_2011082400.tar .

and then transfer it to the local machine.

• The mass store on jet uses the wrapper scripts, but on zeus, access to the HSMS is all via hsi and htar. Note that recent stuff (e.g. from May onward) is already being copied to the HSMS on zeus.
• Note that to use hsi on jet or zeus, it is necessary to load the module for hpss:  'module load hpss'.

• Note that when using hsi, the output is normally directed to the screen since it starts another process. Doing a normal redirect of an hsi command to try to store a listing in a file doesn't work. To redirect this to standard output, you have to invoke it with the -P option:
  • hsi -P "ls /HFIP/hwrfv3/Weiguo.Wang/hwrf-baseline-basinscale_final" > listing_available_on_zeus_hsms
  • More details at http://www.mgleicher.us/GEL/hsi/hsi_man_page.html

-P ......... popen command mode. This option is similar to the -O option, but causes all output to be directed to stdout, which is normally redirected to a process that starts HSI with the popen(3) system call. It also results in setting "quiet" (no extraneous messages) mode, disabling verbose response message, and disabling interactive file transfer messages.

Update: CISL has some simple examples of bbcp  using yellowstone's data transfer nodes:

https://www2.cisl.ucar.edu/docs/transfer/bbcp