Page History

...

/path/to/aws s3 sync --size-only --profile=jcsda-usaf-aws-us-east-2 /work2/noaa/jcsda/role-jcsda/static/ s3://r2d2.static.s3.us-east-2/

Adding Files for R2D2 Archive (such as background files)

Requirements:

This procedure requires the use of R2D2 Admin functions, therefore you will need:

• Recent version of R2D2 on the host the data is located on
• Make sure your r2d2, solo, and venv directories are accessible by the jedipara user. Hint:
  

  Code Block
  language bash theme Midnight
  chmod g+rX jedi-bundle chmod o+rX jedi-bundle chmod -R g+rX r2d2 chmod -R g+rX solo chmod -R g+rX venv

Steps (example using Discover):

1. Log into the HCP where the data to be ingested is located, become the jedipara or role-jcsda user depending on HPC, and go to the location for R2D2 archive.
   

   Code Block
   language bash theme Midnight
   ssh -XY <user_name>@login.nccs.nasa.gov sudo -iu jedipara cd /discover/nobackup/projects/jcsda/s2127

2. Set up your venv, using setup.sh, that was created in the requirements section. You might need to update $JEDI_ENV.
   

   Code Block
   language bash theme Midnight
   vi setup.sh # Verify JEDI_ENV location source setup.sh

3. Log into a screen session so no work will be lost if you get logged out

   Code Block
   language bash theme Midnight
   screen -S r2d2_ingest

4. Verify that your venv is still loaded, if not re-load following the same steps as #2.
5. Check your r2d2 admin utility access. If this does not work make sure you followed the requirements.

   Code Block
   language bash theme Midnight
   python3 >>> from r2d2.util.admin_util import AdminUtil

6. Use the "AdminUtil.move_experiment" function to move the experiment given in the r2d2-ingest ticket to oper and the correct data_store. You can refer to the r2d2 code to ceck for arguments needed and how to use them.

   Code Block
   language bash theme Midnight
   >>> AdminUtil.move_experiment(source_experiment='e8254a', target_experiment='oper', ensemble_data_store_type='geosensemble')

7. To get the data to the other HPCs, on your origin system you will sync to S3 and then sync the data stores.

   Code Block
   language bash theme Midnight
   /path/to/aws s3 sync --size-only --profile=jcsda-usaf-aws-us-east-2 /discover/nobackup/projects/jcsda/s2127/r2d2-archive-nccs/ s3://r2d2-archive-jcsda-usaf-aws-us-east-2/ python3 >>> from r2d2.util.admin_util import AdminUtil >>> AdminUtil.sync_data_stores(source_data_store='r2d2-archive-nccs', target_data_store='r2d2-archive-jcsda-usaf-aws-us-east-2')

8. Log into MSU's Orion and sync from S3. Then you can proceed to log into the other HPCs and sync NWSC/SSEC systems from MSU using the sync scripts in ~/bin.

   Code Block
   language bash theme Midnight
   ssh -Y <user_name>@orion-login.hpc.msstate.edu sudo -su role-jcsda # Load venv to get aws modules or load modules following jedi-docs aws s3 sync --size-only --profile=jcsda-usaf-aws-us-east-2 s3://r2d2-archive-jcsda-usaf-aws-us-east-2/ /work2/noaa/jcsda/role-jcsda/r2d2-archive-msu/

9. Run the rsync from the other HPCs. There is a script located inside jedipara's ~/bin  directory the can be used to perform the rsync. Make sure the username is yours instead of the most recent team members. If you get an ssh error, you can remove the machine from known hosts and try again.
   

   Code Block
   language bash theme Midnight
   ssh -XY <user_name>@<hpc> sudo -iu jedipara cd bin vi rsync-r2d2-archive-from-msu-<user-name>.sh        # Edit to your Orion username bash rsync-r2d2-archive-from-msu-<user-name>.sh

Installing the r2d2 v3 server

...