Research Repository for Data and Diagnostics

Start here: https://github.com/JCSDA-internal/r2d2/blob/develop/TUTORIAL.md

R2D2 User Guide by Eric: google doc

R2D2 Admin Guide by Eric: google doc

GitHub repositories:

• https://github.com/JCSDA-internal/r2d2 (R2D2 V2 and R2D2 V3 server: feature/cleanup1)
• https://github.com/JCSDA-internal/r2d2-ingest (R2D2 data ingest request system)
• https://github.com/JCSDA-internal/r2d2-data (localhost R2D2 database)
• https://github.com/JCSDA-internal/r2d2-client (R2D2 V3 client: feature/cleanup1)

...

1. Log into the HCP where the data to be ingested is located, become the jedipara or role-jcsda user depending on HPC, and go to the location for R2D2 archive.
   

   Code Block
   language bash theme Midnight
   ssh -XY <user_name>@login.nccs.nasa.gov sudo -iu jedipara cd /discover/nobackup/projects/jcsda/s2127

2. Set up your venv, using setup.sh, that was created in the requirements section. You might need to update $JEDI_ENV.
   

   Code Block
   language bash theme Midnight
   vi setup.sh # Verify JEDI_ENV location source setup.sh

3. Log into a screen session so no work will be lost if you get logged out

   Code Block
   language bash theme Midnight
   screen -S r2d2_ingest

4. Verify that your venv is still loaded, if not re-load following the same steps as #2.
5. Check your r2d2 admin utility access. If this does not work make sure you followed the requirements.

   Code Block
   language bash theme Midnight
   python3 >>> from r2d2.util.admin_util import AdminUtil

   Use the "AdminUtil.move_experiment" function to move the experiment given in the r2d2-ingest ticket to oper and the correct data_store. You can refer to the r2d2 code to ceck for arguments needed and how to use them.
6. Before moving an entire experiment, you can check which files will be moved by using R2D2Data.search, looping over all of the R2D2Data items (analysis, bias_correction, diagnostic, feedback, forecast, media, observation).
   

   Code Block
   Code Block
   language bash theme Midnight
   >>> from r2d2 import R2D2Data >>> AdminUtilR2D2Data.move_experiment(source_experimentsearch(item='e8254aforecast', target_experiment='oper', ensemble_data_store_type='geosensemble')

7. To get the data to the other HPCs, on your origin system you will sync to S3 and then sync the data stores.

   Code Block
   language bash theme Midnight
   /path/to/aws s3 sync --size-only --profile=jcsda-usaf-aws-us-east-2 /discover/nobackup/projects/jcsda/s2127/r2d2-archive-nccs/ s3://r2d2-archive-jcsda-usaf-aws-us-east-2/ python3 >>> from r2d2.util.admin_util import AdminUtil >>> AdminUtil.sync_data_stores(source_data_store='r2d2-archive-nccs', target_data_store='r2d2-archive-jcsda-usaf-aws-us-east-2')

8. e65ab2') # Returns: [{'forecast_index': 3207928, 'model': 'geos', 'experiment': 'e65ab2', 'file_extension': 'nc4', 'resolution': 'c90', 'domain': '', 'file_type': 'bkg', 'step': 'PT3H', 'tile': -9999, 'member': -9999, 'date': datetime.datetime(2022, 2, 15, 6, 0), 'create_date': datetime.datetime(2024, 8, 23, 20, 52, 47), 'mod_date': datetime.datetime(2024, 8, 23, 20, 52, 47)}]

9. Use the "AdminUtil.move_experiment" function to move the experiment given in the r2d2-ingest ticket to oper and the correct data_store. You can refer to the r2d2 code to check for arguments needed and how to use themLog into MSU's Orion and sync from S3. Then you can proceed to log into the other HPCs and sync NWSC/SSEC systems from MSU using the sync scripts in ~/bin.

   Code Block
   language bash theme Midnight
   >>> AdminUtil.move_experiment(source_experiment='<expid>', target_experiment='oper', ensemble_data_store_type='<data_store_type>')

10. To get the data to the other HPCs, on your origin system you will sync to S3 and then sync the data stores.

    Code Block
    language bash theme Midnight
    /path/to/ssh -Y <user_name>@orion-login.hpc.msstate.edu sudo -su role-jcsda # Load venv to get aws modules or load modules following jedi-docs aws s3 sync --size-only --profile=jcsda-usaf-aws-us-east-2 /discover/nobackup/projects/jcsda/s2127/r2d2-archive-nccs/ s3://r2d2-archive-jcsda-usaf-aws-us-east-2/ /work2/noaa/jcsda/role-jcsda/r2d2-archive-msu/ python3 >>> from r2d2.util.admin_util import AdminUtil >>> AdminUtil.sync_data_stores(source_data_store='r2d2-archive-nccs', target_data_store='r2d2-archive-jcsda-usaf-aws-us-east-2')

11. Log into MSU's Orion, rsync from S3 and sync the data_stores. Then you can proceed to log into the other HPCs and sync NWSC/SSEC systems from MSU using the sync scripts in ~/bin and sync the data_stores.

    Code Block
    language bash theme Midnight
    ssh -Y <user_name>@orion-login.hpc.msstate.edu sudo -su role-jcsda # Load venv to get aws modules or load modules following jedi-docs # To sync archive: aws s3 sync --size-only --profile=jcsda-usaf-aws-us-east-2 s3://r2d2-archive-jcsda-usaf-aws-us-east-2/ /work2/noaa/jcsda/role-jcsda/r2d2-archive-msu/ # To sync data_stores: python3 >>> from r2d2.util.admin_util import AdminUtil >>> AdminUtil.sync_data_stores(source_data_store='r2d2-archive-jcsda-usaf-aws-us-east-2', target_data_store='r2d2-archive-msu')

12. Finally you will run the rsync and sync the data_stores from the other data hubs. There is a script located inside jedipara's ~/bin  directory the can be used to perform the rsync. Make sure the username is yours instead of the most recent team members. If you get an ssh error, you can remove the machine from known hosts and try again. Note, if you need to find the value for data_hub that is stored in R2D2 you can use: R2D2Index.search(item='data_hub'). Run the following steps for each data hub:
    

    Code Block
    language bash theme Midnight
    ssh -XY <user_name>@<hpc> sudo -iu jedipara cd bin vi rsync-r2d2-archive-from-msu-<user-name>.sh        # Edit to your Orion username # Run the rsync for archive: bash rsync-r2d2-archive-from-msu-<user-name>.sh # Sync the data_stores python3 >>> from r2d2.util.admin_util import AdminUtil >>> AdminUtil.sync_data_stores(source_data_store='r2d2-archive-jcsda-usaf-aws-us-east-2', target_data_store='r2d2-archive-<data_hub>')

    Run the rsync from the other HPCs. There is a script located inside jedipara's ~/bin  directory the can be used to perform the rsync. Make sure the username is yours instead of the most recent team members. If you get an ssh error, you can remove the machine from known hosts and try again.
    

    Code Block
    language bash theme Midnight
    ssh -XY <user_name>@<hpc> sudo -iu jedipara cd bin vi rsync-r2d2-archive-from-msu-<user-name>.sh        # Edit to your Orion username bash rsync-r2d2-archive-from-msu-<user-name>.sh

Adding observation files to R2D2 Archive

...