Page History

...

• https://github.com/JCSDA-internal/r2d2 (R2D2 V2 and R2D2 V3 server: feature/cleanup1)
• https://github.com/JCSDA-internal/r2d2-ingest (R2D2 data ingest request system)
• https://github.com/JCSDA-internal/r2d2-data (localhost R2D2 database)
• https://github.com/JCSDA-internal/r2d2-client (R2D2 V3 client: feature/cleanup1)

Related GitHub repositories:

• https://github.com/JCSDA-internal/diag-plots (R2D2 V3: feature/r2d2v3)
• https://github.com/JCSDA-internal/ewok (R2D2 V3: feature/r2d2v3)
• https://github.com/JCSDA-internal/skylab (R2D2 V3: feature/r2d2v3)

...

R2D2 is a lightweight Python API, an SQL schema, and a live, production, cloud-based MySQL database server that Skylab utilizes for data assimilation experiments.The  The current R2D2 is version 2 which uses Python MySQL Connector to execute queries directly to the remote, production MySQL database located on us-east-2 on r2d2.jcsda.org on port 3306.   The new R2D2 is version 3 which uses a standardized client / server architecture using REST HTTP API calls well-defined by an OpenAPI 3.0 specification matching R2D2's SQL schema.Both  Both the client and REST server APIs for R2D2 V3 are item-based and not function-based as in the R2D2 V2 database connector API.The  The SQL schema for V2 and V3 are identical. 

...

1. Log into the HCP where the data to be ingested is located, become the jedipara or role-jcsda user depending on HPC, and go to the location for R2D2 archive.
   

   Code Block
   language bash theme Midnight
   ssh -XY <user_name>@login.nccs.nasa.gov sudo -iu jedipara cd /discover/nobackup/projects/jcsda/s2127

2. Set up your venv, using setup.sh, that was created in the requirements section. You might need to update $JEDI_ENV.
   

   Code Block
   language bash theme Midnight
   vi setup.sh # Verify JEDI_ENV location source setup.sh

3. Log into a screen session so no work will be lost if you get logged out

   Code Block
   language bash theme Midnight
   screen -S r2d2_ingest

4. Verify that your venv is still loaded, if not re-load following the same steps as #2.
5. Check your r2d2 admin utility access. If this does not work make sure you followed the requirements.

   Code Block
   language bash theme Midnight
   python3 >>> from r2d2.util.admin_util import AdminUtil

6. Use the "AdminUtil.move_experiment" function to move the experiment given in the r2d2-ingest ticket to oper and the correct data_store. You can refer to the r2d2 code to ceck for arguments needed and how to use them.

   Code Block
   language bash theme Midnight
   >>> AdminUtil.move_experiment(source_experiment='e8254a', target_experiment='oper', ensemble_data_store_type='geosensemble')

7. To get the data to the other HPCs, on your origin system you will sync to S3 and then sync the data stores.

   Code Block
   language bash theme Midnight
   /path/to/aws s3 sync --size-only --profile=jcsda-usaf-aws-us-east-2 /discover/nobackup/projects/jcsda/s2127/r2d2-archive-nccs/ s3://r2d2-archive-jcsda-usaf-aws-us-east-2/ python3 >>> from r2d2.util.admin_util import AdminUtil >>> AdminUtil.sync_data_stores(source_data_store='r2d2-archive-nccs', target_data_store='r2d2-archive-jcsda-usaf-aws-us-east-2')

8. Log into MSU's Orion and sync from S3. Then you can proceed to log into the other HPCs and sync NWSC/SSEC systems from MSU using the sync scripts in ~/bin.

   Code Block
   language bash theme Midnight
   ssh -Y <user_name>@orion-login.hpc.msstate.edu sudo -su role-jcsda # Load venv to get aws modules or load modules following jedi-docs aws s3 sync --size-only --profile=jcsda-usaf-aws-us-east-2 s3://r2d2-archive-jcsda-usaf-aws-us-east-2/ /work2/noaa/jcsda/role-jcsda/r2d2-archive-msu/

9. Run the rsync from the other HPCs. There is a script located inside jedipara's ~/bin  directory the can be used to perform the rsync. Make sure the username is yours instead of the most recent team members. If you get an ssh error, you can remove the machine from known hosts and try again.
   

   Code Block
   language bash theme Midnight
   ssh -XY <user_name>@<hpc> sudo -iu jedipara cd bin vi rsync-r2d2-archive-from-msu-<user-name>.sh        # Edit to your Orion username bash rsync-r2d2-archive-from-msu-<user-name>.sh

Adding observation files to R2D2 Archive

Prerequisites:

• IODA formated observation files to be moved and synced, these must be VERIFIED by the observation team
• Access to R2D2 and the feature/new_scripts branch

Steps:

1. The easiest way is to log into two Orion shells, one as user role-jcsda and the other as your own user
2. As your own user, update your copy of r2d2-ingest code from https://github.com/JCSDA-internal/r2d2-ingest
   

   Code Block
   language bash theme Midnight
   ssh -Y <user_name>@orion-login.hpc.msstate.edu cd /work2/noaa/jcsda/<user_name>/JEDI/jedi-bundle # Where ever you have or keep jedi-bundle git clone https://github.com/JCSDA-internal/r2d2-ingest.git cd r2d2-ingest git checkout feature/new_scripts

3. As role-jcsda, use the environment from your individual setup of jedi/r2d2-ingest. There are example setup.sh scripts in /work2/noaa/jcsda/role-jcsda to use and refer to.
   

   Code Block
   language bash theme Midnight
   ssh -Y <user_name>@orion-login.hpc.msstate.edu sudo -su role-jcsda cd /work2/noaa/jcsda/role-jcsda vi setup.sh # Edit JEDI_ENV to your location

4. Usually the observation team will give you the location to their log file which contains the output from their local R2D2 store. Inside r2d2-ingest, scripts_v2/parse/parse_r2d2_obs_log.py, will dump the r2d2 indexes into a text file. Therefore, update parse_r2d2_obs_log.py or a file that supports your needs with the information from the observation team in the r2d2-ingest issue ticket. Then run parse_r2d2_obs_log.py (or other). Question for Eric: is this done as your own user?
5. Update scripts_v2/parse/move_parsed_obs.py to point to the new text file.
6. Once a txt file is generated for the experiment, start a screen session as the role-jcsda user to move the parsed obs to R2D2's archive data store.

   Code Block
   language bash theme Midnight
   # Inside role-jcsda session screen -S move_obs # Verify your $JEDI_ENV is correct echo $JEDI_ENV python3 $JEDI_ENV/jedi-bundle/r2d2-ingest/scripts_v2/parse/move_parsed_obs.py

7. After that completes, follow the steps to sync R2D2's archive and data_stores across all of our HPCs. WARNING, you must use the screen command (or nohup). The full obs ingest and sync could take hours and background files could even take days!

Installing the r2d2 v3 server

...