Research Repository for Data and Diagnostics

GitHub repositories:

https://github.com/JCSDA-internal/r2d2 (R2D2 V2 and R2D2 V3 server: feature/cleanup1)
https://github.com/JCSDA-internal/r2d2-ingest (R2D2 data ingest request system)
https://github.com/JCSDA-internal/r2d2-data (localhost R2D2 database)
https://github.com/JCSDA-internal/r2d2-client (R2D2 V3 client: feature/cleanup1)

Related GitHub repositories:

https://github.com/JCSDA-internal/diag-plots (R2D2 V3: feature/r2d2v3)
https://github.com/JCSDA-internal/ewok (R2D2 V3: feature/r2d2v3)
https://github.com/JCSDA-internal/skylab (R2D2 V3: feature/r2d2v3)

Documentation:

R2D2 Planning Meeting Notes:

https://docs.google.com/document/d/1eZ38vtVOiN6wl1sXZb9jIx-jF6YVZPFDWUtw7oMXktA/edit

News Releases:

Table of Contents

About

R2D2 is a lightweight Python API, an SQL schema, and a live, production, cloud-based MySQL database server that Skylab utilizes for data assimilation experiments. The current R2D2 is version 2 which uses Python MySQL Connector to execute queries directly to the remote, production MySQL database located on us-east-2 on r2d2.jcsda.org on port 3306. The new R2D2 is version 3 which uses a standardized client / server architecture using REST HTTP API calls well-defined by OpenAPI 3.0.

Procedures

Adding Files to EWOK_STATIC_DATA

Requirements:

Access to Orion since files are synced from there, although this can be worked around.
Access to jedi-para or jedipara or role-jcsda on HPCs

Steps:

Log into Orion and sudo to the role-jcsda account.
Copy static files from staging location to the decided $EWOK_STATIC_DATA location. The staging location is usually given in the work ticket and is the location where the JCSDA team member has placed the data. File names can be renamed if needed. Make sure to match the permissions as the other files in $EWOK_STATIC_DATA directory.
Run the rsync from the other HPCs. There is a script located inside jedipara's ~/bin directory the can be used to perform the rsync. Make sure the username is yours instead of the most recent team members. If you get an ssh error, you can remove the machine from known hosts and try again. An example of the script on Discover is located at /home/jedipara/bin/rsync-ewok-static-from-orion.sh.
Check off each machine in the R2D2-data ticket as you sync.

R2D2 V3: Client / Server Implementation

Installing the r2d2 server

Note: This installation process needs to be updated and combined

cd r2d2
python3 -m pip install -e .
cd server
python3 -m pip install -e .

Starting the r2d2 server

cd r2d2/server/app

$ pwd
.../r2d2/server/app

Then

$ run_r2d2_app --port=8080 --debug

You should see this output. This means that the server is running.


 * Serving Flask app 'app.app'
 * Debug mode: on
WARNING: This is a development server. Do not use it in a production deployment. Use a production WSGI server instead.
 * Running on all addresses (0.0.0.0)
 * Running on http://127.0.0.1:8080
 * Running on http://192.168.10.66:8080
Press CTRL+C to quit
 * Restarting with stat
 * Debugger is active!
 * Debugger PIN: 912-946-391

Installing the r2d2 client

cd r2d2-client
python3 -m pip install -e .

How to use the r2d2-client generator

from generator import Generator
Generator().generate(client_output_path='/Users/eric2/jedi/jedi-bundle/r2d2-client/src/r2d2_client/r2d2_client.py')

or 

Generator(selected_item='observation').generate(client_output_path='/Users/eric2/jedi/jedi-bundle/r2d2-client/src/r2d2_client/r2d2_client.py')

How to launch the Swagger Editor for editing app.yaml using a localhost Docker container

docker pull swaggerapi/swagger-editor
docker run -d -p 80:8080 swaggerapi/swagger-editor