This guide covers administrative tasks for the R2D2 API server, including managing, updating, fixing, and monitoring the API, database, and AWS infrastructure. It’s meant for JEDI Infrastructure team members and JCSDA AWS admins who need to keep things running smoothly, debug issues, and ensure uptime. In addition to a system overview administration guide, this document will contain playbook entries and work logs to maintain history and procedures of administrative work. If this guide is missing entries, please author them as needed.

Quick Reference

Infrastructure Overview

R2D2’s HTTP API is hosted on AWS utilizing the following components. For more info see this Feb 7, 2025 slide deck.


Common Tasks

Deploy Cloud Formation Updates

  1. Copy the updated prod.yaml config to our infrastructure-as-code bucket and get the versioned URL for the file as shown here. 


    aws s3 cp server/cfn/prod.yaml s3://jcsda-usaf-iac-artifacts/r2d2/prod.yaml
    file_version=$(aws s3api list-object-versions \
                       --bucket jcsda-usaf-iac-artifacts \
                       --prefix "r2d2/prod.yaml"  \
                   | jq -r '.Versions[] | select(.IsLatest==true) | .VersionId')
    echo "https://jcsda-usaf-iac-artifacts.s3.us-east-2.amazonaws.com/r2d2/prod.yaml?versionId=${file_version}"
  2. Go to the CloudFormation stack "r2d2-api-prod" in the AWS console (or use quick-ref link above) and click "Update".
  3. In the update menu, select "Replace existing template" and enter the URL generated in the first step. Change any parameters and review deploy options.
    1. Review the change log and ensure that EC2 instances and RDS instances are not replaced.
  4. Monitor the CloudFormation rollout in console and ensure it executes to completion.

Build and publish API server binaries

1) Build the docker image

git clone -b feature/prod-run https://github.com/jcsda-internal/r2d2.git
cd r2d2
docker build -f server/docker/Dockerfile.app \
     --platform=linux/amd64 \
     -t 747101682576.dkr.ecr.us-east-2.amazonaws.com/r2d2-server:prod .

2) Get local credentials for our ECR repository.

aws ecr get-login-password --region us-east-2 \
| docker login --username AWS \
               --password-stdin 747101682576.dkr.ecr.us-east-2.amazonaws.com

3) Push the docker image

docker push 747101682576.dkr.ecr.us-east-2.amazonaws.com/r2d2-server:prod

Update a Running API Server Instance

This is the process needed to push updated code to an API Server instance. Note that you can build the binary on the instance, but if you do, you should also push it to ECR to keep a persistent copy of our server code.

# From your developer machine log into the server
aws ssm start-session --target i-026c4881c032096e0 --region us-east-2

# This process needs to be completed as root
sudo su -

docker pull 747101682576.dkr.ecr.us-east-2.amazonaws.com/r2d2-server:prod

# Stop the running service.
systemctl stop r2d2.service

# Remove the current container.
docker rm r2d2-api-service

# Re-start the prod service container.
docker run -d --name r2d2-api-service -p 80:80 \
           -e MYSQL_USER='r2d2' \
           -e MYSQL_DATABASE='r2d2' \
           -e MYSQL_HOST='r2d2-api-prod-rdsinstance-eja4eofrohvy.cg24vilqoa8w.us-east-2.rds.amazonaws.com' \
           747101682576.dkr.ecr.us-east-2.amazonaws.com/r2d2-server:prod \
           run_r2d2_app --port 80 --nodebug

# Restart the systemd service.
systemctl start r2d2.service


Accessing the RDS SQL database

The RDS database cannot be accessed from the public internet and must be accessed from one of our dev servers using a ssm shell login.

1) Log into the dev server (see quick reference)

2) Fetch the server password from the parameters in our cloud formation stack (see link in quick refrence)

3) From the dev server execute this command, using the password retrieved above

mysql -h r2d2-api-prod-rdsinstance-eja4eofrohvy.cg24vilqoa8w.us-east-2.rds.amazonaws.com -u admin -p


Load "dev" server data onto r2d2 database

This should not be done! This procedure was used to test the R2D2 database prior to release and is documented here for historical context and in case some of these steps have useful analogs for other work.

For the localhost server container, can we pre-load r2d2-data and the r2d2 mysql user so
that end users just download the container and execute it on port 8080? Basically execute
the setup database script from r2d2/scripts.


Update crontabs on all platforms (experiment scrubber)
 - Platforms include derecho, discover, hercules, orion, s4
  - Update venv used by crontabs to use r2d2-client and not r2d2



Create username and api keys for all users
 - We will have to massage the user database table since some humans have more than one username
 - We need an api key generator and a way to send these api keys to the user


Go to dev server dump dev database into IAC bucket
 - Access astromech-nonprod
     $ aws ssm start-session --target i-0e97feb060f886e43 --region us-east-2
     $ sudo su ubuntu && cd ~
 - Use password "local-server-root-password" (only works on dev server)
 - DATE=$(date "+%Y%m%dT%H%M%SZ")
 - mysqldump -h 127.0.0.1 -u root -p --databases r2d2 > r2d2-dev-backup-${DATE}.sql
 - aws s3 cp r2d2-dev-backup-20250205T171114Z.sql s3://jcsda-usaf-iac-artifacts/r2d2/


Log into prod server and load database with dev data
 - Login and load data
   - ssm login command (see quick ref)
   - sudo su ubuntu && cd ~
   - aws s3 cp  s3://jcsda-usaf-iac-artifacts/r2d2/r2d2-dev-backup-20250205T171114Z.sql ./r2d2-dev-backup.sql
 - Get password cloud formation console parameters
 - Log into server
   - mysql -h r2d2-api-prod-rdsinstance-eja4eofrohvy.cg24vilqoa8w.us-east-2.rds.amazonaws.com -u admin -p
 - Run these sql commands
      CREATE DATABASE r2d2;
      USE r2d2;
      source /home/ubuntu/r2d2-dev-backup.sql