This guide covers administrative tasks for the R2D2 API server, including managing, updating, fixing, and monitoring the API, database, and AWS infrastructure. It’s meant for JEDI Infrastructure team members and JCSDA AWS admins who need to keep things running smoothly, debug issues, and ensure uptime. In addition to a system overview administration guide, this document will contain playbook entries and work logs to maintain history and procedures of administrative work. If this guide is missing entries, please author them as needed.
Quick Reference
- AWS Resources for R2D2 Prod
- Cloud formation config: //r2d2/server/cfn/prod.yaml
- Cloud formation stack: arn:aws:cloudformation:us-east-2:747101682576:stack/r2d2-api-prod/c15cb090-d2dc-11ef-a026-06c957b621e9
- API Server EC2 Instances: ec2 filtered search
- Load Balancer: arn:aws:elasticloadbalancing:us-east-2:747101682576:loadbalancer/app/r2d2-prod-load-balancer/f526daf391077f70
- SSL certificate (for r2d2-api.jcsda.org): arn:aws:acm:us-east-2:747101682576:certificate/be95f8b1-4162-4615-9ed5-29ed897b527b
- Database
- Admin commands
- Login to a server instance:
aws ssm start-session --target i-026c4881c032096e0 --region us-east-2
- Login to a server instance:
Infrastructure Overview
R2D2’s HTTP API is hosted on AWS utilizing the following components. For more info see this Feb 7, 2025 slide deck.
- Application Load Balancer - API Server routing, load balancing, and SSL termination.
- EC2 Compute Hosts - R2D2 API server execution environment.
- AWS Systems Manager - shell access and administration.
- AWS Relational Database Service - R2D2 database.
Common Tasks
Deploy Cloud Formation Updates
- Copy the updated prod.yaml config to our infrastructure-as-code bucket and get the versioned URL for the file as shown here.
aws s3 cp server/cfn/prod.yaml s3://jcsda-usaf-iac-artifacts/r2d2/prod.yaml
file_version=$(aws s3api list-object-versions \
--bucket jcsda-usaf-iac-artifacts \
--prefix "r2d2/prod.yaml" \
| jq -r '.Versions[] | select(.IsLatest==true) | .VersionId')
echo "https://jcsda-usaf-iac-artifacts.s3.us-east-2.amazonaws.com/r2d2/prod.yaml?versionId=${file_version}" - Go to the CloudFormation stack "r2d2-api-prod" in the AWS console (or use quick-ref link above) and click "Update".
- In the update menu, select "Replace existing template" and enter the URL generated in the first step. Change any parameters and review deploy options.
- Review the change log and ensure that EC2 instances and RDS instances are not replaced.
- Monitor the CloudFormation rollout in console and ensure it executes to completion.
Build and publish API server binaries
1) Build the docker image
git clone -b feature/prod-run https://github.com/jcsda-internal/r2d2.git cd r2d2 docker build -f server/docker/Dockerfile.app \ --platform=linux/amd64 \ -t 747101682576.dkr.ecr.us-east-2.amazonaws.com/r2d2-server:prod .
2) Get local credentials for our ECR repository.
aws ecr get-login-password --region us-east-2 \ | docker login --username AWS \ --password-stdin 747101682576.dkr.ecr.us-east-2.amazonaws.com
3) Push the docker image
docker push 747101682576.dkr.ecr.us-east-2.amazonaws.com/r2d2-server:prod
Update a Running API Server Instance
This is the process needed to push updated code to an API Server instance. Note that you can build the binary on the instance, but if you do, you should also push it to ECR to keep a persistent copy of our server code.
# From your developer machine log into the server aws ssm start-session --target i-026c4881c032096e0 --region us-east-2 # This process needs to be completed as root sudo su - docker pull 747101682576.dkr.ecr.us-east-2.amazonaws.com/r2d2-server:prod # Stop the running service. systemctl stop r2d2.service # Remove the current container. docker rm r2d2-api-service # Re-start the prod service container. docker run -d --name r2d2-api-service -p 80:80 \ -e MYSQL_USER='r2d2' \ -e MYSQL_DATABASE='r2d2' \ -e MYSQL_HOST='r2d2-api-prod-rdsinstance-eja4eofrohvy.cg24vilqoa8w.us-east-2.rds.amazonaws.com' \ 747101682576.dkr.ecr.us-east-2.amazonaws.com/r2d2-server:prod \ run_r2d2_app --port 80 --nodebug # Restart the systemd service. systemctl start r2d2.service