Websites: https://experiments.jcsda.org/ and https://skylab.jcsda.org/
| Table of Contents |
|---|
The JEDI web applications, https://experiments.jcsda.org and https://skylab.jcsda.org, run on two individual AWS EC2 instances in the jscda-noaa account in us-east-1. They are voila applications running on port 8866 on an nginx web server on ports 80/443. The web apps are simply python notebooks in the diag-plots repository with r2d2-client as the only JCSDA dependency. These packages are located in /home/ubuntu. A python virtual environment located in /home/ubuntu/venv_web_app is used to install the required python packages and execute the voila application as a service.
Code in the header for the files diag-plots/src/jedi_data_viewer/jedi_data_viewer_experiments.py and diag-plots/src/jedi_data_viewer/jedi_data_viewer_skylab.py allow the programmatic assignment of R2D2's required environmental variables which configure the client.
# Hack to set env for contacting server before importing r2d2 below # I need to figure out how to set env vars in voila os.environ['R2D2_USER'] = 'r2d2-web-app' os.environ['R2D2_HOST'] = 'aws-us-east-1' os.environ['R2D2_COMPILER'] = 'unknown' os.environ['R2D2_API_KEY'] = '6a8bacf2-d6ae-11f0-9286-06d3b1fbe489' os.environ['R2D2_LOG_LEVEL'] = 'SILENT' |
For activating the web app's virtual environment in /home/ubuntu/venv_web_app and exporting R2D2's environmental variables for the client's configuration, execute source /home/ubuntu/setup.sh.
#!/bin/bash source /home/ubuntu/venv_web_app/bin/activate export R2D2_USER=r2d2-web-app export R2D2_HOST=aws-us-east-1 export R2D2_COMPILER=unknown export R2D2_API_KEY=6a8bacf2-d6ae-11f0-9286-06d3b1fbe489 export R2D2_LOG_LEVEL=SILENT |
The Nginx configuration files are located at /etc/nginx/sites-enabled/experiments.jcsda.org and /etc/nginx/sites-enabled/skylab.jcsda.org. Be sure to replace the word "experiments" with "skylab" for the other web app's file.
server {
server_name experiments.jcsda.org;
proxy_buffering off;
location / {
proxy_pass http://localhost:8866;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "upgrade";
proxy_read_timeout 86400;
}
client_max_body_size 100M;
error_log /var/log/nginx/error.log;
listen 443 ssl; # managed by Certbot
ssl_certificate /etc/letsencrypt/live/experiments.jcsda.org/fullchain.pem; # managed by Certbot
ssl_certificate_key /etc/letsencrypt/live/experiments.jcsda.org/privkey.pem; # managed by Certbot
include /etc/letsencrypt/options-ssl-nginx.conf; # managed by Certbot
ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem; # managed by Certbot
}
server {
if ($host = experiments.jcsda.org) {
return 301 https://$host$request_uri;
} # managed by Certbot
listen 80;
server_name experiments.jcsda.org;
return 301 https://$host$request_uri;
# return 404; # managed by Certbot
} |
The Voila configuration files are located at /usr/lib/systemd/system/voila.service.
[Unit] Description=Voila [Service] Type=simple PIDFile=/run/voila.pid ExecStart=/home/ubuntu/venv_web_app/bin/python3.10 -m voila --no-browser --show_traceback=False diag-plots/src/jedi_data_viewer/skylab_data_viewer.ipynb User=ubuntu WorkingDirectory=/home/ubuntu/ Restart=always RestartSec=10 [Install] WantedBy=multi-user.target |
The error logs are located at /var/log/nginx/error.log and can be accessed by executing
journalctl -u voila.service |
Please perform the following command line actions to update the develop branch for each web app.
cd ~ source setup.sh cd r2d2-client git pull python3 -m pip install . cd ~ cd diag-plots git pull python3 -m pip install . |
Restart the following services and run:
sudo systemctl restart nginx.service sudo systemclt daemon-reload sudo systemclt restart voila.service |
The certificate should be automatically updated and managed via CertBot and Let's Encrypt. If you need to manually update the certificate you can follow this procedure.
sudo certbot --nginx |
sudo apt update sudo apt upgrade sudo apt install build-essential libssl-dev libffi-dev |
wget https://www.python.org/ftp/python/3.10.19/Python-3.10.19.tgz tar -xvf Python-3.10.19.tgz cd Python-3.10.19 ./configure --enable-optimizations --with-ensurepip=install nproc make -j 8 sudo make altinstall /usr/local/bin/python3.10 --version |
The following command line will create the virtual environment for each web app.
cd~ /usr/local/bin/python3.10 -m venv venv_web_app |
import ipywidgets as widgets
out = widgets.Output(layout={'border': '1px solid black'})
display(out)
from jedi_data_viewer_experiments import JediDataViewerExperiments
jedi_data_viewer_experiments = JediDataViewerExperiments(out)
with self._out: print('My message here') |
Issue: the web pages will fail to load/ timeout
Explanation: Each time the website is hit a new python kernel is opened and will stay open forever. Therefore, when the instance memory is full of these python kernels the page will go down.
Solution: Reboot the EC2 Instance (see procedure above)
Instructions from https://docs.google.com/document/d/1vqK-qAxBGt_I9j6VG1SSLtPhQp89PzV4Y32yuWLWbUg/edit
Voila documentation: https://voila.readthedocs.io/en/stable/deploy.html#running-voila-on-a-private-server
WARNING: These instructions are very old and some steps might be outdated or simply not correct for the current web apps.
Start a new EC2 instance using the console:
Create nginx Service:
Install nginx (enter in SSH’d terminal):
sudo apt install nginx sudo systemctl status nginx |
Create the file /etc/nginx/sites-enabled/{website_name}.jcsda.org with the following content:
server {
listen 80;
server_name {website_name}.jcsda.org;
proxy_buffering off;
location / {
proxy_pass http://localhost:8866;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "upgrade";
proxy_read_timeout 86400;
}
client_max_body_size 100M;
error_log /var/log/nginx/error.log;
} |
Enable and start the nginx service.
sudo systemctl enable nginx.service sudo systemctl start nginx.service |
Install pip
sudo apt update && sudo apt install python3-pip |
Clone Git Repo and Install Dependencies:
ssh-keygen -t ed25519 -C "{email}@ucar.edu" |
cat /home/ubuntu/.ssh/id_ed25519.pub |
From instance terminal, clone feature/voila branch. Note: the branch will change to “develop” in the future - unsure if this is the case right now.
git clone -b feature/voila git@github.com:JCSDA-internal/jtd.git |
Install dependencies
sudo python3 -m pip install --no-cache-dir -r jtd/voila/requirements.txt |
Create Voila Service:
Create /usr/lib/systemd/system/voila.service with the contents:
[Unit] Description=Voila [Service] Type=simple PIDFile=/run/voila.pid ExecStart=voila --no-browser --show_traceback=False jtd/voila/app.ipynb User=ubuntu WorkingDirectory=/home/ubuntu/ Restart=always RestartSec=10 [Install] WantedBy=multi-user.target |
Enable and start the voila.service:
sudo systemctl enable voila.service sudo systemctl start voila.service |
Restart nginx
sudo systemctl restart nginx.service |
Enable access to https:
sudo snap install core; sudo snap refresh core sudo snap install --classic certbot sudo ln -s /snap/bin/certbot /usr/bin/certbot sudo certbot --nginx |