...

About

Our The JEDI web applications, https://experiments.jcsda.org and https://skylab.jcsda.org, run on two individual AWS EC2 instances in the jscda-noaa account in us-east-1. They are Voila voila applications running on port 8866 on an nginx web server on ports 80/443. The web apps are simply python notebooks in the diag-plots repository with r2d2 and solo -client as the only dependenciesJCSDA dependency. These packages are located in /home/ubuntu. Python3 virtual environments, e.i. "venv" are not used for the web apps. All python code is installed to the system python. It was the only way we could figure out to get the notebooks to work with Viola and nginx over https.

The Viola middleware service strips the cells from each notebook at runtime and displays only the user interface graphics required for the drop down menus and image viewing. Every time someone visits one of these web pages, a new Python3 kernel is started in memory. Over time, this leads to OutOfMemoryErrors from the OS and the web apps freeze up and the instance will have to be rebooted as detailed below.

The error logs are located at /var/log/nginx/error.log and can be accessed by executing

journalctl -u voila.service

The https nginx configurations are at /etc/nginx/sites-enabled/experiments.jcsda.org and /etc/nginx/sites-enabled/skylab.jcsda.org.

WARNING: Do not change the https nginx configurations unless you know what you are doing.

server {
    server_name experiments.jcsda.org;
    proxy_buffering off;
    location / {
            proxy_pass http://localhost:8866;
            proxy_set_header Host $host;
            proxy_set_header X-Real-IP $remote_addr;
            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;

            proxy_http_version 1.1;
            proxy_set_header Upgrade $http_upgrade;
            proxy_set_header Connection "upgrade";
            proxy_read_timeout 86400;
    }
    client_max_body_size 100M;
    error_log /var/log/nginx/error.log;

    listen 443 ssl; # managed by Certbot
    ssl_certificate /etc/letsencrypt/live/experiments.jcsda.org/fullchain.pem; # managed by Certbot
    ssl_certificate_key /etc/letsencrypt/live/experiments.jcsda.org/privkey.pem; # managed by Certbot
    include /etc/letsencrypt/options-ssl-nginx.conf; # managed by Certbot
    ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem; # managed by Certbot
}

server {
    if ($host = experiments.jcsda.org) {
        return 301 https://$host$request_uri;
    } # managed by Certbot

    listen 80;
    server_name experiments.jcsda.org;
    return 301 https://$host$request_uri;
    # return 404; # managed by Certbot
}

Procedures

Reboot EC2 Instance

...

Restart the following services and run:

sudo systemctl restart nginx.service
sudo systemclt daemon-reload
sudo systemclt restart voila.service

...

Verify the website is back up and running.

A python virtual environment located in /home/ubuntu/venv_web_app is used to install the required python packages and execute the voila application as a service.

R2D2 Configuration 

In Web App

Code in the header for the files diag-plots/src/jedi_data_viewer/jedi_data_viewer_experiments.py and diag-plots/src/jedi_data_viewer/jedi_data_viewer_skylab.py allow the programmatic assignment of R2D2's required environmental variables which configure the client. 

# Hack to set env for contacting server before importing r2d2 below
# I need to figure out how to set env vars in voila
os.environ['R2D2_USER'] = 'r2d2-web-app'
os.environ['R2D2_HOST'] = 'aws-us-east-1'
os.environ['R2D2_COMPILER'] = 'unknown'
os.environ['R2D2_API_KEY'] = '6a8bacf2-d6ae-11f0-9286-06d3b1fbe489'
os.environ['R2D2_LOG_LEVEL'] = 'SILENT'

On EC2 Instance

For activating the web app's virtual environment in /home/ubuntu/venv_web_app and exporting R2D2's environmental variables for the client's configuration, execute source /home/ubuntu/setup.sh. 

#!/bin/bash
  
source /home/ubuntu/venv_web_app/bin/activate

export R2D2_USER=r2d2-web-app
export R2D2_HOST=aws-us-east-1
export R2D2_COMPILER=unknown
export R2D2_API_KEY=6a8bacf2-d6ae-11f0-9286-06d3b1fbe489
export R2D2_LOG_LEVEL=SILENT

Nginx Configuration

The Nginx configuration files are located at /etc/nginx/sites-enabled/experiments.jcsda.org and /etc/nginx/sites-enabled/skylab.jcsda.org. Be sure to replace the word "experiments" with "skylab" for the other web app's file. 

server {
    server_name experiments.jcsda.org;
    proxy_buffering off;
    location / {
            proxy_pass http://localhost:8866;
            proxy_set_header Host $host;
            proxy_set_header X-Real-IP $remote_addr;
            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;

            proxy_http_version 1.1;
            proxy_set_header Upgrade $http_upgrade;
            proxy_set_header Connection "upgrade";
            proxy_read_timeout 86400;
    }
    client_max_body_size 100M;
    error_log /var/log/nginx/error.log;

    listen 443 ssl; # managed by Certbot
    ssl_certificate /etc/letsencrypt/live/experiments.jcsda.org/fullchain.pem; # managed by Certbot
    ssl_certificate_key /etc/letsencrypt/live/experiments.jcsda.org/privkey.pem; # managed by Certbot
    include /etc/letsencrypt/options-ssl-nginx.conf; # managed by Certbot
    ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem; # managed by Certbot
}

server {
    if ($host = experiments.jcsda.org) {
        return 301 https://$host$request_uri;
    } # managed by Certbot

    listen 80;
    server_name experiments.jcsda.org;
    return 301 https://$host$request_uri;
    # return 404; # managed by Certbot
}

Voila Configuration

The Voila configuration files are located at /usr/lib/systemd/system/voila.service. 

[Unit]
Description=Voila

[Service]
Type=simple
PIDFile=/run/voila.pid
ExecStart=/home/ubuntu/venv_web_app/bin/python3.10 -m voila --no-browser --show_traceback=False diag-plots/src/jedi_data_viewer/skylab_data_viewer.ipynb
User=ubuntu
WorkingDirectory=/home/ubuntu/
Restart=always
RestartSec=10

[Install]
WantedBy=multi-user.target

Voila Logging

The error logs are located at /var/log/nginx/error.log and can be accessed by executing

journalctl -u voila.service

Procedures

Update the Source Code

Please perform the following command line actions to update the develop branch for each web app. 

cd ~
source setup.sh
cd r2d2-client
git pull
python3 -m pip install .
cd ~
cd diag-plots
git pull
python3 -m pip install .

Reboot EC2 Instance

1. Log into the AWS Console in us-east-1 using the jcsda-noaa account.
2. Navigate to the EC2 Instance dashboard.
3. Click the checkbox next to "skylab.jcsda.org" or "experiments.jcsda.org" depending on which site is down. 
4. Click the drop down arrow next to "Instance state" in the top bar and select "Reboot instance". This might take some time to stop and start. Once it says running again, verify the website is reachable and try to connect to the instance via ssh as the ubuntu user.
5. If those fail, then you will need to stop/start the instance. That will change the public IP address so more steps will need to be taken in order to update with the new IP. Proceed to steps 6-X.
6. Under "Instance state" click on "Stop instance" to shut it down. If this takes longer then a few minutes, do a "Force stop instance".
7. Start the instance again, click "Instance state" and "Start instance". After it is back running, try to connect to the instance via ssh as the ubuntu user.
8. Copy the new  “Public IPv4 address” and navigate to the Route 53 dashboard. Click "Hosted zones" then "jcsda.org". Scroll down to experiments.jcsda.org or skylab.jcsda.org and click "Edit record". Paste the new IP where the old one is and click "Save".
9. Try loading the web page. If it still does not load, try with a different browser. If that does not load then you will need to restart services, so proceed with steps 9-X.
10. Log back into the instance as user ubuntu via ssh. 

11. Restart the following services and run:

    Code Block
    sudo systemctl restart nginx.service sudo systemclt daemon-reload sudo systemclt restart voila.service

12. Verify the website is back up and running.

Updating the certificate

The certificate should be automatically updated and managed via CertBot and Let's Encrypt. If you need to manually update the certificate you can follow this procedure.

sudo certbot --nginx

Installing Python 3.10 on Ubuntu 20.04

Updates to OS

sudo apt update
sudo apt upgrade
sudo apt install build-essential libssl-dev libffi-dev 

Download, Build, and Test

wget https://www.python.org/ftp/python/3.10.19/Python-3.10.19.tgz
tar -xvf Python-3.10.19.tgz
cd Python-3.10.19
./configure --enable-optimizations --with-ensurepip=install
nproc
make -j 8 
sudo make altinstall
/usr/local/bin/python3.10 --version

Create Virtual Environment for the Web App

The following command line will create the virtual environment for each web app. 

cd~
/usr/local/bin/python3.10 -m venv venv_web_app

Troubleshooting

How to run the web apps on a locally

How to print to output from ipywidgets

import ipywidgets as widgets
out = widgets.Output(layout={'border': '1px solid black'})
display(out)

from jedi_data_viewer_experiments import JediDataViewerExperiments
jedi_data_viewer_experiments = JediDataViewerExperiments(out)

with self._out: print('My message here')

Web page fails to load

Issue: the web pages will fail to load/ timeout

Explanation: Each time the website is hit a new python kernel is opened and will stay open forever. Therefore, when the instance memory is full of these python kernels the page will go down.

Solution: Reboot the EC2 Instance (see procedure above)

Launch New Instance Running Voila and Nginx [OUTDATED]

Updating the certificate

The certificate should be automatically updated and managed via CertBot and Let's Encrypt. If you need to manually update the certificate you can follow this procedure.

...

Run:

sudo certbot --nginx

...

Instructions from https://docs.google.com/document/d/1vqK-qAxBGt_I9j6VG1SSLtPhQp89PzV4Y32yuWLWbUg/edit

Voila documentation: https://voila.readthedocs.io/en/stable/deploy.html#running-voila-on-a-private-server

WARNING: These instructions are very old and some steps might be outdated or simply not correct for the current web apps. 

Start a new EC2 instance using the console:

...

1. Create /usr/lib/systemd/system/voila.service with the contents:
   

   Code Block
   [Unit] Description=Voila [Service] Type=simple PIDFile=/run/voila.pid ExecStart=voila --no-browser --show_traceback=False jtd/voila/app.ipynb User=ubuntu WorkingDirectory=/home/ubuntu/ Restart=always RestartSec=10 [Install] WantedBy=multi-user.target

    

2. Enable and start the voila.service:
   

   Code Block
   sudo systemctl enable voila.service sudo systemctl start voila.service

    

3. Restart nginx
   

   Code Block
   sudo systemctl restart nginx.service

    

Enable access to https:


Code Block
sudo snap install core; sudo snap refresh core sudo snap install --classic certbot sudo ln -s /snap/bin/certbot /usr/bin/certbot sudo certbot --nginx

4. Check access on https://{website_name}.jcsda.org, it should be up and running

Troubleshooting

How to run the web apps on a locally

How to print to output from ipywidgets

import ipywidgets as widgets
out = widgets.Output(layout={'border': '1px solid black'})
display(out)

from jedi_data_viewer_experiments import JediDataViewerExperiments
jedi_data_viewer_experiments = JediDataViewerExperiments(out)

with self._out: print("")

Web page fails to load

Issue: the web pages will fail to load/ timeout

Explanation: Each time the website is hit a new python kernel is opened and will stay open forever. Therefore, when the instance memory is full of these python kernels the page will go down.

...

1. =multi-user.target

    

2. Enable and start the voila.service:
   

   Code Block
   sudo systemctl enable voila.service sudo systemctl start voila.service

    

3. Restart nginx
   

   Code Block
   sudo systemctl restart nginx.service

    

4. Enable access to https:
   

   Code Block
   sudo snap install core; sudo snap refresh core sudo snap install --classic certbot sudo ln -s /snap/bin/certbot /usr/bin/certbot sudo certbot --nginx

   
   

5. Check access on https://{website_name}.jcsda.org, it should be up and running