Websites: https://experiments.jcsda.org/ and https://skylab.jcsda.org/

About

Our web applications, https://experiments.jcsda.org and https://skylab.jcsda.org run on two individual AWS EC2 instances in us-east-1. They are Voila applications running on port 8866 on an nginx web server on ports 80/443. The web apps are simply python notebooks in the diag-plots repository with r2d2 and solo as the only dependencies. These packages are located in /home/ubuntu. Python3 virtual environments, e.i. "venv" are not used for the web apps. All python code is installed to the system python. It was the only way we could figure out to get the notebooks to work with Viola and nginx over https.

The Viola middleware service strips the cells from each notebook at runtime and displays only the user interface graphics required for the drop down menus and image viewing. Every time someone visits one of these web pages, a new Python3 kernel is started in memory. Over time, this leads to OutOfMemoryErrors from the OS and the web apps freeze up and the instance will have to be rebooted as detailed below.

The error logs a Voila application with python notebooks and each run from their own instance. The logs on these instances are located at /var/log/nginx/error.log and can be accessed by executing

journalctl -u voila.

...

service

The https nginx configurations are at /etc/nginx/sites-enabled/experiments.jcsda.org and /etc/nginx/sites-enabled/skylab.jcsda.org.

WARNING: Do not change the https nginx configurations unless you know what you are doing.

server {
    server_name experiments.jcsda.org;
    proxy_buffering off;
    location / {
            proxy_pass http://localhost:8866;
            proxy_set_header Host $host;
            proxy_set_header X-Real-IP $remote_addr;
            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;

            proxy_http_version 1.1;
            proxy_set_header Upgrade $http_upgrade;
            proxy_set_header Connection "upgrade";
            proxy_read_timeout 86400;
    }
    client_max_body_size 100M;
    error_log /var/log/nginx/error.log;

    listen 443 ssl; # managed by Certbot
    ssl_certificate /etc/letsencrypt/live/experiments.jcsda.org/fullchain.pem; # managed by Certbot
    ssl_certificate_key /etc/letsencrypt/live/experiments.jcsda.org/privkey.pem; # managed by Certbot
    include /etc/letsencrypt/options-ssl-nginx.conf; # managed by Certbot
    ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem; # managed by Certbot
}

server {
    if ($host = experiments.jcsda.org) {
        return 301 https://$host$request_uri;
    } # managed by Certbot

    listen 80;
    server_name experiments.jcsda.org;
    return 301 https://$host$request_uri;
    # return 404; # managed by Certbot
}

Procedures

Reboot EC2 Instance

...

The certificate should be automatically updated and managed via CertBot and Let's Encrypt. If you need to manually update the certificate you can follow this procedure.

1. Log into the EC2 Instance as user ubuntu.

2. Run:
   

   Code Block
   sudo certbot --nginx

   
   

Launch new instance running Voila and NGINX

TODO, see Instructions from https://docs.google.com/document/d/1vqK-qAxBGt_I9j6VG1SSLtPhQp89PzV4Y32yuWLWbUg/edit

Voila documentation: https://voila.readthedocs.io/en/stable/deploy.html#running-voila-on-a-private-server

WARNING: These instructions are old and some steps might be outdated or simply not correct for the current web apps.

Start a new EC2 instance using the console:

1. From https://console.aws.amazon.com/ec2/v2/home?region=us-east-1, click on the blue “Launch Instance” button
2. Scroll down to Ubuntu Server 20.04 LTS (HVM), SSD Volume Type and click the blue “Select” button Note: The user name is ubuntu, unlike other EC2 instance, where the user name is ec2-user
3. Select t2.large, and click “Next: Configure Instance Details”
4. Keep the default options and click “Next: Add Storage” Note: An IAM role will need to be added to the instance in order to access private buckets. For Phase 0, the bucket that contains the pre-generated plots, jtd.jcsda, is public.
5. Update the size to 50GB, and click “Next: Add Tags”. Note: It is good practice to add tags.
6. Click “Next: Configure Security Groups”. 
7. Change the radio button to “Select an existing security group”, and select sg-023d069fd080b4c15, or “dashboard”, which allows global HTTP and HTTPS access, and SSH access from NCAR IP addresses.
8. Click launch and a key pair popup will appear. Choose “Create a new key pair” unless you want to use an old one of yours.
9. Find your new instance in the console and edit the name field.

Create nginx Service:

1. SSH into the instance (Select instance from console, click the Connect button, copy and paste the example ssh line into your computer’s terminal). Make sure you’re on the NCAR VPN since you’ve set up the security group.

2. Install nginx (enter in SSH’d terminal):
   

   Code Block
   sudo apt install nginx sudo systemctl status nginx

   
   

3. Connect {website_name}.jcsda.org to IPv4 on Route 53. (Go to https://console.aws.amazon.com/route53/v2/hostedzones#ListRecordSets/Z1F2XBLX8SOJLO, click create record, name it, keep the type as “A” and enter the instance’s public IPv4 as the Value)

4. Create the file /etc/nginx/sites-enabled/{website_name}.jcsda.org with the following content:
   

   Code Block

   server { listen 80; server_name {website_name}.jcsda.org; proxy_buffering off; location / { proxy_pass http://localhost:8866; proxy_set_header Host $host; proxy_set_header X-Real-IP $remote_addr; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; proxy_http_version 1.1; proxy_set_header Upgrade $http_upgrade; proxy_set_header Connection "upgrade"; proxy_read_timeout 86400; } client_max_body_size 100M; error_log /var/log/nginx/error.log; }

   
   

5. Enable and start the nginx service.
   

   Code Block
   sudo systemctl enable nginx.service sudo systemctl start nginx.service

   
   

6. Install pip
   

   Code Block
   sudo apt update && sudo apt install python3-pip

   
   

Clone Git Repo and Install Dependencies:

1. From the instance terminal, create a new SSH key. Press Enter 3 times, in order to use default filename and no passcode.
   

   
   

   Code Block
   ssh-keygen -t ed25519 -C "{email}@ucar.edu"

   
   

2. Add SSH Key to Github. 
   1. Go to Github.com, click "Settings" under account picture in upper righthand corner
   2. Click on "SSH and GPG keys"
   3. Click "New SSH Key" and then from instance terminal, enter:
      

      
      

      Code Block
      cat /home/ubuntu/.ssh/id_ed25519.pub

      
      

   4. Copy and paste what was returned in step 4 in the Key section

3. From instance terminal, clone feature/voila branch. Note: the branch will change to “develop” in the future - unsure if this is the case right now.
   

   Code Block
   git clone -b feature/voila git@github.com:JCSDA-internal/jtd.git

   
   

4. Install dependencies
   

   Code Block
   sudo python3 -m pip install --no-cache-dir -r jtd/voila/requirements.txt

   
   

Create Voila Service:

1. Create /usr/lib/systemd/system/voila.service with the contents:
   

   Code Block
   [Unit] Description=Voila [Service] Type=simple PIDFile=/run/voila.pid ExecStart=voila --no-browser --show_traceback=False jtd/voila/app.ipynb User=ubuntu WorkingDirectory=/home/ubuntu/ Restart=always RestartSec=10 [Install] WantedBy=multi-user.target

   
   

2. Enable and start the voila.service:
   

   Code Block
   sudo systemctl enable voila.service sudo systemctl start voila.service

   
   

3. Restart nginx
   

   Code Block
   sudo systemctl restart nginx.service

   
   

4. Enable access to https:
   

   Code Block
   sudo snap install core; sudo snap refresh core sudo snap install --classic certbot sudo ln -s /snap/bin/certbot /usr/bin/certbot sudo certbot --nginx

   
   

5. Check access on https://{website_name}.jcsda.org, it should be up and running

Troubleshooting

Web page fails to load

...