Websites: https://experiments.jcsda.org/ and https://skylab.jcsda.org/

Table of Contents

About

Our web applications, experiments.jcsda.org and skylab.jcsda.org run on AWS EC2 instances in us-east-1. They are a Voila application with python notebooks and each run from their own instance. The logs on these instances are located at /var/log/nginx/error.log. The r2d2, solo, and diag-plots repositories are needed and are located inside the home directory.

Procedures

Reboot EC2 Instance

  1. Log into the AWS Console in us-east-1 using the jcsda-noaa account.
  2. Navigate to the EC2 Instance dashboard.
  3. Click the checkbox next to "skylab.jcsda.org" or "experiments.jcsda.org" depending on which site is down. 
  4. Click the drop down arrow next to "Instance state" in the top bar and select "Reboot instance". This might take some time to stop and start. Once it says running again, verify the website is reachable and try to connect to the instance via ssh as the ubuntu user. If those fail, then you will need to stop/start the instance. That will change the public IP address so more steps will need to be taken in order to update with the new IP. Proceed to steps 5-X.
  5. Under "Instance state" click on "Stop instance" to shut it down. If this takes longer then a few minutes, do a "Force stop instance".
  6. Start the instance again, click "Instance state" and "Start instance". After it is back running, try to connect to the instance via ssh as the ubuntu user.
  7. Copy the new  “Public IPv4 address” and navigate to the Route 53 dashboard. Click "Hosted zones" then "jcsda.org". Scroll down to experiments.jcsda.org or skylab.jcsda.org and click "Edit record". Paste the new IP where the old one is and click "Save".
  8. Try loading the web page. If it still does not load, try with a different browser. If that does not load then you will need to restart services, so proceed with steps 9-X.
  9. Log back into the instance as user ubuntu via ssh. 

  10. Restart the following services and run:

    sudo systemctl restart nginx.service
sudo systemclt daemon-reload
sudo systemclt restart voila.service


  11. Verify the website is back up and running.

Updating the certificate

The certificate should be automatically updated and managed via CertBot and Let's Encrypt. If you need to manually update the certificate you can follow this procedure.

  1. Log into the EC2 Instance as user ubuntu.

  2. Run: 

    sudo certbot --nginx


Launch new instance running Voila and NGINX

Instructions from https://docs.google.com/document/d/1vqK-qAxBGt_I9j6VG1SSLtPhQp89PzV4Y32yuWLWbUg/edit

Voila documentation: https://voila.readthedocs.io/en/stable/deploy.html#running-voila-on-a-private-server

WARNING: these instructions are old and some steps might be outdated.

Start a new EC2 instance using the console:

  1. From https://console.aws.amazon.com/ec2/v2/home?region=us-east-1, click on the blue “Launch Instance” button
  2. Scroll down to Ubuntu Server 20.04 LTS (HVM), SSD Volume Type and click the blue “Select” button Note: The user name is ubuntu, unlike other EC2 instance, where the user name is ec2-user
  3. Select t2.large, and click “Next: Configure Instance Details”
  4. Keep the default options and click “Next: Add Storage” Note: An IAM role will need to be added to the instance in order to access private buckets. For Phase 0, the bucket that contains the pre-generated plots, jtd.jcsda, is public.
  5. Update the size to 50GB, and click “Next: Add Tags”. Note: It is good practice to add tags.
  6. Click “Next: Configure Security Groups”. 
  7. Change the radio button to “Select an existing security group”, and select sg-023d069fd080b4c15, or “dashboard”, which allows global HTTP and HTTPS access, and SSH access from NCAR IP addresses.
  8. Click launch and a key pair popup will appear. Choose “Create a new key pair” unless you want to use an old one of yours.
  9. Find your new instance in the console and edit the name field.

Create nginx Service:

  1. SSH into the instance (Select instance from console, click the Connect button, copy and paste the example ssh line into your computer’s terminal). Make sure you’re on the NCAR VPN since you’ve set up the security group.

  2. Install nginx (enter in SSH’d terminal):

    sudo apt install nginx
sudo systemctl status nginx


  3. Connect {website_name}.jcsda.org to IPv4 on Route 53. (Go to https://console.aws.amazon.com/route53/v2/hostedzones#ListRecordSets/Z1F2XBLX8SOJLO, click create record, name it, keep the type as “A” and enter the instance’s public IPv4 as the Value)

  4. Create the file /etc/nginx/sites-enabled/{website_name}.jcsda.org with the following content:

    server {
    listen 80;
    server_name {website_name}.jcsda.org;
    proxy_buffering off;
    location / {
            proxy_pass http://localhost:8866;
            proxy_set_header Host $host;
            proxy_set_header X-Real-IP $remote_addr;
            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;

            proxy_http_version 1.1;
            proxy_set_header Upgrade $http_upgrade;
            proxy_set_header Connection "upgrade";
            proxy_read_timeout 86400;
    }

    client_max_body_size 100M;
    error_log /var/log/nginx/error.log;
}


  5. Enable and start the nginx service.

    sudo systemctl enable nginx.service
sudo systemctl start nginx.service


  6. Install pip

    sudo apt update && sudo apt install python3-pip


Clone Git Repo and Install Dependencies:

  1. From the instance terminal, create a new SSH key. Press Enter 3 times, in order to use default filename and no passcode.


    ssh-keygen -t ed25519 -C "{email}@ucar.edu"


  2. Add SSH Key to Github. 
    1. Go to Github.com, click "Settings" under account picture in upper righthand corner
    2. Click on "SSH and GPG keys"
    3. Click "New SSH Key" and then from instance terminal, enter:


      cat /home/ubuntu/.ssh/id_ed25519.pub


    4. Copy and paste what was returned in step 4 in the Key section

  3. From instance terminal, clone feature/voila branch. Note: the branch will change to “develop” in the future - unsure if this is the case right now.

    git clone -b feature/voila git@github.com:JCSDA-internal/jtd.git


  4. Install dependencies

    sudo python3 -m pip install --no-cache-dir -r jtd/voila/requirements.txt


Create Voila Service:

  1. Create /usr/lib/systemd/system/voila.service with the contents:

    [Unit]
Description=Voila

[Service]
Type=simple
PIDFile=/run/voila.pid
ExecStart=voila --no-browser --show_traceback=False jtd/voila/app.ipynb
User=ubuntu
WorkingDirectory=/home/ubuntu/
Restart=always
RestartSec=10

[Install]
WantedBy=multi-user.target


  2. Enable and start the voila.service:

    sudo systemctl enable voila.service
sudo systemctl start voila.service


  3. Restart nginx

    sudo systemctl restart nginx.service


  4. Enable access to https:

    sudo snap install core; sudo snap refresh core
sudo snap install --classic certbot
sudo ln -s /snap/bin/certbot /usr/bin/certbot
sudo certbot --nginx


  5. Check access on https://{website_name}.jcsda.org, it should be up and running

Troubleshooting

Web page fails to load

Issue: the web pages will fail to load/ timeout

Explanation: Each time the website is hit a new python kernel is opened and will stay open forever. Therefore, when the instance memory is full of these python kernels the page will go down.

Solution: Reboot the EC2 Instance (see procedure above)