I just talked with John. They setup 4 and 14 today, but 14 never came online. Both 14 and 11 are Vipers, so John wonders if that's part of the problem. John said he had poor cell coverage for his third-party cell phone, but Matt has Verizon and his signal was adequate. It seems unlikely that this is a signal strength problem. Both 11 and 14 had some trouble during the outside testing, as noted in JIRA issues ISFS-99 and ISFS-103.
Site 12 went offline today around 11:13 am MDT, and at the moment it has not come back. Site 12 is a Titan.
Maybe we'll have to implement something to restart the cell modem interface if it appears to be down... However, in the meanwhile, here are a couple things to try. The first is to unplug the USB modem and then plug it back in. The second achieves something similar but without unplugging anything, and it has worked once before. Log into the DSM and enter these commands:
sudo su
ifdown eth1
rmmod cdc_ether
rmmod usbnet
modprobe cdc_ether
exit
Finally, John will check on the space on the media disk on each DSM. Probably there is plenty of space, but we don't need any of the pre-project data, and we should make sure to give ourselves as much time as possible to record offline until we can visit the site and get the Internet connection up.