The network to the outside world wasn't working from about 0800-1200 this morning. The LAN was all fine, so we didn't lose any data. Somehow the cradlepoint on the top of rim died. Power cycling from rimup with vio 7 0; vio 7 1 brought it back. The "router_check" script on rimup should have brought it back up, but apparently ran once at 0800 and not again. Gordon is investigating why this didn't work.
None of our data were affected by this issue. Hopefully it was late enough that it didn't impact the just-finished IOP operations.
Comment by Gordon: The above was to due to a bug in the crontab entry that checks the internet connection:
*/20 * * * * net_check.sh eth0 192.168.0 192.168.0.5 && router_check.sh 7 www.google.com
For some reason it appears that the ethernet interface on the router died this morning, such that it didn't respond to pings from the DSM. The above crontab entry does not power cycle the router if the DSM can't ping it. The idea is not to power cycle the router and modems if the problem is at our end.
Changed it to the following, which will do a net_check.sh and router_check.sh every 10 minutes:
*/10 * * * * net_check.sh eth0 192.168.0 192.168.0.5; router_check.sh 7 www.google.com