On Thursday afternoon, JohnM added crontab entries to power this station's router down between 02Z and 14Z to conserve power. Our intent is to do the same to the other stations if this works.
On Thursday night, the crontab executed, but the router stayed up. Later that night, I logged into isfs3 and executed the command manually, but the router still stayed up. We thus concluded that there was a faulty connection or jumper controlling the power.
However, on Friday night the crontab executed and the router went down. Unfortunately, playa1's rsync took much longer than expected, so the rsync from hiland3 didn't start until after 02Z and thus failed. Having not seen data coming in, JohnH (and later DaveW) were tasked with rebooting this station to revive it on Saturday morning.
On Saturday afternoon, I changed the crontab to turn off the station at 03Z to give it more time.
On Saturday night, the crontab again executed and rsync worked.
Ling and I have discussed some options. We have several ideas:
1. rsync should run in parallel on all stations, rather than serially to better define the time window and also allow us to turn off all stations by 01Z.
2. even better would be to have the rsync script (started on porter?) execute the command to turn off the router once the rsync has finished. The only reason no to do this would be if we wanted staff to be able to log into stations after ~01Z.
3. to fill in the night data, another the rsync script should run in the morning as well. The evening rsync script would run faster since only half of the data would have to be transferred.
It would be best if Gordon would make these changes, since we think that the rsync script runs under crontab on his account.