We had a partial network outage in the ISFS trailer that started around 10pm local time on Sunday night. There was a great deal of activity going on in the trailer, that seemed to confuse and complicate the issue, but the initial evidence was that Dave W. lost his wireless connection, and we had to switch him over to a wired connection. Then we noticed that the wired connections in the rear of the trailer that serve the ceilometer, sounding, and data manager computers were not connected to the network... Diagnosis began working from the back to the front, with rebooting the hubs in the rear (linksys) then the middle (d-link), but hesitated at rebooting the white belkin netgear wireless router... Other computers on the d-link (middle) are downstream from the belkin, and DID have outside internet connection. So I was reluctant to risk breaking something that was working, and risk loosing connection to Dave's (critical) laptop during our IOP.
After much diagnosis, I realized there was no other option but to reset the white belkin netgear wireless router, and I finally did about 1:30am, and all seems well after that. I reset the connections to the APG systems in the back, and the backlog of latent connections on Nagios began to clear out. (correction- At 2:15am, still several Nagios services reporting as critical, but are wholeskycamera related to network loss.
Hope I didn't screw things up too much- I blame the lack of sleep!
Thanks, Tim