Overcast in the morning with sunshine in the afternoon and a high in the mid-60s.
Morning check of the systems showed that data from the lidar at ISS1 was not being reported. I went over in the late morning to investigate. The lidar laptop was responding slowly, however response picked up when the lidar's Ethernet cable was plugged directly into the laptop's USB to Ethernet dongle. Slower response was seen when the Ethernet cable was reconnected/rerouted thru the switch. No real reason for the slowness through the switch is known. Even though the laptop was slow to update, the lidar was deemed to be running. The data error was seen to resolve itself with time, though it has reappeared late in the day again.
One thing I did notice while at the site was a lack of lights flashing on the (blue) Ethernet cable on the switch that ran to/from the data manager. I found that the connector was loose in its socket because the plug on the end of the cable is missing it's snap-in retainer. I replaced the cable with a (white) cable found in the long Ethernet cable tub in the back of the trailer and a solid connection was seen.
Finally, it looks the the WV sensor at ISS2, which looked good all day today, is acting up again when systems were checked after dinner tonight.
I can head back out to both sites tomorrow to check things again, though not sure what I can do besides check cables and/or power cycle instruments.
Clear and sunny day with a high in the mid to upper 60's and some late haze forming along the coastal region.
Nagios showed an issue with the WV sensor at ISS2. It had gone offline yesterday for several hours before coming back late in the day, then was off again today. Laura headed off with Chris to s16(?) while I checked out the WV issue.
I stopped at ISS1 on my way out and found an Nagios issue with the ceilometer. I checked the network cable connections and power cycled the ceilometer by unplugging and re-plugging its power cable. Neither cleared the error so Bill was texted. Bill did some probing and the system came back online with no clear reason as to what the issue was.
I also notice a message window on the lidar laptop saying it (the laptop/client?) had lost communication with the server. I noticed that some of the cables plugged into the Ethernet switch were not sitting in their jacks square due to lateral strain from the relatively inflexible network cable coming in from outside (I believe it was the DSM Ethernet cable). A similar problem was seen at ISS1 in CFACT (reported in Ops 2/15) and I suspected the same issue. I pulled the Ethernet switch off the wall and set it on the countertop surface in a manner that there is no lateral forces on the Ethernet cables plugging into the switch (image below). I also re-plugged all the Ethernet cables into the switch. Shortly thereafter, the communication error on the lidar laptop went away. I may revisit the site tomorrow and move the two construction screws that hold the switch onto the wall over to be above the hole in the countertop where the rigid Ethernet cables come in from the outside, to eliminate any cross force on the plugs.
Ethernet switch on the countertop after re-plugging connections to remove lateral forces. The black Ethernet cables with the red plug is an outdoor cable (used for the DSM, I believe) and was pulling sideways on the switch and may have been causing connectivity issues. I may remount the switch on the wall above the hole in the countertop to alleviate the issue and get the switch back off the countertop surface.
Mostly clear but hazy today.
Laura and I visited iss2 and iss3 today. At iss2 I gave Laura a tour of the site and a refresher on the MW41 system. I also backed up the sounding system config for this project. At iss3 I backed up the sounding system config and got the temperature probe working again (I had to power cycle it and clear the log, which was taking too long to download).
I leave tomorrow, so this is the last blog post from me!
Very hazy today, especially on the coast. Windy and cool at Rancho Alegre.
IOP 6 went very well, all launches were successful. We launched on a modified schedule, at 10, 1, 4, and 5:30. Chris is now an expert balloon launcher. Chris also scavenged enough horseshoes for us to play a couple rounds in between soundings!
Laura arrived today, and tomorrow we'll visit iss2 and do a refresher on soundings. I'm also planning a visit to iss3 to look at the temp probe. No IOP planned for tomorrow or Saturday.
Very hazy today, getting gradually more overcast as the day went on.
IOP is on for tomorrow with modified schedule–our last launch is at 5:30, not 7. Chris and I will take care of the launches tomorrow.
All quiet for ISS today, so I went to Gaviota Peak with Chris for a maintenance visit.
Hazy and warm inland, more and more overcast as the day went on. Windy further west along the coast.
John arrived today, Lou leaves tomorrow. No IOP tomorrow but likely will have one on Thursday.
I went to iss3 to look into why we weren't getting ceilometer and all sky cam data. When I arrived at the site the ceilometer computer was on and responsive but somewhat sluggish, with something like 5 copies of each of the ceilometer programs running. Rsync server was not responsive. I tried to open task manager to kill rsync server but got this error message:
Tried restarting and shutting down the computer from the start menu, but both times after you confirmed shutdown/restart it just stayed running. Eventually power cycled via the power button. After that I started the ceilometer software and tested that rsyncs to the data manager work.
Unfortunately it looks like the ceilometer software was not saving data while the rsyncs weren't working, so we lost ~17hrs of data.
The iss3 container was full of earwigs which were getting in through a gap in the edge of the screen of the window behind the sounding computer. I taped over the screen edges which will hopefully keep them out.
Warm, calm, and some high cirrus at Rancho Alegre today. EOP 2 went well, with all balloon launches successful. Saw some deer and turkeys on an evening walk. There's a tree down across the road to the site (as of this afternoon), but it's not hard to drive around it.
No IOP or EOP expected for tomorrow or Wednesday.
After getting back to the hotel this evening noticed in nagios that the data from the ceilometer and all sky camera at iss3 stopped a couple hours ago. I can still ping the ceilometer pc, but rsyncs from it fail. I'm guessing rsync server stopped (either on its own or because the computer rebooted). It's possible the ceilometer computer is still collecting data and just not transferring it, but if the computer rebooted or hung I'm not sure that's the case. I'm planning to visit iss3 tomorrow to get it working again.
Another clear and warm day. All good at all three sites this morning when I checked nagios.
Lou went to iss3 to drop off some cleaning supplies for teardown. I spent the day with Chris going to Little Pine for a maintenance visit.
Clear today, calm in the morning and getting windier through the afternoon. Morning data check showed all good at all 3 sites, and we didn't hear about any problems from the students at iss3. Our launches at iss2 all went smoothly.
Possible EOP on Monday, but that won't be decided until tomorrow.
Another quiet day for ISS. No issues at morning data check so worked from hotel. IOP is on for tomorrow so we will be launching at iss2.
In the afternoon helped Mack go recover a dropsonde off the ridge near Romero Saddle!
A quiet day for ISS. I arrived yesterday evening, Liz left this morning. Thanks Liz!
Morning data check showed no issues, so Lou and I spent the day working on other stuff from the hotel. The potential IOP for Friday has now been pushed back to Saturday.
This morning Nagios showed that the ISS3 sounding computer was down so we suspected it's the same issues we have had with the UPS. When Lou and I got there the UPS was off so I turned it back on and everything booted up fine. We had stopped at the ACE and grabbed a digital clock to plug into the UPS to see what the time difference is the next time. There had been no indication of a power outage as the other electric clock in the trailer read the correct time.
Next we went to ISS2 since the plots had not updated for the CL51. We went to make sure the files were still saving since Nagios did not indicate any issues with data. Once we verified the files were still saving, I called Isabel to help fix the issue. She found that her work the other day created a bug in the script and she was able to get everything fixed.
I head back to Boulder tomorrow and Isabel safety made it to Santa Barbara to take my spot.
Today was warm with high clouds and some wind later on.
Last night the 4am and 7am launches did not happen due to miscommunications and LAX not approving the launches.
This morning while checking Nagios I noticed the all sky camera and CL31 at ISS3 had not reported data for a while. Lou and I spent the morning there troubleshooting. It seemed as if the computer was frozen or working extremely slowly, so we restarted it. I also did a hard restart on the CL31. We realized we both didn't quite remember how to restart the CL31 software so we contacted Gary and Isabel. Between the both of them and us physically in the trailer we were able to resolve the issues and data had started coming back in.
ISS2 and ISS1 do not have any pressing issues. There is a hard drive filling up (81%) at ISS1 for the profiler however.
Today started out clear and calm in the morning with clouds moving in in the afternoon. Hasn't been much wind at all other than a very light breeze every so often.
Today ended EOP1 and IOP4 started.
One of the students notified us that the outside light of ISS3 burned out last night, so I went and replaced it after the first sounding. Other than that all systems are running well.
Sunny with very few high scattered clouds throughout the day. There were some slight breezes throughout the day but became all calm by evening.
Today we held EOP #1 which will lead into IOP #4 tomorrow. The PI's are wanting to see the transition into sundowner wind conditions, so essentially balloons will be launched for a 48hr period with the same sounding schedule we have been doing.
Everything for ISS ran smoothly with no issues arising.