Today was mostly clear with periods of high clouds and quite warm temperatures. Some sites saw fairly strong winds throughout the day, while it was very windy in the afternoon and evening in town. This morning all sensors were operating nominally, other than the few known issues (S16 IRGA, S11 Victron, S9 TP01). Today I collected soil samples, measured boom angles, and cleaned radiometers at S2 and S4. Tomorrow I will continue doing round 2 of soil samples.
Today we transitioned from EOP 1 to IOP 4. The ISFS network was in good shape, so it was another light day. Gary remotely rebooted S18, which resolved the usb disk issue. I visited S5 to troubleshoot the sonic and generally check out the station. A simple power cycle on the port seemed to do the trick and we have been getting reasonable data since. I also re-tensioned some of the guy wires, so the tower is much more plumb than we left it. Before I left I remembered to measured the boom angle, which was 136 magnetic. Later in the day I collected a soil sample at s1 and measured the boom angle at S3 at 136 magnetic.
Everything else behaved pretty well today, but Steve did notice that the TP01 at S9 is reporting likely bad data. I will consider replacing it if I make it to S9 for soils in the next few days.
Today was EOP 1, which is planned to continue directly into IOP 4 tomorrow, leaving me without support from the ISS crew. Fortunately all ISFS systems are functioning reasonably well and today was a light day. Some ongoing issues are summarized below.
- S5: CSAT ldiag remains at 1 since we raised the tower. I will look for another spare in the Pod tomorrow, as I suspect we need to swap the sonic to fix this issue. I may visit the site tomorrow to visually inspect things.
- S11: No Vbatt.
- S16: Bad CO2.
- S18: According to Nagios the raw_data has not been rsync’d in >7 hrs. It also seems like the usb disk is not being found on the DSM - confirmed by nothing returned by lsu command. It seems like another visit to s18 is in order to reboot the DSM.
The weather today started mostly sunny, quickly changed to overcast with drizzle, before clearing and becoming breezy in the afternoon. All sites except s17 measured some very light rain today.
During the morning overcast and drizzle, Liz and I set out to finish repairs at s5. We dressed cables on the EC150 boom, again inspected the tower, and raised the mast before noon. We had one 70 pound sandbag on the northwest tripod leg while raising and the leg barely budged, but added a 2nd sandbag to further secure it and could add a 3rd as well if necessary. We also had to re-tension all guy wires and added zip ties to help keep them secure. At this point the tower is up and reporting data, however since then there have been no CO2, H2O, or CSAT winds coming in. I hope this is due to the weather, since the ldiag is 1 and it was precipitating when we left the site.
After breaking for lunch, we visited s18 to troubleshoot the NR01 that stopped reporting a few days ago. Everything looked visibly normal, so we reseated and loosened the NR01 cable to no avail. However, a power cycle of the mote brought the radiation back up. This time we saw about a dozen goats - so the streak of seeing new animals at every visit continues.
I had noticed that s4 was offline and unreachable since sometime this afternoon, so we paid the site a visit at the end of the day. I could not connect via usb cable, so I connected to the WiFi instead. Data was coming in and looked normal, save for nothing coming from the NR01. A couple of mote power cycles seemed to do the trick, but the site was still off the network when we left. It is now back up, as of 10 pm and the NR01 is reporting.
A few other notes:
- S16: TRH seems to have been reporting NA’s during the middle of the day, but is now alright.
- A scattering of NA’s across several variables at multiple sites, likely due to the weather.
- Jacquie left this morning and I'd like to thank her for all the help during the first week of my ops shift. Thanks Jacquie!
Our initial plan for the day was to investigate why s5 fell off the net yesterday evening and check on the radiometer at s18. Unfortunately, we never made it to s18.
As it turns out, the tripod at s5 must have blown over in the ~20 m/s winds it faced yesterday. Steve does not recall this ever happening before, but the stakes on the “northwest” leg pulled right out of the fairly loose, sandy soil. Upon visual inspection the only damage was to the EC100, which was pretty smashed after landing on a rock. I suspect the fall may not have been all that violent, since I found no other signs of damage. That said, we spent the afternoon replacing the CSAT, EC150, EC100, and Gill and lugging supplies up and down the mountainside.
After some discussion, we secured the footpad with a 70 pound bag of sand and will add more tomorrow when we have the energy to haul them up the hill. We can consider adding a set of guy wires secured to screw anchors, but for now I think the sand will be sufficient. We left the tower lowered and resting on the mast stand, but powered on so that I could check the data remotely. Tomorrow we will visit the site again to dress cables, raise the mast, and further secure the footpad.
This evening I also noticed that the Vbatt for s11 has been NA all day, but not the rest of the site. Restarting Nidas remotely did not recover the Victron stream.
No new issues this morning, so Jacquie and I decided to take care of a couple of small tasks and begin round 2 of soil sampling.
- S18: Reseated wetness sensor and radiometer cables. The wetness cable did not appear loose, but the radiometer cable did. Either way we could see an immediate jump in wetness values, suggesting the problem is fixed. We took a soil sample and admired the parade of farm animals wandering the site - steer, llama, and two goats. Unfortunately, this evening it appears that the radiometer is out. The soils are fine, so there could be a problem with the radiometer cable. It was very taught and barely made it to the mote, so perhaps it got pulled out in this evening’s high winds?
- S10: At Isabel’s suggestion we swapped the Victron data cable to the DSM. Restarting Nidas brought up the data stream and it has apparently remained up. Fingers crossed that it is fixed. We also took a soil sample.
- S9: Took a soil sample.
- S5: Off the network after 1930 PDT. May visit tomorrow to check it out and take a soil sample.
Strong winds developed across some of the western, coastal sites in the late afternoon. Winds reached ~20 m/s at s5 before it dropped off the network.
IOP3 began today, with no glaring issues across the array. Jacquie and I had an easy morning and handled the 4 and 7 pm radiosonde launches at ISS2.
Ongoing issues:
- S10: The Victron continues to drop out regularly, but it also comes back up by restarting Nidas remotely (done at 20:50 tonight).
- S18: The wetness sensor needs to be reseated. We will visit tomorrow afternoon, where we will also start the 2nd round of soil samples.
- S16: CO2 is bad more often than not. Jacquie will show me how to replace the desiccant on one of our spare EC150's at base so that I can do it at some point in the future.
This morning’s data check found no issues, save for the power monitor at s10 being offline again. Steve restarted it with a ddn/dup this afternoon, but it appears that swapping the Victron may have only helped resolve part of the problem. As long as we can bring it back up remotely, I don’t see any reason to mess with it further.
Jacquie went with Lou to troubleshoot the profiler at ISS3, leaving me to work on the Leica at the base. The GPS antenna is still unable to attain a reasonable level of accuracy, improving to only 3-4m in the horizontal after more than an hour. I also still can’t upload the necessary geoid files to create the proper coordinate system for this location. A half day of troubleshooting yielded little progress and ended with multiple user support tickets being sent. Hopefully they will be able to help.
The forecast calls for continuing strong winds over the next few days, with conditions favorable for a potential western Sundowner tomorrow, so Bill and I spent a few minutes securing loose containers/lids and foam pads that could blow around in stronger winds.
An IOP has been called starting tomorrow (Wednesday). Jacquie and I will be doing the 4 and 7 pm soundings from ISS2, and may visit s18 to reseat the wetness sensor.
Saw that s6 wetness has been reading about 0.004 the entire project. Just reloaded its calibration coefficients (which would include the Wetness gain) as:
@NR029:COF 17.55e-6 0.0 0.0 0.0 19.39e-6 0.0 0.0 0.0 14.12e-6 0.0 0.0 0.0 13.87e-6 0.0 0.0 0.0 0.00335061106138849 0.000277187139537418 -9.71375961660885e-07 4.97046649968123e-06 0.0 1.0 0.0 0.0
and now get wetness values of about 0.25. From before and after readings, it appears that the A/D counts were the same, thus we could reapply this coefficient in post analysis if desired. Easier would just be to apply a gain adjustment in cal_files where before 0.003928 was from A/D counts of 16066 and after 0.251624 came from 16081. I.e., need a gain of 64 prior to now (15:30 PDT 12 Apr).
Another fairly light day today, as Steve left in the morning. Thanks Steve for sharing your knowledge during our overlap!
No pressing issues in the morning data check, other than NA’s coming from the sonic at s12 and later resolving itself, so we decided to swap the Victron at s10 and visit s9 on the way. During these visits the weather was pleasant and skies mostly clear, but we ended up having localized areas of clouds and light to moderate precipitation in the afternoon. Most of the western sites measured rainfall while none of the eastern sites did. Strong winds developed in the afternoon in advance of a cold front, which dropped temperatures quite a bit by the evening. As the winds picked up they blew around some unsecured plastic tub lids at the ISFS base that Bill had to clean up. We will have to secure the site in advance of more winds forecast the next few days.
S9:
- Cleaned radiometer 1036-1038 (Note that fan sounds loud)
- Measured boom angle twice, once from in close and once from the road - both 316 magnetic out from the boom or 136 into the boom.
- One turnbuckle nut has loosened completely, so I retensioned the guy slightly and tightened the nut. We also added zip ties to the turnbuckles.
- Noticed that the lightning rod is loose. We should tighten the next time we drop this tower and should be careful to not stand under the tower when it’s windy.
S10:
- Cleaned radiometer at 1120
- Measured sonic orientation at 137 magnetic into the boom
- We found that the current Victron data cable had a loose connection due to part of the port having broken off. We swapped out this one with the spare and all seems nominal. We shutdown the DSM at 1128 to replace Victron.
Just to mention that I've now looked at the radiometer cleanings we've done at s1, s8, and s18 over the past 2 days and cannot discern a different between the readings before and after the cleaning. Thus, the data before the cleanings can be used without any correction.
Mostly took today off, in prep for IOP tomorrow.
s18: (orientation for Matt)
- guy tension okay. added cable ties
- has been working fine with a 1A fuse, but changed to 3A (didn't have a 2A)
- took site photos (finally finishing this task!)
- cleaned radiometers at 1600
- shot boom angle: 315magnetic
s10: At the beginning of the day, found the Victron output dead as usual and got it back by ddn/dup. (Recall from a post several days back that upgrading the firmware on this Victron and checking that the USB cable was seated didn't help.). In the afternoon it was dead again, but didn't respond to two different ddn/dups. Getting worse...
I leave tomorrow. Thanks for all the fish!
Orientation day for Matt. Jacquie went off with Bill. Isabel ended her shift. Thanks for everything, Isabel!!
No particular agenda, so...
s1:
- Quite windy (12m/s) at the beginning of the visit, with whitecaps near Gaviota. Died down a bit toward the end of the visit
- Tried to use GPS for boom angle, but Leica had large errors. Matt will work on this in the coming days.
- Cleaned radiometers at 10:45
- cable tied turnbuckles, but were okay
- In lieu of the GPS, shot boom angle of 303magnetic, and phone says declination is +12degrees
s12:
- tightened turnbuckles 2.5 turns, then resecured locknuts and cable tied
- releveled manfrotto (pitched down a few degrees)
- shot boom angle 321magnetic, and confirmed with shots from back, front, and side (tripod lower brace)
- pushed back in the adhesive square for the FTDI board and propped with a stick(!)
s8:
- one inner guy turnbuckle was loose. Resecured and tightened. Cable tied outer guys
- shot boom angle 317magnetic
- cleaned radiometers (about 1600(?))
s6:
- turnbuckles fine. Cable tied outer.
- radiometer fine.
- boom angle 315magnetic
We're getting the idea that some stations were set up at NW magnetic and some NW true. It doesn't really matter as long as we know what it was.
A light day, introducing Jacquie to the ISFS set-up:
s15: (revisiting to check on tower issues)
- Found the last turnbuckle nut, so retensioned guys and secured all turnbuckles with locknuts (wrench tight). Then added back cable ties for safety.
- Were going to drop the mast to reinstall the lightning rod, but found that the screw holding it on itself was snapped. Thus, couldn't fix and thus left mast in place.
- Propped the disconnected Gsoil up on a rock to protect the electrical taping
s14: (test of EC150 dessicant bottle replacement)
- Dropped mast from about 13:30-14:00
- Cleaned a bunch of bird droppings, including from the bottom EC150 lens
- Changed dessicant bottles
- Afterwards, have noticed a small step change in the EC150 readings (the bird cleaning), followed by a slow drift to larger value (the effect of the bottle?). We'll have to see how the h2o and co2 values change in the coming days. The dessicant is supposed to take 24hrs to completely settle.
- Also cleaned radiometers.
Matt arrived in the evening.
Stephan de Wekker drove by s11 and noticed that it (like s9 did last week) blew off vertical. He straightened it and we'll visit today to secure it better, but we'll have to determine when this happened (undoubtedly during the Sundowner event of IOP 1/2) and remove the data.