Very clear and cold this morning, with temperatures in the low teens. The rest of the day stayed mostly clear with calm winds before scattered cirrus moved in by the late afternoon. The temperature rose to just above freezing in the afternoon.
The morning data check showed no new data issues, other than a couple of HRXL’s not reporting. We reset the HRXL at CC, SP, and DCS around 10:15. This usb reset seems to be necessary every 1-2 days at these three sites.
Since there were no other pressing issues, we decided to lower the PRS tower to replace the 32m thermocouple and retrieve an EC100 serial number for Dan. We started around 11:15 am and finished before 12:30 pm. Everything looked good when we left the site, but as is common the site dropped off the network shortly thereafter. It came back around 4:40 pm and everything was reporting normally.
In the afternoon we lowered the DCS tower to replace the 7m and 32m thermocouples. Unfortunately we weren’t able to connect to the DSM to verify their function and the tower dropped from the network while we were there as well. Now that the site is back on the network, we can confirm that the new thermocouple at 32m is not reporting. We will plan on replacing it again tomorrow morning and may have to do more troubleshooting if it's not the thermocouple itself that is bad. The previous 32m thermocouple was reporting intermittently and at times with obviously bad data, so it's possible there is another problem at play.
We ended the day after the sounding and a quick data check, in which we found no other issues. Tomorrow we will swap batteries at DC and replace the thermocouple again at DCS.
This morning started fairly warm (upper 20’s) and mostly cloudy, but quickly began to clear from the north. There was light snow reported last night between 2-4 am, but we saw no new snow on the ground in the morning. The day was mostly clear, with periods of low-mid level stratocumulus, and breezy winds.
The Rohn tower at DCS was off the net this morning starting around 5 am, but came back on its own after 9 am, likely due to a modem reset. The HRXL at SP needed to be reset this morning, which we did around 10:20 am. We also discovered that the 7m thermocouple at DCS is broken. Later in the day the 32m thermocouple failed as well. We will replace these along with the 32m at PRS tomorrow, per Sebastian’s request.
In the afternoon we visited UP to connect the NR01 fan, per Dan’s request, and tested a faster method of taking Leica measurements of the CSAT boom angles. We used the GS16 and field controller to measure two points along the boom and one point along the same bearing away from the boom. This process was significantly faster than measuring the points with the multi-station, so hopefully we will get useful angles from it. We repeated this process at PC and SH as well.
Each of the sites we visited today had shown suspicious behavior in snow depth on 23-24 Jan., as reported by Jacquie (https://wiki.ucar.edu/display/CFACTISFS/CFACT+Data+Matrix?focusedCommentId=475761174#comment-475761174), but we could see nothing visibly wrong. We did not further investigate, as the data now look reasonable.
In the late afternoon Liz discovered that the 449 profiler wasn’t working, so she spent the last part of the day (except for when we launched the afternoon sounding) troubleshooting the problem. I finished the day providing moral support as she successfully corrected the issue.
Looking at the DC data, it appears that it lasted for 6 days after the power went out. A day before it died, the battery voltage (Vbatt.dc) dropped below 12.8V. I would use this as a threshold to trigger the need for battery swaps.
In contrast, Vbatt.up and Vbatt.cc (the other solar-powered sites) have never dropped below 13.1. Thus, 2 racks of solar panels seem to be sufficient to keep a station running – albeit without much sensor heating.
Today was another beautiful day in Heber Valley. The temps started off in the teens but as the day went on with mostly clear skies and the sun shinning it warmed up to the mid 30s.
It was Matts first day at CFACT so I decided to give him a day to settle in before taking him around. We just toured ISS1, Base, the sounding site and later went to the storage unit. Tomorrow depending on soil moisture data and if no other major issues arise we will begin the second set of soil samples. Matt also wants to begin taking Leica measurements as well.
Today we listened in on the CFACT daily meeting and learned that we will probably not have any IOPs until next week due to very low to no chances of fog the rest of the week. However, they have just stated no IOPs for the next 24 hrs. to be conservative. At some time this week they will also test out the tether sonde as this has been approved (only to 250ft).
Since Isabel gave me a great refresher on doing ISFS checks I was able to do the same for Matt for a few things plus show him the super cool new DSM dashboard (P.S. I really like this new feature). Today I just had to do a usb reset on HRLXs at CC and DCS and they came back up.
Glad I got to be involved in the ISFS meeting today!
After the discussion of power at DC during the meeting and Dan and I discussing a little more this evening, plus Steve's informative post we decided to stick to battery swaps. Its not much of an issue for us to spend 20-30 mins about once a week to change batteries (drive/swap time), and there is a possibility of the electrician getting back to Dan.
After the sounding today Matt and I went to the storage unit to check for PPE numbers on the EC150s and we were able to find the one Dan is looking for. Matt has the images and will be sending those.
I have glanced at Tsoil and Qsoil values. It seems that after 8 Jan or so the ground started freezing and reported Qsoil values started becoming smaller (since the EC-5 probes don't respond to the dielectric value of ice, only liquid water). At many of the sites, there actually is an interesting large diurnal cycle to Qsoil as water melts and later freezes. In any case, I think gravimetric sampling to compare to the EC-5 Qsoil measurements is hopeless until Tsoil values return to be positive.
It does bring up the interesting question of whether we should continue gravimetric samples to actually measure Qsoil, now that the EC-5 data is hard to interpret. I would ask the PIs. In the past, I've argued that frozen soil moisture isn't going anywhere, i.e. evaporating, so it didn't really matter what Qsoil was. However, in this melt/freeze situation, it is more complicated.
BTW, the values we have been using are volumetric fraction, mislabeled as volumetric %. I've changed the config, but after I started a recalculation of noqc_instrument. noqc_geo should be correctly labeled.
Another beautiful day in the Heber valley. Clear and cold this morning (15 degrees when we left the hotel!) but warmed up and got hazier in the afternoon.
This morning, after some rass troubleshooting at iss1, we went to DCS while it was frozen enough to drive in, so I could bring the ladder and get the last couple EC100 serial numbers. We also checked out the tsoil that had stopped reporting yesterday–visually everything was fine, after unplugging and replugging the binder connections it is again. Glad we didn't have to try burying another one in the frozen ground.
Spent the afternoon showing Liz how I've been checking data for ISFS. Matt is arriving this evening to take over from me.
Heard from the PIs that there will not be an IOP starting tomorrow, so replacing the 32m thermocouple at PRS is not urgent, but will need to happen before the next IOP.
No new data issues today. Reset some HRXLs (sp, dcs, cc–the usual ones), noticed at one point that prsr was off the net but it came back successfully on its own.
For anyone's information, last night I tweaked all the snow depths and boom angles with the measurements so far and reran the 5-minute statistics for both noqc_instrument and geo up to yesterday. Clearly, we don't have a merged data set yet, so there still will be data gaps. The next steps would be to restart the real-time statsproc to use the new cal_files and to recreate the qctables and webplots. I can do this later today.
Low overcast in the morning, burned off to clear by noon.
Started by visiting DC, which had been off the net for 12 hours by the time we checked data. At DC there was no power at the outlet on the fence and the batteries were dead, so presumably the power failed a couple days ago. We talked to one of the people at the stables, who told us the breaker for the power outlet has been tripping since last week, and lately is tripping immediately after resetting it. We unplugged our power cord from the outlet and tried the breaker again, but it still tripped, so we think the problem is somewhere between the breaker and the outlet, not in any of our equipment. Dan is going to talk to the electrician about it. We replaced the batteries at DC with fresh ones, which got it going again. We'll probably have to swap batteries here every couple days until we get the power to the outlet fixed.
After that I took Liz to the last two satellite sites of our tour:
- CC: measured HRXL height
- DC: measured HRXL height. The EC100 box was open when we got there, so either we left it open last time we were there or someone else came to investigate. Either way the EC100 has been reporting data the entire time, the lights are all green, and it doesn't look like any snow got into the box.
This afternoon I visited PRS while Liz did the sounding. I got HRXL height, compass boom angles, one more EC100 serial number, and site photos. Refilled the gas in the generator since I was thinking about it. Got prst back on the net since I was there anyway, but it was just a modem problem so it would have come back up. I did some more digging into the port 7 problem on prsr but didn't really figure anything out, see Jira for more details: https://jira.ucar.edu/browse/ISFS-529
We also got the soil corer working again by doing some filing so the hole in the handle is now large enough to fit the weight rod again.
Talked to Sebastian today, who was wondering about thermocouple status. Last I checked only the 32m thermocouple at PRS was broken, out of the ones Sebastian didn't remove before the snow the other night. Sebastian put all the thermocouples back that he removed the other day (2m and below). We'll want to lower the tower at PRS to replace the 32m thermocouple before the next IOP, which sounds like will probably be starting at noon on Monday.
Not much else to report data-wise. I used usbreset to get the HRXL at dcsr reporting again this morning. As of now dcst is off the net, but if it's just a modem problem it will probably come back up in half an hour. The tsoil at dcs stopped reporting this afternoon, so I tried power cycling the mote with pio, but that didn't bring it back. We'll try to head over there tomorrow.
Morning lightly snowing and low clouds, cleared to mostly sunny and windy by this afternoon.
Heard from PIs that there will be no IOPs for next 72 hours, so the soonest an IOP could start would be noon on Monday. As of today the 32m thermocouple at PRS is broken, so before the next IOP we will need to lower the tower and replace it.
Liz and I took advantage of the down day and had a nice morning checking out the Soldier Hollow nordic center. For future reference: it's $15 for a day pass and $25 to rent skis (classic or skate), and there's a fun variety of trails! We decided it was practice for having to ski in for SOS.
Data checks all looked good this morning, and I haven't noticed anything being off the net. Gary figured out a better way to get HRXL data back than rebooting the dsm, hooray! https://jira.ucar.edu/browse/ISFS-527
Liz and I got the soil corer weight unstuck from the handle but it's still not fully functional, see my comment on yesterday's post.
Low clouds in the morning, gradually clearing as the day went on but remained hazy. Some snow possible tonight and tomorrow.
Heard from Sebastian that the PIs are taking a down day tomorrow, no project meeting. Liz and I plan to take a semi-down day as well. Looks like there probably won't be an IOP before Sunday (don't quote me on that). Sebastian said he had taken down all the thermocouples he could reach (his rake, plus the 1m and 2m ones) at both supersites, but didn't mention anything about wanting us to take down the others.
In the morning Liz and I visited a bunch of satellite sites so she will know how to get to them:
- SH: measured HRXL height
- MW: got boom angle, measured HRXL height, said hi to the horses
- UP: measured HRXL height, checked on status of power drop. Nothing has happened since our last visit.
- PC: measured HRXL height
- MH: measured HRXL height, got sonic serial numbers + sonic height
Went to DCS to do a soil sample in the late afternoon while the soil was warmest (ha). The good news: got the sample, which completes round 1 of soil samples. The bad news: got the rod + weight part of the soil corer stuck in the handle:
I had left the rod + weight on while using the sledge on it because I hadn't wanted to bang up the top of the pipe too much, but apparently this was worse. I think either the rod is slightly bent or the top of the pipe got beaten out of shape enough that the rod no longer comes out. Liz and I might spend more time trying to get it unstuck tomorrow.
Also while at DCS I got some boom angles, site photos, and the HRXL height.
Nothing else new to report data-wise. Netcdfs stopped generating on barolo and Steve fixed by restarting dsm_server, and MW and and UP went off the net but came back up on their own. Nidas didn't start at MW after reboot until I logged in and started it manually, see jira: https://jira.ucar.edu/browse/ISFS-533. HRXLs at SP, CC, and DCS were offline at some point. I just rebooted these dsms and they're now back (for the next however long), so they can measure the potential snowfall tonight.
We had always wanted the EC150 to be operating in automatic heater mode. According to our SE wiki entry, the heater volts should be set to -2. One (7m.dcs?) that I checked several days ago seemed to be frosting up and had the setting at -1, which I don't know the meaning of.
Just to be sure, I've just reset all 3 levels of both TTs to be -2 via:
rs x
<cr><cr><cr>
P<cr>
2718<cr>
S<cr>
M -2 <cr>
Y<cr>
I used a really wide screen and looked for ASCII messages at the end of the lines to confirm my entries, but basically you are typing in the blind (and have to do this quickly before the EC150 times out) while binary messages are streaming by.
If we see frosting-up behavior on other EC150s, we can repeat this on the other sensor positions.
Sunny and clear in the morning, but getting cloudier as the day went on. Cool and windy by the late afternoon.
IOP starting at noon today (though it was not officially decided on until after noon today).
Data check looked good this morning, without even any broken thermocouples. At some point today I rebooted dcsr and cc to bring back HRXLs. pc was off the net for a couple hours this afternoon but seemed to come back fine.
Got a ubiquiti link working between the elementary school and the base trailer today. Internet is now much faster, if you use ISS-GUEST rather than the 5G modem's wifi network.
Not much to report, I mostly took the day off!
Data looked OK this morning except that we had the same problem with netcdfs not being generated on barolo. Restarting dsm_server and statsproc once again fixed that issue. PC and SP both went off the net during the day but came back after the 4hr timeout in the dsm_watch_network script, so it must have been a modem problem. SP once again didn't have nidas running after it rebooted, see jira: https://jira.ucar.edu/browse/ISFS-533
Rebooted cc and dcsr to bring back HRXLs this evening. Now all HRXLs are reporting but we'll see how long that lasts.
Potential of an IOP tomorrow but conditions are uncertain so they are delaying making the call until tomorrow. Not ideal for us since it's not a lot of notice, but in this case all thermocouples are still working and there are no other sensor issues, so we don't have much to do to prepare.
Some high clouds and sunny.
Morning data check showed no data from HRXLs at cc, sp, dcs. Rebooted the dsms which brought the HRXLs back to working at all 3 sites (for now). Also this morning updated the dsm_watch_network script to reboot the dsm after 4 hours off the net instead of 12, to make it easier to keep an eye on data.
This morning Liz and I went to PRS so I could test out my fix for the port problems on prsr (and to pay a visit to the cows). With the udev rule set so the HRXL comes up as ttyHRXL port 0 is back to functional, so plugged the 2m ec100 back in to prsr. Tried the ott in port 7 on prsr again, but still not getting data from it through that port, even with dsm_port_config showing that the port is set correctly. So ott is still connected to prsg, port 6.
This afternoon visited SP and took site photos, boom angle, sonic height, serial numbers, and a soil sample. Pretty hard to get soil cores from the frozen ground but Liz got it done with the sledge!
No IOP tomorrow but there's talk on slack of a possible IOP on Wednesday.
Mostly clear with some haze, colder morning than the last few. IOP starting at noon today looking at non-fog cases.
Liz arrived yesterday and joined us this morning.
This morning Liz and I visited DCS to replace the broken thermocouple at 7m before the IOP started. We took advantage of the cold morning to drive most of the way to the site, so we could refill the gas in the generator. Tower lowering and raising went smoothly and everything is reporting after replacing the broken thermocouple. Tower was down between ~10am and 11:45 local. It looks like the missing data problem from the 7m EC150 resolved itself while we were lowering the tower and was likely just frost (sorry Steve). Got some more ec100 serial numbers while the tower was down. Everything was good when we left the site but dcst has not been on the net since 10:15 this morning (while we were on site–but I can't think of what we did that would have an effect on the modem). Assuming it's a modem problem and will come back up later tonight, since I was logged in and checking data after it went off the net.
Memorial hill was off the net this morning when I checked data, but came back on its own before I had time to visit it.