Today was much like yesterday, but even warmer, with mostly clear skies and temperatures rising into the mid 40’s. The winds were calm most of the day, but did begin to pick up in the late afternoon ahead of the approaching system.
The morning data check showed an issue with soils and radiation at DCS, in particular the tsoils were missing and the other variables from DCSG had dropped out at some point in the morning. We initially thought it was the typical modem problem, however further investigation revealed that all affected variables were from sensors running through the soil mote. We decided to change the PIC board for the tsoils because we have had some previous issues with that sensor, however we will keep an eye on this site in case we need to swap out the mote. We changed the tsoil PIC from TS17 to TS23 around 12:15 pm, which will require changing coefficients at some point.
Otherwise we spent the day showing John some of the ISFS peculiarities and tools for checking the data. We also visited DC to check on the work the electrician was supposed to have finished, but were disappointed to find out that nothing had been done. For now the site is still on battery power, but the charge seems to be lasting much longer now that we reseated the thermocouple.
Finally, we requested a quote from a place in town to fix the broken weld on the soil corer. Hopefully we can get it turned around pretty quickly. It should also be noted that Sebastian removed all the thermocouples from both PRS and DCS below 2m this afternoon.
Today was the warmest day we’ve had in the past week, with temperatures warming into the upper 30’s during the day, under clear skies and calm winds. There was a bit more haze today however, especially concentrated in the BL when looking towards Provo Canyon.
The morning data check showed no issues, so we decided to take advantage of the nice weather to visit a handful of sites for soil samples. We were able to collect samples at SH, PC, UP, and DCS, although it was quite the challenge at DCS because of the hardness of the frozen soil. We managed to break the weld on half of the handle with the sledge hammer, but the corer is still usable. This does beg the question as to whether or not continuing to collect soil samples in the extremely frozen ground is worth the effort and breaking equipment. Despite this question, we did buy new rags for the soil kit and cleaned/organized both soil tool boxes, which was long overdue.
Liz also spoke to Sebastian today, who informed us that we will not need to remove the thermocouples from the telescoping towers due to the expected weakening of this week’s forecast weather system. Finally, Liz will be picking up John at the train station in Provo tonight and we will begin showing him the ropes tomorrow.
The weather today was almost exactly the same as the past few days, i.e. cold, clear, and calm. The morning data check showed no issues and no HRXL problems, which means the script that Gary wrote to automatically reset the usb seems to be working. Thanks for that Gary!
Since there were no issues to attend to, we decided to spend the day visiting a few of the sites I had not yet been to so that we could take Leica measurements and soil samples. We started at SP, where we successfully measured the boom angle, but could not take a soil sample. The ground was very frozen and we may have hit a rock when driving in the corer, as its cutting edge bent. We will try to fix it as best we can, but at least we have the corer from the other soil kit. We were able to take a soil sample at the next site we visited, MW, as well as measure the boom angle there.
In the afternoon we took a trip to DC to troubleshoot the increased battery drain we’ve been noticing. We swapped batteries and reseated the thermocouple wire because it had not been reporting. We were not able to pinpoint why the site keeps draining batteries so quickly. In comparing the load between all sites, I am curious as to why some sites show an increase in load during the day (DC, MW, UP), while others show an increase over the night (CC, LC, MH, PC, SH, SP). I would expect the increase overnight if heaters are turning on, but am not sure what would cause the opposite behavior at certain sites.
On our way back to the base we drove past CC and LC so that Liz could show me where to find them. After the daily sounding we walked to DCS to troubleshoot the tsoil sensor that stopped reporting. We tried reconnecting the cables, as Isabel and Liz previously had restored them at DCS in this way, which fixed the problem.
We also saw a "change engine oil soon" message in the ISFS truck today.
This morning was a bit colder than the previous few, having temperatures in the single digits when we left the hotel, with clear skies. The daytime high temperature again peaked just above freezing.
The morning data check showed no issues, save for the usual HRXL not reporting at CC, SP, and DCS. We reset them around 9:45 am. We then took a quick break for some cross-country skiing at Soldier Hollow, in preparation for next winter’s SOS campaign.
In the afternoon we met with Sebastian to join in on some tethered balloon training, as well as lead a radiosonde training session for a few students. We also noticed that the batteries we installed at DC yesterday morning were being drained quickly, down from 13V to 12.5V in 24 hours. Previously at the site the batteries took about 3.5 days to decrease that amount. We had hoped this would be a non-issue, as the electrician should have restored the power to the site today. Unfortunately, when we stopped by DC at the end of the day to plug in the site and swap batteries we found that the work had not been done. Hopefully this battery swap will last longer then yesterday’s.
In the evening, I noticed that ncharts, qctables, and web plots were not updating (as of 2pm or so), but all the sites were reporting data. I believe this is a dsm_server issue on barolo and emailed Steve and Isabel to restart the process.
Assuming all goes well, tomorrow we are planning to visit a couple of the remaining sites that I have not seen yet in preparation for Liz’s return to Boulder. We will measure boom angles and collect soil samples as time permits.
Another clear and cold morning today, with temperatures again in the low teens. The rest of the day was very similar to yesterday, with beautiful clear skies and a high around freezing. Winds were mostly calm, but increased in the afternoon. The weekend looks to be a bit warmer and sunny, before the next weather system moves in around Tuesday.
As is customary at this point, we had to reset HRXL’s at CC, SP, and DCS first thing in the morning. The morning data check showed no other new issues, but the battery at DC had dropped to near 12.5V, necessitating a battery swap. While we were there, Liz helped troubleshoot the electrical issues with the electrician and I took a soil sample and measured the boom angle with the Leica GPS antenna. The ground was quite frozen and covered in about 6 inches of snow. For the electrical, it turns out there was a short at a junction outside of the barn. The electrician planned on bypassing that section as a temporary fix, so we will hopefully have power at the site restored by the time of this blog post.
In the afternoon we returned to DCS to troubleshoot the thermocouple at 32m. Unfortunately the problem was with the thermocouple itself, forcing us to lower the tower to replace it. This did the trick and we are now getting data. We only have one spare thermocouple left at that site, but there is discussion about removing all the thermocouples from the towers before next week's forecast snow, which would alleviate this concern. We also decided to dress the cables at the tower to facilitate lowering and raising the tower. Thanks to Liz for doing much of the untangling and taping despite freezing hands! This should make life much easier for future visits to DCS.
We wrapped up the day with the 4:15 pm sounding and a data check, finding no issues. Tomorrow is a down day for the project.
Very clear and cold this morning, with temperatures in the low teens. The rest of the day stayed mostly clear with calm winds before scattered cirrus moved in by the late afternoon. The temperature rose to just above freezing in the afternoon.
The morning data check showed no new data issues, other than a couple of HRXL’s not reporting. We reset the HRXL at CC, SP, and DCS around 10:15. This usb reset seems to be necessary every 1-2 days at these three sites.
Since there were no other pressing issues, we decided to lower the PRS tower to replace the 32m thermocouple and retrieve an EC100 serial number for Dan. We started around 11:15 am and finished before 12:30 pm. Everything looked good when we left the site, but as is common the site dropped off the network shortly thereafter. It came back around 4:40 pm and everything was reporting normally.
In the afternoon we lowered the DCS tower to replace the 7m and 32m thermocouples. Unfortunately we weren’t able to connect to the DSM to verify their function and the tower dropped from the network while we were there as well. Now that the site is back on the network, we can confirm that the new thermocouple at 32m is not reporting. We will plan on replacing it again tomorrow morning and may have to do more troubleshooting if it's not the thermocouple itself that is bad. The previous 32m thermocouple was reporting intermittently and at times with obviously bad data, so it's possible there is another problem at play.
We ended the day after the sounding and a quick data check, in which we found no other issues. Tomorrow we will swap batteries at DC and replace the thermocouple again at DCS.
This morning started fairly warm (upper 20’s) and mostly cloudy, but quickly began to clear from the north. There was light snow reported last night between 2-4 am, but we saw no new snow on the ground in the morning. The day was mostly clear, with periods of low-mid level stratocumulus, and breezy winds.
The Rohn tower at DCS was off the net this morning starting around 5 am, but came back on its own after 9 am, likely due to a modem reset. The HRXL at SP needed to be reset this morning, which we did around 10:20 am. We also discovered that the 7m thermocouple at DCS is broken. Later in the day the 32m thermocouple failed as well. We will replace these along with the 32m at PRS tomorrow, per Sebastian’s request.
In the afternoon we visited UP to connect the NR01 fan, per Dan’s request, and tested a faster method of taking Leica measurements of the CSAT boom angles. We used the GS16 and field controller to measure two points along the boom and one point along the same bearing away from the boom. This process was significantly faster than measuring the points with the multi-station, so hopefully we will get useful angles from it. We repeated this process at PC and SH as well.
Each of the sites we visited today had shown suspicious behavior in snow depth on 23-24 Jan., as reported by Jacquie (https://wiki.ucar.edu/display/CFACTISFS/CFACT+Data+Matrix?focusedCommentId=475761174#comment-475761174), but we could see nothing visibly wrong. We did not further investigate, as the data now look reasonable.
In the late afternoon Liz discovered that the 449 profiler wasn’t working, so she spent the last part of the day (except for when we launched the afternoon sounding) troubleshooting the problem. I finished the day providing moral support as she successfully corrected the issue.
Looking at the DC data, it appears that it lasted for 6 days after the power went out. A day before it died, the battery voltage (Vbatt.dc) dropped below 12.8V. I would use this as a threshold to trigger the need for battery swaps.
In contrast, Vbatt.up and Vbatt.cc (the other solar-powered sites) have never dropped below 13.1. Thus, 2 racks of solar panels seem to be sufficient to keep a station running – albeit without much sensor heating.
Today was another beautiful day in Heber Valley. The temps started off in the teens but as the day went on with mostly clear skies and the sun shinning it warmed up to the mid 30s.
It was Matts first day at CFACT so I decided to give him a day to settle in before taking him around. We just toured ISS1, Base, the sounding site and later went to the storage unit. Tomorrow depending on soil moisture data and if no other major issues arise we will begin the second set of soil samples. Matt also wants to begin taking Leica measurements as well.
Today we listened in on the CFACT daily meeting and learned that we will probably not have any IOPs until next week due to very low to no chances of fog the rest of the week. However, they have just stated no IOPs for the next 24 hrs. to be conservative. At some time this week they will also test out the tether sonde as this has been approved (only to 250ft).
Since Isabel gave me a great refresher on doing ISFS checks I was able to do the same for Matt for a few things plus show him the super cool new DSM dashboard (P.S. I really like this new feature). Today I just had to do a usb reset on HRLXs at CC and DCS and they came back up.
Glad I got to be involved in the ISFS meeting today!
After the discussion of power at DC during the meeting and Dan and I discussing a little more this evening, plus Steve's informative post we decided to stick to battery swaps. Its not much of an issue for us to spend 20-30 mins about once a week to change batteries (drive/swap time), and there is a possibility of the electrician getting back to Dan.
After the sounding today Matt and I went to the storage unit to check for PPE numbers on the EC150s and we were able to find the one Dan is looking for. Matt has the images and will be sending those.
I have glanced at Tsoil and Qsoil values. It seems that after 8 Jan or so the ground started freezing and reported Qsoil values started becoming smaller (since the EC-5 probes don't respond to the dielectric value of ice, only liquid water). At many of the sites, there actually is an interesting large diurnal cycle to Qsoil as water melts and later freezes. In any case, I think gravimetric sampling to compare to the EC-5 Qsoil measurements is hopeless until Tsoil values return to be positive.
It does bring up the interesting question of whether we should continue gravimetric samples to actually measure Qsoil, now that the EC-5 data is hard to interpret. I would ask the PIs. In the past, I've argued that frozen soil moisture isn't going anywhere, i.e. evaporating, so it didn't really matter what Qsoil was. However, in this melt/freeze situation, it is more complicated.
BTW, the values we have been using are volumetric fraction, mislabeled as volumetric %. I've changed the config, but after I started a recalculation of noqc_instrument. noqc_geo should be correctly labeled.
Another beautiful day in the Heber valley. Clear and cold this morning (15 degrees when we left the hotel!) but warmed up and got hazier in the afternoon.
This morning, after some rass troubleshooting at iss1, we went to DCS while it was frozen enough to drive in, so I could bring the ladder and get the last couple EC100 serial numbers. We also checked out the tsoil that had stopped reporting yesterday–visually everything was fine, after unplugging and replugging the binder connections it is again. Glad we didn't have to try burying another one in the frozen ground.
Spent the afternoon showing Liz how I've been checking data for ISFS. Matt is arriving this evening to take over from me.
Heard from the PIs that there will not be an IOP starting tomorrow, so replacing the 32m thermocouple at PRS is not urgent, but will need to happen before the next IOP.
No new data issues today. Reset some HRXLs (sp, dcs, cc–the usual ones), noticed at one point that prsr was off the net but it came back successfully on its own.
Low overcast in the morning, burned off to clear by noon.
Started by visiting DC, which had been off the net for 12 hours by the time we checked data. At DC there was no power at the outlet on the fence and the batteries were dead, so presumably the power failed a couple days ago. We talked to one of the people at the stables, who told us the breaker for the power outlet has been tripping since last week, and lately is tripping immediately after resetting it. We unplugged our power cord from the outlet and tried the breaker again, but it still tripped, so we think the problem is somewhere between the breaker and the outlet, not in any of our equipment. Dan is going to talk to the electrician about it. We replaced the batteries at DC with fresh ones, which got it going again. We'll probably have to swap batteries here every couple days until we get the power to the outlet fixed.
After that I took Liz to the last two satellite sites of our tour:
- CC: measured HRXL height
- DC: measured HRXL height. The EC100 box was open when we got there, so either we left it open last time we were there or someone else came to investigate. Either way the EC100 has been reporting data the entire time, the lights are all green, and it doesn't look like any snow got into the box.
This afternoon I visited PRS while Liz did the sounding. I got HRXL height, compass boom angles, one more EC100 serial number, and site photos. Refilled the gas in the generator since I was thinking about it. Got prst back on the net since I was there anyway, but it was just a modem problem so it would have come back up. I did some more digging into the port 7 problem on prsr but didn't really figure anything out, see Jira for more details: https://jira.ucar.edu/browse/ISFS-529
We also got the soil corer working again by doing some filing so the hole in the handle is now large enough to fit the weight rod again.
Talked to Sebastian today, who was wondering about thermocouple status. Last I checked only the 32m thermocouple at PRS was broken, out of the ones Sebastian didn't remove before the snow the other night. Sebastian put all the thermocouples back that he removed the other day (2m and below). We'll want to lower the tower at PRS to replace the 32m thermocouple before the next IOP, which sounds like will probably be starting at noon on Monday.
Not much else to report data-wise. I used usbreset to get the HRXL at dcsr reporting again this morning. As of now dcst is off the net, but if it's just a modem problem it will probably come back up in half an hour. The tsoil at dcs stopped reporting this afternoon, so I tried power cycling the mote with pio, but that didn't bring it back. We'll try to head over there tomorrow.
For anyone's information, last night I tweaked all the snow depths and boom angles with the measurements so far and reran the 5-minute statistics for both noqc_instrument and geo up to yesterday. Clearly, we don't have a merged data set yet, so there still will be data gaps. The next steps would be to restart the real-time statsproc to use the new cal_files and to recreate the qctables and webplots. I can do this later today.
Morning lightly snowing and low clouds, cleared to mostly sunny and windy by this afternoon.
Heard from PIs that there will be no IOPs for next 72 hours, so the soonest an IOP could start would be noon on Monday. As of today the 32m thermocouple at PRS is broken, so before the next IOP we will need to lower the tower and replace it.
Liz and I took advantage of the down day and had a nice morning checking out the Soldier Hollow nordic center. For future reference: it's $15 for a day pass and $25 to rent skis (classic or skate), and there's a fun variety of trails! We decided it was practice for having to ski in for SOS.
Data checks all looked good this morning, and I haven't noticed anything being off the net. Gary figured out a better way to get HRXL data back than rebooting the dsm, hooray! https://jira.ucar.edu/browse/ISFS-527
Liz and I got the soil corer weight unstuck from the handle but it's still not fully functional, see my comment on yesterday's post.
Low clouds in the morning, gradually clearing as the day went on but remained hazy. Some snow possible tonight and tomorrow.
Heard from Sebastian that the PIs are taking a down day tomorrow, no project meeting. Liz and I plan to take a semi-down day as well. Looks like there probably won't be an IOP before Sunday (don't quote me on that). Sebastian said he had taken down all the thermocouples he could reach (his rake, plus the 1m and 2m ones) at both supersites, but didn't mention anything about wanting us to take down the others.
In the morning Liz and I visited a bunch of satellite sites so she will know how to get to them:
- SH: measured HRXL height
- MW: got boom angle, measured HRXL height, said hi to the horses
- UP: measured HRXL height, checked on status of power drop. Nothing has happened since our last visit.
- PC: measured HRXL height
- MH: measured HRXL height, got sonic serial numbers + sonic height
Went to DCS to do a soil sample in the late afternoon while the soil was warmest (ha). The good news: got the sample, which completes round 1 of soil samples. The bad news: got the rod + weight part of the soil corer stuck in the handle:
I had left the rod + weight on while using the sledge on it because I hadn't wanted to bang up the top of the pipe too much, but apparently this was worse. I think either the rod is slightly bent or the top of the pipe got beaten out of shape enough that the rod no longer comes out. Liz and I might spend more time trying to get it unstuck tomorrow.
Also while at DCS I got some boom angles, site photos, and the HRXL height.
Nothing else new to report data-wise. Netcdfs stopped generating on barolo and Steve fixed by restarting dsm_server, and MW and and UP went off the net but came back up on their own. Nidas didn't start at MW after reboot until I logged in and started it manually, see jira: https://jira.ucar.edu/browse/ISFS-533. HRXLs at SP, CC, and DCS were offline at some point. I just rebooted these dsms and they're now back (for the next however long), so they can measure the potential snowfall tonight.