Over the last ~7 days we’ve noticed that the sonic at LC has been intermittent during the overnight hours. During these periods we’re seeing an ldiag=1, although the sonic does still report values sporadically. As it turns out, the pattern of ldiag=1 from roughly 8 pm to 9 am has been consistent throughout the project, however the missing data started just within the last week or so.
In looking at the DSM dashboard today, I found diagbits=0, 1 and 5 at various times while ldiag=1. I haven’t looked at past data to see if this pattern has been consistent. According to the EC-100 manual these represent “amplitude is too low”, “amplitude is too high”, and “sonic head calibration signature error”, respectively.
It’s pretty clear that this is not an icing/obstruction issue, so the best course of action may be swapping out the sonic. Since we do not have a spare, Steve has suggested swapping out the EC-100 box to see if that helps. John and Jacquie will try this in the next couple of days.
Another lovely day around Heber City, with clear skies, light haze, and calm winds. The warming trend continues, as the high reached into the low 40’s this afternoon. This pattern looks to continue all week, along with increasing haze.
We spent the day showing Jacquie around and introducing her to our daily routine, in particular collecting soil samples at SP and PRS (the last of round 2 of soils) and lowering the PRS tower to replace the thermocouple at 17m. We lowered the tower from about 1:25 - 2:25 pm and everything went off without a hitch. While there I took a couple photos of the trailer license plate holder showing that there is no license plate on the trailer. It's possible that it's somewhere less obvious, but none of us could find it. We also took a quick tour of some sites, visiting UP, PC, and SH. We ended the day with the afternoon sounding and a quick discussion with Steve about the sonic at LC. I am writing a separate blog post with a bit more detail about that issue.
Jacquie will be taking over the blog posts tomorrow, as I head back home. Thanks to her for taking up the torch and big thanks to John for his help over the last week.
Today was almost a repeat of yesterday’s weather, with clear skies and highs in the upper 30’s. The afternoon saw a moderate northerly wind develop, as well as some light haze. There were no issues with ISFS sensors today, so John and I took a nice hike up Memorial Hill to measure the CSAT boom angle and enjoy the views.
In the afternoon, I paid a visit to the PRS to take a full set of Leica measurements. The wind picked up to 5-8m/s by the time I arrived, making it difficult to get accurate measurements of the higher booms (not to mention rather cold standing in the field after a couple of hours). I will have to see how the measurements look, but I’m not confident given the amount of sway in the boom at 32m. I attempted to take 2-3 measurements of each point on the 32m and 17m levels, but found the target had occasionally drifted out of the field of view in the few seconds between scans. The rest of the measurements went okay, although I had to set up in an area between some fencing to keep the cows from getting too interested in the Leica. Between this and the general difficulty in getting a perfectly clear view of all necessary sensors, this site proved challenging. I spent about 2.5 hours taking the ~56 individual measurements.
I ended the day helping John with the afternoon sounding. Jacquie arrived this afternoon, so tomorrow we will spend the day showing her around.
Today was another clear, calm, and mild day, with some high clouds moving in in the late afternoon and a bit more haze than yesterday. The daytime high ended up being in the low 40’s - the start of a forecast warming trend over the next week that should see high’s near 50. The snow continues to melt and some fields are nearly clear of snow.
We began the day at the ISS2 sounding trailer troubleshooting a power failure - the GFI tripped on the outlet. Afterwards we visited LC and CC to collect soil samples and measure the boom angles with the GPS. While at LC we visually inspected the sonic because of the intermittency of the wind measurements. I expected to see frost, as there was considerable frost at the ISS2 site, but there was none at LC. From what we could see, there were no obvious obstructions on the transducers, but we did not have a ladder. The sonic status LED inside the EC100 box was solid red, which is not surprising since the ldiag flag has been at 1 during the periods of missing data such as when we were at the site this morning. It’s pretty consistent at this point that ldaig=1 from shortly after sunset to shortly after sunrise. Perhaps a change of the sonic in in order?
A number of sites dropped off the network over the course of the day, but according to Nagios they have all recovered. It also appears that the 17m thermocouple at PRS broke this morning, so we will have to replace it before the next IOP. Otherwise everything is running smoothly. We still need to collect a 2nd round of soil samples at SP and PRS, which I will probably wait until Jacquie arrives to do, and I need GPS boom angles from MH and Leica measurements from PRS, both of which I may do tomorrow before Jacquie arrives.
Today was clear and warmer, with a daytime high just above freezing and a bit more haze than the past few days. Winds were very calm all day and based on the afternoon sounding were calm into the free atmosphere above the BL.
John and I spent the first part of the day doing some outreach with a local high school science class, discussing the CFACT project and giving a tour of the Upper Provo site. John potentially set up another outreach opportunity for next Friday where he would give some middle school students a tour of the Deer Creek Super Site and ISS trailers.
Throughout the day there continued to be no issues with ISFS sensors, so I decided to take full Leica measurements of the DCS towers. I took 4 measurements per sonic (7, 17, 32, 1, 2, 3m) and 1 measurement per TRH (7, 17, 32, 1, 2, 3m), in addition to 4 GPS positions and one trailer point for reference. The entire process of 35 measurements and setup took a little over 2 hours.
Jacquie pointed out in a blog post that the data from the sonic at LC is rather sporadic, a trend that I’ve been monitoring myself. It seems to be happening overnight, so I suspect it’s an icing/moisture issue - ldiag = 1 during most (all?) of these periods. We will likely pay a visit to the site to collect a soil sample and measure the boom angle by GPS in the next couple of days, so we will assess the sonic at that time as well.
Today the PI’s announced there will be no IOP’s for the next 48 hours and no science meeting tomorrow, making it a hard down day.
Not much change in the weather today. Partly cloudy in the morning that became mostly clear the rest of the day with light northerly winds. The daytime high stayed below freezing again. The next few days look to stay cold, but slowly begin a warming trend into next week.
The morning data check showed no issues. An ephemeral fog IOP began at noon and by then all ISFS instruments seemed to be functioning properly. Sebastian plugged in his thermocouples at PRS and DCS in the late afternoon and they seem to all be functional as well as of this writing. The only real news to report from today is that the power returned to the MW site, as evidenced by the increase in Vbatt and Icharge at the site around 8:40 am. With this development all sites are back on AC or solar power and we will not need to swap batteries. We will continue to keep an eye on the Vbatt for all sites in case any issues crop back up. We also picked up the repaired soil corer T-bar from the welding shop, so we can return to collecting soil samples.
Due to the IOP we launched soundings at 2:15 and 4:15 pm and continued to check on the status of the ISFS instrumentation throughout the day.
Today we awoke to a dusting of snow that fell sometime after midnight last night. This morning was colder than the past few days with partly cloudy skies that slowly cleared throughout the day. The winds at the surface were from the north and quite cold, while the high temperature stayed below freezing.
The morning data check showed no issues, except that the battery at DC was getting low. We swapped out batteries at the site before noon, plugged the site into the AC power that had finally been restored, and took an additional Leica measurement. We then dropped off the T-bar to the soil corer at a local welding shop to get it fixed. It should be done by the end of the week or sooner.
John and I spent a bit of time working on the lidar, which has a moisture problem and software issue keeping it offline. We used the soil oven to bake another pack of desiccant and added it to the lidar, which will hopefully help with the moisture problem.
There is an IOP scheduled for tomorrow beginning at noon. As of this evening there are no issues with the ISFS.
There was finally a slight change in the weather today to colder and partly cloudy with breezy winds. The clouds mostly stayed over the mountains to the west.
The morning data check showed that MW was off the network. At first I assumed it was the typical modem issue, but Nagios showed that it had been offline for 9 hours (since around midnight). A visit to the site revealed that the power had shut off and the batteries had drained over the last few days. We think it’s likely that the breaker has been tripped, however we could not find a tripped circuit in the exterior breaker box meaning that the breaker we need to access is probably in the garage. Liz coordinated with Dan to call a contact for the site, who revealed that the owner is aware of the issue and is in the process of getting it fixed. In the meantime we swapped batteries and will monitor their charge going forward.
In troubleshooting the MW site, I also noticed that NCharts was not showing any data after Jan 31. I contacted Gary and he was able to fix the problem quickly, which was an error in the NCharts configuration for the project end date.
In the afternoon I dropped Liz off at the airport, returning in time to help John with the afternoon sounding. Many thanks to Liz for all her help with ISFS, especially helping me get back up to speed with supporting an ISFS project. Thanks Liz!
The PI’s were discussing having an IOP tomorrow (Wednesday), but have now decided to consider an IOP starting on Thursday at noon.
All of the CFACT modems need to be switched to a different Verizon data plan to avoid excessive charges, and that is going to require changing their static IP addresses. I don't know exactly when this will happen, but the change was approved this morning. At worst, all the modems stop connecting until the DSM script restarts them, either in 15 minutes or with a reboot at 4 hours. More likely, the impact will be minimal. The modems should reconnect on their own, get assigned a new IP address, and the dynamic DNS update script will update the dyndns.org address for each DSM automatically. If you happen to have hardcoded the IP address somewhere, such as in a ssh config or browser bookmark, then it will need to be replaced with the DNS name, like cc-isfs.dyndns.org.
Nagios will tell us if any DSMs do not come back as expected.
This is a good time to remind everyone who connects to the DSMs to be careful about authentication. If ssh reports that a host key has changed, then it's entirely possible it's because the previous IP address for a DSM has been assigned to someone else's host, and it's not a good idea to start sending credentials to an unknown host. 
If you're ever not sure, one way to confirm a DSM's address is to first log in through the ssh tunnel, since only our DSMs are able to authenticate with eol-rt-data to create the tunnel port.
Today was much like yesterday, but even warmer, with mostly clear skies and temperatures rising into the mid 40’s. The winds were calm most of the day, but did begin to pick up in the late afternoon ahead of the approaching system.
The morning data check showed an issue with soils and radiation at DCS, in particular the tsoils were missing and the other variables from DCSG had dropped out at some point in the morning. We initially thought it was the typical modem problem, however further investigation revealed that all affected variables were from sensors running through the soil mote. We decided to change the PIC board for the tsoils because we have had some previous issues with that sensor, however we will keep an eye on this site in case we need to swap out the mote. We changed the tsoil PIC from TS17 to TS23 around 12:15 pm, which will require changing coefficients at some point.
Otherwise we spent the day showing John some of the ISFS peculiarities and tools for checking the data. We also visited DC to check on the work the electrician was supposed to have finished, but were disappointed to find out that nothing had been done. For now the site is still on battery power, but the charge seems to be lasting much longer now that we reseated the thermocouple.
Finally, we requested a quote from a place in town to fix the broken weld on the soil corer. Hopefully we can get it turned around pretty quickly. It should also be noted that Sebastian removed all the thermocouples from both PRS and DCS below 2m this afternoon.
Today was the warmest day we’ve had in the past week, with temperatures warming into the upper 30’s during the day, under clear skies and calm winds. There was a bit more haze today however, especially concentrated in the BL when looking towards Provo Canyon.
The morning data check showed no issues, so we decided to take advantage of the nice weather to visit a handful of sites for soil samples. We were able to collect samples at SH, PC, UP, and DCS, although it was quite the challenge at DCS because of the hardness of the frozen soil. We managed to break the weld on half of the handle with the sledge hammer, but the corer is still usable. This does beg the question as to whether or not continuing to collect soil samples in the extremely frozen ground is worth the effort and breaking equipment. Despite this question, we did buy new rags for the soil kit and cleaned/organized both soil tool boxes, which was long overdue.
Liz also spoke to Sebastian today, who informed us that we will not need to remove the thermocouples from the telescoping towers due to the expected weakening of this week’s forecast weather system. Finally, Liz will be picking up John at the train station in Provo tonight and we will begin showing him the ropes tomorrow.
The weather today was almost exactly the same as the past few days, i.e. cold, clear, and calm. The morning data check showed no issues and no HRXL problems, which means the script that Gary wrote to automatically reset the usb seems to be working. Thanks for that Gary!
Since there were no issues to attend to, we decided to spend the day visiting a few of the sites I had not yet been to so that we could take Leica measurements and soil samples. We started at SP, where we successfully measured the boom angle, but could not take a soil sample. The ground was very frozen and we may have hit a rock when driving in the corer, as its cutting edge bent. We will try to fix it as best we can, but at least we have the corer from the other soil kit. We were able to take a soil sample at the next site we visited, MW, as well as measure the boom angle there.
In the afternoon we took a trip to DC to troubleshoot the increased battery drain we’ve been noticing. We swapped batteries and reseated the thermocouple wire because it had not been reporting. We were not able to pinpoint why the site keeps draining batteries so quickly. In comparing the load between all sites, I am curious as to why some sites show an increase in load during the day (DC, MW, UP), while others show an increase over the night (CC, LC, MH, PC, SH, SP). I would expect the increase overnight if heaters are turning on, but am not sure what would cause the opposite behavior at certain sites.
On our way back to the base we drove past CC and LC so that Liz could show me where to find them. After the daily sounding we walked to DCS to troubleshoot the tsoil sensor that stopped reporting. We tried reconnecting the cables, as Isabel and Liz previously had restored them at DCS in this way, which fixed the problem.
We also saw a "change engine oil soon" message in the ISFS truck today.
This morning was a bit colder than the previous few, having temperatures in the single digits when we left the hotel, with clear skies. The daytime high temperature again peaked just above freezing.
The morning data check showed no issues, save for the usual HRXL not reporting at CC, SP, and DCS. We reset them around 9:45 am. We then took a quick break for some cross-country skiing at Soldier Hollow, in preparation for next winter’s SOS campaign.
In the afternoon we met with Sebastian to join in on some tethered balloon training, as well as lead a radiosonde training session for a few students. We also noticed that the batteries we installed at DC yesterday morning were being drained quickly, down from 13V to 12.5V in 24 hours. Previously at the site the batteries took about 3.5 days to decrease that amount. We had hoped this would be a non-issue, as the electrician should have restored the power to the site today. Unfortunately, when we stopped by DC at the end of the day to plug in the site and swap batteries we found that the work had not been done. Hopefully this battery swap will last longer then yesterday’s.
In the evening, I noticed that ncharts, qctables, and web plots were not updating (as of 2pm or so), but all the sites were reporting data. I believe this is a dsm_server issue on barolo and emailed Steve and Isabel to restart the process.
Assuming all goes well, tomorrow we are planning to visit a couple of the remaining sites that I have not seen yet in preparation for Liz’s return to Boulder. We will measure boom angles and collect soil samples as time permits.
Another clear and cold morning today, with temperatures again in the low teens. The rest of the day was very similar to yesterday, with beautiful clear skies and a high around freezing. Winds were mostly calm, but increased in the afternoon. The weekend looks to be a bit warmer and sunny, before the next weather system moves in around Tuesday.
As is customary at this point, we had to reset HRXL’s at CC, SP, and DCS first thing in the morning. The morning data check showed no other new issues, but the battery at DC had dropped to near 12.5V, necessitating a battery swap. While we were there, Liz helped troubleshoot the electrical issues with the electrician and I took a soil sample and measured the boom angle with the Leica GPS antenna. The ground was quite frozen and covered in about 6 inches of snow. For the electrical, it turns out there was a short at a junction outside of the barn. The electrician planned on bypassing that section as a temporary fix, so we will hopefully have power at the site restored by the time of this blog post.
In the afternoon we returned to DCS to troubleshoot the thermocouple at 32m. Unfortunately the problem was with the thermocouple itself, forcing us to lower the tower to replace it. This did the trick and we are now getting data. We only have one spare thermocouple left at that site, but there is discussion about removing all the thermocouples from the towers before next week's forecast snow, which would alleviate this concern. We also decided to dress the cables at the tower to facilitate lowering and raising the tower. Thanks to Liz for doing much of the untangling and taping despite freezing hands! This should make life much easier for future visits to DCS.
We wrapped up the day with the 4:15 pm sounding and a data check, finding no issues. Tomorrow is a down day for the project.