Blog

statsproc_redo

For anyone's information, last night I tweaked all the snow depths and boom angles with the measurements so far and reran the 5-minute statistics for both noqc_instrument and geo up to yesterday.  Clearly, we don't have a merged data set yet, so there still will be data gaps.  The next steps would be to restart the real-time statsproc to use the new cal_files and to recreate the qctables and webplots.  I can do this later today.


Ops 1/22

Low overcast in the morning, burned off to clear by noon.

Started by visiting DC, which had been off the net for 12 hours by the time we checked data. At DC there was no power at the outlet on the fence and the batteries were dead, so presumably the power failed a couple days ago. We talked to one of the people at the stables, who told us the breaker for the power outlet has been tripping since last week, and lately is tripping immediately after resetting it. We unplugged our power cord from the outlet and tried the breaker again, but it still tripped, so we think the problem is somewhere between the breaker and the outlet, not in any of our equipment. Dan is going to talk to the electrician about it. We replaced the batteries at DC with fresh ones, which got it going again. We'll probably have to swap batteries here every couple days until we get the power to the outlet fixed.

After that I took Liz to the last two satellite sites of our tour:

  • CC: measured HRXL height
  • DC: measured HRXL height. The EC100 box was open when we got there, so either we left it open last time we were there or someone else came to investigate. Either way the EC100 has been reporting data the entire time, the lights are all green, and it doesn't look like any snow got into the box.

This afternoon I visited PRS while Liz did the sounding. I got HRXL height, compass boom angles, one more EC100 serial number, and site photos. Refilled the gas in the generator since I was thinking about it. Got prst back on the net since I was there anyway, but it was just a modem problem so it would have come back up. I did some more digging into the port 7 problem on prsr but didn't really figure anything out, see Jira for more details: https://jira.ucar.edu/browse/ISFS-529

We also got the soil corer working again by doing some filing so the hole in the handle is now large enough to fit the weight rod again.

Talked to Sebastian today, who was wondering about thermocouple status. Last I checked only the 32m thermocouple at PRS was broken, out of the ones Sebastian didn't remove before the snow the other night. Sebastian put all the thermocouples back that he removed the other day (2m and below). We'll want to lower the tower at PRS to replace the 32m thermocouple before the next IOP, which sounds like will probably be starting at noon on Monday.

Not much else to report data-wise. I used usbreset to get the HRXL at dcsr reporting again this morning. As of now dcst is off the net, but if it's just a modem problem it will probably come back up in half an hour. The tsoil at dcs stopped reporting this afternoon, so I tried power cycling the mote with pio, but that didn't bring it back. We'll try to head over there tomorrow.

Ops 1/21

Morning lightly snowing and low clouds, cleared to mostly sunny and windy by this afternoon.

Heard from PIs that there will be no IOPs for next 72 hours, so the soonest an IOP could start would be noon on Monday. As of today the 32m thermocouple at PRS is broken, so before the next IOP we will need to lower the tower and replace it.

Liz and I took advantage of the down day and had a nice morning checking out the Soldier Hollow nordic center. For future reference: it's $15 for a day pass and $25 to rent skis (classic or skate), and there's  a fun variety of trails! We decided it was practice for having to ski in for SOS.

Data checks all looked good this morning, and I haven't noticed anything being off the net. Gary figured out a better way to get HRXL data back than rebooting the dsm, hooray! https://jira.ucar.edu/browse/ISFS-527

Liz and I got the soil corer weight unstuck from the handle but it's still not fully functional, see my comment on yesterday's post.

Ops 1/20

Low clouds in the morning, gradually clearing as the day went on but remained hazy. Some snow possible tonight and tomorrow.

Heard from Sebastian that the PIs are taking a down day tomorrow, no project meeting. Liz and I plan to take a semi-down day as well. Looks like there probably won't be an IOP before Sunday (don't quote me on that). Sebastian said he had taken down all the thermocouples he could reach (his rake, plus the 1m and 2m ones) at both supersites, but didn't mention anything about wanting us to take down the others.

In the morning Liz and I visited a bunch of satellite sites so she will know how to get to them:

  • SH: measured HRXL height
  • MW: got boom angle, measured HRXL height, said hi to the horses
  • UP: measured HRXL height, checked on status of power drop. Nothing has happened since our last visit.
  • PC: measured HRXL height
  • MH: measured HRXL height, got sonic serial numbers + sonic height

Went to DCS to do a soil sample in the late afternoon while the soil was warmest (ha). The good news: got the sample, which completes round 1 of soil samples. The bad news: got the rod + weight part of the soil corer stuck in the handle:

I had left the rod + weight on while using the sledge on it because I hadn't wanted to bang up the top of the pipe too much, but apparently this was worse. I think either the rod is slightly bent or the top of the pipe got beaten out of shape enough that the rod no longer comes out. Liz and I might spend more time trying to get it unstuck tomorrow.

Also while at DCS I got some boom angles, site photos, and the HRXL height.

Nothing else new to report data-wise. Netcdfs stopped generating on barolo and Steve fixed by restarting dsm_server, and MW and and UP went off the net but came back up on their own. Nidas didn't start at MW after reboot until I logged in and started it manually, see jira: https://jira.ucar.edu/browse/ISFS-533. HRXLs at SP, CC, and DCS were offline at some point. I just rebooted these dsms and they're now back (for the next however long), so they can measure the potential snowfall tonight.


We had always wanted the EC150 to be operating in automatic heater mode.  According to our SE wiki entry, the heater volts should be set to -2.  One (7m.dcs?) that I checked several days ago seemed to be frosting up and had the setting at -1, which I don't know the meaning of.

Just to be sure, I've just reset all 3 levels of both TTs to be -2 via:

rs x

<cr><cr><cr>

P<cr>

2718<cr>

S<cr>

M -2 <cr>

Y<cr>

I used a really wide screen and looked for ASCII messages at the end of the lines to confirm my entries, but basically you are typing in the blind (and have to do this quickly before the EC150 times out) while binary messages are streaming by.


If we see frosting-up behavior on other EC150s, we can repeat this on the other sensor positions.


Ops 1/19

Sunny and clear in the morning, but getting cloudier as the day went on. Cool and windy by the late afternoon.

IOP starting at noon today (though it was not officially decided on until after noon today).

Data check looked good this morning, without even any broken thermocouples. At some point today I rebooted dcsr and cc to bring back HRXLs. pc was off the net for a couple hours this afternoon but seemed to come back fine.

Got a ubiquiti link working between the elementary school and the base trailer today. Internet is now much faster, if you use ISS-GUEST rather than the 5G modem's wifi network.

Ops 1/18

Not much to report, I mostly took the day off!

Data looked OK this morning except that we had the same problem with netcdfs not being generated on barolo. Restarting dsm_server and statsproc once again fixed that issue. PC and SP both went off the net during the day but came back after the 4hr timeout in the dsm_watch_network script, so it must have been a modem problem. SP once again didn't have nidas running after it rebooted, see jira: https://jira.ucar.edu/browse/ISFS-533

Rebooted cc and dcsr to bring back HRXLs this evening. Now all HRXLs are reporting but we'll see how long that lasts.

Potential of an IOP tomorrow but conditions are uncertain so they are delaying making the call until tomorrow. Not ideal for us since it's not a lot of notice, but in this case all thermocouples are still working and there are no other sensor issues, so we don't have much to do to prepare.

Ops 1/17

Some high clouds and sunny.

Morning data check showed no data from HRXLs at cc, sp, dcs. Rebooted the dsms which brought the HRXLs back to working at all 3 sites (for now). Also this morning updated the dsm_watch_network script to reboot the dsm after 4 hours off the net instead of 12, to make it easier to keep an eye on data.

This morning Liz and I went to PRS so I could test out my fix for the port problems on prsr (and to pay a visit to the cows). With the udev rule set so the HRXL comes up as ttyHRXL port 0 is back to functional, so plugged the 2m ec100 back in to prsr. Tried the ott in port 7 on prsr again, but still not getting data from it through that port, even with dsm_port_config showing that the port is set correctly. So ott is still connected to prsg, port 6.

This afternoon visited SP and took site photos, boom angle, sonic height, serial numbers, and a soil sample. Pretty hard to get soil cores from the frozen ground but Liz got it done with the sledge!

No IOP tomorrow but there's talk on slack of a possible IOP on Wednesday.

Ops 1/16

Mostly clear with some haze, colder morning than the last few. IOP starting at noon today looking at non-fog cases.

Liz arrived yesterday and joined us this morning.

This morning Liz and I visited DCS to replace the broken thermocouple at 7m before the IOP started. We took advantage of the cold morning to drive most of the way to the site, so we could refill the gas in the generator. Tower lowering and raising went smoothly and everything is reporting after replacing the broken thermocouple. Tower was down between ~10am and 11:45 local. It looks like the missing data problem from the 7m EC150 resolved itself while we were lowering the tower and was likely just frost (sorry Steve). Got some more ec100 serial numbers while the tower was down. Everything was good when we left the site but dcst has not been on the net since 10:15 this morning (while we were on site–but I can't think of what we did that would have an effect on the modem). Assuming it's a modem problem and will come back up later tonight, since I was logged in and checking data after it went off the net.

Memorial hill was off the net this morning when I checked data, but came back on its own before I had time to visit it.

Ops 1/15

Once again things mostly working in the morning. LC had been off the net overnight but came back up on its own this morning. Snow sensor at DCS and CC were not reporting, tried rebooting both dsms, which looks like it did nothing at DCS and only brought the snow sensor back briefly at CC. Annoying, but I haven't figured out anything else to try about these.

Spent the morning on ISS, then visited some satellite sites in the afternoon:

  • PC: took a soil sample.
  • MW: site photos, sonic height, ec100 serial numbers, soil sample. Took about 4 tries to get the soil corer deep enough in the frozen ground to get a sample, the horses were very entertained. Forgot a compass to get the boom angle.
  • SH: site photos, sonic height, ec100 serial numbers. Took a look in the pumphouse again, looks like they've replaced the leaky hose.

IOP starting tomorrow at noon. Sebastian has requested that we replace the broken 7m thermocouple at DCS if we have time, which as of now I think we will, so aiming to do that in the morning.

Isabel had noticed that qctables hadn't updated today.  (Eventually) realized that the netcdf files hadn't been updated since 21:52 last night and that "dsp" wasn't working on barolo.  killed and restarted dsm_server, which seems to have gotten things unstuck.  We'll see if qctables looks okay in a few minutes.

Ops 1/14

Some clouds and haze, warmest day since I've been here.

Data check this morning (once we got data going again) didn't show any major issues. SP had been offline, but came back up when the modem script triggered a reboot. After reboot, though, it wasn't collecting data until I manually started nidas, more details on jira: https://jira.ucar.edu/browse/ISFS-533

Other than that not much to report, I spent most of the day getting caught up on ISS work.

Ops 1/13

Clear and cold in the morning, warmer and cloudier in the afternoon.

Did some blog catch-up of site pics, serial numbers, etc this morning. Data check showed HRXL at cc and dcs were down, but rebooting the dsms brought both of them back up. No other new sensor problems.

Did more troubleshooting at PRS with help onsite from Bill (thanks Bill!) and help over the phone from Chris (thanks Chris!). Turns out the problem on prsr with getting good data immediately after boot but then getting nothing is that at boot the port is configured correctly, but after a minute or so it gets configured for RS232 and therefore you don't get any more sonic data. This is due to interference with the HRXL usb-serial, which comes up as ttyUSB0, which apparently conflicts with ttyDSM0. I added a udev rule for the usb-serial at this site so it comes up as ttyHRXL, which seems to have solved the problem. I don't have anything plugged into port 0 right now, but checking with dsm_port_config shows that it remains set for rs485. Didn't do any looking at port 7, which was not reporting ott data the other day.

Also at PRS tried out the stepladder we found in the closet in the base trailer, but it's not tall enough to reach the 3m thermocouples. Got serial numbers from the 2m ec100 but couldn't reach the box for the 3m one.

Gary submitted a Jira issue that this sensor was reporting every 10s, rather than 60.  I've just changed it by sending it the command "CS/I/60".


2 Comments  · 
Tear-down prep

Timing:

  • Steve already there; Clayton arrives 2/23 (last day of ops)
    • strip all towers of instruments, pack in base
      • remove heating stuff from CSATs
      • good luck removing soils!
    • provide some assistance to remove PI sensors
      • sort out NCAR vs. UU radiometers/ventillators
      • return HRXLs
      • return CS125s (DC is ISS's, I think)
    • return to remove satellite infrastructure
  • Dan, Tony, Chris arrive 2/28ish
    • coil up supersite power cables, pack in Penske for Boulder
    • help remove cc solar panels (hope we can drive in)
    • remove fencing
    • pull TTs out to a staging area

Need to arrange:

  • Trailer Transit for base to SWEX
  • Trailer Transit for TTs back to Boulder
  • Penske from Boulder to bring SWEX infrastructure
    • PAM tripods; booms; clamps
    • more job boxes/batteries?
    • more solar panels?
  • Penske in Heber to return to Boulder
    • power cables/transformers
    • Rohn towers
    • thermocouple motes
    • anything dead (solar panels with bad connectors, etc.)
    • v2.7 stuff, Kipp&Zonens, ventillators

Issues:

  • Do we leave ventillation on NR01s? (I'd like to, but don't know wiring)