Cold (14C) when we arrived 8:40 ish. Above freezing and sloppy as we left about 13:40. Lots of blue sky. Making snow by the hotel.
Isabel and Steve
Tasks:
- Seal & Peel all sonics, in a semi-random order (bottom to top). Initially, stuff sollidified in the caulk tube, so we cut off the spouts. Hopefully, this will help.
- Cleaned 9m radiometers from 0957-1005 (times from plot of rawdata, now entered into cal_files). Used isopropyl alcohol since it was cold. Domes looked clean already, with lots of fan ventilation, though the air didn't feel warm to me.
- d DCDC changed to 20W to power Xstream's Ubiquiti
- Tried to upload v4 software to snow pillows, but got a "Failed to read the controller ID" error on the last pillow (#3). Unable to fix, so left it running v3 code.
- Gary uploaded change to NIDAS to pre-pend a message type prefix, that should allow us to parse the snow pillow data, that seems to work.
- Also overnight, added a cal file for the CSAT3BH, so winds should be correct (though it hasn't been added to statsproc yet).
- Installed the wifi repeater, connected to cm. Had to go through the set-up menu again (Gary had done in the lab) to get it to connect to the Cradlepoint, but now up.
- Nevertheless, one of Ethan's Pis still won't connect. We cycled power on both Pis on uw by unplugging and plugging in their USB-C power cables. (Twice on 4, reseated SD card, LEDs on ethernet same on 3 and 4, so we think it is still getting data from the lidar.)
- Discovered that uw DSM wasn't getting GPS messages (rs G had no response). Power cycle brought everything back. systemctl restart gpsd didn't work.
- Also realized that the uw Apogee data have been quite erratic for the last week. Reseated its screw connector and both Binder connectors on the popsicle stick (also pio cycled power somewhere in there), but still see bad values. P.S. Downwind's apogee also is odd now.
- Soil core taken.
- Back at RMBL, removed 8x CSAT3A cassettes for the shop to work on. Stored CSAT3BH box in white quonset tent.
Left Boulder around 10, got to the site by 3:30 with a stop at RMBL to grab the ladder and try to get into the lab building (no luck). The road to RMBL is mostly clear, the road to kettle ponds is mostly still covered in snow. There's at least a chance that the current snowpack at the site will be there all winter, so we're using the official paths to access towers and guy points while minimizing tracks inside the triangle of d, ue, uw. Currently the site is mostly snow-covered, some parts significantly deeper than others:
Took guy wire tensions on all towers (added to tower page – they appear to be holding well) and climbed uw to add the heated sonic at 3m. I'll let Steve fill in the other stuff he took care of while I was making a hash of cable management for the CSAT3BH.
Other tasks:
- Flipped 1m.uw sonic head about 16:40. Followed Jacquie's suggestion to power down during the change, so it would be apparent in the data. Will update the cal file after finishing this blog.
- Moved the network switch power on cb and cl to the DCDC output, to avoid inadvertent pio power cycles (especially with the snow pillows on cb).
Lost sunlight around 4:30 and was getting too cold to work by 5:30. Back out tomorrow to do the rest of the site visit tasks.
Just changed the data rate for the snow pillows to 0.002s, rather than 0.02s. I only did this on cb's config – haven't pushed this to git. (I'm hoping to sample this snow event without further data gaps in the pillow measurements.)
Sometime, we'll want to add prompting for the other pillow outputs as well (e.g. SWE).
We haven't yet gotten into a mode of regular daily status messages, so just some general notes here:
- I've been looking at QC tables several times a day and have noticed no problems, with the exception of the snow pillows
- Snow pillows (no particular pattern of which pillow) have stopped reporting several times a day. They always come up with a power cycle (pio, using a type-ahead to avoid permanently losing the network connection). We are hoping that a software upgrade will fix this, and the cable and software may be in the mail to us now.
- The CSAT3BH (minus one cable connection) is on the NCAR loading dock as I write this. We hope to configure it in the coming days and take a quick trip back to Crested Butte to install it late next week (along with the software upgrade).
- The nanobarometers that are sealed off have large pressure variations, due to temperature changes. This is confusing on both ncharts and QCtables, but is normal.
Found that at least one Tsnow is sending corrupt data messages, presumably the same issue we saw in the previous 2 projects on the soil wisard sensors. This is happening even though both the Tsnows and the soils are now running versions of the PIC code with double buffering (v2.38 and v1.08, respectively). For the moment, I've set limits on the Tsnow data in sensor_catalog to remove wildly incorrect values, though I guess if the really bad corrupt messages are happening, that more subtle corruption also can occur.
After this change, I restarted both statsprocs...
Pillows 2&4 stopped last night, brought back with power cycle. Other sensors still running.
Hoping that this visit can occur while the road is still open...
Just to put this in one place: (number of minutes)
- DONE Flip uw.1m sonic (the head to the cassette) (5)
- 3/4 DONE Upgrade software on snow pillows (20)
- DONE Install CSAT3BH (15)
- DONE Seal all CSAT cassette mounts (120)
- DONE Clean radiometers (2m NR01 and 9m K&Zs) (10)
- DONE Another soil sample (10)
- DONE Install wifi extender, attached to cm dcdc power (and swap in 20W dcdc converter) (15)
- DONE In cb and cl, move the ethernet switch power from bank1 to the dcdc terminal on the power panel (bring extra dcdc converters?) (5)
- DONE Check new pillow prompting code that Gary may install (unknown)
- DONE Read guy tensions (20)
- Remove cassettes from spare CSAT3As, if time allows (15)
Total: (235 min = 4h, but could do some tasks in parallel)
What to bring:
- individual allen wrench set with 5/32" (on shelf)
- snow pillow cable and windows laptop (on shelf)
- CSAT3BH and mount and mounting bolts (on shelf)
- Seal and Peel and caulk gun (grab another gun from home)
- Kim wipes and alcohol (on shelf)
- soil kit (on shelf)
- wifi extender and cables and dcdc converter (from Gary)
- more dcdc converters (have 1 extra 20W for d to power Xtream's Ubiquiti and 2 extra 10W just in case cb and cl don't have them)
- my tool bag (in car)
- tire chains (in truck)
From the wind direction data, I was pointed to a problem with this sonic, and photos show that it is upside down. This time, I think the "cassette" was incorrectly positioned – probably a year ago, but we hadn't noticed it. It would be a 5-min fix to remount, especially since it is low on the tower, but Chris and I decided to come home, rather than add over an hour of driving with a loaded pickup over the rutted road. We'll need to address this in our next site visit.
This also brings up the question of whether we should have sealed the cassette mounts as Campbell had advised last spring. We'll need to consider this in the coming weeks.
In the meantime, I changed the cal_file to "flipped", so the data now appear to be correct.
I just used the boom angles that I calculated last night from the Leica scan to set Vazimuth for all of the sonics in the cal files. I then just restarted statsproc for noqc_geo. Hopefully, they'll be right!
...but a "pio BANK1 0; sleep 5; pio BANK1 1" revived them.
It is curious that data still appear as 0 in qctables, even though rs shows no data coming in. Maybe just the timing of me looking at it...
Weather: even warmer, same breezes came up during the day
People: everyone left the site by 4. We're done with set-up!
Tasks:
- Helped Ethan get his Pis connected and powered
- Swapped the ec150.3m.c since it was reading high
- Documented tensions (ue had loosened some, but all still okay – see table elsewhere in this blog)
- Took soil core at 1130 (see comment below)
- Capped off unused pressure ports
- Packed up and removed the on-site staging pile
- We're hoping for the best!
Weather: another nice day, clear and cool, warrming, then clouds, then some wind, then chilly when the direct sun is gone
People: no change
Bunch of misc tasks:
- Read all serial numbers
- Finished Leica shooting all sensor positions (today was just the PIs)
- Finished dressing cables
- Added snow pressure port tubing at c
- Swapped 5m.c EC150, since it was reading way offscale
- Finally figured out that the snow pillow problem was our prompting. Isabel, with advice from Gary, figured out how to add "offset" to the configs to prompt these sensors as they need to be. Now working!
- Deployed the rest of Ethan's lidars, though he has some Pi3 issues. We leant him one of our Pi3s out of a spare DSM.
- Jessica's students deployed the Tsnows at d and ue. They won't deploy Tsnows at uw, but instead will log data from them using their laptop manually episodically. I promised to give them a D9 to Bulgin cable for this (mote to laptop with a real serial port). I haven't mentioned powering the mote...
- The temperature screen was set up and demonstrated for about an hour – interesting images!
- Pulled the haul line on the 20 m tower and rotated haul boom to the downwind side.
Our todo for tomorrow (then planning to leave Sat):
- Further help Ethan debug his connectivity issues
- Take a gravimetric sample
- Re-measure guy tensions
- Clean up the equipment pile
At some point, we have been asked to log some of the other messages (at least SWE) coming out of the snow pillows, but that can be set up from Boulder.
Saw that these Tsnows were not being logged by the mote. "rs" to the mote and "si" to rescan the I2C devices, which brought them back.