Blog from September, 2016

This list for my own use (though others are welcome to comment/add)...Steve

sonics: have Gordon/Kate shoot a third set; take median of the 3 for boom angles; put into QC files

TRH: check T against a neighbor anytime Ifan=0 to see if fan was really off.  If on, reset Ifan minValue.

TRH: add offset from postcal to one sensor that read off

(DONE, on eddy) Pirga: adjust offset

Qsoil: just use median of gravimetrics everywhere (need to take more)

Cvsoil: just use median of values taken (need to get one working from Steve S.)

tc: correct based on Tc using Tom's method; use when TRH dies

Rad: remove cleaning periods?; remove wetness-not-dry

Wetness: convert S15 to water-on-surface flag

(DONE, on eddy) Rain: remove test tips

IRGA: remove cases with water on lens (there should be a diag flag for this?)

(DONE) Remove all S3 data from ~1800 9/30 to ?? 10/1 while testing/configuring Pi DSM

on 10/3, tower rebooted (probably low power) with wrong date; 2 misnamed data files were created.
(Not only are the files misnamed, all the recorded timetags are wrong as well.  chrony was running, but simply logged that the PPS/NMEA inputs were hugely different – it didn't correct the system clock.  We would have to adjust the timetags (ideally, using the valid GPS messages in the data to synchronize, though we don't have PPS) of all samples.  Gordon doesn't have a utility to do this and recommends simply tossing the 1.5 hours of data in these files.  Unfortunately, winds are from the north to northeast during this period at 4 m/s, which could be good data.)

(DONE, on eddy) Qsoil has some downward spikes in early Oct – add a min threshold of 60(!).

Pirga: s15 Changed(?) to internal on 21 Oct 12:00, added offset of 2.6mb.  Bad data while this change was made from 11:25-12:00.  s13 had level shifts during rain on 22 Oct.

Went in knowing that there were lots of issues, related to the high water from the recent storm.  Out from trailer 1300-1700 with Cristina:

S3: Station dead due to water getting into DSM through duct-taped Gordon hole.  Obvious bits of corrosion on power interface board.  Wiped off water on board and replaced blown main fuse, but new fuse blew within a few seconds (DSM LEDs were briefly all green.)  Removed DSM for debugging/replacement.  Capped cables by putting into an empty water bottle.  Strapped lid down on battery box.

S1: Power monitor board blown due to shorting with obvious corrosion.  Replaced with spare we had brought.  Strapped lid down on battery box.

S11: Power monitor board blown due to shorting with obvious corrosion.  Battery cooler lid was ajar.  Left in place.  Strapped lid down on battery box.

S7: Power monitor board obviously had been wet, but still reporting.  However, gain on Vbatt was 0.  Changed with vbg=10 and now get a voltage (though calibration probably isn't correct).  Strapped lid down on battery box.

S15: Water everywhere, but station was working.  Tried cycling power on both sides of Bluetooth, but could get data to flow.  Rebooting DSM brought back the Bluetooth connection.  During this visit, the blue Bulgin cap for the console port blew away.  We need to replace this.  (Why aren't spares in the box or laptop case or toolbag?)

We noticed that Vbatt was only about 12.0V and likely wouldn't last the night.  No charging happened today due to stormy weather.  We decided to swap in our one spare battery.  Found that the entire battery box was mostly filled with water, so dumped that out.  (Hope the others in the stations aren't similarly filled!)  Even though we tried to keep the charge controller going, it reset and lost its settings.  Set md=1 again and now seems to be happy.

So I've now taken just one core each at S9 and S15 (the only sites with Qsoil sensors).  The results were: EC-5 67%/65% and Gravimetric 90%/89%, which could be interpreted as the EC-5 having a bias of 23%.  A bias or gain change is plausible, since we are in saline conditions, though I don't know if the marsh exceeds the EC5 salinity operating range (8 dS/m for mineral soils).  (Seawater is supposed to be 54 dS/m, so 8/54 = 15% seawater certainly could be exceeded in the marsh.)  I'm actually impressed that the two EC-5 probes read as close as they do, that the two gravimetric samples were close, and that the higher EC-5 reading was associated with the higher gravimetric reading.  This is especially true given the difficultly in obtaining a gravimetric core, with lots of sample handling issues.

Clearly, it takes more than one point to develop a calibration.  Usually, we take a sample in wet conditions (shortly after rain) and dry conditions, to have a large calibration range.  I had hoped that data from high-tide and low-tide would produce such a variation; however, during the entire experiment here, the EC-5 probes have varied in time by only 1%.  It would be unprecedented for the accuracy of our gravimetric readings to be 1%, usually due to spatial heterogeneity, and I don't see why here would be any different.  Thus, all of our samples here will be effectively at only one soil moisture level, which will make it impossible to distinguish a gain from an offset.

Perhaps all this is a good thing:  if Qsoil doesn't vary with time, it can be treated as constant and set to the average value of the gravimetric readings, essentially ignoring the EC-5 data.  Since they will be important, we should still take more gravimetric samples.

Once we get the TP01 probes sampling again, we could try looking at the bulk soil heat capacity measurements to confirm/deny that Qsoil is constant.  (I do wonder if the TP01 heat pulse method itself will be valid in "soils" with flowing water...)

At 0310 this morning, S9's TRH Ifan jumped to 330mA and stayed there for 30min.  There was heavy rain at the time.  If the PIC electronics had been working properly, this couldn't have happened.  (The PIC would have shut it down after one sample.)  However, Iload doesn't show a jump in current at all, indicating that this high Ifan reading wasn't real.

About 0340, the sensor stopped reporting completely, though Iload if anything increased a bit.

I manually connected to the sensor at 0810.  Initially, rserial showed that no data were coming in and no response to R, thus the PIC was hung.  With tio 2 0/1, got the PIC running again.  Ifan was immediately 0 (didn't have one sample high and get turned off by the PIC).  Iload if anything got lower at exactly this time, hinting that the fan might actually be off now.  

However, the temperature reading cooled during the first 5 min after this restart.  Usually, I would say that the sensor had heated up in the sun and that the cooling indicates a fan that was stopped and is now running.  Rsw.in only shows 20 W/m2, though (it is raining, with lots of cloud), so it is hard to believe that the sensor would have that much radiation heating.  The only other source of heat could be a stuck fan, but I don't quite believe this either.

5 min after being restarted, the temperature agrees with the adjacent S1.

Ultimately, we don't know if the fan is running or not to determine if we have good T,RH data.  (But errors would be small in the cloudy, windy conditions now.)

We had several reasons to go out, especially that TRH.S11 was reading Ifan=0, even after restarts.  Went in&out via the tiki cut, but did hit bottom once on the way out.  Wind direction seemed perfect for a wake to hit S2.  However, I've looked at the data from this period, e.g. from 12:40-13:00 when the wind direction of 70deg should have directly hit S1, and see no evidence of a wake in the raw data or w-spectra.  If anything, w high-frequencies are somewhat lower than at S3, for example.

S13: shot boom angles.  discovered that the monopod interferes, so shot other angles without it.

S11: (0953?) shot boom angles and replaced TRH sensor.  Turned out that fan was on, despite reading 0, so it is a current monitoring issue.  Data QC could allow last night's data to be okay.  It took a while for the new TRH sensor to cool down.

S14: shot boom angles (via the African Queen route)

S3: (1059?) shot boom angles

S2: shot boom angles

S1: (1112) shot boom angles (now the 3rd set) and cleaned radiometer.

(S4:) From the boat, saw osprey sitting on the crossarm opposite the Gill.  The bird's body was a bit taller than the Gill.  (We are not completely sure that this is when we saw it – could have been the previous trip.  In any case, I haven't been able to find an effect of it in the data for both trips (which surprises me).)

S9: (1126) cleaned radiometer (but now raining)

S6: shot boom angles

S15: shot boom angles and (1143) cleaned radiometer (again wet)

Is this a record for number of sites (9) in one trip?

With a break in the rain, and mid-level tide, we took a quick run into the marsh today:

S6: shot boom angles (but the CSAT angles were not correct)

S7: shot boom angles (same issue with CSAT)

S15: shot boom angles (same issue)

         collected soil core (horrendous!  corer was rusted, roots tenacious, gnats all over)

         checked out S1's old Qsoil wisard board and probe.  Definitively isolated problem to EC-5 probe.  Red wisard board is okay.

S8: shot boom angles (same issue)

Made it out under the bridge with 4cm to spare.

Around 4pm yesterday, data stopped being processed in Boulder.  There are no UDP or netcdf files for last night and webplots of course weren't created.  (UDP data were saved here by eddy and presumably all data are on the stations.)  I found that, although dsm_server was running, data_stats, for example, wasn't showing anything.

Thus, I killed dsm_server on barolo and let crontab restart it.  Now UDP files are being created, but I still didn't see netcdf files.  So, I killed statsproc as well and the next 15min crontab cycle restarted it.  We're now creating netcdf files again.

It still is a puzzle to me how dsm_server could hang in a state of not relaying data...

Noticed that TRH.42m Ifan was at 0, indicating that it finally went over 80mA last night.  However, when I tried to connect (either rs or minicom), it didn't respond, nor to eio cycles.  Tried rebooting the box and it never came up.  Walked to the DSM and had no response on the console.  Finally power cycled the DSM and everything came back up, including TRH.42m with Ifan now 63mA.

As mentioned before, we'll keep watching this sensor.

It was annoying to have tower DSM flake out like this, though.

The USB disk on S12 died over the weekend.  The rsyncs of s12 from barolo started to fail, and then upon logging into s12 found that the sda disk no longer showed up in lsusb.  So it looks like the disk has just died.  I have changed vertex.xml to comment out the <mount> element, and instead created a link from /media/usbdisk/projects to /home/daq/data/projects, so now the raw archive files are being written to the home partition:

daq@s12:~/isfs/projects/VERTEX/ISFS/config$ lsu
Filesystem      Size  Used Avail Use% Mounted on
/dev/mmcblk0p2  856M  2.4M  793M   1% /home
total 780
-r--r--r-- 1 daq eol 795677 Sep 26 23:01 s12_20160926_225143.dat

With almost 800M available and writing 54M per 12 hours, that means there is room for 7 days of data.  If I change the rsync configuration to remove more data and only leave the most recent 2 days of data at the site, then S12 could be offline for almost 5 days before it starts losing data.  So maybe you'd prefer to go without a new USB disk, it certainly seems feasible.

If you choose to replace it with another 8GB disk, then once you insert it, it can be formatted remotely on the DSM, and then the configuration needs to be switched back to using the mount option.

We have noticed that the tower Ifan values are getting rather large (more than 60mA).  Here is the plot over the past month, where especially 42 and 49m (of course ) are at times running significantly higher than the others.  The only <possible> action that we could take would be to turn them off to conserve potential run time on the fans.  However, stopping the fans might allow stuff to build up more and humidity to do more damage (since the fans wouldn't be warm).  Furthermore, the sonic temperatures appear to be <reasonably> close to the TRHs, so if a sensor dies, a tower temperature profile could still be constructed.

Thus, we won't take any action here and hope the TRHs last.  At the rate they are going, they should still be under 80mA for another month.  (If not, the current threshold could be raised.)