Blog

Summary: Typical day with intermittent rain and light winds and partly cloudy.  A boat trip saw the DSM placed at S3, the TRH and crossarm reinstalled at S2, TRH replacements done at S9 and S14, and S15 getting some work to get it reporting again. 

Actions past 24 hours:

• Yesterday (10/3) Worked on DSM for S3 and modified TRH's for swaps at S9 and S14.  The modified TRH's bypass the Ifan pcb so they won't read any current but they work otherwise.

To dos:

• Reshoot all boom angles (just because...Have done most sites so far)
• Soil core samples (took some from S15, could redo S9, S1 waiting for a new Qsoil)
• Install TP01 when fixed
• Fix WWW plots that die too frequently

Station Status:

S3: Pi DSM is working.

S15: Also working.

Sensor Status:

T/RH:  Lots of issues: 42m now Ifan=0, don't know if this is real;
P: ok; Pirga still lower by about 1mb
2D Gill: ok
csat u,v:  S3 looked bad 2 days ago when initially restarted through Pi.  Now not sure.  
csat ldiag: ok
csat w, tc: ok
EC150: ok
motes: S15 was up at least a bit yesterday, but not sure when we left.
Wetness: ok
radiation: ok
Tsoil: ok
Gsoil: ok
Qsoil: S9 and S15 ok; S1 is dead (a new EC-5 sensor may be coming from Boulder)
Cvsoil: still need to have John/Steve S come up with a fix (in Boulder)

Rainr: ok
Vbatt: ok

This visit was to take care of numerous instrument and infrastructure issues:

S3: Installed Pi DSM with old Cradlepoint and old Verizon modem (after lots of testing/configuration yesterday and this morning).  Station is now completely up.  This is a bit surprising, since we had some CSAT3 rebooting errors a few days ago, but it looks okay now (knock on wood).  I think we forgot to dress the cell modem external antenna cable.  Also finished installation of raised DSM with a bottom support rod.

S2: Reinstalled TRH crossarm/cable/sensor that I had removed Sunday to get the TRH off.  (I ended up drilling out the setscrew, but didn't damage the helicoil much.)  Note that fan is wired always on and reports Ifan=0.  Raised the DSM.

S14: Swapped out TRH.  Found that the fuse was blown and replaced it stealing from the GPS port.  Raised the DSM.  Also raised the battery box.

S9: Swapped out TRH.  This fan also always on with Ifan=0.

S15: Powered off, then on with only cell modem connected to USB.  It worked.  Plugged in Bluetooth to mote.  It worked.  Found that data had successfully been recorded on the on-board SD card.  Powered off, moved SD card to Acer, copied raw_data files to Acer, reinstalled SD card on Titan, powered up again (with both modem and Bluetooth), and everything came up again.

So now, the only sensor issues are Qsoil.s1 (which should arrive in the mail today) and the TP01s.

We'll go back to Home Depot for more support rods and battery box shims.

At the moment, this is up.  With hints from syslog during a crash, and then Googling, I've done:

/boot/cmdline.txt – added "dwc_otg.fiq_fix_enable=0", though this may not have been needed (update: yes, it is...)

/etc/default/avahi-daemon – set detect parameter to 0 to turn off avahi .local checking (couldn't find where unicast .local is being defined, so just turned the check off)

3 stations were dead this morning, so a trip to the marsh (on our day off):

S15: USB modem and USB stick both not talking, despite power cycle.  Reconfigured DATAMNT to send data to compact flash instead of modem, removed stick, removed USB hub – modem now directly plugged in.  cycled power, still don't have cell connection, but at least saving data locally.  Note that this was 2 visits, the second was when we actually got local storage working

S7: console not responding.  Power cycle brought back cell.  USB stick took a while to mount, but now okay

S3: pi in bad non-connecting mode.  removed and brought back to base.  Now plugged into cradlepoint, but c-point not connecting to cell modem.  Will work on tomorrow.

S2: verified that TRH fan is dead, but couldn't remove to bring back because setscrew frozen in place.  Will need kroil next visit.

update: brought kroil, but still wouldn't budge.  Brought entire crossarm back to base and eventually drilled out the setscrew.

This list for my own use (though others are welcome to comment/add)...Steve

sonics: have Gordon/Kate shoot a third set; take median of the 3 for boom angles; put into QC files

TRH: check T against a neighbor anytime Ifan=0 to see if fan was really off.  If on, reset Ifan minValue.

TRH: add offset from postcal to one sensor that read off

(DONE, on eddy) Pirga: adjust offset

Qsoil: just use median of gravimetrics everywhere (need to take more)

Cvsoil: just use median of values taken (need to get one working from Steve S.)

tc: correct based on Tc using Tom's method; use when TRH dies

Rad: remove cleaning periods?; remove wetness-not-dry

Wetness: convert S15 to water-on-surface flag

(DONE, on eddy) Rain: remove test tips

IRGA: remove cases with water on lens (there should be a diag flag for this?)

(DONE) Remove all S3 data from ~1800 9/30 to ?? 10/1 while testing/configuring Pi DSM

on 10/3, tower rebooted (probably low power) with wrong date; 2 misnamed data files were created.
(Not only are the files misnamed, all the recorded timetags are wrong as well.  chrony was running, but simply logged that the PPS/NMEA inputs were hugely different – it didn't correct the system clock.  We would have to adjust the timetags (ideally, using the valid GPS messages in the data to synchronize, though we don't have PPS) of all samples.  Gordon doesn't have a utility to do this and recommends simply tossing the 1.5 hours of data in these files.  Unfortunately, winds are from the north to northeast during this period at 4 m/s, which could be good data.)

(DONE, on eddy) Qsoil has some downward spikes in early Oct – add a min threshold of 60(!).

Pirga: s15 Changed(?) to internal on 21 Oct 12:00, added offset of 2.6mb.  Bad data while this change was made from 11:25-12:00.  s13 had level shifts during rain on 22 Oct.

Went in knowing that there were lots of issues, related to the high water from the recent storm.  Out from trailer 1300-1700 with Cristina:

S3: Station dead due to water getting into DSM through duct-taped Gordon hole.  Obvious bits of corrosion on power interface board.  Wiped off water on board and replaced blown main fuse, but new fuse blew within a few seconds (DSM LEDs were briefly all green.)  Removed DSM for debugging/replacement.  Capped cables by putting into an empty water bottle.  Strapped lid down on battery box.

S1: Power monitor board blown due to shorting with obvious corrosion.  Replaced with spare we had brought.  Strapped lid down on battery box.

S11: Power monitor board blown due to shorting with obvious corrosion.  Battery cooler lid was ajar.  Left in place.  Strapped lid down on battery box.

S7: Power monitor board obviously had been wet, but still reporting.  However, gain on Vbatt was 0.  Changed with vbg=10 and now get a voltage (though calibration probably isn't correct).  Strapped lid down on battery box.

S15: Water everywhere, but station was working.  Tried cycling power on both sides of Bluetooth, but could get data to flow.  Rebooting DSM brought back the Bluetooth connection.  During this visit, the blue Bulgin cap for the console port blew away.  We need to replace this.  (Why aren't spares in the box or laptop case or toolbag?)

We noticed that Vbatt was only about 12.0V and likely wouldn't last the night.  No charging happened today due to stormy weather.  We decided to swap in our one spare battery.  Found that the entire battery box was mostly filled with water, so dumped that out.  (Hope the others in the stations aren't similarly filled!)  Even though we tried to keep the charge controller going, it reset and lost its settings.  Set md=1 again and now seems to be happy.

So I've now taken just one core each at S9 and S15 (the only sites with Qsoil sensors).  The results were: EC-5 67%/65% and Gravimetric 90%/89%, which could be interpreted as the EC-5 having a bias of 23%.  A bias or gain change is plausible, since we are in saline conditions, though I don't know if the marsh exceeds the EC5 salinity operating range (8 dS/m for mineral soils).  (Seawater is supposed to be 54 dS/m, so 8/54 = 15% seawater certainly could be exceeded in the marsh.)  I'm actually impressed that the two EC-5 probes read as close as they do, that the two gravimetric samples were close, and that the higher EC-5 reading was associated with the higher gravimetric reading.  This is especially true given the difficultly in obtaining a gravimetric core, with lots of sample handling issues.

Clearly, it takes more than one point to develop a calibration.  Usually, we take a sample in wet conditions (shortly after rain) and dry conditions, to have a large calibration range.  I had hoped that data from high-tide and low-tide would produce such a variation; however, during the entire experiment here, the EC-5 probes have varied in time by only 1%.  It would be unprecedented for the accuracy of our gravimetric readings to be 1%, usually due to spatial heterogeneity, and I don't see why here would be any different.  Thus, all of our samples here will be effectively at only one soil moisture level, which will make it impossible to distinguish a gain from an offset.

Perhaps all this is a good thing:  if Qsoil doesn't vary with time, it can be treated as constant and set to the average value of the gravimetric readings, essentially ignoring the EC-5 data.  Since they will be important, we should still take more gravimetric samples.

Once we get the TP01 probes sampling again, we could try looking at the bulk soil heat capacity measurements to confirm/deny that Qsoil is constant.  (I do wonder if the TP01 heat pulse method itself will be valid in "soils" with flowing water...)

At 0310 this morning, S9's TRH Ifan jumped to 330mA and stayed there for 30min.  There was heavy rain at the time.  If the PIC electronics had been working properly, this couldn't have happened.  (The PIC would have shut it down after one sample.)  However, Iload doesn't show a jump in current at all, indicating that this high Ifan reading wasn't real.

About 0340, the sensor stopped reporting completely, though Iload if anything increased a bit.

I manually connected to the sensor at 0810.  Initially, rserial showed that no data were coming in and no response to R, thus the PIC was hung.  With tio 2 0/1, got the PIC running again.  Ifan was immediately 0 (didn't have one sample high and get turned off by the PIC).  Iload if anything got lower at exactly this time, hinting that the fan might actually be off now.  

However, the temperature reading cooled during the first 5 min after this restart.  Usually, I would say that the sensor had heated up in the sun and that the cooling indicates a fan that was stopped and is now running.  Rsw.in only shows 20 W/m2, though (it is raining, with lots of cloud), so it is hard to believe that the sensor would have that much radiation heating.  The only other source of heat could be a stuck fan, but I don't quite believe this either.

5 min after being restarted, the temperature agrees with the adjacent S1.

Ultimately, we don't know if the fan is running or not to determine if we have good T,RH data.  (But errors would be small in the cloudy, windy conditions now.)

We had several reasons to go out, especially that TRH.S11 was reading Ifan=0, even after restarts.  Went in&out via the tiki cut, but did hit bottom once on the way out.  Wind direction seemed perfect for a wake to hit S2.  However, I've looked at the data from this period, e.g. from 12:40-13:00 when the wind direction of 70deg should have directly hit S1, and see no evidence of a wake in the raw data or w-spectra.  If anything, w high-frequencies are somewhat lower than at S3, for example.

S13: shot boom angles.  discovered that the monopod interferes, so shot other angles without it.

S11: (0953?) shot boom angles and replaced TRH sensor.  Turned out that fan was on, despite reading 0, so it is a current monitoring issue.  Data QC could allow last night's data to be okay.  It took a while for the new TRH sensor to cool down.

S14: shot boom angles (via the African Queen route)

S3: (1059?) shot boom angles

S2: shot boom angles

S1: (1112) shot boom angles (now the 3rd set) and cleaned radiometer.

(S4:) From the boat, saw osprey sitting on the crossarm opposite the Gill.  The bird's body was a bit taller than the Gill.  (We are not completely sure that this is when we saw it – could have been the previous trip.  In any case, I haven't been able to find an effect of it in the data for both trips (which surprises me).)

S9: (1126) cleaned radiometer (but now raining)

S6: shot boom angles

S15: shot boom angles and (1143) cleaned radiometer (again wet)

Is this a record for number of sites (9) in one trip?