Sensor status:
T: ok
RH: ok
Ifan: ok
spd: ok
P: ok
co2/h2o: ok
csat u,v,w: ok
csat ldiag: ok
soils: ok
Wetness: ok
Rsw/Rlw/Rpile: ok
Voltages: ok
sstat outputs: ok, ok
We had to recover the DSM and batteries for use in PECAN, so I removed these from ehs from about 1018-1038 today. The rest of the station is still in place, but we no longer are collecting data.
bao and the tower sensors are still up, and currently scheduled to start being removed 1 June. (Actually, I don't know if this extension to 1 June has officially been approved, but I am assuming that this will be the case.)
I introduced a bug in the R code yesterday such that it would try to create an infinitely long netcdf file. This is the code that adds the derived quantities to the netcdf files.
As a results ncharts was hanging, and probably also the R plots weren't working.
The bug is now fixed, and things should return to normal.
Sensor status:
T: ok
RH: ok
Ifan: ok
spd: ok
P: ok
co2/h2o: ok
csat u,v,w: ok
csat ldiag: ok
soils: Gsoil/Cvsoil.ehs has a lot of missing data during noontime
Wetness: ok
Rsw/Rlw/Rpile: ok
Voltages: ok
sstat outputs: ok, ok
Sensor status:
T: ok
RH: ok
Ifan: ok
spd: ok
P: ok
co2/h2o: ok
csat u,v,w: ok
csat ldiag: ok
soils: ok
Wetness: ok
Rsw/Rlw/Rpile: ok
Voltages: ok
sstat outputs: ok, ok
Sensor status:
T: T.300m missing
RH: T.300m missing
Ifan: ok
spd: ok
P: ok
co2/h2o: ok
csat u,v,w: ok
csat ldiag: ok
soils: ok
Wetness: ok
Rsw/Rlw/Rpile: ok
Voltages: ok
sstat outputs: ok, ok
Dan noticed that, as predicted a few logbook entries ago, T.300m was bad again. I manually restarted it at about 9 this morning. Since then, Gordon has written a script that checks tower TRH data every 10s and restarts the sensor if needed.
I also note the METCRAX-II logbook comment: "TRH.40m.rim restarted" where we found that the (bad) data are correctable. My (manual) code to implement this for METCRAXII is at $ISFF/projects/METCRAXII/ISFF/R/fixSHT.qq. It isn't clear that it is worth the effort to implement this fix for the 15 hours of data that are missing during this educational deployment...
If we decide to fix these data, we will need the following coefficients:
Sensor ID3 I2C ADD: 12 data rate: 1 (secs) fan(0) max current: 80 (ma)\n2015 04 09 15:01:26.7912 0.08542 44 \rresolution: 12 bits 1 sec MOTE: off\r\n
2015 04 09 15:01:26.8472 0.05602 28 calibration coefficients:\r\n
2015 04 09 15:01:26.8755 0.02831 21 Ta0 = -4.112729E+1\r\n
2015 04 09 15:01:26.9047 0.02921 21 Ta1 = 4.153065E-2\r\n
2015 04 09 15:01:26.9348 0.03005 21 Ta2 = -5.198994E-7\r\n
2015 04 09 15:01:26.9555 0.02074 21 Ha0 = -7.871138E+0\r\n
2015 04 09 15:01:26.9848 0.02932 21 Ha1 = 6.237115E-1\r\n
2015 04 09 15:01:27.0148 0.02994 21 Ha2 = -5.446227E-4\r\n
2015 04 09 15:01:27.0348 0.01999 21 Ha3 = 8.683383E-2\r\n
2015 04 09 15:01:27.0648 0.03001 21 Ha4 = 7.886339E-4\r\n
2015 04 09 15:01:27.0947 0.02994 21 Fa0 = 3.222650E-1\r\n
P.S. There now is a version of fixSHT.qq in the CABL/ISFF/R directory that implements this fix, though we now how to figure out how to apply this fix in our data flow. One possibility is to bundle it with the wind-direction computing script to write to both high-rate and 5-min NetCDF files.
Ultimately, of course, the solution is to fix the sensor/microprocessor. This is an intermittent problem that has affected at least 2 different sensors. Since it is a bit slip, it likely is an issue communicating between the PIC and the SHT. Thus, it appears that either the PIC, SHT, or the interface circuit is marginal in some aspect. Is the timing too fast? Is a pull-up resistor needed? Is some timing wait state needed?
From 20-22 UTC on 8 April, the wind direction held steady, at all levels, at ~ 340-350 deg.
Recall that the NW booms point to 334 deg and the SE booms point to 154 deg.
This flow condition results in a prolonged and distinct tower wake (or mast effect) on the sonics on the SE boom. Wind speeds at the SE booms during this time period are 4-5 ms-1 slower than those measured on the NW booms!
This is a perfect example of mast effects for our classes!
Sensor status:
T: ok
RH: ok
Ifan: ok
spd: ok
P: ok
co2/h2o: ok
csat u,v,w: ok
csat ldiag: "ldiag.nw.250m" "ldiag.nw.300m" "ldiag.se.300m" were all flagged from ~3:00 to 8:30 this morning
soils: ok aside from Qsoil
Wetness: ok
Rsw/Rlw/Rpile: ok
Voltages: ok
sstat outputs: ok, ok (uptime of 2 hr 21 min @ bao)
Sensor status: Note: "Qsoil.ehs" is returning bad data; "Tsoil.0.6cm.ehs" from 9:45 to 11:40 is NA
T: ok
RH: ok
Ifan: ok
spd: ok
P: ok
co2/h2o: ok
csat u,v,w: ok
csat ldiag: ok
soils: "Qsoil.ehs" is returning bad data; "Tsoil.0.6cm.ehs" from 9:45 to 11:40 is NA
Wetness: ok
Rsw/Rlw/Rpile: ok
Voltages: ok
sstat outputs: ok, ok
Sensor status:
T: ok
RH: ok
Ifan: ok
spd: missing spd.5m.bao from 16:40 to 22:30 during moisture event; also, a couple NA's at nw.250m, nw.300m, se.200m
P: ok
co2/h2o: wet but recovered. h2o.5m.bao is a bit lower than it was before the wetness event
csat u,v,w: same as 'spd'
csat ldiag: ok now
soils: Qsoil.ehs has lots of NA data
Wetness: ok
Rsw/Rlw/Rpile: ok
Voltages: ok
sstat outputs: ok, ok
I just noticed that 4 days ago, Tsoil.0.6cm.ehs revived itself, after being offscale since almost the beginning of the project!
In prep for a post-mortem on this project, I wanted to start a page of issues that have occurred:
| Issue | Action Taken | Future Actions |
|---|---|---|
| USB sticks died (all? were purple 8G Transcend) | Switched to Pocketec, Compact Flash, New stick | Replace all sticks with new ones recently purchased |
| GPS timing odd | Widened range of acceptable time lag | Upgrade software on all receivers (3.0-3.7 -> 3.9) |
| 200m DSM needed 150m GPS, but it died | Changed to use 250m? GPS | None? |
| Qsoil reading 0x8000 | none | Fix Check connector at tear-down |
| CSAT3 broken upon installation | Repaired at CSI – bad coax cable in head/box | Unknown - new handling procedure? |
| XBee mote not reliable over 3m distance | Changed to serial mote (tried changing power) | Change to all Bluetooth motes? |
| TRH went to large values | eio power cycle | Unknown |
| TRH fan controller died | Replaced | Check; see if systematic issue for all |
| TRH sensor died | None yet | Improve rain vulnerability? Lightning issue? |
| Lost data connection when eol-rt-data changed | eol-rt-data configuration restored crontab script fixed | SIG checks with Gordon (he tells them when using?) SIG avoids "snapshots" |
| Odd Gsoil at ehs | none | Don't install in darkhorse shadow! |
| EC150 bad data for a week | Cleaned bird scat | Install bird diverters? |
| tsoil.bao mote data intermittent | none; came back | Unknown |
| Mast issues: guy wires; winch; clamp positions; tight nubs | Various in-field adjustments | Systematically check each component for all masts |
| AC powering failed (probably loose connections) | Changed to solar power | Test this option (needed in future) |
| Picklefork Tsoil died (then revived!) | none (had installed duplicate picklefork) | New picklefork design |
| Picklefork completely died (bao) | none | Diagnose Create new picklefork probe (in process) |
| EC150 data garbled on ttyS4 at bao, OK on ttyS4 at ehs | Moved to Emerald serial port on bao | Diagnose. Seems to be a rs422 signal level or termination problem |
DSM networking didn't come up after power drop happened to 150m and 300m | Manually switched off/on | Add battery or ultra cap to main power line? Identify network interface weakness? |
| Mote communications failed on bao Tsoil | Enabled XBee watchdog | Change to Bluetooth Enable all watchdogs New whip antenna location? (birds/wind knock down) |
| New Tirga fan died | Replaced | Better rain protection? |
| New Tirga has radiation error | Installed double-shield | Might be acceptable now |
TRH at 300m has been reporting bad T and RH since about 15:20 today (April 1, no fooling):
rs 5
TRH3 166.68 222.66 29 0 5364 983 92\r\n
TRH3 166.68 223.29 28 0 5364 981 88\r\n
TRH3 166.68 223.60 28 0 5364 980 89\r\n
TRH3 166.68 223.60 29 0 5364 980 92\r\n
Power cycled it with
eio 5 0
eio 5 1
which brought it back:
rs 5
...
TRH3 13.55 29.97 30 0 1339 61 94\r\n
TRH3 13.55 29.97 29 0 1339 61 90\r\n
TRH3 13.50 29.97 30 0 1338 61 94\r\n
Sensor status:
T: ok
RH: ok
Ifan: ok
spd: ok
P: ok
co2/h2o: ok
csat u,v,w: ok
csat ldiag: couple flags
Tsoil: ok
Wetness: ok
Rsw/Rlw/Rpile: ok
Voltages: ok
sstat outputs: ok, ok