Weather: low clouds burning off this morning, potential shower in the afternoon
Summary: Rain event last night starting at 8:30pm. Lost soila and soilb. A lot of low signal diagnostics
from sonics during storm.
Sonics: ok, low signal diagnostics on all sonics during storm
TRH, P: ok
Wind Speed & Dir: ok
Soil, Rainr: rain event around 8:30pm. Soila out at ~11:00pm, Soilb out at ~8:30pm
Vmote (rad, a, a2, b): 13.0, NA, 13, NA
Radiation, Wetness: ok
H2O, CO2: ok
System (GPS, Vdsm): ok
Pond:ok
Here in Tuscaloosa, we've seen spectacular lightning to the south of us. Haven't seen evidence of it in the data yet, but our site could be getting more rain soon.
P.S. Just after I wrote this, the WWW plots updated with 1/2" of rain and a temperature drop of 4 degC in 10min.
P.P.S. And now the 32m sonic is not able to trigger properly. Tried eio 7 0/1, but didn't help.
Just in case there is an effect on the data, here is my attempt to reconstruct tours of the tower and radiometer stands:
6/19 1530-1630 Program managers tour.
6/23 1130-1400 Ann Marie and sons tour.
6/27 1300-1500 Karsten and Lindsey tour.
These events could have spikes on the 2m chemistry inlet, as people were standing near the tower base. There also could be higher Rlw.out or Rlw.in.
We had rain during lunch. It was pretty heavy for a while in Marion, but AABC didn't seem to get much. However, temperatures dropped by 8 degC, so things are very nice. (We've turned off the trailer AC.)
Steve S was on the tower from ~0915-0930 to dry off the krypton and 14m sonic. All are now working again (though sonic may have revived itself a bit earlier). Kryptons are still only about 0.1V. Steve noted that there was an obvious water drop hanging on the sonic.
Weather: overcast but starting to burn off, possible rain in the afternoon
Summary: Rain event last night at 11:30pm. 13.9m sonic went bad, IDIAG low signal, at that time and has not recovered.
8m licor CO2 seems high compared to others, will monitor.
Sonics: 13.9m sonic out at ~11:30pm.
TRH, P: ok
Wind Speed & Dir: ok
Soil, Rainr: rain last night at ~11:30pm. Tsoila.4.4cm bad.
Vmote (rad, a, a2, b): 13.0, 12.9, 13.1, 12.4
Radiation, Wetness: ok
H2O, CO2: ok, licor 8m CO2 looks high, will monitor
System (GPS, Vdsm): ok
Pond:ok
The gravimetric tests of the Qsoils are not so hot.
soil.b gravimetric readings have all been within 1% of each other (no change in soil moisture), whereas the EC5 data show a 4% change (still not much change). With such a tight clustering, we can't fit a line, but the data suggest an offset of ~-7%.
soil.a gravimetric readings are quite variable (38 - 51%), but the range in EC5 values are half as much. The middle data point doesn't agree at all, though a line with slope=1 and offset of -8% fits the remaining (2) samples.
Thus, the indication is that both EC5s are reading about 8% low. Clearly, we need a few more samples, preferably in a wider range of soil moisture conditions.
P.S. With the 4th sample taken on 3 July, there is no change to the above statements. An offset of 8% seems to be the best we can do.
The 4-component radiometers near the top of the tower are above the 38m EC150/CSAT. Steve and I agree that these are at (about) 40m. We don't know why the config shows them at 36m.
Oncley up to 38m to do LAI measurements, then back down through 2m (see LAI entry)
1027 - Oncley cleaned krypton at 2m, then up to 20m using the wetness sensor as the indicator of cleaning times (both krypton/IRGA and PARs), then Oncley back down.
1110 - Semmer up to 44m to do breath test on "funnel" inlet. Did 5 breaths that had delay times of 8 or 9 seconds.
1128-1129 cleaned 44m Li7500
1132 cleaned 40m rad
1138-1140 cleaned 38m EC150
1143-1147 cleaned 32m Li7500 and PAR
1148-1153 cleaned 26m Li7500 and PAR
Semmer off tower at 1159
1159-1200 Oncley cleaned sawhorse rads (forgot to do PAR)
To see if it helps avoid the infrequent situations where PC104 interrupts cease to be serviced, I changed the second emerald board on the low DSM to use IRQ4:
ddn set_emerald -e /dev/emerald1 0x140 4 /etc/init.d/emerald start dup
After that, here's the current configuration. Since the config is saved in the EEPROM on the emerald, it should come up in this state after a reboot:
root@low root# set_emerald /dev/emerald0 current port config: port 0x100 irq 3 port 0x108 irq 3 port 0x110 irq 3 port 0x118 irq 3 port 0x120 irq 3 port 0x128 irq 3 port 0x130 irq 3 port 0x138 irq 3 root@low root# set_emerald /dev/emerald1 current port config: port 0x140 irq 4 port 0x148 irq 4 port 0x150 irq 4 port 0x158 irq 4 port 0x160 irq 4 port 0x168 irq 4 port 0x170 irq 4 port 0x178 irq 4 root@low root# cat /proc/tty/driver/serial serinfo:1.0 driver revision: 0: uart:XScale mmio:0x40100000 irq:38 tx:3573 rx:0 RTS|DTR 1: uart:XScale mmio:0x40200000 irq:37 tx:20 rx:22438852 RTS|CTS|DTR 2: uart:XScale mmio:0x40700000 irq:36 tx:0 rx:2926614 RTS|DTR 3: uart:16550A mmio:0x14300010 irq:116 tx:5235 rx:11378936 fe:19 RTS|DTR 4: uart:16550A mmio:0x14300000 irq:115 tx:27 rx:20225583 RTS|CTS|DTR 5: uart:ST16654 port:F1000100 irq:3 tx:0 rx:0 6: uart:ST16654 port:F1000108 irq:3 tx:0 rx:0 7: uart:ST16654 port:F1000110 irq:3 tx:0 rx:80133446 fe:11 RTS|CTS|DTR 8: uart:ST16654 port:F1000118 irq:3 tx:0 rx:411388 RTS|CTS|DTR 9: uart:ST16654 port:F1000120 irq:3 tx:15 rx:22437904 RTS|CTS|DTR 10: uart:ST16654 port:F1000128 irq:3 tx:0 rx:2947936 RTS|DTR 11: uart:ST16654 port:F1000130 irq:3 tx:0 rx:427094 RTS|CTS|DTR 12: uart:ST16654 port:F1000138 irq:3 tx:15 rx:19230482 fe:1 RTS|CTS|DTR 13: uart:ST16654 port:F1000140 irq:4 tx:0 rx:2945680 RTS|DTR 14: uart:ST16654 port:F1000148 irq:4 tx:0 rx:80119263 fe:17 RTS|CTS|DTR 15: uart:ST16654 port:F1000150 irq:4 tx:0 rx:410093 RTS|DTR 16: uart:ST16654 port:F1000158 irq:4 tx:27 rx:20204969 fe:11 RTS|CTS|DTR|DSR|CD|RI 17: uart:ST16654 port:F1000160 irq:4 tx:0 rx:0 18: uart:ST16654 port:F1000168 irq:4 tx:0 rx:2917645 RTS|DTR 19: uart:ST16654 port:F1000170 irq:4 tx:15 rx:19234163 fe:1 RTS|CTS|DTR 20: uart:ST16654 port:F1000178 irq:4 tx:0 rx:2491823 RTS|CTS|DTR root@low root# irqs Counting interrupts over 5 seconds ... IRQ Interrupt Type Total Int Int/sec ------------------------------------------------------ 3: ISA serial: 742 148.4 4: ISA serial: 819 163.8 24: GPIO-l eth0: 58 11.6 25: GPIO-l GPIO1-PC104: 1412 282.4 36: SC serial: 15 3 37: SC serial: 102 20.4 42: SC ost0: 509 101.8 114: GPIO isp116x-hcd:usb1: 92 18.4 115: GPIO serial: 227 45.4 116: GPIO serial: 100 20 120: GPIO pata_pcmcia: 8 1.6
Steve O. took a soil sample yesterday. Data has been entered in the soil table log entry.
Weather: Overcast
Summary: Things are looking good. We may do a LAI measurement today if it stays overcast
Sonics: ok
TRH, P: ok, pressure bump at about 2:30 this morning
Wind Speed & Dir: ok
Soil, Rainr: tsoila.4.4cm out, some rain events last night from WXT and OTT
Vmote (rad, a, a2, b): 13.1, 12.9, 13.1, 12.6
Radiation, Wetness: ok
H2O, CO2: ok
System (GPS, Vdsm): ok
Pond: ok
11:45 Low lost communication with its emerald boards around 11:00.
Rebooted low and everything back up.
Information for Gordon:
root@low root# irqs
Counting interrupts over 5 seconds ...
IRQ Interrupt Type Total Int Int/sec
------------------------------------------------------
24: GPIO-l eth0: 2 0.4
36: SC serial: 15 3
37: SC serial: 102 20.4
42: SC ost0: 508 101.6
114: GPIO isp116x-hcd:usb1: 290 58
115: GPIO serial: 226 45.2
116: GPIO serial: 103 20.6
end of dmesg
i2c i2c-0: i2c_pxa: timeout waiting for bus free
i2c i2c-0: i2c_pxa: timeout waiting for bus free
i2c i2c-0: i2c_pxa: timeout waiting for bus free
i2c i2c-0: i2c_pxa: timeout waiting for bus free
i2c i2c-0: i2c_pxa: timeout waiting for bus free
i2c i2c-0: i2c_pxa: timeout waiting for bus free
handle_IRQ_event called 4 times for IRQ 3
handle_IRQ_event called 4 times for IRQ 3
root@low root#
root@low root#
Added by Gordon, Jun 26:
/var/log/isfs/kernel has those dmesg messages, with timetags
Jun 23 16:49:50 low kernel: i2c i2c-0: i2c_pxa: timeout waiting for bus free Jun 23 16:49:53 low last message repeated 5 times Jun 25 09:27:58 low kernel: handle_IRQ_event called 4 times for IRQ 3 Jun 25 17:46:14 low kernel: handle_IRQ_event called 4 times for IRQ 3
Those were the only messages before the reboot, and they occurred at least 23 hours earlier, which means the problem is not due to a kernel oops, or any other atypical event that the kernel could detect. It is just the good ol' situation where there seems to be a very small possibility that a PC104 interrupt can be missed, and not retriggered, even though the PC104 IRQ interrupt line is high, such that the interrupt handler is never again called.
I believe restarting the dsm process with a ddn/dup, which closes and re-opens the serial ports, will bring it back too
I just updated the xml on the low DSM so that every sensor has a timeout. The dsm process should then close and reopen each port after detecting the timeout, which should also help to recover from this situation more quickly.
Seems that I need to install a PC104 interrupt watchdog module. There is some indication this has happened on the aircraft, also quite infrequently. A test is being setup out at RAF.
When the PC104 interrupts are being handled, the irqs listing looks like so, showing 275 interrupts/sec from the Emerald cards:
root@low root# irqs Counting interrupts over 5 seconds ... IRQ Interrupt Type Total Int Int/sec ------------------------------------------------------ 3: ISA serial: 1376 275.2 24: GPIO-l eth0: 62 12.4 25: GPIO-l GPIO1-PC104: 1376 275.2 36: SC serial: 15 3 37: SC serial: 101 20.2 42: SC ost0: 509 101.8 114: GPIO isp116x-hcd:usb1: 90 18 115: GPIO serial: 228 45.6 116: GPIO serial: 102 20.4
Weather: mostly sunny, slight chance of rain in the afternoon
Summary: Tsoila.4.4cm bad, some low signal diag on 38.m sonic this morning
H2o at 38m action between 4:00 and 6:00.
Sonics: ok, some recent ldiag activity at ~9:00am on 38m
TRH, P: ok
Wind Speed & Dir: ok
Soil, Rainr: ok, tsoila.4.4cm
Vmote (rad, a, a2, b): 13.1, 13.0, 13.2, 12.7
Radiation, Wetness: ok
H2O, CO2: ok, H2O at 38m showed some action between 4:00 and 6:00 this morning
System (GPS, Vdsm): ok
Pond:ok
9:30 - 10:00 Moved the base mote from about 3m up the tower down to ground level to see if that improves the reception from the rad and soil motes.
15:00 Moved it again, since soil.b seemed to have a lot of missed data (gaps of 40--120s). Now seems okay, but still in 10.0/0.2 sec bursts.