The 4-component radiometers near the top of the tower are above the 38m EC150/CSAT. Steve and I agree that these are at (about) 40m. We don't know why the config shows them at 36m.
Oncley up to 38m to do LAI measurements, then back down through 2m (see LAI entry)
1027 - Oncley cleaned krypton at 2m, then up to 20m using the wetness sensor as the indicator of cleaning times (both krypton/IRGA and PARs), then Oncley back down.
1110 - Semmer up to 44m to do breath test on "funnel" inlet. Did 5 breaths that had delay times of 8 or 9 seconds.
1128-1129 cleaned 44m Li7500
1132 cleaned 40m rad
1138-1140 cleaned 38m EC150
1143-1147 cleaned 32m Li7500 and PAR
1148-1153 cleaned 26m Li7500 and PAR
Semmer off tower at 1159
1159-1200 Oncley cleaned sawhorse rads (forgot to do PAR)
To see if it helps avoid the infrequent situations where PC104 interrupts cease to be serviced, I changed the second emerald board on the low DSM to use IRQ4:
ddn set_emerald -e /dev/emerald1 0x140 4 /etc/init.d/emerald start dup
After that, here's the current configuration. Since the config is saved in the EEPROM on the emerald, it should come up in this state after a reboot:
root@low root# set_emerald /dev/emerald0 current port config: port 0x100 irq 3 port 0x108 irq 3 port 0x110 irq 3 port 0x118 irq 3 port 0x120 irq 3 port 0x128 irq 3 port 0x130 irq 3 port 0x138 irq 3 root@low root# set_emerald /dev/emerald1 current port config: port 0x140 irq 4 port 0x148 irq 4 port 0x150 irq 4 port 0x158 irq 4 port 0x160 irq 4 port 0x168 irq 4 port 0x170 irq 4 port 0x178 irq 4 root@low root# cat /proc/tty/driver/serial serinfo:1.0 driver revision: 0: uart:XScale mmio:0x40100000 irq:38 tx:3573 rx:0 RTS|DTR 1: uart:XScale mmio:0x40200000 irq:37 tx:20 rx:22438852 RTS|CTS|DTR 2: uart:XScale mmio:0x40700000 irq:36 tx:0 rx:2926614 RTS|DTR 3: uart:16550A mmio:0x14300010 irq:116 tx:5235 rx:11378936 fe:19 RTS|DTR 4: uart:16550A mmio:0x14300000 irq:115 tx:27 rx:20225583 RTS|CTS|DTR 5: uart:ST16654 port:F1000100 irq:3 tx:0 rx:0 6: uart:ST16654 port:F1000108 irq:3 tx:0 rx:0 7: uart:ST16654 port:F1000110 irq:3 tx:0 rx:80133446 fe:11 RTS|CTS|DTR 8: uart:ST16654 port:F1000118 irq:3 tx:0 rx:411388 RTS|CTS|DTR 9: uart:ST16654 port:F1000120 irq:3 tx:15 rx:22437904 RTS|CTS|DTR 10: uart:ST16654 port:F1000128 irq:3 tx:0 rx:2947936 RTS|DTR 11: uart:ST16654 port:F1000130 irq:3 tx:0 rx:427094 RTS|CTS|DTR 12: uart:ST16654 port:F1000138 irq:3 tx:15 rx:19230482 fe:1 RTS|CTS|DTR 13: uart:ST16654 port:F1000140 irq:4 tx:0 rx:2945680 RTS|DTR 14: uart:ST16654 port:F1000148 irq:4 tx:0 rx:80119263 fe:17 RTS|CTS|DTR 15: uart:ST16654 port:F1000150 irq:4 tx:0 rx:410093 RTS|DTR 16: uart:ST16654 port:F1000158 irq:4 tx:27 rx:20204969 fe:11 RTS|CTS|DTR|DSR|CD|RI 17: uart:ST16654 port:F1000160 irq:4 tx:0 rx:0 18: uart:ST16654 port:F1000168 irq:4 tx:0 rx:2917645 RTS|DTR 19: uart:ST16654 port:F1000170 irq:4 tx:15 rx:19234163 fe:1 RTS|CTS|DTR 20: uart:ST16654 port:F1000178 irq:4 tx:0 rx:2491823 RTS|CTS|DTR root@low root# irqs Counting interrupts over 5 seconds ... IRQ Interrupt Type Total Int Int/sec ------------------------------------------------------ 3: ISA serial: 742 148.4 4: ISA serial: 819 163.8 24: GPIO-l eth0: 58 11.6 25: GPIO-l GPIO1-PC104: 1412 282.4 36: SC serial: 15 3 37: SC serial: 102 20.4 42: SC ost0: 509 101.8 114: GPIO isp116x-hcd:usb1: 92 18.4 115: GPIO serial: 227 45.4 116: GPIO serial: 100 20 120: GPIO pata_pcmcia: 8 1.6
Steve O. took a soil sample yesterday. Data has been entered in the soil table log entry.
Weather: Overcast
Summary: Things are looking good. We may do a LAI measurement today if it stays overcast
Sonics: ok
TRH, P: ok, pressure bump at about 2:30 this morning
Wind Speed & Dir: ok
Soil, Rainr: tsoila.4.4cm out, some rain events last night from WXT and OTT
Vmote (rad, a, a2, b): 13.1, 12.9, 13.1, 12.6
Radiation, Wetness: ok
H2O, CO2: ok
System (GPS, Vdsm): ok
Pond: ok
11:45 Low lost communication with its emerald boards around 11:00.
Rebooted low and everything back up.
Information for Gordon:
root@low root# irqs
Counting interrupts over 5 seconds ...
IRQ Interrupt Type Total Int Int/sec
------------------------------------------------------
24: GPIO-l eth0: 2 0.4
36: SC serial: 15 3
37: SC serial: 102 20.4
42: SC ost0: 508 101.6
114: GPIO isp116x-hcd:usb1: 290 58
115: GPIO serial: 226 45.2
116: GPIO serial: 103 20.6
end of dmesg
i2c i2c-0: i2c_pxa: timeout waiting for bus free
i2c i2c-0: i2c_pxa: timeout waiting for bus free
i2c i2c-0: i2c_pxa: timeout waiting for bus free
i2c i2c-0: i2c_pxa: timeout waiting for bus free
i2c i2c-0: i2c_pxa: timeout waiting for bus free
i2c i2c-0: i2c_pxa: timeout waiting for bus free
handle_IRQ_event called 4 times for IRQ 3
handle_IRQ_event called 4 times for IRQ 3
root@low root#
root@low root#
Added by Gordon, Jun 26:
/var/log/isfs/kernel has those dmesg messages, with timetags
Jun 23 16:49:50 low kernel: i2c i2c-0: i2c_pxa: timeout waiting for bus free Jun 23 16:49:53 low last message repeated 5 times Jun 25 09:27:58 low kernel: handle_IRQ_event called 4 times for IRQ 3 Jun 25 17:46:14 low kernel: handle_IRQ_event called 4 times for IRQ 3
Those were the only messages before the reboot, and they occurred at least 23 hours earlier, which means the problem is not due to a kernel oops, or any other atypical event that the kernel could detect. It is just the good ol' situation where there seems to be a very small possibility that a PC104 interrupt can be missed, and not retriggered, even though the PC104 IRQ interrupt line is high, such that the interrupt handler is never again called.
I believe restarting the dsm process with a ddn/dup, which closes and re-opens the serial ports, will bring it back too
I just updated the xml on the low DSM so that every sensor has a timeout. The dsm process should then close and reopen each port after detecting the timeout, which should also help to recover from this situation more quickly.
Seems that I need to install a PC104 interrupt watchdog module. There is some indication this has happened on the aircraft, also quite infrequently. A test is being setup out at RAF.
When the PC104 interrupts are being handled, the irqs listing looks like so, showing 275 interrupts/sec from the Emerald cards:
root@low root# irqs Counting interrupts over 5 seconds ... IRQ Interrupt Type Total Int Int/sec ------------------------------------------------------ 3: ISA serial: 1376 275.2 24: GPIO-l eth0: 62 12.4 25: GPIO-l GPIO1-PC104: 1376 275.2 36: SC serial: 15 3 37: SC serial: 101 20.2 42: SC ost0: 509 101.8 114: GPIO isp116x-hcd:usb1: 90 18 115: GPIO serial: 228 45.6 116: GPIO serial: 102 20.4
Weather: mostly sunny, slight chance of rain in the afternoon
Summary: Tsoila.4.4cm bad, some low signal diag on 38.m sonic this morning
H2o at 38m action between 4:00 and 6:00.
Sonics: ok, some recent ldiag activity at ~9:00am on 38m
TRH, P: ok
Wind Speed & Dir: ok
Soil, Rainr: ok, tsoila.4.4cm
Vmote (rad, a, a2, b): 13.1, 13.0, 13.2, 12.7
Radiation, Wetness: ok
H2O, CO2: ok, H2O at 38m showed some action between 4:00 and 6:00 this morning
System (GPS, Vdsm): ok
Pond:ok
9:30 - 10:00 Moved the base mote from about 3m up the tower down to ground level to see if that improves the reception from the rad and soil motes.
15:00 Moved it again, since soil.b seemed to have a lot of missed data (gaps of 40--120s). Now seems okay, but still in 10.0/0.2 sec bursts.
~10:00 Steve S. just replaced the battery at soil.b. V went from 11.3 to 13.2, but it is dropping a bit.
~10:30 to 12:30 Spent time replacing the batteries at rad, soila and soila2 with one large battery.
I noted yesterday that an extension cord going to pond had been chewed on (a bit of copper was showing). Since it was raining, decided not to do anything then. Taped over it this morning (unplugged cable, taped, plugged back in) from 09:02-09:14.
During this process, I note that Vdsm went from 12.4 (plugged in) to 11.9 (just battery). This tells me that the power supply voltage that charges the battery is a bit low. I went back to check if this power supply voltage could be adjusted, but it can't (at least without taking the supply apart, which I didn't want to do because it is quite hot). It has been running for weeks like this, so I would expect it to keep running.
Weather: Clear skies this morning, possible showers in the afternoon
Summary: Tsoil.a at 4.4cm bad, low batteries on rad, soil.a2, and soil.b. EC150 & Licor 43.9m
had some events last night. Qsoil.a to be monitored.
Sonics: ok
TRH, P: ok
Wind Speed & Dir: ok
Soil, Rainr: ok, low batteries and tsoil.a 4.4cm bad. Minor precipitation activity during the night.
Qsoil.a showing suspicious activity since yesterday morning.
Vmote (rad, a, a2, b): 11.5, 11.9, 11.5, 11.2.
Radiation, Wetness: ok
H2O, CO2:ok, EC150 & 43.9m Licor had some strange action last night, ~3:00 to 7:00.
System (GPS, Vdsm): ok
Pond: ok
Averaged the GPS latitude, longitude and altitude for Jun 12 00-08Z, an arbitrary choice of time.
Note the altitudes are not very accurate, especially lower on the tower. Can't imagine that there is an altitude difference of 19 meters between pond and low. A 30 meter difference between pond and high is getting close.
Google map, with terrain overlay and 40 foot contours, indicates the pond site is something like 170 feet altitude = 52 meters.
gps |
lat |
stddev |
lon |
stddev |
alt (m) |
stddev |
|---|---|---|---|---|---|---|
low |
32.69467 |
0.000022 |
-87.24881 |
0.000022 |
73.4 |
3.8 |
mid |
32.69459 |
0.000019 |
-87.24878 |
0.000000 |
76.8 |
3.4 |
high |
32.69456 |
0.000019 |
-87.24880 |
0.000009 |
84.8 |
1.9 |
pond |
32.69603 |
0.0 |
-87.25576 |
0.0 |
54.7 |
2.2 |
11:20 - The 13.9m sonic is having problems. I assume it is due to the recent rain. Will monitor.
UPDATE, 11:50 - 13.9m looking good, idiag normal
10:58 - Tsoil.b has been replaced with the new probe Steve O. installed yesterday. The only good signal from the old probe (Tsoil.1.9cm.a) is 0.15C lower, which indicates that the new probe is <mostly> settled in. Note that the rain this morning should have helped "heal" the soil as well.
9:25 - The internet link to the fish hatchery has been acting up so we are doing a test to try and
isolate the cause. Since Gordon had success with lowering the power on the AP24 we decided to
shut it down.
UPDATE, 10:30 : We went through a process of shutting down each interant and checking Paul's link.
There was no improvement. We then shut down the SODAR and again not success. Paul will
call his provider and we will start getting things back on the air.
UPDATE, 10:45 : Everything back on the air