June 26, 2012
11:00am - 2:05pm MDT
Removal of all sensors on the Turbulence Tower today. Sonics, Licors, TRHs, pressure, four-component, batteries and dsm. Cables have remained on tower. Kept beacon on also.
Since we left cables on tower we had to mark cables on the dsm side so we know what instrument it was and which port. This was our organization codes using colored paper clips, zip ties and electrical tape. All levels have a sonic, TRH and Licor which are bundled together. This is the code for the bundle.
CSAT marked with zip tie
TRH marked with tape
Licor marked with nothing.
Heights are marked with paper clip colors.
45m - Green
30m - Pink
15m - Blue
8m - Yellow
2m - White
Additional random cables.
Ethernet - 1 SMALL zip tie, no clip
Rad22m - 2 SMALL zip ties, no clip
Pressure - 3 SMALL zip ties, no clip
dsm was taken down at 11:35am.
WE NEED PRESSURE TUBING!
We have all the sensors and the data system running in the staging area. We've added one Licor 7500 to what was last on the tower, for a total of 4 Licors. The data cables, and Licor power cables are different however, since the originals were left on the tower.
After setting it up, I increased the 4 Licor 7500s from 10 samples/sec at 9600 baud to 20 samples/sec at 19200 baud.
The spurious interrupts are happening, sometimes spiking to above 100:
Jun 28 21:27:02 manitou kernel: viper_irq_handler: irq=2 spurious= 326001, #/sec=102 Jun 28 21:27:21 manitou kernel: viper_irq_handler: irq=2 spurious= 328001, #/sec=109 Jun 28 21:29:06 manitou kernel: viper_irq_handler: irq=2 spurious= 338001, #/sec=102 Jun 28 21:29:46 manitou kernel: viper_irq_handler: irq=2 spurious= 342001, #/sec=106 Jun 28 21:29:56 manitou kernel: viper_irq_handler: irq=2 spurious= 343001, #/sec=102 Jun 28 21:30:38 manitou kernel: viper_irq_handler: irq=2 spurious= 347001, #/sec=103 Jun 28 21:30:48 manitou kernel: viper_irq_handler: irq=2 spurious= 348001, #/sec=101 Jun 28 21:31:28 manitou kernel: viper_irq_handler: irq=2 spurious= 352001, #/sec=111 Jun 28 21:31:58 manitou kernel: viper_irq_handler: irq=2 spurious= 355001, #/sec=104 root@manitou root# uptime 21:44:36 up 1:43, 1 user, load average: 0.16, 0.11, 0.09
The kernel issues a "spurious" message when they occur more often than 100/sec. The system came up at 21:44 - 1:43 = 20:01. At the time of the last message above, the system had been up for 1 hour 31 minutes. 355001 interrupts in 1 hour 31 minutes averages to 65/sec.
The above test is with the same kernel (2.6.16.28-arcom1-2-viper #1 PREEMPT Wed Sep 16 17:04:19 MDT 2009) and CPU (viper 4) as was deployed on the tower.
manitou:/dev/ttyS9 1 20 14 2012 06 28 23:13:12.975 06 28 23:13:26.005 1.00 0.938 1.043 19 19 manitou:/var/tmp/gps_pty0 1 30 29 2012 06 28 23:13:12.574 06 28 23:13:26.574 2.00 0.154 0.882 72 73 manitou:/dev/ttyS1 1 100 285 2012 06 28 23:13:12.624 06 28 23:13:26.820 20.01 0.046 0.054 12 12 manitou:/dev/ttyS5 1 120 15 2012 06 28 23:13:12.893 06 28 23:13:26.489 1.03 0.965 0.979 30 30 manitou:/dev/ttyS6 1 200 285 2012 06 28 23:13:12.654 06 28 23:13:26.852 20.00 0.044 0.058 12 12 manitou:/dev/ttyS7 1 210 285 2012 06 28 23:13:12.619 06 28 23:13:26.814 20.01 0.042 0.059 56 56 manitou:/dev/ttyS8 1 220 14 2012 06 28 23:13:13.516 06 28 23:13:26.256 1.02 0.974 0.986 29 29 manitou:/dev/ttyS10 1 300 285 2012 06 28 23:13:12.652 06 28 23:13:26.854 20.00 0.041 0.061 12 12 manitou:/dev/ttyS11 1 310 285 2012 06 28 23:13:12.621 06 28 23:13:26.825 19.99 0.045 0.056 49 49 manitou:/dev/ttyS12 1 320 15 2012 06 28 23:13:13.098 06 28 23:13:26.775 1.02 0.971 0.986 29 29 manitou:/dev/ttyS19 1 330 3 2012 06 28 23:13:10.785 06 28 23:13:20.787 0.20 5.000 5.002 56 56 manitou:/dev/ttyS13 1 400 285 2012 06 28 23:13:12.658 06 28 23:13:26.858 20.00 0.046 0.054 12 12 manitou:/dev/ttyS14 1 410 286 2012 06 28 23:13:12.593 06 28 23:13:26.845 20.00 0.047 0.052 49 49 manitou:/dev/ttyS15 1 420 14 2012 06 28 23:13:13.374 06 28 23:13:26.024 1.03 0.969 0.980 30 30 manitou:/dev/ttyS20 1 500 286 2012 06 28 23:13:12.622 06 28 23:13:26.869 20.00 0.047 0.053 12 12 manitou:/dev/ttyS17 1 510 285 2012 06 28 23:13:12.619 06 28 23:13:26.824 19.99 0.042 0.058 49 49 manitou:/dev/ttyS18 1 520 14 2012 06 28 23:13:13.284 06 28 23:13:25.990 1.02 0.970 0.984 30 30
Gordon Jun 27, 2012
After removing the sensors and data system, we're testing the set up back in Boulder, to see if we can duplicate the problem of not being able to sample 5 licors.
In setting up the test, noticed that the fan on TRH012 does not run, but the TRH does output data. Checking the data archive, that unit is data id 1,420, which was the 30 meter TRH. Will check with Chris in case he noticed whether the fan was running when he removed the sensor.
If the fan was out while the unit was on the tower, then T,RH at 30m data should be treated with suspicion. By looking at the profiles, one might be able to determine when the fan went out.
According to Ned, it is very likely that the fan connection was damaged when the unit was removed from the tower.
The entire turbulence tower was down from May 31 until June 8. During this time, we were able to ping the datalogger but unable to log in. Richard Oakes from the USFS cycled the power on the datalogger at about 2pm on June 8, which restored access to the system.
All sensors came happily back to life, with the exception of the 2m LiCor that Gordon was able to resuscitate via sending ASCII commands to the sensor.
Comments added by Gordon:
From a phone conversation with Richard Oakes: He recalled a hail and lightning storm in the vicinity of the MFO during the afternoon and early evening of May 31, and that his wife noticed a lightning strike that she thought was in the vicinity of the turbulence tower that afternoon. His record indicates just 0.01 inch of precip on May 31.
To bring back the 2m licor, I used the commands shown in https://wiki.ucar.edu/x/CAK9B:
adn minicom ttyS2 ctrl-a w (enable line wrap) ctrl-a e (enable local echo) ctrl-a f (send BREAK) (Outputs (BW 10) (Delay 0) (RS232 (Freq 10.0))) ctrl-a q aup