Blog

I should have put in the logbook last night that, when Gordon and I went to the tower last night to replace M 4m TRH, there was very light frost on the ground.

Below is a short segment from mote_dump 11,42 with only Tsoil data.  Note that values of 94.4x cycle through each soil depth measurement about every 6 samples.  I don't yet know whether this is a Mote, Sensor, Software, or interference issue.  Other sensors on this mote don't exhibit this behavior.  Clearly, these "spikes" are easy to remove, so I won't worry about it excessively.

2012 10 05 04:03:42.9490       0 11,0xaa23      16      94.44       6.09       7.44       8.34

2012 10 05 04:03:43.9481       0 11,0xaa23      16       4.27       6.09       7.44       8.34

2012 10 05 04:03:44.9409       0 11,0xaa23      16       4.27       6.09       7.44       8.34

2012 10 05 04:03:45.9333       0 11,0xaa23      16       4.27       6.09       7.44       8.34

2012 10 05 04:03:46.9452       0 11,0xaa23      16       4.27       6.09       7.44       8.34

2012 10 05 04:03:47.9494       0 11,0xaa23      16       4.27       6.08       7.44       8.34

2012 10 05 04:03:48.9374       0 11,0xaa23      16       4.27       6.08       7.44      94.48

2012 10 05 04:03:49.9325       0 11,0xaa23      16       4.27       6.08       7.44       8.34

2012 10 05 04:03:50.9556       0 11,0xaa23      16       4.27       6.08       7.44       8.34

2012 10 05 04:03:51.9416       0 11,0xaa23      16       4.27       6.08       7.44       8.34

2012 10 05 04:03:52.9437       0 11,0xaa23      16       4.27       6.08       7.44       8.34

2012 10 05 04:03:53.9469       0 11,0xaa23      16       4.27       6.08       7.44       8.34

2012 10 05 04:03:54.9419       0 11,0xaa23      16       4.26       6.08      94.46       8.34

2012 10 05 04:03:55.9379       0 11,0xaa23      16       4.26       6.08       7.44       8.34

2012 10 05 04:03:56.9471       0 11,0xaa23      16       4.27       6.08       7.44       8.34

2012 10 05 04:03:57.9372       0 11,0xaa23      16       4.27       6.08       7.44       8.34

2012 10 05 04:03:58.9442       0 11,0xaa23      16       4.26       6.08       7.44       8.34

2012 10 05 04:03:59.9304       0 11,0xaa23      16       4.26      94.43       7.44       8.34

2012 10 05 04:04:00.9344       0 11,0xaa23      16       4.26       6.08       7.44       8.34

2012 10 05 04:04:01.9574       0 11,0xaa23      16       4.26       6.08       7.44       8.34

2012 10 05 04:04:02.9295       0 11,0xaa23      16       4.26       6.08       7.44       8.34

2012 10 05 04:04:03.9566       0 11,0xaa23      16       4.26       6.08       7.44       8.34

2012 10 05 04:04:04.9329       0 11,0xaa23      16       4.26       6.08       7.44       8.34

2012 10 05 04:04:05.9278       0 11,0xaa23      16      94.44       6.08       7.44       8.34

2012 10 05 04:04:06.9958       0 11,0xaa23      16       4.26       6.08       7.44       8.34

2012 10 05 04:04:07.9301       0 11,0xaa23      16       4.26       6.08       7.44       8.34

2012 10 05 04:04:08.9280       0 11,0xaa23      16       4.26       6.08       7.44       8.34

2012 10 05 04:04:09.9381       0 11,0xaa23      16       4.26       6.08       7.44       8.34

2012 10 05 04:04:10.9363       0 11,0xaa23      16       4.26       6.08       7.44       8.34

2012 10 05 04:04:11.9353       0 11,0xaa23      16       4.26       6.08       7.44      94.48

2012 10 05 04:04:12.9273       0 11,0xaa23      16       4.26       6.08       7.44       8.34

2012 10 05 04:04:13.9513       0 11,0xaa23      16       4.26       6.08       7.44       8.34

2012 10 05 04:04:14.9475       0 11,0xaa23      16       4.26       6.08       7.44       8.34

2012 10 05 04:04:15.9566       0 11,0xaa23      16       4.26       6.08       7.44       8.34

2012 10 05 04:04:16.9329       0 11,0xaa23      16       4.26       6.08      94.46       8.34

A19 replaced TRH lower.  Initially came up fine, then died within 20minutes.  Recycled power and came up again, but later died again.  This is the unit that had been taken to Boulder and tested okay.

A19 started to remove GPS (to swap in order to reconfigure), but decided against.  In the process, popped out the SD card that brought the station down for about half hour while I tried to debug.

A18 TRH lower won't send data.  Removed.  Later replaced sensor and tested on A9 and saw that it was still dead.  Likely is a bad connection.  Note that this is the sensor that I spliced the chewed-up cable.

A11 Unplugged Tsoil.g, that is obviously bad.  However, tried plugging in 2 different probes that I had brought from Boulder and both gave odd readings (temperatures between 0 and 300 degrees, changing sporatically).  John has taken one of the probes back to Boulder, while I'll try to determine if the other can be made to work.  Any Tsoil.g data from this morning should be deleted (even "good" values were taken in air).

A15 moved to new location.  Sonic is now at 2m (up from 1m).  Boom angle shot and added to table.

A17 moved to new location.  Sonic is now at 0.5m (down from 1m).  Boom angle shot and added to table.

A8 John removed the mote that is non-functional to take to Boulder.

M Shot 4m boom angle (and reshot 2, 3, 5 to check).  Angles off since yesterday.

Later:

M replaced serializer in 10m CSAT (removed from MFO 853(?)).  This brought back the krypton.

A18 found that my splice of TRH lower had accumulated water and oxidized the Orange wire.  (The problem was because I couldn't find encapsulating heat shrink and had relied on normal heat shrink to protect the splice.)  I repaired, wrapped with electrician's tape, and stuffed the splice into the boom.  Now this works.

A11 Power had dropped.  Possible reason is that the power supply (that I replaced the fuse on yesterday that likely had blown during the lightning) is weak.  Another possibility is a bad battery (PCAPS leftover).  Another is the charge controller.  Removed all of this (and brought back to the base for testing).  In the meantime, stretched a DC power cable from M to A11's DSM.

Known issues still remaining:

A19 lower TRH still bad

A8 charge controller isn't keeping the battery up.  Battery voltage was only 12.0 at 1630, despite good sun in the afternoon.

A8 mote wasn't operating.  John should be able to fix in Boulder

A11 Tsoil.g is dead and replacements didn't work (will keep on testing)

Much later:

M 15m TRH not reporting.  Replaced and now okay.

These shot today with DataScope after calibrating and with "north" set to 0.

Notes:

- Above 3m on M may be less accurate due to compass dip with elevation angle

- I only checked the N-S markings on 3 Handars and one (a Vaisala) I couldn't find the marks.  Thus, I could be off by +/- 120 degrees.  

( I checked markings on all 5 handars - stations 1-3, and 5-6 for the boom angles I shot on 11/20. KK )

- All angles are the median of 3 or more readings.  Some of the angles seemed not to converge well, so I report them here only to 1 degree.  Others are generally within +/- 0.3 degree. 

P.S.  As long as I have this big table, might as well add the tilts to them.  Note that many gully stations are not on a planar surface, so there are sometimes significant departures from the planar fit.  We may or may not decide to use these.  Most make sense, with lean azimuths close to +/-90 degrees for the sloping sites and opposite signed on the south-facing and north-facing slopes.

10/22/2012:  We shot sonic angles with the Data Scope with 'no cal' correction.  Handars were shot South Leg to North Leg (marked on head).

10/29/2012: Data scope was calibrated before shooting angles. Handars were shot going from South to North

11/26: Data scope was calibrated using camera tripod, "good = 8".   Didn't complete all shots: main tower getting dark.  Went back out 11/27 and filled in missing.   Note some are significantly different than other's.  jm

11/30/2012: Data scope on tripod, used John's calibration.

Started with Main:

Installed upper DSM; GPS PPS board needed to be reseated

Tried to get serializer on 5m to work -- couldn't (this was S/N 537, I believe), thus swapped 2m (S/N 536, that I reenabled serializer) and 5m.

Added krypton to 5m.

Then decided to go east-to-west.  Everywhere, shot boom angles and taped TRH boom ends on A & C towers:

OSU Sodar: Replaced power supply and both batteries.  Plugged in SODAR power cable -- sounds like it is functioning.

A18: cycled power on mote (that brought back TRH fans?), but still no TRH data :(

C: Replaced 0.5m TRH -- now good

A19: noted that lower TRH is missing (spare is in trailer, tested as good by SRS)

OSU tower: Replaced power supply

A8: Replaced battery (station now up); TRH motes still down, didn't come up with various fixes

A11: Replaced grass mote battery (that was wet and 5.6V) and plugged in all sensors

A13: Reseated loose CSAT Bulgin cable -- now CSAT okay

A3: Gordon added a barometer cable and added GPS

A6: Gordon added a CSAT data cable

October 2, 2012

I started to look more closely to the damage from the lightening strike at SCP on September 27, 2012.  This is what I found...

CSAT0855, KH201393, KH201389 are good.  The sonic did have a bad serializer.

CSAT0856, Licor1163, Licor1167, TRH037, TRH012, TRH013, TRH006, TRH003, TRH004 are all BAD.  With the four of the TRHs five had bad fan monitor boards.

DSMs stacks were both bad...but this is just looking at power card.  Can't do much without it.  MainLo had a blown stack, power panel and possibly a serial panel.  Viper and serial may be okay.  MainHi did have a blown Viper(Eth port and reg) and power panel (5V reg and lands), stack.  Serial panel was okay on MainHi.

More to come...

We found (by trial-and-error) that the "tio" command (setting power on ports 0-4 on titans) has at least one port numbering wrong:

tio 2 0; tio 2 1 works for ttyS2 (CSAT)

tio 3 0; tio 3 1 works for ttyS1! (the TRH mote on plain A stations)

We're not sure at this point whether it is a cabling or software issue.  We also haven't tested the other ports.

In running through status tonight (still working on it), have found several TRHs with stopped fans.  All have responded to control-R.  See Ifan for times.

On site: Larry, Kurt, Steve, (at 6pm, Kurt left and Gordon arrived)

- re-installed many sensors on main tower:

* krypton at 15m

* licors at 1 and 2m (both needed power rewired from BNC connectors)

* sonic at 5m (has a serial cable, but appears to have been disabled)

* sonic at 2m (thought I was switching off serializer, but killed -- corrected about 6:30)

* all missing TRHs

* 5m nanobarometer (now in RS422 mode)

* etherant

* main power supply

A11: blown fuse on power supply had caused battery to drain and charge controller to turn off output.  Fixed (and swapped batteries with main, that wasn't so drained).

Added ground stake to OSU center tower.  Also replaced etherant.  (Turned out we shouldn't have...)

Tried to revive C 0.5m TRH by cycling power, but didn't fix.  Will replace tomorrow.

At about 6:30 added (new) M21 DSM and got back almost everything up to 5m.

The nanobarometer RS232 interface died during the lightning event.  However, its RS422 port and the sensor itself is still alive.  Thus, I have just rewired it for RS422 and we should be able to configure the DSM interface to use RS422.  Note that it no longer appears to need prompting.  Also, the output format is slightly different -- we'll have to see if it can be parsed with the old format.

Gordon, Sep 30, 12:15
Built a new kernel for titans, with a two-line fix that I hope will fix the issue where pc104 interrupts cease to be serviced.

Installed it on Aph3 from Boulder, and rebooted it. It came up and data from the emerald serial ports 5 (TRH24), 6 (TRH51) ,7 (Handar) (as well as the sonic on 2) is coming in. Previously the issue was seen within an hour or two. Wait and see...

Didn't work. Emerald data quit after about 1/2 hour. So, then made another small change to the interrupt handler, installed the new kernel and rebooted Aph3 around 12:58 pm. It's been up for about 6 hours and still getting Emerald data, which is longer than it had ever run before, so things might be fixed.

Note that the barometer on port 1 is not reporting, needs a cable.

Gordon, Oct 2

The above attempts didn't fix the problem. Emerald data still died after several hours. It took several more iterations, but I think it has finally been fixed.

Gordon, Sep 29

One of the Ethernats being used by OSU was knocked out by the lightning strike. Chris says the power light does not come on, and the status light on the injector goes out when the Etherant is plugged in.

We have pulled EantA from the main tower. Its status is unknown, likely dead.

By my record. OSU was using EantD, 00:20:F6:05:16:CD at the Sodar, and EantJ 00:20:F6:05:24:4F at the tower.

I presume it was the one at the OSU tower that died, but as now, Sep 29, these Etherants are reporting:

interface wireless registration-table print 
 # INTERFACE        RADIO-NAME       MAC-ADDRESS       AP  SIGNAL... TX-RATE UPTIME              
 0 wlan1-Int-Ant                     00:20:F6:05:1E:D5 no  -26dBm... 11Mbps  4d23h2m11s          
 1 wlan1-Int-Ant                     00:20:F6:05:24:85 no  -32dBm... 11Mbps  1d3h51m6s           
 2 wlan1-Int-Ant                     00:20:F6:05:24:4F no  -95dBm... 11Mbps  1h33m13s            
 3 wlan1-Int-Ant                     00:20:F6:05:24:56 no  -95dBm... 11Mbps  3m4s                

Unfortunately the AP24 doesn't show the radio names. From my earlier post I thought EantJ was on the OSU tower, but I guess not, since EantJ is currently reporting, so it must have been on the Sodar at that time. From the registration table printout above, here is what is currently reporting:

Gordon, Sep 29.

The DSMs on the main tower stopped reporting on Sep 27, 21:03:05 UTC (15:03:05 MDT). This was probably the moment just before the lightning strike.

Two of the sensors at C also ceased reporting at this time:

data_stats isfs_20120928_000000.dat.bz2 
C20:/var/tmp/gps_pty0     20     30     28952 2012 09 27 20:00:00.613  09 27 23:59:59.735    2.01  0.085   6.849   51   73
C20:/dev/ttyS1            20     50     14341 2012 09 27 20:00:00.446  09 27 23:59:59.296    1.00  0.842   7.001   20   20
C20:/dev/ttyS2            20     60     11172 2012 09 27 20:00:00.646  09 27 23:59:59.726    0.78  0.949   6.421   36   36
C20:/dev/ttyS5            20    100    286914 2012 09 27 20:00:00.020  09 27 23:59:59.940   19.92  0.001   5.850   12   12
C20:/dev/ttyS6            20    110     11171 2012 09 27 20:00:00.306  09 27 23:59:59.826    0.78  0.917   7.211   38   38
C20:/dev/ttyS7            20    150      2925 2012 09 27 20:00:00.306  09 27 21:03:05.006    0.77  1.278   2.581   37   37
C20:/dev/ttyS8            20    200    286910 2012 09 27 20:00:00.021  09 27 23:59:59.900   19.92  0.003   5.850   12   12
C20:/dev/ttyS9            20    250     11176 2012 09 27 20:00:01.126  09 27 23:59:58.626    0.78  1.073   6.722   36   38
C20:/dev/ttyS10           20    300      3772 2012 09 27 20:00:00.986  09 27 21:03:04.576    1.00  0.952   2.023   20   20

Id 20,150 is the 1.5m TRH. 20,300 is the Vaisala PTB.

The 1.5m TRH (TRH34) resumed reporting 7.5 hours later on Sep 28 04:29 UTC (Sep 27 22:29 MDT). I believe this was a miraculous resurrection , with no human intervention.

This morning three TRHs are reporting at C:

data_dump -i 20,-1 | fgrep TRH
2012 09 29 16:02:03.8064 0.01448 20, 110      35 TRH19 18.57 58.28 0 0 1460 114 0\r\n
2012 09 29 16:02:04.1965 0.005246 20, 250      38 TRH28 17.02 63.57 33 0 1420 125 104\r\n
2012 09 29 16:02:04.2564 0.01515 20, 150      35 TRH34 18.84 56.44 0 0 1468 110 0\r\n

These are the 1m, 1.5 and 2.5 TRHs. The 0.5m is not reporting. From a run of data_stats on the C20_20120928_120000.dat archive file it quit on Sep 28, 14:54 MDT.

Sept 29

A lightning strike on the afternoon of Sept 27 knocked out power to the base trailer and damaged equipment on the 20m main tower.

In the real-time archive, the DSMs at the main tower ceased reporting on Sep 27 21:03:05 UTC (15:03:05 MDT).

Station 12 also quit reporting at that time. A TRH and barometer at C20 and a TRH at Ah6 also quit at that time.

Kurt restored power mid-morning Sept 28 and we assessed damage to the main tower.  Our best guess is that the lightning struck near the base trailer and a current surge flowed down the power cable to DSM M21, trashing the charge controller, the PC104 stack, and the power interface panel.  Many of the sensors associated with M21 were found inoperative, but M21 appears to have acted as an expensive 'fuse', limiting damage to M22 and its associated sensors.  Gordon wrote:

"The most 'striking' damage was in the battery box at the main NCAR tower, the power interface panel on the lower DSM at the tower, and to systems at the OSU tower.  In general, the sensors and DSM higher on the main NCAR tower suffered less damage.

"The breaker in the conference center for power to the trailer was tripped, as was the breaker in the trailer which (I believe) is the circuit for power to the transformers at 'C' and 'M'.  The breaker that was tripped in the trailer is the lowest one on the panel, and is labelled 'class'  Kurt saw burn marks on one lug of a power cable connector at the trailer. I believe he said it was the socket at the end of the power cable from the conference center.

"All sensors except for one at the C site are reporting.  Sodars are OK, I hear.  The Picarro was not connected to AC.  It apparently suffered damage via the serial cable from the main tower DSM. So the damage was on the systems connected to the AC circuit to the main and OSU tower.  I think Tom and Kurt agree that it seems the lightning didn't strike either tower but the surge came in from the AC circuit.

"I like Kurt's suggestion that if a lightning storm threatens, that the power cables to the transformers be disconnected at the trailer. (Probabaly a good idea to throw the "class" breaker first).  That will shut down the Sodars and Piccarro. The ISFS systems and sensors at C and M should still run from their batteries.  I believe everything on the OSU tower is on battery?"

The sensor damage on the main tower was determined as in the following tables.  Sensors marked NG (Not Good) were removed and transported to Boulder.  We also found that one of the four sensors at each of the two soil sites were bad and brought down its mote if connected.  I recall that the soil temperature probe was bad at Grass, but the connector labels at Cactus were inadequate to determine the bad sensor (not the TP01 in my recollection).  The bad soil sensors were disconnected from the motes but not disinterred.

data_stats isfs_20120927_200000.dat.bz2 indicates that sensors at other sites quit reporting at 21:03:05, including two sensors at C: the 1.5m TRH and the barometer. The TRH resumed reporting on Sep 28 04:29 UTC.

Station 12 quit reporting at 21:03:05, but is now working after replacing the charging controller. However data from the TRHs at 12 via the bluetooth mote is not coming in.

0.5m TRH at Ah6 also stopped.

Other Damaged Equipment

• WIFI antenna (EantA) on main tower. Unknown status, probably dead.
• WIFI antenna (EantD) on OSU tower. Unknown status.
• USB disk drive (pocketec) on M21 DSM is not recognized by host systems. This contained 21 hours of data for Sep 27, but we have the same data that was received in real-time over the network.
• 5-port network switch in M21 is toast

Pocketec USB disk drive in Mu22, upper DSM on NCAR tower, is OK.
*

Gordon, Sep 29.

Ap12 had not worked well since it was deployed. It died early every morning due to low voltage. During the night the measured station voltage, Vdsm, would dive down to a cutoff at 11.3 V, from an apparent healthy 14 V when being charged during the day.

Replaced the battery on Sep 26. Also checked that the lugs on the controller were tight.

The station was dead the evening of Sep 27, so it was feared it had been damaged in the lightning storm. The measured voltage at the interface panel was about 3 V when the system was turned on, and 12 V when off.

Brought the system to the trailer and could not find anything wrong. Voltages all OK. Decided that the problem must be that the charge controller could not deliver sufficient power. Kurt replaced the controller and the system powered up at 16:48 MDT (per the times in the archive). It has run through the night.

Gordon, later on Sep 29:

Turns out that 12 was affected by the lightning. It quit reporting, both via network an to its local storage right at the time of the strike, Sep 27, 21:03:05 UTC. After the charge controller was replaced, all sensors except the bluetooth mote serving the TRH's resumed reporting.