Gordon, Oct 7
Last night noticed from cockpit that the fan current for the 2m TRH on 5 was 0. Logged in an restarted it with a ctrl-R from rs 5 around 8:40 pm (time determined from the data).
Here's a plot of the 5 minute averages when it died:
Will look at the 1 second data to see if it ever reported a value over 50 mA.
Testing of our one spare Tsoil shows that it reads correctly less than 20% of the time. Nevertheless, it is better than nothing, so I've installed it as Tsoil.g. Hopefully, I'll bring up a working spare tomorrow
qctables shows that lately (not last week), A4 0.5m TRH was the highest RH. Replaced SHT. A10 0.5m TRH was the lowest RH. Replaced SHT.
Also, Gordon noticed that A17 0.5mTRH was coming in as S/N87, with id=0x11. This SHT was labeled 068! Also, Gordon's serial number logbook entry shows a different S/N 2 weeks ago. Using mote_dump, we don't see a 0x10 sensor, however statsproc has been creating NetCDF files with data for both 0.5m and 2m TRHs. This is all very odd. Replaced this SHT with 060 and data now coming in ok.
From the log output of mote_dump, here are the serial number messages from the TRH mote at 17 from Sep 20 through Oct 6:
2012 09 20 19:52:34.913, mote=17, sensorType=0x11 SN=43, typeName=TRH 2012 09 20 19:55:21.532, mote=17, sensorType=0x10 SN=68, typeName=TRH 2012 09 20 20:24:40.643, mote=17, sensorType=0x11 SN=87, typeName=TRH 2012 09 20 20:25:12.672, mote=17, sensorType=0x11 SN=43, typeName=TRH 2012 09 20 20:42:35.153, mote=17, sensorType=0x11 SN=87, typeName=TRH 2012 09 20 20:43:05.852, mote=17, sensorType=0x11 SN=43, typeName=TRH 2012 10 06 19:06:56.001, mote=17, sensorType=0x11 SN=87, typeName=TRH 2012 10 06 19:09:37.901, mote=17, sensorType=0x11 SN=43, typeName=TRH 2012 10 06 19:27:18.501, mote=17, sensorType=0x11 SN=87, typeName=TRH 2012 10 06 19:27:33.401, mote=17, sensorType=0x11 SN=43, typeName=TRH 2012 10 06 19:47:10.631, mote=17, sensorType=0x10 SN=60, typeName=TRH
My best guess is that the SN=87 records are somehow incorrect, in that there wasn't really an 87 at that location.
I also see the same issue at 19, where it is claiming that TRH 127 is present. In today's /var/log/isfs/isfs.log from statsproc, 17 and 19 are the only stations exhibiting this issue:
2012 10 06 17:13:53.194, mote=19, sensorType=0x10 SN=46, typeName=TRH 2012 10 06 18:01:52.954, mote=19, sensorType=0x11 SN=65, typeName=TRH 2012 10 06 20:49:41.294, mote=19, sensorType=0x10 SN=46, typeName=TRH 2012 10 06 20:49:41.294, mote=19, sensorType=0x11 SN=65, typeName=TRH 2012 10 07 02:31:43.613, mote=19, sensorType=0x10 SN=46, typeName=TRH 2012 10 07 02:31:43.613, mote=19, sensorType=0x11 SN=65, typeName=TRH 2012 10 07 09:17:06.213, mote=19, sensorType=0x11 SN=127, typeName=TRH 2012 10 07 09:32:27.243, mote=19, sensorType=0x11 SN=65, typeName=TRH 2012 10 07 09:33:50.863, mote=19, sensorType=0x11 SN=127, typeName=TRH 2012 10 07 09:35:32.293, mote=19, sensorType=0x11 SN=65, typeName=TRH
NIDAS only logs the serial numbers in a run the first time they are encountered, and when they change.
Gordon, Oct 6
The "flux" laptop, a Dell Precision M6400 has died twice in the last few days, including last night. The symptom is that it is completely powered down, with nothing indicated in /var/log/messages.
Replaced it a Dell Latitude D820, that I'm calling "gully". As of about 2pm all should be working.
Started falling about an hour ago (~0900). The first band was light and left <1cm on the ground. Now there is a second band with larger flakes that added another 1cm and still piling up.
The wetness sensor doesn't appear to be registering this yet. Its voltage has only changed from 0.265 to 0.269 in the last hour, though the timing of these mV changes are about right. Apparently, the snow crystal structure prevents its water from making good contact with the sensor?
This an example of data from the 2nd spare Tsoil, installed in the air at A11.grass. Clearly, it is worse than any of the others. The odd thing is that I recall getting similar bad numbers from the probe that John just tested in Boulder and found reasonable values in all but one position (except for the 94.xx values).
2012 10 05 21:17:56.2154 0 11,0xa920 16 1.95 28.16 1.95 2.04
2012 10 05 21:17:57.1902 0 11,0xa920 16 0 -158.61 -130.56 -112.64
2012 10 05 21:17:58.1995 0 11,0xa920 16 1.94 28.16 1.94 2.04
2012 10 05 21:17:59.2075 0 11,0xa920 16 1.94 0 -158.61 -135.68
2012 10 05 21:18:00.2097 0 11,0xa920 16 1.92 28.16 1.92 2.04
2012 10 05 21:18:01.2587 0 11,0xa920 16 1.91 2.02 0 -166.29
2012 10 05 21:18:02.2169 0 11,0xa920 16 1.46 28.16 93.62 2
2012 10 05 21:18:03.3258 0 11,0xa920 16 1.92 2.02 2.08 28.16
2012 10 05 21:18:04.1999 0 11,0xa920 16 0 -281.49 -137.88 -122.88
2012 10 05 21:18:05.2599 0 11,0xa920 16 1.92 2.02 0 -163.73
2012 10 05 21:18:06.2011 0 11,0xa920 16 1.92 2.02 28.16 1.92
2012 10 05 21:18:07.2298 0 11,0xa920 16 1.92 0 -163.73 -138.24
2012 10 05 21:18:08.2312 0 11,0xa920 16 1.94 0 -158.61 -133.12
2012 10 05 21:18:09.2063 0 11,0xa920 16 1.92 28.16 1.92 2.02
2012 10 05 21:18:10.1931 0 11,0xa920 16 0 -163.73 -138.24 -120.32
2012 10 05 21:18:11.2165 0 11,0xa920 16 1.93 28.16 1.93 2.03
2012 10 05 21:18:12.2116 0 11,0xa920 16 0 -161.17 -135.68 -117.76
2012 10 05 21:18:13.1944 0 11,0xa920 16 1.94 2.04 28.16 1.94
2012 10 05 21:18:14.2569 0 11,0xa920 16 0 -158.61 -133.12 92.16
2012 10 05 21:18:15.1988 0 11,0xa920 16 1.95 2.05 28.16 1.95
2012 10 05 21:18:16.2079 0 11,0xa920 16 1.97 0 -150.93 -125.44
2012 10 05 21:18:17.1983 0 11,0xa920 16 0 -150.93 -122.88 -107.52
2012 10 05 21:18:18.2468 0 11,0xa920 16 0 -145.81 -120.32 -192
Looking at Ifan, saw that 2.5m on C and 0.5m on A3 were off. Both responded to control-R just now.
A11: This had been powered by an AC-powered, open-frame power supply, through a Sunsaver to a PCAPS battery. On Tues, replaced a blow fuse in the power supply and swapped in a relatively healthy battery. Yesterday found that the voltage was low and decided to test components at the base. In the meantime, wired a DC cable from the M tower battery box using a DC splitter cable. This arrangement is working fine. In the base, was able to charge the battery overnight through the Sunsaver using the bench Tenma supply. Tenma current draw ran for several hours limited at 3A, but had dropped to 1A in the morning. Conclude that Sunsaver and battery are okay, and thus that the open-frame supply is suspicious and may have sustained more damage due to lightning than just a blown fuse. If we continue to run this way, the AC cable to the cooler could be removed.
A8: battery had dropped to 11.8V (from 12.0V yesterday). Voltage on solar input was reading 20V even though pretty cloudy (had been 21V yesterday in nearly full sun). This all seemed odd. Replaced Sunsaver charge controller with an identical Sunsaver that was removed from A11. Now Solar in dropped to 12.1 with battery also at 12.1. "load disconnect" was on, but went off after about 20min with battery voltage now 12.4. Station is now up. I think this definitively demonstrates that the old Sunsaver is bad (not charging).
M21/M22/A11: These DSMs were now all powered through a Lambda brick power supply (6.5A rating, adjusted to 13.6V no load) through an old MorningStar. The Lambda was running VERY hot (too hot to touch after 1/2 second), even before we added A11. Got a call this morning from Kurt/John/Chris recommending that we remove the charge controllers from all AC-powered stations. Used wire nuts to connect input/battery/load wires together. Adjusted Lambda down to 12.8V, as per instructions. Blew breaker on M21 DSM during its boot sequence. Moved M22's power from M21 Aux to another head of the Y cable. (Now the Y splitter is directly connected to power cables to each of M21, M22, and A11.) This all appears to work and the Lamba brick is now cool.
C: Haven't yet changed. Our instructions are to remove its charge controller as well.
I should have put in the logbook last night that, when Gordon and I went to the tower last night to replace M 4m TRH, there was very light frost on the ground.
Below is a short segment from mote_dump 11,42 with only Tsoil data. Note that values of 94.4x cycle through each soil depth measurement about every 6 samples. I don't yet know whether this is a Mote, Sensor, Software, or interference issue. Other sensors on this mote don't exhibit this behavior. Clearly, these "spikes" are easy to remove, so I won't worry about it excessively.
2012 10 05 04:03:42.9490 0 11,0xaa23 16 94.44 6.09 7.44 8.34
2012 10 05 04:03:43.9481 0 11,0xaa23 16 4.27 6.09 7.44 8.34
2012 10 05 04:03:44.9409 0 11,0xaa23 16 4.27 6.09 7.44 8.34
2012 10 05 04:03:45.9333 0 11,0xaa23 16 4.27 6.09 7.44 8.34
2012 10 05 04:03:46.9452 0 11,0xaa23 16 4.27 6.09 7.44 8.34
2012 10 05 04:03:47.9494 0 11,0xaa23 16 4.27 6.08 7.44 8.34
2012 10 05 04:03:48.9374 0 11,0xaa23 16 4.27 6.08 7.44 94.48
2012 10 05 04:03:49.9325 0 11,0xaa23 16 4.27 6.08 7.44 8.34
2012 10 05 04:03:50.9556 0 11,0xaa23 16 4.27 6.08 7.44 8.34
2012 10 05 04:03:51.9416 0 11,0xaa23 16 4.27 6.08 7.44 8.34
2012 10 05 04:03:52.9437 0 11,0xaa23 16 4.27 6.08 7.44 8.34
2012 10 05 04:03:53.9469 0 11,0xaa23 16 4.27 6.08 7.44 8.34
2012 10 05 04:03:54.9419 0 11,0xaa23 16 4.26 6.08 94.46 8.34
2012 10 05 04:03:55.9379 0 11,0xaa23 16 4.26 6.08 7.44 8.34
2012 10 05 04:03:56.9471 0 11,0xaa23 16 4.27 6.08 7.44 8.34
2012 10 05 04:03:57.9372 0 11,0xaa23 16 4.27 6.08 7.44 8.34
2012 10 05 04:03:58.9442 0 11,0xaa23 16 4.26 6.08 7.44 8.34
2012 10 05 04:03:59.9304 0 11,0xaa23 16 4.26 94.43 7.44 8.34
2012 10 05 04:04:00.9344 0 11,0xaa23 16 4.26 6.08 7.44 8.34
2012 10 05 04:04:01.9574 0 11,0xaa23 16 4.26 6.08 7.44 8.34
2012 10 05 04:04:02.9295 0 11,0xaa23 16 4.26 6.08 7.44 8.34
2012 10 05 04:04:03.9566 0 11,0xaa23 16 4.26 6.08 7.44 8.34
2012 10 05 04:04:04.9329 0 11,0xaa23 16 4.26 6.08 7.44 8.34
2012 10 05 04:04:05.9278 0 11,0xaa23 16 94.44 6.08 7.44 8.34
2012 10 05 04:04:06.9958 0 11,0xaa23 16 4.26 6.08 7.44 8.34
2012 10 05 04:04:07.9301 0 11,0xaa23 16 4.26 6.08 7.44 8.34
2012 10 05 04:04:08.9280 0 11,0xaa23 16 4.26 6.08 7.44 8.34
2012 10 05 04:04:09.9381 0 11,0xaa23 16 4.26 6.08 7.44 8.34
2012 10 05 04:04:10.9363 0 11,0xaa23 16 4.26 6.08 7.44 8.34
2012 10 05 04:04:11.9353 0 11,0xaa23 16 4.26 6.08 7.44 94.48
2012 10 05 04:04:12.9273 0 11,0xaa23 16 4.26 6.08 7.44 8.34
2012 10 05 04:04:13.9513 0 11,0xaa23 16 4.26 6.08 7.44 8.34
2012 10 05 04:04:14.9475 0 11,0xaa23 16 4.26 6.08 7.44 8.34
2012 10 05 04:04:15.9566 0 11,0xaa23 16 4.26 6.08 7.44 8.34
2012 10 05 04:04:16.9329 0 11,0xaa23 16 4.26 6.08 94.46 8.34
A19 replaced TRH lower. Initially came up fine, then died within 20minutes. Recycled power and came up again, but later died again. This is the unit that had been taken to Boulder and tested okay.
A19 started to remove GPS (to swap in order to reconfigure), but decided against. In the process, popped out the SD card that brought the station down for about half hour while I tried to debug.
A18 TRH lower won't send data. Removed. Later replaced sensor and tested on A9 and saw that it was still dead. Likely is a bad connection. Note that this is the sensor that I spliced the chewed-up cable.
A11 Unplugged Tsoil.g, that is obviously bad. However, tried plugging in 2 different probes that I had brought from Boulder and both gave odd readings (temperatures between 0 and 300 degrees, changing sporatically). John has taken one of the probes back to Boulder, while I'll try to determine if the other can be made to work. Any Tsoil.g data from this morning should be deleted (even "good" values were taken in air).
A15 moved to new location. Sonic is now at 2m (up from 1m). Boom angle shot and added to table.
A17 moved to new location. Sonic is now at 0.5m (down from 1m). Boom angle shot and added to table.
A8 John removed the mote that is non-functional to take to Boulder.
M Shot 4m boom angle (and reshot 2, 3, 5 to check). Angles off since yesterday.
Later:
M replaced serializer in 10m CSAT (removed from MFO 853(?)). This brought back the krypton.
A18 found that my splice of TRH lower had accumulated water and oxidized the Orange wire. (The problem was because I couldn't find encapsulating heat shrink and had relied on normal heat shrink to protect the splice.) I repaired, wrapped with electrician's tape, and stuffed the splice into the boom. Now this works.
A11 Power had dropped. Possible reason is that the power supply (that I replaced the fuse on yesterday that likely had blown during the lightning) is weak. Another possibility is a bad battery (PCAPS leftover). Another is the charge controller. Removed all of this (and brought back to the base for testing). In the meantime, stretched a DC power cable from M to A11's DSM.
Known issues still remaining:
A19 lower TRH still bad
A8 charge controller isn't keeping the battery up. Battery voltage was only 12.0 at 1630, despite good sun in the afternoon.
A8 mote wasn't operating. John should be able to fix in Boulder
A11 Tsoil.g is dead and replacements didn't work (will keep on testing)
Much later:
M 15m TRH not reporting. Replaced and now okay.
These shot today with DataScope after calibrating and with "north" set to 0.
Notes:
- Above 3m on M may be less accurate due to compass dip with elevation angle
- I only checked the N-S markings on 3 Handars and one (a Vaisala) I couldn't find the marks. Thus, I could be off by +/- 120 degrees.
( I checked markings on all 5 handars - stations 1-3, and 5-6 for the boom angles I shot on 11/20. KK )
- All angles are the median of 3 or more readings. Some of the angles seemed not to converge well, so I report them here only to 1 degree. Others are generally within +/- 0.3 degree.
P.S. As long as I have this big table, might as well add the tilts to them. Note that many gully stations are not on a planar surface, so there are sometimes significant departures from the planar fit. We may or may not decide to use these. Most make sense, with lean azimuths close to +/-90 degrees for the sloping sites and opposite signed on the south-facing and north-facing slopes.
10/22/2012: We shot sonic angles with the Data Scope with 'no cal' correction. Handars were shot South Leg to North Leg (marked on head).
10/29/2012: Data scope was calibrated before shooting angles. Handars were shot going from South to North
11/26: Data scope was calibrated using camera tripod, "good = 8". Didn't complete all shots: main tower getting dark. Went back out 11/27 and filled in missing. Note some are significantly different than other's. jm
11/30/2012: Data scope on tripod, used John's calibration.
Position |
Angle |
Reshoot 10/4 |
10/22 |
10/22 |
10/29 |
11/20 |
11/26 |
11/30 |
Comments |
Lean |
Lean Azimuth |
|---|---|---|---|---|---|---|---|---|---|---|---|
A1 CSAT |
112.4 |
|
113.5 |
112.7 |
111.8 |
109.3 |
109 |
105.6 |
|
1.6 |
-28.2 |
A1 Handar |
337.9 |
|
42.2 |
41.2 |
33.0 |
36.2 |
39.5 |
39.2 |
may not be N-S (SO/CG/SS?) |
|
|
A2 CSAT |
113.4 |
|
117.2 |
116.2 |
112.5 |
110.0 |
110.0 |
109.7 |
|
2.6 |
-29.5 |
A2 Handar |
292 |
|
112.7 |
112.4 |
108.4 |
108.9 |
107.9 |
107.2 |
approximate(SO/CG/SS?) |
|
|
A3 CSAT |
113.2 |
|
114.7 |
114.4 |
114.2 |
111.0 |
111.0 |
107.9 |
|
2.1 |
-8.1 |
A3 Handar |
338.6 |
|
42.0 |
42.3 |
36.2 |
35.4 |
42.0 |
41.2 |
shot angle from south to north as |
|
|
A4 CSAT |
110.4 |
|
113.7 |
113.0 |
108.8 |
105.6 |
106.0 |
103.7 |
|
5.6 |
-79.2 |
A5 CSAT |
135.7 |
|
134.9 |
133.9 |
132.0 |
132.0 |
130.4 |
130.7 |
|
2.8 |
8.9 |
A5 Handar |
320 |
|
139.1 |
139.7 |
137.1 |
141.6 |
133.9 |
133.8 |
shot angle from south to north |
|
|
A6 CSAT |
107.5 |
|
108.9 |
108.5 |
107.4 |
104.4 |
106.6 |
105.1 |
|
2.9 |
-5.2 |
A6 Handar |
333.6 |
|
40.7 |
39.5 |
32.2 |
34.4 |
37.8 |
38.9 |
shot angle from south to north as |
|
|
A7 CSAT |
82.9 |
|
87.4 |
89.5 |
83.2 |
80.9 |
83.2 |
79.7 |
|
1.5 |
-39.8 |
A8 CSAT |
116.4 |
|
115.4 |
113.5 |
111.2 |
109.3 |
110.2 |
109.1 |
1m height (kk) |
1.3 |
-25.0 |
A8 CSAT 3M |
|
|
|
|
|
115.2 |
112.3 |
111.9 |
3m height (kk) |
|
|
A9 CSAT |
86.4 |
|
90.0 |
89.9 |
84.7 |
86.4 |
86.7 |
86.1 |
|
5.1 |
-76.7 |
A10 CSAT |
97.9 |
|
100.0 |
101.0 |
98.7 |
94.6 |
96.8 |
96.0 |
|
3.6 |
-67.9 |
A11 CSAT |
104.3 |
|
104.9 |
103.9 |
100.0 |
101.2 |
100.3 |
99.4 |
|
1.6 |
30.7 |
A12 CSAT |
108.0 |
|
105.2 |
106.2 |
101.6 |
100.7 |
102.0 |
97.3 |
|
3.4 |
79.6 |
A13 CSAT |
110.4 |
|
109.9 |
110.2 |
103.9 |
107.0 |
104.5 |
103.9 |
|
2.9 |
77.6 |
A14 CSAT |
111.8 |
|
109.5 |
109.4 |
107.7 |
107.4 |
105.3 |
104.4 |
|
3.4 |
67.8 |
A15 CSAT |
93.5 |
|
|
|
|
|
|
|
first location |
3.5 |
-64.0 |
A15 CSAT |
|
88.3 |
95.8 |
94.8 |
86.6 |
85.9 |
87.0 |
87.5 |
new location (and now at 2m) |
3.9 |
-74.9 |
A16 CSAT |
98.8 |
|
99.7 |
97.0 |
94.4 |
92.6 |
93.1 |
90.0 |
|
1.1 |
-44.8 |
A17 CSAT |
91.2 |
|
|
|
|
|
|
|
first location |
5.0 |
45.8 |
A17 CSAT |
|
88.7 |
90.4 |
90.5 |
85.9 |
85.9 |
87.5 |
87.9 |
new location (and now at 0.5m) |
2.0 |
42.9 |
A18 CSAT |
82.2 |
|
81.1 |
79.6 |
77.8 |
74.6 |
78.3 |
77.9 |
|
3.5 |
-56.3 |
A19 CSAT |
91.1 |
|
91.0 |
88.0 |
85.1 |
88.2 |
84.8 |
83.5 |
|
4.3 |
74.8 |
C 1m CSAT |
96.4 |
|
94.9 |
96.1 |
89.5 |
88.7 |
90.7 |
90.7 |
|
0.7 |
2.0 |
C 2m CSAT |
92.3 |
|
94.4 |
93.9 |
92.4 |
88.2 |
90.2 |
90.3 |
|
1.6 |
47.0 |
C Handar |
353.0 |
|
171.3 |
170.1 |
173.4 |
173.5 |
170.6 |
168.6 |
shot angle from south to north |
|
|
M 0.5m CSAT |
98.1 |
|
102.1 |
102.0 |
94.6 |
99.2 |
96.6 |
95.8 |
|
1.5 |
-45.7 |
M 1m CSAT |
101.0 |
|
101.0 |
102.2 |
93.4 |
98.3 |
96.5 |
95.7 |
|
1.9 |
-21.0 |
M 2m CSAT |
102.5 |
98.5 |
101.9 |
103.9 |
99.9 |
97.4 |
98.0 |
96.6 |
obviously reshoots don't agree! |
1.8 |
-13.4 |
M 3m CSAT |
101.8 |
97.8 |
101.8 |
103.7 |
99.1 |
99.9 |
96.8 |
96.5 |
obviously reshoots don't agree! |
0.8 |
-23.8 |
M 4m CSAT |
|
97.3 |
104.1 |
103.6 |
99.6 |
99.2 |
96.8 |
96.2 |
|
1.3 |
21.9 |
M 5m CSAT |
102.4 |
101.2 |
104.3 |
104.0 |
99.3 |
99.7 |
97.3 |
97.4 |
|
1.1 |
8.2 |
M 6m Handar |
003.9 |
|
72.6 |
70.8 |
69.0 |
63.7 |
66.2 |
66.2 |
shot angle from south to north as |
|
|
M 8m Handar |
002.1 |
|
68.8 |
69.0 |
65.4 |
63.2 |
65.2 |
64.0 |
|
|
|
M 10m CSAT |
103.1 |
|
100.8 |
103.7 |
94.2 |
99.6 |
97.5 |
96.6 |
|
1.0 |
22.6 |
M 15m Handar |
001 |
|
66.6 |
61.7 |
66.5 |
61.0 |
63.9 |
65.8 |
not too accurate |
|
|
M 20m CSAT |
101 |
|
110.9 |
96.7 |
98.9 |
99.3 |
96.5 |
94.6 |
not too accurate |
1.8 |
-11.6 |
Started with Main:
Installed upper DSM; GPS PPS board needed to be reseated
Tried to get serializer on 5m to work -- couldn't (this was S/N 537, I believe), thus swapped 2m (S/N 536, that I reenabled serializer) and 5m.
Added krypton to 5m.
Then decided to go east-to-west. Everywhere, shot boom angles and taped TRH boom ends on A & C towers:
OSU Sodar: Replaced power supply and both batteries. Plugged in SODAR power cable -- sounds like it is functioning.
A18: cycled power on mote (that brought back TRH fans?), but still no TRH data :(
C: Replaced 0.5m TRH -- now good
A19: noted that lower TRH is missing (spare is in trailer, tested as good by SRS)
OSU tower: Replaced power supply
A8: Replaced battery (station now up); TRH motes still down, didn't come up with various fixes
A11: Replaced grass mote battery (that was wet and 5.6V) and plugged in all sensors
A13: Reseated loose CSAT Bulgin cable -- now CSAT okay
A3: Gordon added a barometer cable and added GPS
A6: Gordon added a CSAT data cable
October 2, 2012
I started to look more closely to the damage from the lightening strike at SCP on September 27, 2012. This is what I found...
CSAT0855, KH201393, KH201389 are good. The sonic did have a bad serializer.
CSAT0856, Licor1163, Licor1167, TRH037, TRH012, TRH013, TRH006, TRH003, TRH004 are all BAD. With the four of the TRHs five had bad fan monitor boards.
DSMs stacks were both bad...but this is just looking at power card. Can't do much without it. MainLo had a blown stack, power panel and possibly a serial panel. Viper and serial may be okay. MainHi did have a blown Viper(Eth port and reg) and power panel (5V reg and lands), stack. Serial panel was okay on MainHi.
More to come...
We found (by trial-and-error) that the "tio" command (setting power on ports 0-4 on titans) has at least one port numbering wrong:
tio 2 0; tio 2 1 works for ttyS2 (CSAT)
tio 3 0; tio 3 1 works for ttyS1! (the TRH mote on plain A stations)
We're not sure at this point whether it is a cabling or software issue. We also haven't tested the other ports.