Just now, temperature is dropping like a stone and winds are 0 many places.
A working list of issues I'm seeing from cockpit -- I'll edit this as they are dealt with...
All sonics: diagbits and ldiag change multiple times per frame. Does this indicate sonic problems?
A11: Vmote.c is low, as if there was no charging.
A11: Tsoil.1.9cm.g at bottom of plot?
A19: TRH 2m was dead. Responded to tio 3 0/1. Not good.
A1 (only): U,V scale only positive
C20: Baro dead. Does not respond to eio 10 0/1
M21: No Picarro (of course)
Both Licors just set to 20 samples/s using:
(Outputs (BW 20) (Delay 0) (RS232 (Freq 20.0)))
Gordon, Oct 7
btmote8, the bluetooth mote at 8, sampling the 2 TRH's, has been out since Sep 25.
The bluetooth radio is a BlueRadios chip, MAC address 00:A0:96:32:00:BC.
I have tried resurrecting it, cleaning the corrosion on the radio board, and testing the radio on 3 different mote boards, using the serial mote cable at station 5. At 5 the power mote is in port 7. I use minicom on port 8 to test btmote8.
The motes come up in serial mode, without pressing the white button twice. Otherwise the mote boards work fine. One can change the ID and update the eeprom. The btradio command switches to the interactive interface to the radio, but the radio does not respond to atsi commands, even if ++CR is used. Ctrl-C exits the interactive menu, and the serial data resumes.
Realized that a simple workaround is possible. Station 8 has 8 available serial ports. Reconfigured its XML to read from /dev/ttyS5 instead of btspp:btmote8. TRHs come up and all looks OK. This change is transparent to the processing configuration XML.
Gordon & Steve Oct 7
We are using a combination of batteries. Most left in the trailer are the new batteries purchased for SCP:
Deka Dominator, 12V Gel, 8G24, 335 CCA, 74 A.H at 20Hr (red label)
Also in use leftover from PCAPS are:
Deka Dominator, 12V Gel, Marine/RV, 8G24M, 410 CCA, 74 A.H at 20Hr (blue label)
Gordon, Oct 7
First noticed from cockpit that system voltage (Vdsm) at 11 was low. 11 gets its power from the supply at the main tower.
As of the last modification (yesterday?) the power system at main just has the gray brick power supply and a battery, no charge controller.
Measured 5.9V on the DC cable going to 11.
Disconnected supply and load from the battery.
Power supply still seems OK: 12.8 V with no load. But it appears that it can't support the load of M21, Mu22 and sensors on the main tower and station 11.
Battery: 9.3 V with no load.
Vased on the real-time archive, he data systems were up, since their power supply cards support a wide range. Surprisingly most sensors stayed up. The PTB on main died first at 10:46 local, the licors quit at 10:55,10:57. Most of the rest was up when I cut power.
Gordon, Oct 7
Last night noticed from cockpit that the fan current for the 2m TRH on 5 was 0. Logged in an restarted it with a ctrl-R from rs 5 around 8:40 pm (time determined from the data).
Here's a plot of the 5 minute averages when it died:
Will look at the 1 second data to see if it ever reported a value over 50 mA.
Testing of our one spare Tsoil shows that it reads correctly less than 20% of the time. Nevertheless, it is better than nothing, so I've installed it as Tsoil.g. Hopefully, I'll bring up a working spare tomorrow
qctables shows that lately (not last week), A4 0.5m TRH was the highest RH. Replaced SHT. A10 0.5m TRH was the lowest RH. Replaced SHT.
Also, Gordon noticed that A17 0.5mTRH was coming in as S/N87, with id=0x11. This SHT was labeled 068! Also, Gordon's serial number logbook entry shows a different S/N 2 weeks ago. Using mote_dump, we don't see a 0x10 sensor, however statsproc has been creating NetCDF files with data for both 0.5m and 2m TRHs. This is all very odd. Replaced this SHT with 060 and data now coming in ok.
From the log output of mote_dump, here are the serial number messages from the TRH mote at 17 from Sep 20 through Oct 6:
2012 09 20 19:52:34.913, mote=17, sensorType=0x11 SN=43, typeName=TRH 2012 09 20 19:55:21.532, mote=17, sensorType=0x10 SN=68, typeName=TRH 2012 09 20 20:24:40.643, mote=17, sensorType=0x11 SN=87, typeName=TRH 2012 09 20 20:25:12.672, mote=17, sensorType=0x11 SN=43, typeName=TRH 2012 09 20 20:42:35.153, mote=17, sensorType=0x11 SN=87, typeName=TRH 2012 09 20 20:43:05.852, mote=17, sensorType=0x11 SN=43, typeName=TRH 2012 10 06 19:06:56.001, mote=17, sensorType=0x11 SN=87, typeName=TRH 2012 10 06 19:09:37.901, mote=17, sensorType=0x11 SN=43, typeName=TRH 2012 10 06 19:27:18.501, mote=17, sensorType=0x11 SN=87, typeName=TRH 2012 10 06 19:27:33.401, mote=17, sensorType=0x11 SN=43, typeName=TRH 2012 10 06 19:47:10.631, mote=17, sensorType=0x10 SN=60, typeName=TRH
My best guess is that the SN=87 records are somehow incorrect, in that there wasn't really an 87 at that location.
I also see the same issue at 19, where it is claiming that TRH 127 is present. In today's /var/log/isfs/isfs.log from statsproc, 17 and 19 are the only stations exhibiting this issue:
2012 10 06 17:13:53.194, mote=19, sensorType=0x10 SN=46, typeName=TRH 2012 10 06 18:01:52.954, mote=19, sensorType=0x11 SN=65, typeName=TRH 2012 10 06 20:49:41.294, mote=19, sensorType=0x10 SN=46, typeName=TRH 2012 10 06 20:49:41.294, mote=19, sensorType=0x11 SN=65, typeName=TRH 2012 10 07 02:31:43.613, mote=19, sensorType=0x10 SN=46, typeName=TRH 2012 10 07 02:31:43.613, mote=19, sensorType=0x11 SN=65, typeName=TRH 2012 10 07 09:17:06.213, mote=19, sensorType=0x11 SN=127, typeName=TRH 2012 10 07 09:32:27.243, mote=19, sensorType=0x11 SN=65, typeName=TRH 2012 10 07 09:33:50.863, mote=19, sensorType=0x11 SN=127, typeName=TRH 2012 10 07 09:35:32.293, mote=19, sensorType=0x11 SN=65, typeName=TRH
NIDAS only logs the serial numbers in a run the first time they are encountered, and when they change.
Gordon, Oct 6
The "flux" laptop, a Dell Precision M6400 has died twice in the last few days, including last night. The symptom is that it is completely powered down, with nothing indicated in /var/log/messages.
Replaced it a Dell Latitude D820, that I'm calling "gully". As of about 2pm all should be working.
Started falling about an hour ago (~0900). The first band was light and left <1cm on the ground. Now there is a second band with larger flakes that added another 1cm and still piling up.
The wetness sensor doesn't appear to be registering this yet. Its voltage has only changed from 0.265 to 0.269 in the last hour, though the timing of these mV changes are about right. Apparently, the snow crystal structure prevents its water from making good contact with the sensor?
This an example of data from the 2nd spare Tsoil, installed in the air at A11.grass. Clearly, it is worse than any of the others. The odd thing is that I recall getting similar bad numbers from the probe that John just tested in Boulder and found reasonable values in all but one position (except for the 94.xx values).
2012 10 05 21:17:56.2154 0 11,0xa920 16 1.95 28.16 1.95 2.04
2012 10 05 21:17:57.1902 0 11,0xa920 16 0 -158.61 -130.56 -112.64
2012 10 05 21:17:58.1995 0 11,0xa920 16 1.94 28.16 1.94 2.04
2012 10 05 21:17:59.2075 0 11,0xa920 16 1.94 0 -158.61 -135.68
2012 10 05 21:18:00.2097 0 11,0xa920 16 1.92 28.16 1.92 2.04
2012 10 05 21:18:01.2587 0 11,0xa920 16 1.91 2.02 0 -166.29
2012 10 05 21:18:02.2169 0 11,0xa920 16 1.46 28.16 93.62 2
2012 10 05 21:18:03.3258 0 11,0xa920 16 1.92 2.02 2.08 28.16
2012 10 05 21:18:04.1999 0 11,0xa920 16 0 -281.49 -137.88 -122.88
2012 10 05 21:18:05.2599 0 11,0xa920 16 1.92 2.02 0 -163.73
2012 10 05 21:18:06.2011 0 11,0xa920 16 1.92 2.02 28.16 1.92
2012 10 05 21:18:07.2298 0 11,0xa920 16 1.92 0 -163.73 -138.24
2012 10 05 21:18:08.2312 0 11,0xa920 16 1.94 0 -158.61 -133.12
2012 10 05 21:18:09.2063 0 11,0xa920 16 1.92 28.16 1.92 2.02
2012 10 05 21:18:10.1931 0 11,0xa920 16 0 -163.73 -138.24 -120.32
2012 10 05 21:18:11.2165 0 11,0xa920 16 1.93 28.16 1.93 2.03
2012 10 05 21:18:12.2116 0 11,0xa920 16 0 -161.17 -135.68 -117.76
2012 10 05 21:18:13.1944 0 11,0xa920 16 1.94 2.04 28.16 1.94
2012 10 05 21:18:14.2569 0 11,0xa920 16 0 -158.61 -133.12 92.16
2012 10 05 21:18:15.1988 0 11,0xa920 16 1.95 2.05 28.16 1.95
2012 10 05 21:18:16.2079 0 11,0xa920 16 1.97 0 -150.93 -125.44
2012 10 05 21:18:17.1983 0 11,0xa920 16 0 -150.93 -122.88 -107.52
2012 10 05 21:18:18.2468 0 11,0xa920 16 0 -145.81 -120.32 -192
Looking at Ifan, saw that 2.5m on C and 0.5m on A3 were off. Both responded to control-R just now.
A11: This had been powered by an AC-powered, open-frame power supply, through a Sunsaver to a PCAPS battery. On Tues, replaced a blow fuse in the power supply and swapped in a relatively healthy battery. Yesterday found that the voltage was low and decided to test components at the base. In the meantime, wired a DC cable from the M tower battery box using a DC splitter cable. This arrangement is working fine. In the base, was able to charge the battery overnight through the Sunsaver using the bench Tenma supply. Tenma current draw ran for several hours limited at 3A, but had dropped to 1A in the morning. Conclude that Sunsaver and battery are okay, and thus that the open-frame supply is suspicious and may have sustained more damage due to lightning than just a blown fuse. If we continue to run this way, the AC cable to the cooler could be removed.
A8: battery had dropped to 11.8V (from 12.0V yesterday). Voltage on solar input was reading 20V even though pretty cloudy (had been 21V yesterday in nearly full sun). This all seemed odd. Replaced Sunsaver charge controller with an identical Sunsaver that was removed from A11. Now Solar in dropped to 12.1 with battery also at 12.1. "load disconnect" was on, but went off after about 20min with battery voltage now 12.4. Station is now up. I think this definitively demonstrates that the old Sunsaver is bad (not charging).
M21/M22/A11: These DSMs were now all powered through a Lambda brick power supply (6.5A rating, adjusted to 13.6V no load) through an old MorningStar. The Lambda was running VERY hot (too hot to touch after 1/2 second), even before we added A11. Got a call this morning from Kurt/John/Chris recommending that we remove the charge controllers from all AC-powered stations. Used wire nuts to connect input/battery/load wires together. Adjusted Lambda down to 12.8V, as per instructions. Blew breaker on M21 DSM during its boot sequence. Moved M22's power from M21 Aux to another head of the Y cable. (Now the Y splitter is directly connected to power cables to each of M21, M22, and A11.) This all appears to work and the Lamba brick is now cool.
C: Haven't yet changed. Our instructions are to remove its charge controller as well.