Blog

10/10/13 at 16:50.  Reseated the up-looking pyranometer shield, which was not completely seated.

10/11/13, twh:

Arrived 4:39 pm.  Cleaned kh2o and reseated all connectors (initially looked okay).

4:49 - 4:53 pm. Cleaned radiometers, wetness sensor.

4:58 - 5:08 pm. Replaced dessicant in pyrgeomters.  The cylinders were screwed in so tight that I wonder whether they had ever been serviced.

10/11/13, twh: 4:14 - 4:20 pm

Cleaned kh2o, radiometers, wetness sensor.

Soil mote stand blown over about 45 degrees to the north.  This stretched the soil sensor wires.  Did it disturb the sensors?.   Mote stand needs a third stake on the south.

We are not sampling the battery voltages at the outlier barometer and sonic stations, and so they have to be measured by hand. We can keep a record here.

NNE
Oct 11: healthy power level: 14.4 V at radio power connector. Data from this barometer is suspect however.

SW
Oct 11: Dead, 4.0 V. Replaced battery. Checked connections. New battery: 16.3 V

SSW1
Oct 11: not great: 11.2 V. Checked connections in battery box. Voltage increased to 11.6

SSW2
Oct 11: healthy: solar 19V, load 14V, battery 14V

SSW3
Oct 11: healthy: 14.7 V at radio power connector

SSW4
Oct 11: healthy: solar 17.7, load 14.2, battery 14.2

Sebastian measured the voltages of the rest of the pressure motes:

WSW at 9:00 am MST 10/12: 13.86 V

W around 9:15: 10.71 V; 11.25 V after fiddling with the connectors to and from charging chip

S at 9:35: 14.18 V

Tom, Gordon arrived at tower in crater floor, 10:30am. Calm, very nice day.

Cleaned radiometers and wetness at 10:40
Replaced dessicant on pyg and pyrs
Level of uplooking KZ pygeometer is good. Pyranometer doesn't have a level.

Checked mounting of zigbee radios in all motes - looked good. All GPS jumpers are removed.

Replaced 1101 krypton with 1394. Cleaned 1394 before mounting. Voltage level 1.1V

At 12:33 checked level on SPN1, which was good.

Went to NNE to check barometer, which has shown some bad data. An easy way to find NNE is to follow the wavy line of hobos in that direction. A hobo is right next to NNE.

Voltages at NNE look good. 14.4 V where the power cable plugs in to the PC board. 3.3V at the other test point. Can't see anything obviously wrong with this barometer. Left NNE at 11:54.

Went to SW-p, which has been out fpr a day or so. Voltage at board power connector was 4.04 V. One connector inside the battery box appeared to be loose. After fixing that, the voltage increased to 7.6 V. So it appears that the is was just a loose connector issue. Replaced the battery. Voltage at radio board was 16.3V.

SSW2-p/c This site has a regular battery and solar panels. The large solar panel had blown over against the other so that it was facing down. Moved it a bit more out of the wind. At the controller: solar 19V, load 14V, battery 14V.

SSW3-p 12:58. 14.67V at the radio power connector.

SSW4-p/c 1:05. At charge controller: solar 17.7, Load 14.2, battery 14.2

SSW1-p 1:20 11.2 V at the radio power connector. Opened up battery box. Seemed the the voltage at the points labeled PV was 0.4V. Fiddled with connectors. Just before closing the lid, saw 11.9 at the battery, 12.3 at the "PV" test points and 11.6 at the radio. Did not replace the battery since it seems that it is now being charged.

Before leaving, checked the motes. SPN1 was not coming in. Tom said the LEDs look OK. Power cycled that mote.

Also noticed that we had failed to reconnect the sensor cables and had left the soil mote turned off when checking whether the radio was seated on the board.

2:00 left floor.

10/10/13

Summary:
TRH at 20m.rim is reporting garbage
P.(nne,sw).flr reporting garbage
kryptons questionable at FLR and NEAR (per SPO)
Motes at FAR and NEAR doing well, FLR rad and SPN1 currently ratty
Tsoil.3.1cm.near has recovered
Rpile.out.far flat-lined

T/RH: 20, 35, 40m T.rim too high after (02:05, 03:45, 02:05), 10/10
                Gordon fixed 40m with software reset at 12:30.
                Tom likewise fixed 35m at 15:25.
                40m RH at NEAR perhaps a bit high.
P: nne.flr went walkabout around 09:00 10/8
     sw.flr went walkabout 05:00 10/10
csat u,v: ok
csat ldiag: some ldiag > 0 during high wind yesterday and this morning with precip
csat w, tc: ok
kh2o: FLR flaky; FLR fluxes noisy but believable
motes: FAR ok 
           NEAR ok
           FLR rad  & SPN1 off and on
Wetness: ok
radiation: Rpile.out.far flat-lined since 10/7 09:00
Tsoil: Tsoil.3.1cm.near has offset since 10/7 12:15
                Spontaneously recovered 10/8 15:25
Gsoil: ok
Qsoil: FLR, FAR were < 0; now > 0 after rain
Cvsoil: ok
2D sonic: ok, FLR wind direction has been corrected

10Oct13, ~16z

Continuing rad/soil mote outages appear related in at least some cases to the 3hourly xbee resets automatically done by the motes.   Because the gps's and the local storage options are now turned off, having radio outages is problematic.   In an attempt to improve the situation the timing was changed:

Flr: id1 & id2 & id17: xbee reset = hourly (xr=3600), gps=off (gr=0)

Near: id3=xbee reset hourly,   id8 & id10=xbee reset off; gps=off(gr=0)

Far: id4 & id22: xbee reset = 2hours (xr=7200),  gps=off (gr=0)

With this mix, hopefully the outages will be less and we can see if the reset timing is still the critical factor instead of hardware issues.

We may want to consider having the dsm's send 'hb' heart-beat commands to the motes and re-enable xbee reset values (which only reset if there is no heart-beat received).

14Oct13, ~1Z GMT

Noticing that Tom reported in the mote status that flr was having additional problems I logged in and rserial'd to the xbee's.

ID1 and 2 were reporting but not 17 as noted.   I tried a 'hb<cr>' command and then a simple <cr> and 17 began reporting.

Further commands issued and normal responses were received.

Changed xbee reset timing: was xr=3600 (1hour), to xr=43200 (12hours).   Maybe they should be turned off, however, what i notice is that things work much better after commands are received from the base radio, so we should consider the cron-'hb' i believe.

14Oct13, ~19Z

flr:  The flr motes had outages last night and at least 2 rebooted.   The setting above hadn't gotten 'eeupdate'd so i reset the xr=43200 and stored the setting.   While interacting with them, id2, which had been 'down', again began sending data.

near:  Daily status reported outage of the spn.   All 3 were reporting when i logged in.  I changed id3 to disable the xbee reset, xr=0, to match the other 2 and not be what appears to be problematic hourly resets.   Gordon's cron 'hb' heartbeat is running so we might try setting xr=12hrs, but not yet.

I refueled the ISFS generator at BASE today at 12:30pm local on thursday the 10th.

I checked fuel levels and found the smaller upper tank empty, and the lower at 1/2, and refueled both, so it should be good for another three days, expect to refuel again on SUNDAY the 13th.

The rest of the site looks like it weathered the storm passage ok, a little evidence of moisture and the expected winds, but no snow down at this altitude.

We (Damien from Leosphere and myself) are here utilizing bench space to work on the Lidar computer replacement.

Let me know(email or cell) if there is anything else I can check on for you guys while I am here this afternoon.

Thanks, Tim

Tom noticed that two trhs are bad on rim.

They are reporting wacky temps and RH:

20m (ttyS15 on rim)
TRH28 140.50 461.21 0 0 4605 2054 0\r\n
TRH28 140.46 461.85 0 0 4604 2052 0\r\n
TRH28 140.46 461.08 0 0 4604 2054 0\r\n

40m (ttyS8 on rimup)
TRH19 139.79 0.00 46 0 4562 1999 144\r\n
TRH19 139.79 0.00 46 0 4562 2001 144\r\n
TRH19 139.82 0.00 46 0 4563 2004 143\r\n
^R\n
\r Sensor ID19   I2C ADD: 12   data rate: 1 (secs)  fan(44) max current: 80 (ma)\n
\rresolution: 12 bits      1 sec MOTE: off\r\n
calibration coefficients:\r\n
Ta0 = -3.993325E+1\r\n
Ta1 =  4.048482E-2\r\n
Ta2 = -2.386631E-7\r\n
Ha0 = -8.241113E+0\r\n
Ha1 =  5.922000E-1\r\n
Ha2 = -4.567768E-4\r\n
Ha3 =  8.744341E-2\r\n
Ha4 =  1.592101E-3\r\n
Fa0 =  3.222650E-1\r\n
TRH19 5.82 60.80 46 0 1138 126 143\r\n
TRH19 5.82 60.80 47 0 1138 126 146\r\n
TRH19 5.86 60.81 47 0 1139 126 146\r\n

Was able to get the 40m TRH to reset and start reporting believable data by sending ctrl-R from rserial as shown above.

That did not work for the 20m TRH (which we've seen before). Power cycling port 15 with eio also does not work

The SPN1 global radiation often is significantly (up to 100 W/m2) higher than Rsw.in on clear days.  This could indicate a reflection problem, though without seeing the exact set-up, I can't imagine how reflections would affect only this sensor.  Plotting the data as an x-y comparison, there appears to be some curvature (non-linear comparison) both at high and low radiation levels, as well as some hysteresis.

Looking at SOAS data, I also see higher SPN1 values than Rsw.in, but this is well modeled by a simple gain factor. The sky was never totally clear to replicate what is being seen in METCRAXII.

At the least, I would check that the SPN1 is level (though I still can't think of how being non-level would increase incoming radiation values).

I note odd values during the last 24 hours in both P.nne.flr and P.sw.flr.

10/9, twh

The high winds today provide an opportunity to adjust the wind directions on the profiles towers.  We only measured the 3m sonic azimuth with the DataScope and assumed that the other sonics on the profile towers were aligned with the lowest level.  For the past 6 hours, the wind speed above 3m at near has been higher than 15 m/s, while that at 3m has been 13-14 m/s.  The following table shows the median wind directions relative to that at the lowest level. These values should be subtracted from the azimuth at each level.  

This adjustment is perhaps more tenuous at RIM because of the complex topography, but the speed and direction plots suggest little terrain influence for winds from 200 degrees (SSW).

See accompanying plots, made with dir.diff.profile for the exact period selected for high wind speeds.

10/31: I redid the analysis with a miminum wind speed of 10 m/s at 10 m.  This gives a much larger data sample (105 hours at NEAR and 150 hours at RIM) and a wider range of wind directions, 190 - 230 deg at NEAR and 178 - 242 deg at RIM, with mean wind directions for both sites near 208 deg.

25m near and 20m rim TRH's have been reporting values of 0 for their fan currents.

I tried rserial to both TRHs and doing a ctrl-R, but that didn't help. The TRH's did not respond with their reboot message. The only effect was to stop the data, which only resumed after hitting CR several times.

I could power cycle the TRH at near with "eio 14 0", "eio 14 1" commands, so that Ifan.25m.near is now about 40.

Power cycling the TRH at rim (eio 15 0/1) didn't work, it continued to report wheh supposidly powered down. That port must not be FET controlled.

Tim bought a replacement for his blown UPS, and has loaned the new unit to us. It's a low-end consumer "CyberPower 425VA" unit.

I've moved some essential peripherals to it from the APC UPS with the dead battery:

• /media/backup external UPS drive
• Hughes satellite modem
• AP24 Alico WIFI access point.
• Dlink 8-port hub
• Netgear router
• Netgear 8-port hub

What's left are laptops with their own batteries, and monitors. These are all on the surge protection side of the UPS's.

During IOP1 the network to the back of the trailer suffered problems. I believe the network switch in the back of the trailer was connecting to a LAN port on the Netgear router, which apparently hung. The WAN port on that router is connected to the Hughes modem. The Netgear also serves the trailer "metcraxii" WIFI.

I've removed all LAN connections on the Netgear router except for one going to the D-link 8-port switch.

We'll connect all other switches to the D-link, and avoid further daisy-chaining the switches and connecting any other systems to the Netgear router and see if that helps.

Spectra of kryptons indicates a noise problem at floor.