Using John's wisardMessageDecoder, I was able to see that Vpile.off values were larger than I thought. I've now (again) expanded the range to +/-50uV and now see data for these values and Lambdasoil (as expected). dsm server, etc was restarted.
Gordon: I now see 4 statsproc processes. This seems like one too many?
During last night's IOP, the users were having to cycle power to the Netgear WiFi box every 15min because of dropped connections. As a test, I have been running today using WiFi for more than 2 hours straight. Nevertheless, I just decided to change the WPA "key lifetime" from 60 -> 120 minutes. Perhaps this will keep the connection going longer?
Obviously, the other logical conclusion is that RF interference from things like the tethersonde that only operate during IOPs might be the source of poor connections?
10/20/13
Summary:
- Everything is working
- IOP4 ended successfully. Warm air intrusions occurred between 10pm and 2am.
Actions past 24 hours:
- Floor: cleaned radiometers (they needed it) and changed P.NNE battery
- Reseated connector to revive SPN1.near.
To dos:
- Take gravimetric soil samples (scales and oven should arrive today with Semmer)
Sensor status:
T/RH: ok
P: ok
csat u,v: ok
csat ldiag: ok
csat w, tc: ok
kh2o: ok. Sebastian cleaned flr yesterday.
motes: ok. SPN1.near died last night, but revived by reseating sensor cable.
Wetness: ok.
radiation: flr cleaned yesterday (and were dirty). Will watch to see if SPN1/psp comparison improves.
Tsoil: ok
Gsoil: ok
Qsoil: Still need to recalibrate far and flr.
Cvsoil: ok
2D sonic: ok
About 9:30 this morning (10/20), Sebastian changed the battery at NNE, in an attempt to solve the (somewhat) erratic data reporting from this barometer. Now I see min/max dTs of 0.73/1.31, which is somewhat better though still a larger range than any of the other crater barometers. Sebastian did note that there was a bit of water in one of the power connectors.
We have not seen a big outage since the event 3? days ago, which caused us to pay attention to this sensor.
About 20:00 last night, SPN1.near died. The mote was in a mode of resetting every 10 seconds, which didn't change in response to "rxb" or "xs" commands.
~10:45 Found that the DIN connector to the SPN1 was slightly pulled out. Reseated, cycled power on mote, and now works. Also measured voltage to SPN1 at 17.1V -- should be high enough! (I note that logged voltages are steadily dropping -- now at 15.3.)
Got this email from Sebastian this evening (19 Oct). See below that these cleanings were good.
Steve,
I cleaned the radiometers and krypton - there was lots of bird droppings on the upfacing pyranometer. It took 3 kim wipes to clean it all up ...
I started at 15:45 with the radiometers and finished by 15:55 with the krypton. But looking at the krypton plot now I think I made it dirty instead of clean! Maybe I should give it another try in the morning?
It's dark now, so I hope NNE will last through the night. I'll go check on it tomorrow morning, or if it drops out at night.
Have a good night, too!
-Sebastian
10/20: Effects:
- kh2oV 1.8 -> 2.2V
- Rsw.in: drop(!) of about 8 W/m^2. (P.S. Some morning and mid-day reductions in Rsw.in are now gone. This appeared to have started between 12--14 Oct -- all at the same time. This is consistent with Sebastian's comment about bird droppings, that would have shaded the sensor for high solar elevation angles.)
- Rpile.in: no effect
- Rsw.out: drop of perhaps 2 W/m^2
- Rpile.out: no effect
- SPN1: so far, better agreement between Rsw.global.flr and Rsw.in.flr (though global does go higher at about noon) and between Rsw.dfs.flr and Rsw.dfs.near (though not much data, since I just restarted near).
Filled both tanks on the ISFS generator today and the next time it needs fuel should be 22 Oct 13.
10/19/13
Summary:
- Everything is working!
- IOP4 begins tonight. Last night was a pretty good case (an "EOP" according to Sebastian), but tonight is supposed to be better.
- ISFS shift change: Tom flies out today; Semmer drives in tomorrow; I'm here
- ISS shift change: Cohn arrived yesterday; Phil leaves tomorrow.
Actions past 24 hours:
- Swapped TRH.30m.near. Temperature profiles have been improved and Ifan is normal.
- Reseated Xbee radio in soil.near. No data outages since.
- Tsoil.4.4cm.near appears to have been fixed magically by the soil mote work!
To dos:
- Swap battery on P.nne and/or check connections.
- Clean flr radiometers
- Take gravimetric soil samples (waiting for scales and oven)
Sensor status:
T/RH: ok. Down to -5 at the bottom of the crater last night.
P: ok, though data_stats shows P.nne with wider min/max sample reporting times than other crater P's.
csat u,v: ok
csat ldiag: ok
csat w, tc: ok. Sustained downdrafts at rim during 2 periods last night
kh2o: ok. An odd shift at near about noon (when we were there).
motes: ok!!! no outages.
Wetness: You've got to be kidding! Dry as a bone for days here... (ok)
radiation: still seeing odd SPN1 data, especially at flr. Those radiometers have never been cleaned -- will do today.
Tsoil: ok
Gsoil: ok
Qsoil: far and flr have been negative except for during the one rain event. Should be recalibrated with gravimetric (or assumed =0!)
Cvsoil: ok (I actually did this calculation yesterday. There was a shift after the rain 1.5 weeks ago, presumably as the probe settled into the soil.)
2D sonic: ok
Due to the change of T.30m.near to the nearup DSM, it was necessary to change configs. We are now on metcraxii2. All seems to be okay so far!
(Thanks, and sorry, Gordon.)
Had noticed that 30m T was about 0.1C high and that Ifan has been bimodal between 32/52 mA. Decided to replace with a spare (32->30).
After replacement, neither the old or new worked, even though the new tested okay when temporarily connected to the P.near port (about 1145??). Deduced that the cable must have been damaged.
Since we only had one spare 5m cable (the others having been used to wire the motes at flr), we couldn't replace the entire 30m of cable. Decided to recable this up to nearup. From about 1430-1530 recovered a 5m length of cable from the bottom of near port 15, then having enough to run down from nearup. Connected to port 11. Changed the nearup config and determined that the replacement TRH was working. (ID 310).
Still need to change the configs on flux to use this new sensor placement.
At about 1230, removed near soil mote since we had seen intermittent data transmissions and I noted visually that the XBee radio was not completely seated on the mote board. Brought it back to the trailer and fixed. Replaced at about 14:20?
10/18/13 ~17:30Z
Near: Soil Mote xbee reset rate was still at xr=3600 (hourly). I thought i'd changed all of them to xr=0 on 10/14, but the 'eeupdate' must not have been received by this mote. The other 2 were at xr=0. ID3(Soil) has been having outages of 1 hour and in the raw data files the comments show that every hour this mote was resetting the radio. Clearly some of those reset were causing lost sync. Meanwhile, the 'hb' command is still going at every 4 hours. This is a bit upside down in a sense: with xr=0 the 'hb' won't matter. Otherwise if we retain the 'hb' every 4 hours the xr should probably be xr=21600 (6hours). For now since they seem to be working with the resets turned off, let's leave them that way.
Far: Both motes have been working well. xr=7200 (2hours) and in the comments the radios are indeed being reset every 2 hours. Apparently these resets aren't causing lost sync. However, the 'hb' command is not only set at 4 hours (ie should be less than xr rate), but it also is being sent to the wrong serial port ttyS7 which is the power mote, not ttyS2 which is the soil/rad port. Since they have been working well I changed the xbee reset rate to xr=0 disabling it.
Yesterday, we see an odd event on kh2o.3m.far from 12:28-12:45, with 2 large level shifts and resultant high variance. Our only explanation is that something (a bug) partially blocked the path (twice). Plotting with RH, this definitely was not a humidity event.
10/18/13
Summary:
Fixed Verizon modem (CradlePoint) yesterday morning.
Tsoil.4.4cm.near flat-lined
T/RH: RH.40m.near and RH.35m.rim perhaps a bit high. Ifan suspicious at near.30m and T a bit high both day and night.
P: P.nne looks okay, although SPO noticed sporadic data in the raw time series yesterday
csat u,v: ok
csat ldiag: ok
csat w, tc: ok
kh2o: ok
motes: FLR OK!
FAR ok NEAR soil out 10/17 03:00-04:00, 09:00-10:00, 14:00-15:00, 20:00-21:00; 10/18 05:00-06:00
Wetness: ok
radiation: ok
Tsoil: Tsoil.4.4cm.near flat-lined at 327.7 degC after around 11:00 10/17
Gsoil: ok
Qsoil: ok
Cvsoil: ok
2D sonic: ok
Went to near to get some insight (pun intended) into why Rsw.in and Rsw.global.in don't agree nearly as well as in SOAS. Didn't solve this problem, but did find that the shield on Rlw.in was cocked -- perhaps even in the field of view of the sensor -- I took a photo. Tom thinks he may have loosened it when he last changed the dessicant (more than a week ago), though he didn't notice it being off when he cleaned 2 days ago. The logical conclusion is that it was loose and recent winds skewed it more (though winds haven't been <that> high in the last 2 days...)
Also decided to clean the radiometers while I was there, since I could imagine a film of dust on the SPN1, though the tissue didn't come away dirty. There were spider webs around. At most, the change of either Rsw.in or Rsw.global was 3 W/m2 -- not enough to explain the large differences we've seen.
Cleaning done about 15:30, though I was standing near the radiometers starting about 15:15. 