Blog from October, 2013

Daily status, October 16

10/16/13

Summary:
Fixed Rlw.out.far pyrgeometer once again.
Today will attempt to hardwire floor Zigbee motes (radiation and soil) to the DSM.

T/RH: RH.40m.near and RH.35m.rim perhaps a bit high.
P: ok
csat u,v: ok
csat ldiag: ok
csat w, tc: ok
kh2o: FLR variance high at night, but also tc'tc'
motes: FAR ok 
           NEAR ok
           FLR all off and on; outages generally last 1:00 - 1:05 hrs.
Wetness: ok
radiation: Rpile.out.far ok
Tsoil: Tsoil.2.5cm.flr & far (linear avg) does not fit profile
Gsoil: ok
Qsoil: ok
Cvsoil: ok
2D sonic: ok

10/16/13 ~1:55Z

logged into flr.   id1 and id2 reporting, not id17.   <cr> restarted id17 connection.

because flr 'hb' cron now runs every hour, set xr=7200 for 2hourly resets if no hb received.

rs 2 again and found id17 not coming in.   played around a bit and then id2 also wasn't coming in.  However, all 3 still responding to any command.
'reboot' command to all 3 motes.     that looks better: all 3 messages coming in at once instead of individually (or not).
let's see if any of this helps....

Service visit to NEAR

10/15/13, twh

Cleaned radiometers around 15:29.

P.S. I don't notice much of an effect on Rsw.in -- certainly no more than 1 W/m2.  This is good.

Service visit to FAR

10/15/13, twh

Replaced Rlw.out.far.  It works!  Cleaned all radiometers.  Left site at 15:15.

At John's suggestion, I just modified the crontab on FLR to send a mote heartbeat every hour.  It had been set for every 4 hours.  Hopefully, this will bring back connections faster if they break.

Daily status, October 15

10/15/13

Summary:
10/14
Replaced floor serializer; kh2o improved
Tried recycling power on Rlw.out.near - no joy
Wack-A-Mote on crater floor
Replaced battery at P.ssw1 and fixed charging at P.w

10/15
Replaced radiation shield at 20m.rim; fan wired backwards

T/RH: RH.40m.near and RH.35m.rim perhaps a bit high.
P: ok
csat u,v: ok
csat ldiag: ok
csat w, tc: ok
kh2o: FLR variance high at night, but also tc'tc'
motes: FAR ok 
           NEAR rad out 10/15 01:35-05:20
           FLR all off and on; now restarting on the hour
Wetness: ok
radiation: Rpile.out.far flat-lined after 10/13, about 16:45
Tsoil: Tsoil.2.5cm.flr & far (linear avg) does not fit profile
Gsoil: ok
Qsoil: ok
Cvsoil: ok
2D sonic: Dir.10m.flr at times appears to be off by 180 deg.

Pulled FAR Rlw.out

Done from about 10:09-10:12.  We'll try to fix it in the base.

P.S. In the base, Tom pulled this sensor apart.  I noticed that the negative (blue) side of the thermopile was electrically connected to the outer conductive ring of the sensor disk due to a misplaced solder blob.  I heated this connection with a soldering iron and removed this contact.  Now there is no connection (infinite resistance) between this contact and the ring.  We'll replace it at far.

Replaced 20m RIM TRH

This done at about 0820 by Gordon (with Steve looking on).  The new sensor took about 10min to cool down, but now looks much better.

The old TRH housing was a bit loose.  I'll take it apart now to check.  Gordon noted that the fan had been running when he removed it.

P.S. It appears that the fan was wired backwards.  When tested, air was blowing down from the plate.  Also noticed that the black/red wires from the fan were opposite those from another spare in the trailer.  Have marked this sensor to be fixed in Boulder (since it isn't obvious to me how to cleanly rewire the small black connector on the end of these wires).

Processing notes

On flux, dsm_server should always be running.

If you make a cal file change, do the following:

  1. dsm_server_restart
  2. statsproc.sh
  3. shutdown cockpit
  4. cockpit.sh

The statsproc.sh script starts 3 statsproc processes. One creates the normal 5 minute, boom normal data. Another creates the 5 minute tilt corrected winds, The third generates 30 second means of speeds, directions, temperatures and pressure for the IOP plots.

To rerun all the covars from the beginning of the project, do:

batch
covar_redo
ctrl-D

That script will run statsproc on the merged data up to 0Z of the current day. It then runs statsproc on today's real-time raw data. Therefore if the network has been congested, the netcdf statistics for today may have gaps.

FYI: the dsm_server process also provides a feed of surface met data over a socket to the GAUS system in the trailer.

Info

See these entries on the SEW wiki:

IOP Plots

On flux, in $ISFF/projects/METCRAXII/ISFF/R is "iop_plots.R", "dataset.R" and a makefile. If you make modifications or add an R file, do a "make" in that directory, then quit/restart R. There is no "synchronize()". You would have to do a "detach(...)", followed by an "attach(...)", so right now it is easiest to quit/restart.

I have not yet created a project.init.R. Would also have to add to $HOME/.Rprofile, to run it automatically.

To run the iop plots, run function "plot.loop()". It reads from dataset("field-30sec") and creates 9 graphics windows.

There is also a script "iop_plots.sh" which should run R and create those plots. There should be a shortcut icon on the desktop called "Rplots" which should run that script.

Updating our R packages

My intention is to put our packages on http://www.eol.ucar.edu/software/R, but currently the web server does not allow downloads from that location. In the meantime, the packages are at http://www.eol.ucar.edu/isf/projects/METCRAXII/isfs/R. I've made a script "update_R" in the project script directory which will fetch and install the packages on Linux.

FLR servicing

Clayton and I were in the crater, starting down at 11:15 and returning to the visitor center about 3.  Tasks done:

FLR tower:

- Swapped serializer, noted voltage returned to 1.68V (rather than 0.05V seen before we set out).  

I've plotted kh2o and tc time series for both far and flr.  tc and kh2o are well correlated in both cases.  Also in both cases kh2o is noisier than tc, mostly noticeable when the variance is low.  I conclude that this serializer is working acceptably well.

- Removed FET jumper on PWR mote serial port.  Verified that we could still talk to it.

SSW1:

- Battery was at 3.3V, even with full sun.  Replaced with spare battery module that we had brought down.  Gordon reported that it was back up.

SW:

- Battery was at 12.48.  Just fine.

WSW:

- Battery at 14.56, slowly dropped when solar panel disconnected, which seemed normal.

W:

- Battery was at 11.28.  Didn't change when charger removed, so guessed that battery wasn't charging.  Opened up box and noticed diode on battery side of charger.  This <seemed> okay, but pulled it out and tried again.  Voltage jumped to 11.68 and continued to slowly climb (in full sun).  Declared it fixed.

Enjoyed our tumbleweed swim.  Enjoyed less our game of boulder rolling.

Daily status, October 14

10/14/13

Summary:
SPO arrived yesterday, will replace floor serializer today.
Wack-A-Mote on crater floor
Rpile.out.near out again, but may be recovering
IOP 3 expected Wednesday evening, 4 & 5 perhaps Friday & Sunday.

T/RH: T.20m.rim often a little high, but also sometimes in line with others.  When does this occur?  At night??
          RH.40m.near and RH.35m.rim perhaps a bit high.
P: ok
csat u,v: ok
csat ldiag: ldiag.near > 0.01 for 3 times for lower dsm (dropped samples sent in real time?)
csat w, tc: ok
kh2o: FLR still flaky; FLR fluxes noisy but believable
motes: FAR ok for past 3 days
           NEAR SPN1 out 10/14 05:25 - 09:15
           FLR all off and on; SPN1 currently missing
Wetness: a bit of dew or frost this morning
radiation: Rpile.out.far flat-lined after 10/13, about 16:45
Tsoil: Tsoil.4.4cm.near has recovered
         Tsoil.2.5cm.flr & far (linear avg) does not fit profile
Gsoil: ok
Qsoil: ok
Cvsoil: ok
2D sonic: Dir.10m.flr at times appears to be off by 180 deg.

Clayton and I refilled the ISFS base trailer generator on Sunday the 13th at approximately 4pm. The lower tank was approximately 60% full, so it could have been pushed to Monday, but it was good training and orientation for Clayton. The new refill by date is now Wednesday at the earliest, but Thursday would be a good bet.

fix inadyn issue

inadyn is a process that runs on the rimup DSM periodically to update the dynamic dns entry for isfs1.dyndns.org. The idea is that it will set the dns entry to the address of the verizon modem on the cradlepoint router. However, the default gateway on that DSM was set to 192.168.0.1, which is the router for the hughes satellite.

As a result, the IP address for isfs1.dyndns.org was being set to the WAN IP address of the Hughes directway from time to time.

I changed the default gateway on that DSM to 192.168.0.5, which is the cradlepoint router.

Also, nearup was also running inadyn to update the DYNDNS entry, which it shouldn't.

So now, the address for isfs1.dyndns.org should now consistently be the Verizon modem address.

At flr, near and far, crontab will now run rserial to send a "hb" followed by a CRNL to the base mote every 4 hours. at 00:00 UTC, 04:00 UTC, etc:

0 */4 * * * echo "hb"$'\r'$'\n' | rserial /dev/ttyS1 localhost > /dev/null 2>&1

The base motes are on ttyS2 at flr, ttyS1 at near and ttyS7 at far.