14-Oct-12 ~11:25-11:40 MDT, 17:25-17:40Z

Radiometer Mote Timing/GPS Reset:  I saw a RIP on a couple parameters and it appeared to me that the mote ID was 'off' when doing mote_dump.   That may be incorrect based on now looking at the 5min covars in qc plots.    Anyway, went to look and discovered it was running a 1hz and the gps was powered on so I backed it off to 5-sec (adjusting sp skip-power to 12 for 1min) and disconnected the gps power.

Radiometers Cleaned: They looked darned good, but i did it anyway

13-Oct-12

Nic visited the site in the afternoon to check the layout, restrictions, etc. prior to taking measurements of opportunity for snow depth/cover/distribution and such.   He took some gps measurements for benchmarking locations.  He's working at NCAR for the next several months.

Nic Wayand
Research Assistant
Mountain Hydrology
University of Washington
email: nicway@u.washington.edu
website: https://students.washington.edu/nicway/ 

14-Oct-12

Personnel: JohnM.

Weather&Ops:

Rain continued in early evening last night (ie early Z of 14th) with some healthy showers around 3zish.   Clearing skies and today is sunny.   Did not freeze overnight and temps should again rise to 15c.   Breezy and pleasant during the day.   By evening (after 0Z 15-Oct-12) winds calmed and with clear skies it could be a good case overnight.

Sensors:

Overall things looking good.   Nearly everything is reporting except 'Grass' soil.  

CSATs still having diag hits this morning presuming some dew issues. 

Krypton voltages have sagged and need to be cleaned after the rain.   Esp. 5m at around .5v.

TRH's fans all seem ok and everyone reporting so the moisture yesterday at least so far didn't manifest in any corrosion or shorting of (as yet uncoated) wagon wheel connection hubs.

Grass Soil:  The mote went down yesterday and I didn't catch it because of not clearing the plot history on cockpit.   Installed mote is the prototype version.   Will investigate.....see log entry

Ah6, A17 trh outages: See other log entry

Flux computer crashed in the middle of the day.   It took me awhile to get things, ie cockpit, going again and had to eventually get ahold of Gordon to determine the proper sequence: script: restart_cp_dsm_server.   NOTE: no data outage on the gully ingest/archive machine, just the cpu we're using to look at the data/status.

13-Oct-12

Weather&Ops:

Some rain went through the area last night and on into today.   So far the periodic showers haven't dropped anything out other than the expected krypton/licor/csat-sampling.    General overcast through the day.   Winds brisk from roughly w/nw.   Temps above freezing roughly 3-10C

Personnel: JohnM, GordanM.    SteveO left yesterday afternoon (10/12) and GordanM left today noon and has pocketec drive to upload high-rate data on Monday.   He generated some 20hz netcdf data for Larry to use.

I haven't figured out the best way to go through the status.  We have been:

- Watching cockpit (about every hour)

- Scanning WWW plots (not exhaustively)

- Scrolling through QCtables

- Using S+ (on Boulder servers) to look at covars for longer-term trends.  

- We do not yet have a way to look at high-rate data in S+ in real-time -- just when data are copied to Boulder via sneakernet.

At the moment, we are only aware of 3 issues:

- Vmote.g dropped quite low last night.  This was just after a change of batteries, so we may have a weak battery.

- The TRH "wagon wheels" that have failed had evidence of corrosion.  We are about to attempt to coat the back of the Binder connectors on all sensors (prior to the next predicted rain event).

- We have had 2 NIDAS issues that Gordon is working on:

  - ntpd started using too much CPU time on A8 last night that blocked all data acquisition

  - A few days ago, we had an Emerald interrupt issue that stopped acquisition of the higher serial ports, I believe on A3.

Wind profiles on the M tower at 0300, 0400, 0700, & 0800 are interesting.  All have winds changing through more than 180 degrees up the tower.  Granted, wind speeds are light (mostly <1 m/s), but the profiles still are very coherent.

Much earlier today (~10:00), John swapped in a different mote (that has an older version of software) into soil.g.  The Tsoil data are now perfect.  It still is somewhat of a mystery as to why this is the case, in part because one of the "spare" Tsoil probes refuses to misbehave on the mote he removed.  About 15:30, we briefly swapped back at grass, and found the same problem.  Thus, we still are not sure of why it is working, or what was wrong, but we now have good Tsoil data from grass for the first time since the lightning strike.  ALL ISFS SCP SENSORS ARE NOW FUNCTIONING!

I just (~20:00 on Oct 11) typed the following to the nanobarometer:

*0100EW*0100PI=50*0100OI=0*0100P4<cr><control-j>

that set the data rate to ~10sps and the output format back to what we used in RS232 mode.

There are settings to set FIR mode, but I don't have this documentation here to tell me how to make sure that it is used.

P.S. I remembered about an hour later that I put all the settings in the software wiki:

http://wiki.eol.ucar.edu/sew/ISFS/SensorInformation/NanoParo

These settings have now (2108) been applied, and the barometer is reporting at ~14sps.

P.P.S.  About another 15min later, decided to test whether these settings are retained during power cycles.  Found that we needed to reinsert the init_string.  This has now been done on the DSM.

Today, Alison was here with 2 people from PES Group to shoot video and interview us about ISFS and SCP.  I didn't get much else done this morning and early afternoon.

This morning, A8 was giving checksum errors.  This was running through a serial port through a cable stolen from A5 last week.  Just changed out this cable.

From about 0830-0840, cleaned: Rlw.in, Rlw.out, Rsw.in, Rsw.out, Li7000.2m, Li7000.1m

As per http://www.ngdc.noaa.gov/geomagmodels/Declination.jsp, the declination for the SCP site is: 8° 43' E.  We will use 8.7 in Vazimuth calculations.  I note that the solar panels (and rad stand) were oriented with the compass set to 12, so they are 3degrees off.  I also note that the GPS RMC messages are showing either 8.8 or 10.0 for this location.

There wasn't obvious damage to the above-ground part of Qsoil.c, but it isn't responding.  About 20min ago, I placed a spare Qsoil poking into the ground from above.  It is reading fine (at 10.76%), but reports as ID29, rather than ID2A of the previous probe, so it isn't being processed correctly.

A restart of dsm_server now finds the new Qsoil (S/N 012).

1505: Realized looking at the data that the old Qsoil (S/N 013) was working (perhaps when it warmed up a bit).  Have just reconnected it.  Also just buried new Qsoil off to the SW.  I'm not totally happy with its installation, but it should be close.  Thus, there are now 2 Qsoils (ID 2A, old and ID 29, new).  I'll let this go like this for another day, then make a decision as to whether to remove one, or to change the .xml to record data from both separately.

Just came back from A19, A18, and C (in that order), where I wiped a few drops off of transducers.  This seems to have lowered the level of diagbits being set.  After cleaning, I blocked the path with my hand for 10s (both to put a flag in the data and to cause the CSAT's software to lose any memory it might have of the wrong sonic signal.  (I don't think it does this in V3, but didn't think it would hurt.)

Most of the instrumentation is working, however there are several issues:

1. The Picarro is now back at the factory being looked at after lightning damaged its serial port.

2. TRHs continue to fail sporadically across the network.  Most revive with a power cycle.  Recently, these outages have only been on I2C TRHs.  In all the recent cases that I can recall (and where I've visited the site), the fans were still running.  Today, A19.2m was brought back to life by replacing the head (but keeping the same SHT).  We've noticed hints of corrosion on the back (top) side of the Binder connector and sometimes on the fan control board.  I would <strongly> recommend potting both of these areas.

3. Tsoil.g still reads mostly garbage through the mote since the lightning strike killed the original Tsoil.g sensor.  The replacement probe that is in the ground reads fine in RS232 mode.  Changing the mote data reporting to 5s didn't help.  Thus, there must be a data format mismatch between the Tsoil and mote.  Steve and John are looking into this.  

4. Qsoil.c died at about midnight last night.  There are no obvious signs of damage above ground.  When it is the only sensor plugged into a mote, it reports na's.  When any other sensor is connected to the mote, even the na's disappear.