17-Oct-12; ~8:00-9:00MST, 14:00-15:00Z

Reset Sampling rates from .78hz to speed up to 1hz on the sensors interfaced as serial devices on stations a1,a2,a3,a5,a6,c20,m21,m22.  See table below.   Those interfaced through the motes are and have been at 1hz.

NOTE: When/if someone changes a serial trh, this will likely need to be done.  

Method:

17-Oct-12

personnel: JohnM, TomH.

Weather / Ops:

Clear but rather windy from the west.   Probably not a good day for what the science needs.

Sensor Status / Issues:

Overall the network is doing very well.  Everything has been staying up.   The Flux computer died again this morning and had to be rebooted but that only effects our ability to monitor data; Gully is ingesting/archiving just fine.

a17: The comms/dsm has been intermittent this day.   Best guess is that it's the comm/blue-tooth.   Checked on this in afternoon and indeed it's been the source.   Possibly the cable: connections ok but I dressed the cable a bit to reduce 'wiggling in the wind.'   Need to watch this.....

a2: 1m CSAT is having a diag bit set (4, bit 2) but the values uvwtc all look good.  The CSAT manual says that bit 2 indicates "poor signal lock", which may be caused by an obstruction in one of the paths.   That was not the case.    I tried shielding it from the strong wind and make a 'sweat-shirt' zero chamber which helped.   Probably signal strength/sensativity settings, need to swap.

TRH's: see other entry about speeding up the data rate on the serial versions, stns 1,2,3,5,6,c20,m21,m22

c20: TRH 1.5m outage ~9:00-10:00MST (15-16Z).   When reprogramming the sensor sampling rate the little devil didn't come back up.   DSM power cycling 'eio 7 0' and 'eio 7 1' did not work (need to check that the jumpers are set on that port for the remote power cycle method), so a manual power cycling was done.    2.5m trh was coated/greased completing the basic protection.

a8: TRH's intermittent this morning.   This appears to be the 'mote \r\n' software bug.   Need to fix and reprogram today.....

To Do:

1. Swap CSAT at A2
2. Swap A17 BTradio/cable.
3. Fix mote \r\n bug and reprogram A8's mote

I added new 'pages' or tabs to cockpit with one or related variables for all stations on each page.  For my use, this makes it easier to scan the status.  I saved this config on the ASTER home directory as "byVar", while the previous config is saved there as "byStn".  I intended to delete the station tabs from the "byVar" config, but I could not do this from the toolbar at the top of the window.  Apparently the station tabs, generated in the xml file, are not classified as 'pages'.

Amazingly, everything appears to be still running okay...

16-Oct-12

Personnel: JohnM.,  TomH.

Weather&Ops:

Remained mostly clear last night with calm/light wind.    Clear sky in am, warmish seasonal temps.  Winds expected to pick up today and cooler tomorrow with wind.   A squall with rain and frontal passage came through at ~20MST in the evening, 2Z on 17th.

Sensors:

Overall things looking good / reporting.   Nearly everything is reporting except the Picarro, which is still awaiting to be repaired at their factory.  

Kryptons: voltages a bit better this morning, but we expect to clean them today.....  Kryptons were cleaned at roughly 10MDT, 16Z.   The voltages did come up a bit to roughly 1.2.   

TRH's maintenance and weatherproofing of binder connectors will continue today causing outages.....see specific log entry table.

15-Oct-12

Because of potential problems with the TRH's dropping out with corrosion/moisture, I'm taking the wagon-wheels apart, cleaning and coating the back side of the connectors with nail polish and greasing around the 'pin-well'.   This means intermittent outages.   Results, summary below.

16-Oct-12 update: all except 'c' 2.5m and 'm' 15m have been done.   In addition all those done on 15th need to be finished with grease on top of the nail polish.   Will update as things proceed......

ToDo: Clean/Coat Upper at 'c','m',1-8,      Add 3rd screw 2m on ap14,    Grease fill all connector wells.

15-Oct-12

Personnel: JohnM.,  TomH arriving today.

Weather&Ops:

Remained mostly clear last night with light wind.    Clear sky in am and seasonal temps.   Mostly calm throughout the day and evening.

Sensors:

Overall things looking good.   Nearly everything is reporting except the Picarro, which is still awaiting to be repaired at their factory.  

CSATs still having diag hits this morning. 

Krypton voltages have sagged and need to be cleaned.

TRH's / fans all seem ok and everyone reporting.   There was a cable problem with at least one of the trh's at a17, see trh entry.

TRH connector maintenance: cleaned and coated trh binders at sites 9-20 causing outages.   See other log entry for times.

14-Oct-12

A6 trh out ~12:00-17:00 MDT (18:00-01:00Z on 15th).   I was checking for corrosion on trh binder connectors in the wagon wheels and on this station decided to clean and coat the connection with nail polish.   Unfortunately I didn't plug the dsm cable back in when I finished.

A17 out ~??-22:00 MDT ( - 04:00Z on 15th).   Cockpit was down for awhile due to a flux computer crash and I didn't see that both trh's at 17 were down.   At the site the mote was blinking normally, however, when I did a 'scan-i2c' it locked up and continually reset indicating further problems.   Presuming corrosion on the connections I brought everything back to the base and swapped in 2 hubs that had been previously cleaned/coated.    I also tried reprogramming the mote with newer code to avoid the \r insertion before a \n value in raw data.   This worked but I couldn't get polling back down to 1sec (2 was fastest) so decided to retain the original code.   After going back and forth outside a few times while using the acer computer to check the mote sensor data before plugging them into the dsm I finally discovered that the 'procomm decoder' version on that machine was old and not able to report the trh's correctly.   After finally updating that utility on the acer and rechecking it, I installed the stuff back on the tower and then things looked good on cockpit, etc.    I don't know if any of the problems SteveO had with trh's could have been due to the faulty procomm decoding while directly talking to a mote for testing.

Cable Problem at A17: Examining the shields/cables from a17 last evening reveals that the lower cable has 3 cuts, with at least one wire severed.    It probably occurred when pulling the cable through the galvanized boom, whether I did that last night or it happened earlier and moisture/vibration eventually caused the problem I don't know.   That shield doesn't work at this point.

Meanwhile, mild green oxidation was observed on all of the trh connections I had checked out earlier in the day.   More later...

Finally, it appears tonight could be a good case for Larry: clear and winds calming down.

14-Oct-12 ~10:50MDT, 16:50Z

Soil.G down ~18:30Z 10/13 until ~16:50Z 10/14.

Based on qc plots the mote died, perhaps due to one of the several rain showers yesterday, but that's uncertain.    Upon visual examination the mote was operating properly (ie 1-sec clock and 5-sec polling/send).   It may be possible that the blue-tooth antenna connector came loose because when swapping in a new mote, it was off (or that happened when i jiggled things).   Anyway, I swapped to test console and it started up correctly: everything good.   When trying the spare mote, sn4 which was here and bad previously, it did not work despite having new software reinstalled.    I ended up putting it back together with what works: the prototype V2 mote as it had been.

14-Oct-12 ~11:25-11:40 MDT, 17:25-17:40Z

Radiometer Mote Timing/GPS Reset:  I saw a RIP on a couple parameters and it appeared to me that the mote ID was 'off' when doing mote_dump.   That may be incorrect based on now looking at the 5min covars in qc plots.    Anyway, went to look and discovered it was running a 1hz and the gps was powered on so I backed it off to 5-sec (adjusting sp skip-power to 12 for 1min) and disconnected the gps power.

Radiometers Cleaned: They looked darned good, but i did it anyway

13-Oct-12

Nic visited the site in the afternoon to check the layout, restrictions, etc. prior to taking measurements of opportunity for snow depth/cover/distribution and such.   He took some gps measurements for benchmarking locations.  He's working at NCAR for the next several months.

Nic Wayand
Research Assistant
Mountain Hydrology
University of Washington
email: nicway@u.washington.edu
website: https://students.washington.edu/nicway/ 

14-Oct-12

Personnel: JohnM.

Weather&Ops:

Rain continued in early evening last night (ie early Z of 14th) with some healthy showers around 3zish.   Clearing skies and today is sunny.   Did not freeze overnight and temps should again rise to 15c.   Breezy and pleasant during the day.   By evening (after 0Z 15-Oct-12) winds calmed and with clear skies it could be a good case overnight.

Sensors:

Overall things looking good.   Nearly everything is reporting except 'Grass' soil.  

CSATs still having diag hits this morning presuming some dew issues. 

Krypton voltages have sagged and need to be cleaned after the rain.   Esp. 5m at around .5v.

TRH's fans all seem ok and everyone reporting so the moisture yesterday at least so far didn't manifest in any corrosion or shorting of (as yet uncoated) wagon wheel connection hubs.

Grass Soil:  The mote went down yesterday and I didn't catch it because of not clearing the plot history on cockpit.   Installed mote is the prototype version.   Will investigate.....see log entry

Ah6, A17 trh outages: See other log entry

Flux computer crashed in the middle of the day.   It took me awhile to get things, ie cockpit, going again and had to eventually get ahold of Gordon to determine the proper sequence: script: restart_cp_dsm_server.   NOTE: no data outage on the gully ingest/archive machine, just the cpu we're using to look at the data/status.

13-Oct-12

Weather&Ops:

Some rain went through the area last night and on into today.   So far the periodic showers haven't dropped anything out other than the expected krypton/licor/csat-sampling.    General overcast through the day.   Winds brisk from roughly w/nw.   Temps above freezing roughly 3-10C

Personnel: JohnM, GordanM.    SteveO left yesterday afternoon (10/12) and GordanM left today noon and has pocketec drive to upload high-rate data on Monday.   He generated some 20hz netcdf data for Larry to use.

I haven't figured out the best way to go through the status.  We have been:

- Watching cockpit (about every hour)

- Scanning WWW plots (not exhaustively)

- Scrolling through QCtables

- Using S+ (on Boulder servers) to look at covars for longer-term trends.  

- We do not yet have a way to look at high-rate data in S+ in real-time -- just when data are copied to Boulder via sneakernet.

At the moment, we are only aware of 3 issues:

- Vmote.g dropped quite low last night.  This was just after a change of batteries, so we may have a weak battery.

- The TRH "wagon wheels" that have failed had evidence of corrosion.  We are about to attempt to coat the back of the Binder connectors on all sensors (prior to the next predicted rain event).

- We have had 2 NIDAS issues that Gordon is working on:

  - ntpd started using too much CPU time on A8 last night that blocked all data acquisition

  - A few days ago, we had an Emerald interrupt issue that stopped acquisition of the higher serial ports, I believe on A3.

Wind profiles on the M tower at 0300, 0400, 0700, & 0800 are interesting.  All have winds changing through more than 180 degrees up the tower.  Granted, wind speeds are light (mostly <1 m/s), but the profiles still are very coherent.

Much earlier today (~10:00), John swapped in a different mote (that has an older version of software) into soil.g.  The Tsoil data are now perfect.  It still is somewhat of a mystery as to why this is the case, in part because one of the "spare" Tsoil probes refuses to misbehave on the mote he removed.  About 15:30, we briefly swapped back at grass, and found the same problem.  Thus, we still are not sure of why it is working, or what was wrong, but we now have good Tsoil data from grass for the first time since the lightning strike.  ALL ISFS SCP SENSORS ARE NOW FUNCTIONING!