20-Oct-12

Weather & Ops

Clear and pretty calm in the am. Wspd at .5m was <2 and 1m ~2.5 with
the coolest site being 11 in the bottom of the confluence. However
temps have been staying up around 10 ± 4ish so maybe still too warm
for preferred event.

Communications are all up and files are growing. Everyone is reachable.
Gully is humming along, disks show /media/isfs0 75% and pocketec was at
40% but swapped out to bring hr data back to Boulder. New pocketec at 1%

Sensors & Issues

Thank goodness for the qc table and plots. Everything appears up.
Very few diags on sonics. Krypton voltages still up. Battery
voltages good, and they should be after all the sun the last few days.

To Dos

Bring Flux computer back on line
Crew change: jm leaving this morning, SteveS/ChrisG arriving tomorrow
morning.
Separate list also on wiki
Mini-List for ChrisG: shoot another set of boom angles, finish
protective greasing trh wagon-wheels for a14-a18. Put high-q
conductive grease on all din connector pins. Keep eye on battery
charging but it seems ok. Clean rad, licors and eventually kh2o. Do
we want a soil sample? One TRH-i2c cable and pc-console cable in base trailer to repair.

19-Oct-12 09:55-10:15 MST (15:55-16:15Z)

CSAT Removed: s/n 833   This is the unit that was glitching in higher winds.   It's data was ok in light winds, but better safe than sorry.   Could be used-as-last-resort if nothing else is available.

CSAT Installed: s/n 671   Gordon confirmed it was looking good from Boulder.

9-Oct-12

personnel: JohnM, TomH.

Weather / Ops:

Winds diminished last night and remain low in the morning, possibly a good night/day for the science.  Clear throughout the day with seasonally warm temps.

Ingest:   Everything remained up throughout the day.

Flux Computer:

The Flux computer was still running cockpit ok this morning allowing status check.   Tom Horst retreived it by 7:30 and took it to Boulder to be rebuilt before the next crew arrives.

Sensor Status / Issues:

The sensors continue to perform well and are coming in.   Stations remain up and running.   The issues noticed:

a2: 1m CSAT was not having any diag errors because the winds have been low.    It was swapped by 10:15MST (16:15Z): s/n 833 removed, s/n 671 installed.

a8: TRH mote data now looking good.  Since code update have not seen any \r\n hits.

Mote \r\n Issues: I don't see any/many of these in the data from other stations and now ID8 is running well.   A more thorough examination confirmed that really only ID8 was a culprit.   That particular mote was a replacement/repair brought up from Boulder after the site had some trh issues.    The problem appears resolved.

To Do:

1. Flux computer getting rebuilt in Boulder

18-Oct-12

AP2 serving Stns A8-A13 lost communications last night.   

Power at the interface board was ok, and so were the connections.    Per Gordon's instructions I did not simply unplug/re-plug the usb to AP2, instead shutdown Gully via "poweroff" and rebooted it.   That worked to re-establish the links with all bluetooths/ap24, etc for all stations.

Another method to check the AP1/AP2 is:     pand_check.sh hci#        where #=0 for AP1 and 1 for AP2.  This kills the pand processes serving those bluetooth interfaced radios and sends the hciconfig command to go down, reset, go up and do a piscan.    This may or may not work but's worth a try before doing the full 'poweroff' noted above.

However during the Gully reboot: /media/usbdisk and archiving did not startup automatically but the dsm_server and rsync processes did.

Gully Fix:

The 'adam_env.sh' script in /home/aster/projects/SCP/ISFS/scripts is run during bootup and had the incorrect DATAMNT device listed.   Gordon fixed this to correctly point to /media/isfs0, so now Gully is good to be rebooted when needed.

18-Oct-12

personnel: JohnM, TomH.,  KurtK/ChrisG came up today to retension the main tower.

Weather / Ops:

Winds diminished over last night but still probably still more than 'science preferred'.  Clear throughout the day with brisk winds and seasonally warm temps.  The winds have dropped down by evening (0z for the 19th), so maybe it'll be a good night tonight.

Ingest:   BlueTooth AP2 serving Stns 8-13 died last night.    Gully is ingesting/archiving AP1, AP3241 for the others ok.  See other log entry on Gully/AP comms.

Chadi Sayde came and worked on Christoph's fiber.   He installed the new transformer with Kurt's help.   The kink in his fiber cable remains but is on the 'turn-around-tower' so they're still getting the lower temp readings.   Kurt/Chris checked and retensioned the main tower guys.

Flux Computer Crashes / Sort-of-Fixes:

The Flux computer' fedora/window mgr died and flux had to be rebooted this morning.   This occurred later in the day as well.   Gordon walked us through 'fixing it up' by removing a 2nd disk, fsck checking the boot disk, updating 'grub' to boot proper fedora linux, startup up automatically better (no 'debug manager', and dsm_server/cockpit autostarts.)    However it still appears that the single disk system is sick and will need to be rebuilt in the near future.   We are able to run with it for diagnostics still pretty well though.

Sensor Status / Issues:

Overall the sensors continue to perform well and are coming in.   Stations remain up and running.   The issues noticed:

a17: The comms/dsm BlueTooth has been working pretty well since last night.   It only shows drop outs when I the Gully system/AP2 was restarted.

a2: 1m CSAT is having a diag bit error continuing.

a8: TRH mote data loss from ~10:00-16:30MST (16:00-22:30Z).   The trh mote was reprogrammed this morning, however, I set it for the incorrect destination port so TRH data was lost until ~16:30MST when it was switched to the serial port.

To Do:

1. A2: Swap CSAT
2. A8: Monitor TRH data to determine if the new software fixed the \r\n bug.   
3. Look for 'trh mote' software crc glitches to determine if others need the updated software as well
4. Nurse Flux and have it rebuilt soon.

17-Oct-12

We've had ups and downs on a17 today, suggesting that the intermittent data dropouts are a bluetooth communication issue.   ping_test.sh does show in sometimes and then some long outages, and of course RTdata.

The station itself did reboot this afternoon and based on the /var/log/isfs/messages files has done so a few times in the last few days.  Looking at the messages does show that the blue_check is finding lots of hci/down, restarts so for sure the BTRadio is dropping in and out.  Since it's been a very windy today the thought was perhaps the usb cabling was loose, however, upon checking at the station that did not appear to be the case.   I did put some cable ties on it and re-positioned the mote box slightly with a bigger tie as well.

Conclusion is we'll keep watching it and perhaps swap the radio if this continues.

It looks like the diagbits for the CSAT at A2 has a value of 4, indicating poor signal lock, possibly caused by an obstruction in the path.

John checked A2 for a physical obstruction to the sonic path around 11 am and found none.  When he was able improvise a zero-wind enclosure with his sweatshirt, the diagbits dropped to zero.  Thus the sonic data around this time will be contaminated by John's testing.

17-Oct-12; ~8:00-9:00MST, 14:00-15:00Z

Reset Sampling rates from .78hz to speed up to 1hz on the sensors interfaced as serial devices on stations a1,a2,a3,a5,a6,c20,m21,m22.  See table below.   Those interfaced through the motes are and have been at 1hz.

NOTE: When/if someone changes a serial trh, this will likely need to be done.  

Method:

17-Oct-12

personnel: JohnM, TomH.

Weather / Ops:

Clear but rather windy from the west.   Probably not a good day for what the science needs.

Sensor Status / Issues:

Overall the network is doing very well.  Everything has been staying up.   The Flux computer died again this morning and had to be rebooted but that only effects our ability to monitor data; Gully is ingesting/archiving just fine.

a17: The comms/dsm has been intermittent this day.   Best guess is that it's the comm/blue-tooth.   Checked on this in afternoon and indeed it's been the source.   Possibly the cable: connections ok but I dressed the cable a bit to reduce 'wiggling in the wind.'   Need to watch this.....

a2: 1m CSAT is having a diag bit set (4, bit 2) but the values uvwtc all look good.  The CSAT manual says that bit 2 indicates "poor signal lock", which may be caused by an obstruction in one of the paths.   That was not the case.    I tried shielding it from the strong wind and make a 'sweat-shirt' zero chamber which helped.   Probably signal strength/sensativity settings, need to swap.

TRH's: see other entry about speeding up the data rate on the serial versions, stns 1,2,3,5,6,c20,m21,m22

c20: TRH 1.5m outage ~9:00-10:00MST (15-16Z).   When reprogramming the sensor sampling rate the little devil didn't come back up.   DSM power cycling 'eio 7 0' and 'eio 7 1' did not work (need to check that the jumpers are set on that port for the remote power cycle method), so a manual power cycling was done.    2.5m trh was coated/greased completing the basic protection.

a8: TRH's intermittent this morning.   This appears to be the 'mote \r\n' software bug.   Need to fix and reprogram today.....

To Do:

1. Swap CSAT at A2
2. Swap A17 BTradio/cable.
3. Fix mote \r\n bug and reprogram A8's mote

I added new 'pages' or tabs to cockpit with one or related variables for all stations on each page.  For my use, this makes it easier to scan the status.  I saved this config on the ASTER home directory as "byVar", while the previous config is saved there as "byStn".  I intended to delete the station tabs from the "byVar" config, but I could not do this from the toolbar at the top of the window.  Apparently the station tabs, generated in the xml file, are not classified as 'pages'.

Amazingly, everything appears to be still running okay...

16-Oct-12

Personnel: JohnM.,  TomH.

Weather&Ops:

Remained mostly clear last night with calm/light wind.    Clear sky in am, warmish seasonal temps.  Winds expected to pick up today and cooler tomorrow with wind.   A squall with rain and frontal passage came through at ~20MST in the evening, 2Z on 17th.

Sensors:

Overall things looking good / reporting.   Nearly everything is reporting except the Picarro, which is still awaiting to be repaired at their factory.  

Kryptons: voltages a bit better this morning, but we expect to clean them today.....  Kryptons were cleaned at roughly 10MDT, 16Z.   The voltages did come up a bit to roughly 1.2.   

TRH's maintenance and weatherproofing of binder connectors will continue today causing outages.....see specific log entry table.

15-Oct-12

Because of potential problems with the TRH's dropping out with corrosion/moisture, I'm taking the wagon-wheels apart, cleaning and coating the back side of the connectors with nail polish and greasing around the 'pin-well'.   This means intermittent outages.   Results, summary below.

16-Oct-12 update: all except 'c' 2.5m and 'm' 15m have been done.   In addition all those done on 15th need to be finished with grease on top of the nail polish.   Will update as things proceed......

ToDo: Clean/Coat Upper at 'c','m',1-8,      Add 3rd screw 2m on ap14,    Grease fill all connector wells.

15-Oct-12

Personnel: JohnM.,  TomH arriving today.

Weather&Ops:

Remained mostly clear last night with light wind.    Clear sky in am and seasonal temps.   Mostly calm throughout the day and evening.

Sensors:

Overall things looking good.   Nearly everything is reporting except the Picarro, which is still awaiting to be repaired at their factory.  

CSATs still having diag hits this morning. 

Krypton voltages have sagged and need to be cleaned.

TRH's / fans all seem ok and everyone reporting.   There was a cable problem with at least one of the trh's at a17, see trh entry.

TRH connector maintenance: cleaned and coated trh binders at sites 9-20 causing outages.   See other log entry for times.

14-Oct-12

A6 trh out ~12:00-17:00 MDT (18:00-01:00Z on 15th).   I was checking for corrosion on trh binder connectors in the wagon wheels and on this station decided to clean and coat the connection with nail polish.   Unfortunately I didn't plug the dsm cable back in when I finished.

A17 out ~??-22:00 MDT ( - 04:00Z on 15th).   Cockpit was down for awhile due to a flux computer crash and I didn't see that both trh's at 17 were down.   At the site the mote was blinking normally, however, when I did a 'scan-i2c' it locked up and continually reset indicating further problems.   Presuming corrosion on the connections I brought everything back to the base and swapped in 2 hubs that had been previously cleaned/coated.    I also tried reprogramming the mote with newer code to avoid the \r insertion before a \n value in raw data.   This worked but I couldn't get polling back down to 1sec (2 was fastest) so decided to retain the original code.   After going back and forth outside a few times while using the acer computer to check the mote sensor data before plugging them into the dsm I finally discovered that the 'procomm decoder' version on that machine was old and not able to report the trh's correctly.   After finally updating that utility on the acer and rechecking it, I installed the stuff back on the tower and then things looked good on cockpit, etc.    I don't know if any of the problems SteveO had with trh's could have been due to the faulty procomm decoding while directly talking to a mote for testing.

Cable Problem at A17: Examining the shields/cables from a17 last evening reveals that the lower cable has 3 cuts, with at least one wire severed.    It probably occurred when pulling the cable through the galvanized boom, whether I did that last night or it happened earlier and moisture/vibration eventually caused the problem I don't know.   That shield doesn't work at this point.

Meanwhile, mild green oxidation was observed on all of the trh connections I had checked out earlier in the day.   More later...

Finally, it appears tonight could be a good case for Larry: clear and winds calming down.

14-Oct-12 ~10:50MDT, 16:50Z

Soil.G down ~18:30Z 10/13 until ~16:50Z 10/14.

Based on qc plots the mote died, perhaps due to one of the several rain showers yesterday, but that's uncertain.    Upon visual examination the mote was operating properly (ie 1-sec clock and 5-sec polling/send).   It may be possible that the blue-tooth antenna connector came loose because when swapping in a new mote, it was off (or that happened when i jiggled things).   Anyway, I swapped to test console and it started up correctly: everything good.   When trying the spare mote, sn4 which was here and bad previously, it did not work despite having new software reinstalled.    I ended up putting it back together with what works: the prototype V2 mote as it had been.