Who: Ned, Christopher
When: December 15, 2011. Arrived 9:51am, Departed 12:00pm
Removed Licor (sn 1176) from 15m.
Note, by Gordon, Jul 13, 2012: There is no sn1176. I think this must have been unit sn1167, which according to the log entry of 4/12/2011, was installed at 15m.
When removing this sensor due to bad chopper motor, it was not running. Rebooted sensor and heard the chopper motor in the head and it sounded bad. Removed the 7m Licor (sn0813) and taped up connections and moved this Licor to the 15m height. Plugged in all connections as normal (signal green cable and power through BNC). All looked (and sounded) great. Replaced the TRH fans at heights 7m and 15m. Both fans sounded REALLY bad. The 2m TRH fan did not sound very good either. Guessing we will be replacing that fan soon.
*Did notice that when the Licor 1176 was at 15m and before removal I checked power at the inside green connector. Voltage was going from 12.9V to 11.4V every six seconds. Odd cycle, but this also seemed odd in general. Checked the power at the 7m height when removing that Licor and it had the same cycle but not as drastic. Took a voltage reading at dsm, voltage was cycling. Took reading at battery bank and did see the cycle but not as drastic. Noticed the battery charger was pulsing to charge the battery bank. It was on the 'deep cycle'. We switched charger to 'conventional cycle' and it all stopped. No more voltage swings. We also wanted to know why the batteries were not taking out that pulsing to the tower. One of the batteries must be bad or wrong charger for batteries. We unplugged the charger to see what the load did on the batteries, checking for bad battery. In 15minutes we dropped from 13.10V to 12.47V. Not much of a drop. Leaning to more of the wrong charger cycle and will monitor the battery bank on this current setting.
15 m Li-COR (sn1176) dropped out of service on November 6, 2011.
Note, by Gordon, Jul 13, 2012: There is no sn1176. I think this must have been unit sn1167, which according to the log entry of 4/12/2011, was installed at 15m.
The diagnostic value dropped to 215 indicating that Bit 5 signaled that the instrument's chopper motor failed. There was an indication a number of hours earlier that the chopper motor was beginning to fail, see:
http://www.eol.ucar.edu/isf/projects/BEACHON_SRM/isfs/qcdata/plots/2011/11/06/licor_20111106.png
Chris and I made a trip to MFO to move the 7m Li-COR (sn 0813) up to 15m and to remove the dead instrument from 15m. The tower is therefore operating again with three levels of water vapor and CO2 fluxes (2m, 16m, 43m). See Chris' entry for details of this instrument swap and timing.
Gordon, Dec 9 3:15 pm
Since Dec 1 22:50 UTC the DSM log file (/var/log/isfs/adam.log) has error messages that the GPS pseudo-terminal device, /var/tmp/gps_pty0, does not exist. Apparently the tee_tty process died at that time.
tee_tty reads the GPS ASCII messages on /dev/ttyS3, and writes them to two pseudo-terminals /dev/gps0 (read by NTP) and /var/tmp/gps_pty0 (read by dsm).
I'm not sure why tee_tty died, but decided that this was an opportunity to upgrade the NIDAS software on this system. The existing version was 5810M, current as of Nov 9, 2010.
Installed the latest and greatest, version 6364.
During this time I noticed that characters were being lost in the ssh session, such that I had to enter characters twice for them to get to the shell if the dsm process was running. I have a faint memory of this happening before. Determined that this is due to tee_tty exiting, leaving the symbolic links to the pseudo-terminals around. The dsm process re-opens the /var/tmp/gps_pty0 every 10 seconds after an error, and so if sshd creates a pseudo-terminal and /var/tmp/gps_pty0 points to it, then the dsm process will be stealing characters from the ssh session. Did a dump of the GPS data and saw my shell commands! Hacked myself 
Installed a new version of tee_tty, which will catch the HUP, INT and TERM signals, and clean up the symbolic links on exit. Tested it and things look good.