Blog from November, 2010

November 30: TRH fan noisy at Playa.

November 30, 2010

Site 2 (ABC): upper radiometers snow/ice covered; lower pyrgeometer has a little frost on it; cleaned

Site 3 (Highland): upper radiometers partially snow/ice covered; cleaned @ 11:20

Site 1 (Playa): Radiometers bare of snow

Site 7 (Riverton): Radiometers bare of snow

Site 5 (E slope): cleaned uplooking radiometers @ 14:53

Cleaned solar panels as needed at all these sites; mostly bare or a little snow/ice near the bottom

Nov 30, 2010.  Visited several sites to install recalibrated kh2o's.

Site 2 (ABC) 1389 head bad; output ~30 mv.  Installed 1393 about 10:30; output ~2 V.

     twh (3/1/11): dat("w") shows mast down ~ 09:15 - 10:35; high-rate w suggests similar period

             (3/2/11): high-rate pressure data more clearly show mast down 09:16 - 10:36

Site 3 (Highland) Arrived ~ 11:15

                          1333 head  bad; output ~30 mv

                          1258 head bad; output ~20 mv

                          1390 head bad; output ~20 mv

                          1525 good!; output 2.2 V

Attached 1133 head to 1525 electronics and warmed head; still bad.

Reattached 1525 head; output 2.3 V.

Reattached 1525 head and left.

   twh (3/1/11): dat("w") suggests mast down ~ 11:25 -  12:35; high-rate w suggests similar period

          (3/2/11): high-rate pressure data clearly shows 11:25 - 12:36

Site 1 (Playa) Installed 1395 around 14:30.  Also tested 1394, which was good.

   twh (3/1/11): dat("w") suggests mast down ~ 14:10 - 14:35; high-rate data suggests 14:07 - 14:34

      (3/2/11): high-rate pressure data clearly show mast down 14:07 - 14:34

Site 7 (Riverton) Installed 1394 around 15:55; output ~2.3 V.

   twh (3/1/11): dat("w") suggests mast down ~ 15:45 - 16:05; high-rate w shows 15:47 - 16:03

           (3/2/11): high-rate data of T-tc also support 15:47 - 16:03; pressure is not clear

Dec 1, 2010:  Completed kh2o installation.

Cleaned 'bad' kh2o's by scrubbing with qtips and distilled water. 

1133 went from 30 mv to 2 V

1258 went from 14 mv to 2 V

1389 went from 23 mv to 2 V

1390 went from 11 mv to 1.7 V.  Designated as spare.

Site 4 (West Valley): 12:10 installed kh2o 1258; 2 V output

    twh (3/1/11): dat("w") suggests mast down ?~ 11:35 - 12:15; high-rate w shows 11:41 -  12:15

           (3/2/11): high-rate pressure data clearly indicate mast down 11:42 - 12:16

Site 6 (West Slope): 14:30 installed kh2o 1389; 2 V output

    twh (3/1/11): mast does not appear to have been lowered

Site 5 (East Slope): 15:50 installed kh2o 1133; 2 V output

   twh (3/1/11): dat("w") suggests mast down ?~ 15:30 - 15:50; high-rate w shows 15:31 - 15:55

          (3/2/11): high-rate data for T-tc and pressure support 15:31 - 15:55

Solar Panels at Site 1 Playa cleared at 1pm MST. Snow covered bottom panel on older set, and 1/3 of lower panel on the newer set. Melting resulted in ice from melted snow forming on lower panels.

Panels at rad sawhorse, and both soil sensor panels were clear at visit.

Approximately 10-12 inches of snow observed at site. Snow damming observed at south facing solar panels, leading to deeper snowdrift at base of panel. Built up snow cleared away from base of panels. Snow is not expected before NCAR ISFS crew arrives.

Snow removal occurred concurrently with installation of UU Mini-SODAR approximately 200m away.

visited site one to determine why the  aux soil mote was down. found that Tsoil was dragging down the mote, so I disconnected Tsoil, and mote now seems to be OK although Qsoil is not reporting.

so...  Tsoil.aux and both Qsoils are no longer reporting at site 1 although both motes are OK

checked radiometer domes and all are clean

I have noticed that at site 2 on two separate occasions, when power to DSM is recycled, the trh data output is in hexadecimal until the sensor itself is rebooted.

moved Viper power cable to another power port on the main power board.

the prop vane azimuths were shot using the datascope earlier this week with the following results

site 1 (magnetic declination set to 12.3 degs per NOAA website, and datascope field calibrated by Chris Golubieski)   1.0 degs

site 6    1.7 degs

neither site has had this boom direction entered into the RMY propvane program as yet.

visited site 2 again today from about 1:30 till 3:00 monitoring power (voltage dropped again this morning to about 11.2 v)

swapped out viper power cable.

wiggling the cable brings the voltage back up to 12.7 v, (where it should be), but that's what was happening yesterday as well, and I repaired the original cable at that time. since the voltage dropped again this morning, right about the time the wind started getting brisk, and shaking the tower, i suspect a cold solder joint in the power distribution board. I had an extra cable, so I decided to try that first.

will monitor situation, and if voltage drops again, i will replace either power panel, or more likely, entire DSM so that the offending board/box can be scrutinized in a more friendly environment

Friday, Nov 19th 11:00 AM MST to approx. 2:00 PM

 I cleaned incoming and outgoing radiometer domes - I noticed during site visit to improve cell connection that the domes (mostly the incoming) had a fine coating of soot. This is a particularly windy site, and the Kennecot copper mine smelter is directly across the highway, about 1-2 miles away. The mine regularly sprays water on the substantial slag heaps to keep the dust down, but it looks to be ineffective. This site may need cleaning more often.

while at the site I had a visit from Randy Finch, an engineer from KSL Radio who regularly visits their building / tower which is about 1/4 mile to the east of our ISFS station. He pointed out to me where the Verizon cell towers (2) are located. Neither are visible to me but their locations are to the NW and SW of our tower.

I positioned site 1's  cellular antenna to minimize blockage.

KSL's tower is a AM radio transmitter around 1.5 MHz

They have a secure building with land line internet connection as well as power, and he offered both to us if we so desired.

Contact info:

Randy Finch

KSL radio

801 556-7565 (cell)

801 575-7637 (transmitter building)

Sat Nov 20

visited site 4 (not reporting as of 7:00 PM last night)

10:30 AM MST cycled power to dsm

station now reporting

PS by Gordon: discovered that this site had a typo in the crontab
entry and the net_check.sh and router_check.sh scripts weren't being
run. Fixed the typo.

/var/log/isfs/messages* and a listing of data files (lsu) indicate that
the DSM was up and taking data, but the router/modem couldn't connect.

If router_check.sh script can't ping google or eol, it power cycles
the router&modem which should have fixed the issue.

Gordon, Nov 19 18:30 MST

At 22:39 UTC (15:39 MST) this TRH started generating the hex gibberish:

2010 11 19 22:35:30.1623  0.9771      29 TRH15 14.10 23.92 5413 732\r\n
2010 11 19 22:38:07.1683     157      29 TRH15 14.07 24.13 5410 738\r\n
2010 11 19 22:38:08.1425  0.9742      29 TRH15 14.05 24.19 5408 740\r\n
2010 11 19 22:39:53.3761   105.2      47 TRH\xff\xff1\xff\xff5 \xff1\xff4.\xff\xff\xff03\xff 24.\xff4\xff8 \xff5\xff40\xff\xff6 74\xff\xff8\r\n
2010 11 19 22:47:55.4524   482.1      42 TRH\xff\xff15 13\xff\xff.99 \xff2\xff4.26\xff \xff540\xff\xff\xff2 74\xff2\xff\r\n

I noticed the gravestones in cockpit after I had installed the new nidas code.

Powering it down and up with eio got it working again at 00:39 UTC Nov 20 (17:39 MST)

eio 5 0
eio 5 1

Perhaps we should do the power cycle of the TRH (port 5) every so often (10 minutes) at this site?

Kurt and Ling were at this site today, Kurt was working on the power inside the box, which had the low voltage problem.

I'll wait to see if the problem returns before doing the watchdog power reset on the TRH.

Nov 19 afternoon.

Kurt and Ling visited playa1 and abc2 today.

Per phone calls:

At playa1, tried to improve the cellular connection. Swapped in a new modem (modem-8 as I recall). Fiddled with the antenna. Sometimes it appeared that the signal was better without an antenna, but later the signal went bad and an antenna helped. So left the site with modem-8 and the external antenna connected. This site still has a typical signal of 28%, -93 dBm, and that seems to be the best we can get with the current setup. It is acceptable, but it would be nice to try a higher gain antenna here.

At abc2, investigated why the system voltage is low. Kurt found that the cable to the VIPER (or DSM) did not have a good connection at the panel, which he fixed. During this time the router reverted to factory defaults. Ling re-installed the router configuration. The system voltage recovered to 12.8 V. This is the second or third time we've seen the CTR350s revert to factory defaults, when they've had a power glitch.

Ling added: The cable from the power board to the VIPER is still questionable. The cable's connector was losing which Kurt fixed, but we still get lower battery voltage sometimes. We suggest to replace the cable completely. Gordon, please bring one of the kind the power cable with you and replace it, when you go to Utah after Thanksgiving. If you have question regards the cable,  please talk to Kurt or me.

Found why inadyn had exited on some systems. If the modem is down, a DNS lookup to the router can return an address of 192.168.0.1 for the dyndns.com server. inadyn then gets an incorrect response back from 192.168.0.1 and exits. Modified the code to sleep and continue.

Also, in a similar way, nidas could get a bad address for eol-rt-data. When it sends a UDP packet to the wrong address it gets a "connection refused", but continues to try that same address. New code will do a new DNS lookup.

Installed the new inadyn and nidas on all systems between 17:30 and 18:15 MST Nov 19.

Today's status:

In general, pretty good by now so John and Ling return today.  (Kurt remains alone for a bit longer.)  Kryptons are being recalibrated in Boulder with shorter pathlengths and will be tested through serializers retrieved (today) from Manitou.  They will be installed by Tom and Gordon just prior to operations 29/30 Nov.  Fixes to the SPN1 and TP01 should also be straightforward.  Problems with Tsoils seem to be caused by inadequate conformal coating of the boards.  We'll probably just apply nail polish to fix.

Here are results from a short sample of about 10 pings to each station from Boulder, with a default ping packet of 56 bytes:

.
Station 1 is the obvious outlier, and it shows the lowest signal level as shown inhttps://wiki.ucar.edu/x/A4giB

Note that starting at 00:00 UTC (17:00 MST) every day, the previous days files are downloaded from each station. This process takes about 1/2 on each station (longer on isfs1). During the downloads the ping times will be longer.