Blog

I have noticed that at site 2 on two separate occasions, when power to DSM is recycled, the trh data output is in hexadecimal until the sensor itself is rebooted.

moved Viper power cable to another power port on the main power board.

the prop vane azimuths were shot using the datascope earlier this week with the following results

site 1 (magnetic declination set to 12.3 degs per NOAA website, and datascope field calibrated by Chris Golubieski)   1.0 degs

site 6    1.7 degs

neither site has had this boom direction entered into the RMY propvane program as yet.

visited site 2 again today from about 1:30 till 3:00 monitoring power (voltage dropped again this morning to about 11.2 v)

swapped out viper power cable.

wiggling the cable brings the voltage back up to 12.7 v, (where it should be), but that's what was happening yesterday as well, and I repaired the original cable at that time. since the voltage dropped again this morning, right about the time the wind started getting brisk, and shaking the tower, i suspect a cold solder joint in the power distribution board. I had an extra cable, so I decided to try that first.

will monitor situation, and if voltage drops again, i will replace either power panel, or more likely, entire DSM so that the offending board/box can be scrutinized in a more friendly environment

Friday, Nov 19th 11:00 AM MST to approx. 2:00 PM

 I cleaned incoming and outgoing radiometer domes - I noticed during site visit to improve cell connection that the domes (mostly the incoming) had a fine coating of soot. This is a particularly windy site, and the Kennecot copper mine smelter is directly across the highway, about 1-2 miles away. The mine regularly sprays water on the substantial slag heaps to keep the dust down, but it looks to be ineffective. This site may need cleaning more often.

while at the site I had a visit from Randy Finch, an engineer from KSL Radio who regularly visits their building / tower which is about 1/4 mile to the east of our ISFS station. He pointed out to me where the Verizon cell towers (2) are located. Neither are visible to me but their locations are to the NW and SW of our tower.

I positioned site 1's  cellular antenna to minimize blockage.

KSL's tower is a AM radio transmitter around 1.5 MHz

They have a secure building with land line internet connection as well as power, and he offered both to us if we so desired.

Contact info:

Randy Finch

KSL radio

801 556-7565 (cell)

801 575-7637 (transmitter building)

Sat Nov 20

visited site 4 (not reporting as of 7:00 PM last night)

10:30 AM MST cycled power to dsm

station now reporting

PS by Gordon: discovered that this site had a typo in the crontab
entry and the net_check.sh and router_check.sh scripts weren't being
run. Fixed the typo.

/var/log/isfs/messages* and a listing of data files (lsu) indicate that
the DSM was up and taking data, but the router/modem couldn't connect.

If router_check.sh script can't ping google or eol, it power cycles
the router&modem which should have fixed the issue.

Gordon, Nov 19 18:30 MST

At 22:39 UTC (15:39 MST) this TRH started generating the hex gibberish:

2010 11 19 22:35:30.1623  0.9771      29 TRH15 14.10 23.92 5413 732\r\n
2010 11 19 22:38:07.1683     157      29 TRH15 14.07 24.13 5410 738\r\n
2010 11 19 22:38:08.1425  0.9742      29 TRH15 14.05 24.19 5408 740\r\n
2010 11 19 22:39:53.3761   105.2      47 TRH\xff\xff1\xff\xff5 \xff1\xff4.\xff\xff\xff03\xff 24.\xff4\xff8 \xff5\xff40\xff\xff6 74\xff\xff8\r\n
2010 11 19 22:47:55.4524   482.1      42 TRH\xff\xff15 13\xff\xff.99 \xff2\xff4.26\xff \xff540\xff\xff\xff2 74\xff2\xff\r\n

I noticed the gravestones in cockpit after I had installed the new nidas code.

Powering it down and up with eio got it working again at 00:39 UTC Nov 20 (17:39 MST)

eio 5 0
eio 5 1

Perhaps we should do the power cycle of the TRH (port 5) every so often (10 minutes) at this site?

Kurt and Ling were at this site today, Kurt was working on the power inside the box, which had the low voltage problem.

I'll wait to see if the problem returns before doing the watchdog power reset on the TRH.

Nov 19 afternoon.

Kurt and Ling visited playa1 and abc2 today.

Per phone calls:

At playa1, tried to improve the cellular connection. Swapped in a new modem (modem-8 as I recall). Fiddled with the antenna. Sometimes it appeared that the signal was better without an antenna, but later the signal went bad and an antenna helped. So left the site with modem-8 and the external antenna connected. This site still has a typical signal of 28%, -93 dBm, and that seems to be the best we can get with the current setup. It is acceptable, but it would be nice to try a higher gain antenna here.

At abc2, investigated why the system voltage is low. Kurt found that the cable to the VIPER (or DSM) did not have a good connection at the panel, which he fixed. During this time the router reverted to factory defaults. Ling re-installed the router configuration. The system voltage recovered to 12.8 V. This is the second or third time we've seen the CTR350s revert to factory defaults, when they've had a power glitch.

Ling added: The cable from the power board to the VIPER is still questionable. The cable's connector was losing which Kurt fixed, but we still get lower battery voltage sometimes. We suggest to replace the cable completely. Gordon, please bring one of the kind the power cable with you and replace it, when you go to Utah after Thanksgiving. If you have question regards the cable,  please talk to Kurt or me.

Found why inadyn had exited on some systems. If the modem is down, a DNS lookup to the router can return an address of 192.168.0.1 for the dyndns.com server. inadyn then gets an incorrect response back from 192.168.0.1 and exits. Modified the code to sleep and continue.

Also, in a similar way, nidas could get a bad address for eol-rt-data. When it sends a UDP packet to the wrong address it gets a "connection refused", but continues to try that same address. New code will do a new DNS lookup.

Installed the new inadyn and nidas on all systems between 17:30 and 18:15 MST Nov 19.

Today's status:

In general, pretty good by now so John and Ling return today.  (Kurt remains alone for a bit longer.)  Kryptons are being recalibrated in Boulder with shorter pathlengths and will be tested through serializers retrieved (today) from Manitou.  They will be installed by Tom and Gordon just prior to operations 29/30 Nov.  Fixes to the SPN1 and TP01 should also be straightforward.  Problems with Tsoils seem to be caused by inadequate conformal coating of the boards.  We'll probably just apply nail polish to fix.

Here are results from a short sample of about 10 pings to each station from Boulder, with a default ping packet of 56 bytes:

.
Station 1 is the obvious outlier, and it shows the lowest signal level as shown inhttps://wiki.ucar.edu/x/A4giB

Note that starting at 00:00 UTC (17:00 MST) every day, the previous days files are downloaded from each station. This process takes about 1/2 on each station (longer on isfs1). During the downloads the ping times will be longer.

10:00 John just called from the site.  He tested the ETI gauge with about 1/2 liter of water and saw 26 tips.  27 tips would have been exact.  Thus, this gauge appears to be working.

Field crew site names:

1 - none

2 - unprintable

3 - happy gravediggers

4 - dog poo

5 - don't touch my dog

6 - cemetary

7 - sex on a stone

Per phone conversation, Dec 17 15:48 MST

The modem at this site was doing many reconnects and registrations with dyndns.org. Ling taped the modem into the router, and saw a good signal level.

Tested the rain gauge, ETI8. Saw no tips from "data_dump -i 4,80 -A" after pouring 1 liter of water in the gauge. John connected a pc to debug, and power cycled the gauge. After the power cycle it registered tips. Poured 1/2 liter (from graduated water bottle) at 22:51:52, and saw a total of 27 tips, as follows, after a delay of 1 minute 35 seconds:

2010 11 17 22:53:12.1234    5.01       9 ETI8 0\r\n
2010 11 17 22:53:17.1336    5.01       9 ETI8 0\r\n
2010 11 17 22:53:22.1438    5.01       9 ETI8 0\r\n
2010 11 17 22:53:27.1535    5.01       9 ETI8 2\r\n
2010 11 17 22:53:32.1734    5.02       9 ETI8 7\r\n
2010 11 17 22:53:37.1836    5.01       9 ETI8 7\r\n
2010 11 17 22:53:42.1938    5.01       9 ETI8 4\r\n
2010 11 17 22:53:47.2039    5.01       9 ETI8 0\r\n
2010 11 17 22:53:52.2149   5.011       9 ETI8 0\r\n
2010 11 17 22:53:57.2251    5.01       9 ETI8 0\r\n
2010 11 17 22:54:02.2445   5.019       9 ETI8 0\r\n
2010 11 17 22:54:07.2547    5.01       9 ETI8 0\r\n
2010 11 17 22:54:12.2649    5.01       9 ETI8 0\r\n
2010 11 17 22:54:17.2751    5.01       9 ETI8 0\r\n
2010 11 17 22:54:22.2853    5.01       9 ETI8 0\r\n
2010 11 17 22:54:27.2954    5.01       9 ETI8 2\r\n
2010 11 17 22:54:32.3064   5.011       9 ETI8 5\r\n
2010 11 17 22:54:37.3262    5.02       9 ETI8 0\r\n
2010 11 17 22:54:42.3360    5.01       9 ETI8 0\r\n
2010 11 17 22:54:47.3462    5.01       9 ETI8 0\r\n

PS: Gordon, Nov 18, 9:30 MST. The frequent reconnects from this site have stopped. At this time the last dyndns registration from site 4 was at 4pm yesterday. Seems that securing the modem with tape solved the problem.

Per phone conversation, Dec 17, pm.

It appears that the CTR350 router had somehow lost its configuration and had reverted to factory defaults. Ling re-configured the CTR350 router at this site, with the configuration file at http://svn.eol.ucar.edu/svn/projects/PCAPS/ISFF/config/ctr350.gws. Ling sees good cellular signal level at this site.

John poured roughly 1 liter of water in the rain gauge at about 21:07:37 UTC. After roughly two minutes it reported tips. Counted a total of 66 tips.  [p.s. 0.01"/tip in a 12" gauge should produce 54 tips for 1 liter.  Thus, this works out to 1.22 liters.  John thinks that he "double poured" when he initially didn't see tips, so 1.22 liters is possible...SPO]

Today's status: