Blog from November, 2010

10:00 John just called from the site.  He tested the ETI gauge with about 1/2 liter of water and saw 26 tips.  27 tips would have been exact.  Thus, this gauge appears to be working.

Field crew site names:

1 - none

2 - unprintable

3 - happy gravediggers

4 - dog poo

5 - don't touch my dog

6 - cemetary

7 - sex on a stone

Per phone conversation, Dec 17 15:48 MST

The modem at this site was doing many reconnects and registrations with dyndns.org. Ling taped the modem into the router, and saw a good signal level.

Tested the rain gauge, ETI8. Saw no tips from "data_dump -i 4,80 -A" after pouring 1 liter of water in the gauge. John connected a pc to debug, and power cycled the gauge. After the power cycle it registered tips. Poured 1/2 liter (from graduated water bottle) at 22:51:52, and saw a total of 27 tips, as follows, after a delay of 1 minute 35 seconds:

2010 11 17 22:53:12.1234    5.01       9 ETI8 0\r\n
2010 11 17 22:53:17.1336    5.01       9 ETI8 0\r\n
2010 11 17 22:53:22.1438    5.01       9 ETI8 0\r\n
2010 11 17 22:53:27.1535    5.01       9 ETI8 2\r\n
2010 11 17 22:53:32.1734    5.02       9 ETI8 7\r\n
2010 11 17 22:53:37.1836    5.01       9 ETI8 7\r\n
2010 11 17 22:53:42.1938    5.01       9 ETI8 4\r\n
2010 11 17 22:53:47.2039    5.01       9 ETI8 0\r\n
2010 11 17 22:53:52.2149   5.011       9 ETI8 0\r\n
2010 11 17 22:53:57.2251    5.01       9 ETI8 0\r\n
2010 11 17 22:54:02.2445   5.019       9 ETI8 0\r\n
2010 11 17 22:54:07.2547    5.01       9 ETI8 0\r\n
2010 11 17 22:54:12.2649    5.01       9 ETI8 0\r\n
2010 11 17 22:54:17.2751    5.01       9 ETI8 0\r\n
2010 11 17 22:54:22.2853    5.01       9 ETI8 0\r\n
2010 11 17 22:54:27.2954    5.01       9 ETI8 2\r\n
2010 11 17 22:54:32.3064   5.011       9 ETI8 5\r\n
2010 11 17 22:54:37.3262    5.02       9 ETI8 0\r\n
2010 11 17 22:54:42.3360    5.01       9 ETI8 0\r\n
2010 11 17 22:54:47.3462    5.01       9 ETI8 0\r\n

PS: Gordon, Nov 18, 9:30 MST. The frequent reconnects from this site have stopped. At this time the last dyndns registration from site 4 was at 4pm yesterday. Seems that securing the modem with tape solved the problem.

Per phone conversation, Dec 17, pm.

It appears that the CTR350 router had somehow lost its configuration and had reverted to factory defaults. Ling re-configured the CTR350 router at this site, with the configuration file at http://svn.eol.ucar.edu/svn/projects/PCAPS/ISFF/config/ctr350.gws. Ling sees good cellular signal level at this site.

John poured roughly 1 liter of water in the rain gauge at about 21:07:37 UTC. After roughly two minutes it reported tips. Counted a total of 66 tips.  [p.s. 0.01"/tip in a 12" gauge should produce 54 tips for 1 liter.  Thus, this works out to 1.22 liters.  John thinks that he "double poured" when he initially didn't see tips, so 1.22 liters is possible...SPO]

Today's status:

Nov 16, around 15:30 MST

Chris installed the sonic head at playa1

Nov 16, 15:50 MST

Phone conversation with John.

Fixed charging issue, one solar panel wasn't connected.

Powered up soil probes. Soil battery was at 12.6 V. Qsoil didn't report initially, cable wasn't fully plugged in.

Cell signal is about 64% of full signal.

No TRH at this site yet.

Nov 16, 14:30 MST

Per phone conversation with John.

On site Ling, John, Kurt.

Communications to this site went out last night, which appears to be due to the USB modem not being fully plugged into the router. USB light was off on the router. After pushing the modem in, comms were re-established.

Updated coefficients in down-looking pyranometer.

Tsoil died at this site at around the same time as the modem went out last night. The Tsoil probe was removed for repair.

No TRH (or krypton) at this site at this time. Everything else looks OK. Cellular levels look good here: around 60% of full signal.

Nov 16

TRH at 5 dropped out for a little over an hour this morning, from 10:11 to 11:22 MST.
I don't think anyone was at the site.

Here's the raw dump:

data_dump -i 5,50 -A isfs_20101116_160000.dat.bz2 

2010 11 16 17:11:41.2109    0.98      29 TRH16 9.65 46.96 4946 1418\r\n
2010 11 16 17:11:42.1917  0.9808      29 TRH16 9.64 46.80 4945 1413\r\n
2010 11 16 17:11:43.1618    0.97      29 TRH16 9.65 46.71 4946 1410\r\n
2010 11 16 18:22:03.8285    4221      21 Ha2 = -1.382833E-6\r\n
2010 11 16 18:22:03.8585 0.03001      21 Ha3 =  3.354407E-2\r\n
2010 11 16 18:22:03.8895 0.03105      21 Ha4 =  3.666422E-5\r\n
2010 11 16 18:22:04.2285   0.339      29 TRH16 8.22 51.75 4804 1573\r\n
2010 11 16 18:22:05.1985    0.97      29 TRH16 8.22 51.72 4804 1572\r\n
2010 11 16 18:22:06.1794  0.9808      29 TRH16 8.22 51.72 4804 1572\r\n

11:20 am MST, Nov 16

From phone conversation with John and Ling:

On site: John, Kurt, Ling

When they removed the SPN1, then the aux soils started coming in. Apparently a faulty I2C device can hang the mote. Mote ID 3 serves the aux soils and spn1 at this site.

They corrected the coefficients on the the pyranometers. Rsw.in and Rsw.out look good.

prep -D Rsw.in#1,Rsw.out#1 sock:porter
2010 11 16 18:26:12.4286         95       15.6
2010 11 16 18:26:17.4287       94.2       15.4
2010 11 16 18:26:27.4291       92.6       15.1
2010 11 16 18:26:32.4293       91.8       14.9

They forgot to bring the sonic head.

TRH013 is installed.

Shadow band radiometer is now working.

Cell modem signal levels: 28-30%, -93 dB

Here is a revised table, now that some things have been fixed (and others have died):

Sites 2,3 and 4 showed dropouts last night around 09:15 UTC (02:15 MST). Data from 3 and 4 resumed about 1/2 hour later, site 2 is still out.

As of this morning we're still not getting data from 2. A host at the IP address returned for isfs2.dyndns.org responds to pings, but reports "connection refused" when ssh-ing to the special port we use. It probably isn't our system at that address.

From the system log (/var/log/isfs/messages) at site 3, the router_check.sh script detected at 09:32 UTC that it couldn't ping eol or google, and successfully power cycled the router and modem. This happened just once. Things worked like they should!

Site 4 is more complicated. The cellular connection is frequently going up and down - several times an hour, which has been going on for the last serveral days. The router detects the connection problem itself and resets the modem, the router_check.sh script is not powering things down.

I think we should try an external antenna at site 4.

Over the weekend, crontab entries were added to all stations, except 6, to power cycle the base motes on /dev/ttyS6:

From crontab -l:

0 */3 * * * /usr/local/isff/scripts/power_cycle_port 6

This will then happen at 00:00, 03:00, ... UTC.

cron logs to /var/log/isfs/messages when it runs, which can be checked to verify the power cycle
script is being run.

Once we have 2-way comms to site 6 we can update its crontab.

Just spoke with John.  He is pretty sure that the radiometers were inadvertently mounted upside down.  This explains the pyrgeometers reporting reversed and Rsw.out being large.  It doesn't explain Rsw.in being VERY large.  Their schedule doesn't allow him to flip them over today, but it is on the list.

While at the site, he also unplugged the offending TP01.aux -- we'll see if data from the other sensors improve.  (So far, not for Tsoil.4.4cm, at least.)

As of noon Nov 14, all 7 stations are set up and reporting.

Cellular comms are excellent at all sites, except for some weirdness at wslope6.

When wslope6 was setup, ChrisG reported that the LEDs on the router and modem indicated that it was having problem establishing a connection. I believe the LEDs looked similar to what he saw with the spare isfs8 system when powering it up in the base trailer. Putting an antenna in the window solved the problem at the trailer.

From http://www.eol.ucar.edu/deployment/field-deployments/field-projects/pcaps/ISFS/FieldNetwork
the LED on the USB760 modem (a little round LED which is about impossible to see in the sunlight) indicates the following:

The logs at dyndns.com show that wslope6 (isfs6.dyndns.org) registered 17 times with dyndns.org between Nov 12 11:40 and 12:01 MST. Each registration was for a different IP address. The last address that it registered with was 75.220.246.241, which is what dyndns is still reporting on Nov 14.

dyndns can blacklist a host if it is registering too often. I believe they send an email to the addresses in the account profile, which is currently my address, but I have not received any reports.

Logging into eol-rt-data and doing a tcpdump indicates that station 6 is reporting from 75.220.122.209. This address was not listed in the logs at dyndns.com. One cannot ping or ssh to either of the two addresses: 75.220.122.209 or 75.220.246.241.

Note that we are getting UDP data from this site from address 75.220.122.209 but cannot login.

My guess is that the signal is marginal at this site. One could bring a PC with wifi to get on the network at this site and point a browser at 192.168.0.1. After logging into the router's admin account, under Status->Modem (as I remember) one can see the signal level. Or, if someone has a Verizon 3G phone, we could get an idea of the signal level at this site.

Perhaps one of the little 6" whip antennas would help at this site, if placed as high as possible on the mast. If we still believe the signal is low, we may need a higher gain antenna.

Update on station 6 mystery. Sun Nov 14, 8:40 pm
~maclean/tcpdump.sh on eol-rt-data now indicates that station 6 is now reporting from 69.98.142.231. That IP is now pingable, but ssh fails with a timeout. dyndns.com still has the same IP address, 75.220.246.241, registered on Nov 12 12:01 pm