Blog

Nov 16, around 15:30 MST

Chris installed the sonic head at playa1

Nov 16, 15:50 MST

Phone conversation with John.

Fixed charging issue, one solar panel wasn't connected.

Powered up soil probes. Soil battery was at 12.6 V. Qsoil didn't report initially, cable wasn't fully plugged in.

Cell signal is about 64% of full signal.

No TRH at this site yet.

Nov 16, 14:30 MST

Per phone conversation with John.

On site Ling, John, Kurt.

Communications to this site went out last night, which appears to be due to the USB modem not being fully plugged into the router. USB light was off on the router. After pushing the modem in, comms were re-established.

Updated coefficients in down-looking pyranometer.

Tsoil died at this site at around the same time as the modem went out last night. The Tsoil probe was removed for repair.

No TRH (or krypton) at this site at this time. Everything else looks OK. Cellular levels look good here: around 60% of full signal.

Nov 16

TRH at 5 dropped out for a little over an hour this morning, from 10:11 to 11:22 MST.
I don't think anyone was at the site.

Here's the raw dump:

data_dump -i 5,50 -A isfs_20101116_160000.dat.bz2 

2010 11 16 17:11:41.2109    0.98      29 TRH16 9.65 46.96 4946 1418\r\n
2010 11 16 17:11:42.1917  0.9808      29 TRH16 9.64 46.80 4945 1413\r\n
2010 11 16 17:11:43.1618    0.97      29 TRH16 9.65 46.71 4946 1410\r\n
2010 11 16 18:22:03.8285    4221      21 Ha2 = -1.382833E-6\r\n
2010 11 16 18:22:03.8585 0.03001      21 Ha3 =  3.354407E-2\r\n
2010 11 16 18:22:03.8895 0.03105      21 Ha4 =  3.666422E-5\r\n
2010 11 16 18:22:04.2285   0.339      29 TRH16 8.22 51.75 4804 1573\r\n
2010 11 16 18:22:05.1985    0.97      29 TRH16 8.22 51.72 4804 1572\r\n
2010 11 16 18:22:06.1794  0.9808      29 TRH16 8.22 51.72 4804 1572\r\n

11:20 am MST, Nov 16

From phone conversation with John and Ling:

On site: John, Kurt, Ling

When they removed the SPN1, then the aux soils started coming in. Apparently a faulty I2C device can hang the mote. Mote ID 3 serves the aux soils and spn1 at this site.

They corrected the coefficients on the the pyranometers. Rsw.in and Rsw.out look good.

prep -D Rsw.in#1,Rsw.out#1 sock:porter
2010 11 16 18:26:12.4286         95       15.6
2010 11 16 18:26:17.4287       94.2       15.4
2010 11 16 18:26:27.4291       92.6       15.1
2010 11 16 18:26:32.4293       91.8       14.9

They forgot to bring the sonic head.

TRH013 is installed.

Shadow band radiometer is now working.

Cell modem signal levels: 28-30%, -93 dB

Here is a revised table, now that some things have been fixed (and others have died):

Sites 2,3 and 4 showed dropouts last night around 09:15 UTC (02:15 MST). Data from 3 and 4 resumed about 1/2 hour later, site 2 is still out.

As of this morning we're still not getting data from 2. A host at the IP address returned for isfs2.dyndns.org responds to pings, but reports "connection refused" when ssh-ing to the special port we use. It probably isn't our system at that address.

From the system log (/var/log/isfs/messages) at site 3, the router_check.sh script detected at 09:32 UTC that it couldn't ping eol or google, and successfully power cycled the router and modem. This happened just once. Things worked like they should!

Site 4 is more complicated. The cellular connection is frequently going up and down - several times an hour, which has been going on for the last serveral days. The router detects the connection problem itself and resets the modem, the router_check.sh script is not powering things down.

I think we should try an external antenna at site 4.

Over the weekend, crontab entries were added to all stations, except 6, to power cycle the base motes on /dev/ttyS6:

From crontab -l:

0 */3 * * * /usr/local/isff/scripts/power_cycle_port 6

This will then happen at 00:00, 03:00, ... UTC.

cron logs to /var/log/isfs/messages when it runs, which can be checked to verify the power cycle
script is being run.

Once we have 2-way comms to site 6 we can update its crontab.

Just spoke with John.  He is pretty sure that the radiometers were inadvertently mounted upside down.  This explains the pyrgeometers reporting reversed and Rsw.out being large.  It doesn't explain Rsw.in being VERY large.  Their schedule doesn't allow him to flip them over today, but it is on the list.

While at the site, he also unplugged the offending TP01.aux -- we'll see if data from the other sensors improve.  (So far, not for Tsoil.4.4cm, at least.)

As of noon Nov 14, all 7 stations are set up and reporting.

Cellular comms are excellent at all sites, except for some weirdness at wslope6.

When wslope6 was setup, ChrisG reported that the LEDs on the router and modem indicated that it was having problem establishing a connection. I believe the LEDs looked similar to what he saw with the spare isfs8 system when powering it up in the base trailer. Putting an antenna in the window solved the problem at the trailer.

From http://www.eol.ucar.edu/deployment/field-deployments/field-projects/pcaps/ISFS/FieldNetwork
the LED on the USB760 modem (a little round LED which is about impossible to see in the sunlight) indicates the following:

The logs at dyndns.com show that wslope6 (isfs6.dyndns.org) registered 17 times with dyndns.org between Nov 12 11:40 and 12:01 MST. Each registration was for a different IP address. The last address that it registered with was 75.220.246.241, which is what dyndns is still reporting on Nov 14.

dyndns can blacklist a host if it is registering too often. I believe they send an email to the addresses in the account profile, which is currently my address, but I have not received any reports.

Logging into eol-rt-data and doing a tcpdump indicates that station 6 is reporting from 75.220.122.209. This address was not listed in the logs at dyndns.com. One cannot ping or ssh to either of the two addresses: 75.220.122.209 or 75.220.246.241.

Note that we are getting UDP data from this site from address 75.220.122.209 but cannot login.

My guess is that the signal is marginal at this site. One could bring a PC with wifi to get on the network at this site and point a browser at 192.168.0.1. After logging into the router's admin account, under Status->Modem (as I remember) one can see the signal level. Or, if someone has a Verizon 3G phone, we could get an idea of the signal level at this site.

Perhaps one of the little 6" whip antennas would help at this site, if placed as high as possible on the mast. If we still believe the signal is low, we may need a higher gain antenna.

Update on station 6 mystery. Sun Nov 14, 8:40 pm
~maclean/tcpdump.sh on eol-rt-data now indicates that station 6 is now reporting from 69.98.142.231. That IP is now pingable, but ssh fails with a timeout. dyndns.com still has the same IP address, 75.220.246.241, registered on Nov 12 12:01 pm

Here is a list of what I know about sensors at each site:

Well...we already know about I2C problems with the TP01 on site5.aux.  I note now that Qsoil is erratic at this site and even Gsoil had a spike yesterday that wasn't seen at any other site.  Tsoils are consistently okay, though.  At the very least, it might be a good idea to reseat these connections in the mote box.

P.S. As of ~1700 on 13 Nov, we're losing Tsoil.4cm as well.  I wonder if gophers are getting to these sensors???  I sure hope not.

I also note that all Qsoils show a small spike when the TP01 is heating.  The amplitude is small enough to be ignored (about 3 A/D counts), but we should be aware of this.

We have found a bug in the humitter code where the output format is incorrect for temperatures between 0 and -1 C.  This causes both T and RH values to be reported as missing.  Examples are:

2010 11 12 12:28:44.0730  0.9708      29 TRH20 0.00 66.96 4004 2164\r\n
2010 11 12 12:28:45.0522  0.9793      30 TRH20- 0.02 66.95 4001 2164\r\n

2010 11 12 13:53:16.4885    0.97      30 TRH20- 0.99 74.28 3906 2430\r\n
2010 11 12 13:53:17.4685    0.98      30 TRH20 -1.01 74.27 3904 2430\r\n

The humitters thus will have to be reprogrammed.  Current thinking is to send all of them back to Boulder for Steve to fix.

This also is an issue for one (TRH11@7m) of the Manitou sensors.

I note that since we are for the first time mounting the barometer outside of the DSM box, Tbaro changes  much faster than before.  In particular, in the past 30minutes this morning, Tbaro has changed at the east sites from being nearly equal to T to being almost 10degrees higher.  I assume this is due to heating by shortwave radiation as the station comes out of the shadow of the mountains. 

The barometer pressure is compensated for changes in its temperature and is still well within its calibrated range, but it is possible that this rate of change could induce temperature gradients within the sensor that makes the Tbaro measurement (and thus any compensation) less valid.  It would be interesting to examine these pressure data in detail during these times to see if an error in pressure can be detected.  (Though it isn't obvious what the reference would be.)

Dynamic DNS

Immediately after the cellular modem connects to the Verizon ISP, the station registers itself with the dynamic name service, dyndns.com.

So, a good remote check on whether the cellular modem and router are working is to login to our account at http://dyndns.com and check the times of the host registrations.

Ask Gordon, Ling or Gary for our account name and password at dyndns.com.

Once you're logged in, click on the My Hosts link. You will see the 7 isfs stations, and two spares, along with the iss systems. The time (MST) that a station last registered with dyndns is shown. Click on Host Update Logs to see a listing of all recent registrations.

ping

Each of the stations should also be reachable via ping:

ping isfs1.dyndns.org