October 31, TWH and SRS:
The 10m kh2o on the main tower was outputting zero voltage. We replaced it (1390) with 1393. Note that we removed and lowered the boom to accomplish this swap, so the 10m sonic level may have changed. The kh2o voltage was 0.1 V, so Steve cleaned it and the voltage went up to 1.2 V. Steve also cleaned the 5m kh2o (1389) and the two Licors while we were at it (~15:30).
October 31, TWH and SRS
Cockpit showed station C20 as RIP. Steve and I found the DSM running but neither Gordon or us could ping the etherant after we unplugged it from the DSM and plugged in back in again. We discovered that the cable had been chewed near its low point between the DSM and the mast. We could not find a spare, so Steve will repair the broken leads.
16:30: Steve found two places where the cable was chewed. He clipped out this section of cable and spliced the two clean ends together. We installed the repaired cable and staked it off the ground. Gordon was able to ping and log into C20, but was puzzled that he could not ping the etherant.
Nov 2, Gordon
Discovered that "flux" was running the dhcpd service. Sheesh... I thought I had turned it off, but it must have been re-enabled as part of the system rebuild.
dhcpd on flux was giving address 192.168.0.163 to 00:20:f6:05:1e:d5, which is pingable. This explains why it didn't respond to 192.168.0.137.
It was also giving out these addresses for the OSU webcam and Sodar:
Nov 2 01:01:17 flux dhcpd: DHCPOFFER on 192.168.0.160 to 00:30:f4:d0:a1:ed via p5p1 Nov 2 10:01:42 flux dhcpd: DHCPOFFER on 192.168.0.161 to 00:50:b6:0b:f5:19 (sodar-47) via p5p1 Nov 2 10:01:43 flux dhcpd: DHCPOFFER on 192.168.0.162 to 00:50:b6:0b:f5:19 (sodar-47) via p5p1
dhcpd has been disabled on flux, and so these systems should receive their old addresses on the next renew.
Weather
Clear skies and light winds. Temperatures are 5C.
Status
Qsoilc went down again. Steve O. believes it is a temperature related problem. Vdsm6 still down.
Tom is coming up today so we should get the 10m krypton repaired. It appears the 10m sonic on the
main tower knew we were going to pay a visit because it has been acting up throughout the night. The
daig has favored a value of 15 verses 0. Since this based on the background it is hard to say if it was continuous through
the night or just a short occurrence. According to Gordon's notes this means a skipped sampled.
Changed the pocketec this morning.
Update.
At about 7:45am the 10m sonic on main started behaving. Maybe some spider was building a web.
Will investigate when we replace the krpyton.
Here is an update from Gordon on the 10m sonic activity: There are a bunch of 4's and 8's around 00:24 UTC (18:24 MDT). Then it
was OK, until 13:14 UTC (07:14) when it started having lots of problems.
(bird poop?)
Afternoon: Site C had gone down. Gordon was able to ping the etherant but no further.
17:30:
kh2o.10m ok, station C20 ok.
Vdsm is RIP for stations 1, 2, 6..
Sonic at 17.1m has been intermittently setting diagabits = 4.
Qsoil.c is okay at the moment.
Weather
Clear skies this morning with light winds. Higher winds last night.
Status
Some activity last night on the sonic diag flags on site 17 and 19. Vdsm6 and krypton@10m down.
Update
Weather
Partly cloudy, light winds last night, temperatures in the single digits.
Status
Overall things looking very good. The Vdsm6 will be down until we replace the mote. Qsoilc is still intermittent. The 10m
krypton will require a tower climb. This will take place possible tomorrow when another person is up here.
Update
Qsoilc: tried moving the DIN connection of the bad Qsoil but it did not make a difference.
Played around with the cactus soil sensors. Can not say for sure what got the sensor running again. Either disconnecting cables
or pushing the scanII2C button. It has been running for the last few hours.
Checked out the tower guywire tensions this afternoon. Inner giuys are around 250. All others above 350.
Information entered in the Main Tower Tension entry.
Weather
Partly cloudy, calm winds through the night, < 5 m/s,. Temperatures stayed around 0c.
Status
Qsoilc down again, Krypton @ 10m down, Vdsm6 down.
Looking at QC plots, the 2m TRH @9 has been consistently high compared to other stations. This agrees with the sonic temp, Tc, at 9
so I assume the TRH is good.
Update
Added conductive grease to cactus soil connectors. Qsoilc came back for a short period. The problem may be in the din connector on
the mote.
Removed Vdsm mote at 6 to test in the base. The unit is not putting out any messages.
Weather:
A little bit of cloud cover, calm winds through out the night. Temps around 3 to 5 C.
Status:
All systems looked good throughout the night except for 10m krypton which is still out. I will try to get up the tower if the RAL person is still here.
The power status at station 6 down. Gordon did a power cycle earlier today which brought it back but it has gone down again, will investigate.
Qsoilc is down.
UPDATE:
Went up the tower to investigate the bad krypton (RAL person still here). Decided to disconnect/connect all cables to krypton. Did not help.
Went to station 6 to look at Vdsm issue. Power cable to mote box was a little loose, reseated and recycled power, no improvement.
Cleaned radiometer domes with methanol. Reseated connectors at soil cactus site. Qsoilc came back.
Gordon Oct 27
The power mote at station 6 has been out since late on Oct 24. It is only providing Vdsm, so it isn't a big priority. When powered down and back up with
eio 7 0 eio 7 1
It reports this to rserial, but nothing else:
ID37:\0x01\0x02Rev1.11 dr=5 EC 100mSec Bres=39999983 Mode=Timed \0x12\0x03\0x04\r ID37:\0x01\0x02Rev1.11 dr=5 EC 100mSec Bres=39999983 Mode=Timed \0x12\0x03\0x04\r
This may be obvious warning, but we all need to pay attention when walking in the field after a snow. All prairie dog holes are covered.
PLEASE BE CAREFUL.
Weather:
Got about 3.0" of snow making for a beautiful day. Not much of a wind and the house stayed warm.
Status:
All systems look great. Fans seemed to take this snow a lot better than the first round. All optical sensors seem to have low signals so I will take a gander at their condition this morning.
Morning Update: Cleaned all Kryptons, Licors and radiometers. After cleaning 10m Krypton, no change to signal level. I thought I cleared off all snow and cleaning water...we may want to keep an eye on this sensor. The reason I could do the Krytpon cleaning is RAL has a snow depth scientist doing measurements today, so he was around to dial 9-1...
I also knocked off snow on some of the solar panels. I will not do that for the rest of the sites. The sun is trying to pop it's head through, so I will let Mother Nature do her thing.
ToDo:
Clean all optic sensors (Licor, Krypton and radiation). Krypton cleaning will depend on a sighter back at the compound. If no sighter, then no cleaning Kryptons.
I will also switch out PockeTecs before leaving.
Gordon Oct 25
Cockpit data was not coming in.
Traced this down a system clock problem on flux. Its clock gradually lost time, and then the dsm_server process started throwing away samples, with this message in /var/log/isfs/isfs.log:
Oct 25 14:54:54 flux dsm_server[30058]: WARNING|sample with timetag in future by 2.305084 secs. time: 2012 Oct 25 20:54:56.861 id=10,20 total future samples=39039001
The problem was in /etc/chrony.conf on flux and gully. The chrony processes were configured to use keys, which were different, and so flux was not able to get time information from gully. Commented out
# keyfile /etc/chrony.keys
statement in both, and restarted chronyd
systemctl restart chronyd.service
chronyd on flux is configured to get time from gully, via "server 192.168.0.12" directive in /etc/chrony.conf on flux.
The server directives in /etc/chrony.conf on gully now look like so:
# server 0.fedora.pool.ntp.org iburst # server 1.fedora.pool.ntp.org iburst # server 2.fedora.pool.ntp.org iburst # server 3.fedora.pool.ntp.org iburst server ntp.colostate.edu server c20 server m21 server m22
chrony was never able to connect any of the fedora.pool.ntp.org servers. The chronyc sources command always returned lines like this for the pool servers:
MS Name/IP address Stratum Poll LastRx Last sample ============================================================================ ^? 64.73.32.134 0 6 10y +0ns[ +0ns] +/- 0ns ^? ns1.your-site.com 0 6 10y +0ns[ +0ns] +/- 0ns ^? 64.73.32.135 0 6 10y +0ns[ +0ns] +/- 0ns ^? cheezum.mattnordhoff.net 0 6 10y +0ns[ +0ns] +/- 0ns
The traffic is probably blocked by a firewall somewhere.
In order to have a reference check for our DSMs, I added ntp.colostate.edu as a chrony server for gully, which works, which means our router is not blocking the traffic.
chronyc sources on gully with the above server configuration looks like so, which indicates that our DSMs (with PPS from a GPS) have better clocks than ntp.colostate.edu:
# chronyc sources 210 Number of sources = 4 MS Name/IP address Stratum Poll LastRx Last sample ============================================================================ ^+ yuma.acns.colostate.edu 2 6 5 +75ms[ +75ms] +/- 173ms ^+ C20 3 6 5 -1281us[-1287us] +/- 3162us ^+ M21 3 6 5 +470us[ +464us] +/- 2506us ^* Mu22 3 6 5 -30us[ -36us] +/- 1799us
chronyd
Weather:
Woke up this morning with about a quarter - half inch of snow. Seemed foggy down in lower areas. Sunshine is now clearing the solar panels.
Status:
Cockpit is getting a lot of dropped packets so it's hard to read with very little to look at this morning. Cockpit plots look better and better as the sun is shining. TRH as A17 is down. The Vdsm at that tower is also down. I will head down to that station and see if the signal cables have pulled. This has happened on this station before. Most Vdsm readings on cockpit are showing RIP: I'm guessing it's from the lack of packets coming through AND the Vdsm is sent every minute so there is more of a chance of it not seeing in cockpit.
MORNING UPDATE: Station 17 was not reporting TRH or Vdsm this morning, but now it is coming in. I rebooted and 'gooped' the DIN connectors. Cables were tight so it may have been a connector coming loose in mote.
-Aph3 TRH fan was fixed. I just rebooted it from gully. All looks good.
AFTERNOON UPDATE: All stations looking good from 'gully' but cockpit is making stations looking bad. Gordon is fixing cockpit so it doesn't throw out data points from the future (a clock error).
- All TRHs are fixed now. We have a total five BUILT TRHs in the base trailer and an additional four probes in a bag located in the dorm. Please noticed if a you have to change out a MoteTRH and it's an odd numbered probe, YOU NEED TO REPLACE WITH AN ODD NUMBER PROBE. There is two in the housings in the Base Trailer.
ToDos:
I will walk to A17 to check on 2mTRH. I will head to base to fix the TRHs in there so we have spares.