Weather
Partly cloudy, calm winds through the night, < 5 m/s,. Temperatures stayed around 0c.
Status
Qsoilc down again, Krypton @ 10m down, Vdsm6 down.
Looking at QC plots, the 2m TRH @9 has been consistently high compared to other stations. This agrees with the sonic temp, Tc, at 9
so I assume the TRH is good.
Update
Added conductive grease to cactus soil connectors. Qsoilc came back for a short period. The problem may be in the din connector on
the mote.
Removed Vdsm mote at 6 to test in the base. The unit is not putting out any messages.
Weather:
A little bit of cloud cover, calm winds through out the night. Temps around 3 to 5 C.
Status:
All systems looked good throughout the night except for 10m krypton which is still out. I will try to get up the tower if the RAL person is still here.
The power status at station 6 down. Gordon did a power cycle earlier today which brought it back but it has gone down again, will investigate.
Qsoilc is down.
UPDATE:
Went up the tower to investigate the bad krypton (RAL person still here). Decided to disconnect/connect all cables to krypton. Did not help.
Went to station 6 to look at Vdsm issue. Power cable to mote box was a little loose, reseated and recycled power, no improvement.
Cleaned radiometer domes with methanol. Reseated connectors at soil cactus site. Qsoilc came back.
Gordon Oct 27
The power mote at station 6 has been out since late on Oct 24. It is only providing Vdsm, so it isn't a big priority. When powered down and back up with
eio 7 0 eio 7 1
It reports this to rserial, but nothing else:
ID37:\0x01\0x02Rev1.11 dr=5 EC 100mSec Bres=39999983 Mode=Timed \0x12\0x03\0x04\r ID37:\0x01\0x02Rev1.11 dr=5 EC 100mSec Bres=39999983 Mode=Timed \0x12\0x03\0x04\r
This may be obvious warning, but we all need to pay attention when walking in the field after a snow. All prairie dog holes are covered.
PLEASE BE CAREFUL.
Weather:
Got about 3.0" of snow making for a beautiful day. Not much of a wind and the house stayed warm.
Status:
All systems look great. Fans seemed to take this snow a lot better than the first round. All optical sensors seem to have low signals so I will take a gander at their condition this morning.
Morning Update: Cleaned all Kryptons, Licors and radiometers. After cleaning 10m Krypton, no change to signal level. I thought I cleared off all snow and cleaning water...we may want to keep an eye on this sensor. The reason I could do the Krytpon cleaning is RAL has a snow depth scientist doing measurements today, so he was around to dial 9-1...
I also knocked off snow on some of the solar panels. I will not do that for the rest of the sites. The sun is trying to pop it's head through, so I will let Mother Nature do her thing.
ToDo:
Clean all optic sensors (Licor, Krypton and radiation). Krypton cleaning will depend on a sighter back at the compound. If no sighter, then no cleaning Kryptons.
I will also switch out PockeTecs before leaving.
Gordon Oct 25
Cockpit data was not coming in.
Traced this down a system clock problem on flux. Its clock gradually lost time, and then the dsm_server process started throwing away samples, with this message in /var/log/isfs/isfs.log:
Oct 25 14:54:54 flux dsm_server[30058]: WARNING|sample with timetag in future by 2.305084 secs. time: 2012 Oct 25 20:54:56.861 id=10,20 total future samples=39039001
The problem was in /etc/chrony.conf on flux and gully. The chrony processes were configured to use keys, which were different, and so flux was not able to get time information from gully. Commented out
# keyfile /etc/chrony.keys
statement in both, and restarted chronyd
systemctl restart chronyd.service
chronyd on flux is configured to get time from gully, via "server 192.168.0.12" directive in /etc/chrony.conf on flux.
The server directives in /etc/chrony.conf on gully now look like so:
# server 0.fedora.pool.ntp.org iburst # server 1.fedora.pool.ntp.org iburst # server 2.fedora.pool.ntp.org iburst # server 3.fedora.pool.ntp.org iburst server ntp.colostate.edu server c20 server m21 server m22
chrony was never able to connect any of the fedora.pool.ntp.org servers. The chronyc sources command always returned lines like this for the pool servers:
MS Name/IP address Stratum Poll LastRx Last sample ============================================================================ ^? 64.73.32.134 0 6 10y +0ns[ +0ns] +/- 0ns ^? ns1.your-site.com 0 6 10y +0ns[ +0ns] +/- 0ns ^? 64.73.32.135 0 6 10y +0ns[ +0ns] +/- 0ns ^? cheezum.mattnordhoff.net 0 6 10y +0ns[ +0ns] +/- 0ns
The traffic is probably blocked by a firewall somewhere.
In order to have a reference check for our DSMs, I added ntp.colostate.edu as a chrony server for gully, which works, which means our router is not blocking the traffic.
chronyc sources on gully with the above server configuration looks like so, which indicates that our DSMs (with PPS from a GPS) have better clocks than ntp.colostate.edu:
# chronyc sources 210 Number of sources = 4 MS Name/IP address Stratum Poll LastRx Last sample ============================================================================ ^+ yuma.acns.colostate.edu 2 6 5 +75ms[ +75ms] +/- 173ms ^+ C20 3 6 5 -1281us[-1287us] +/- 3162us ^+ M21 3 6 5 +470us[ +464us] +/- 2506us ^* Mu22 3 6 5 -30us[ -36us] +/- 1799us
chronyd
Weather:
Woke up this morning with about a quarter - half inch of snow. Seemed foggy down in lower areas. Sunshine is now clearing the solar panels.
Status:
Cockpit is getting a lot of dropped packets so it's hard to read with very little to look at this morning. Cockpit plots look better and better as the sun is shining. TRH as A17 is down. The Vdsm at that tower is also down. I will head down to that station and see if the signal cables have pulled. This has happened on this station before. Most Vdsm readings on cockpit are showing RIP: I'm guessing it's from the lack of packets coming through AND the Vdsm is sent every minute so there is more of a chance of it not seeing in cockpit.
MORNING UPDATE: Station 17 was not reporting TRH or Vdsm this morning, but now it is coming in. I rebooted and 'gooped' the DIN connectors. Cables were tight so it may have been a connector coming loose in mote.
-Aph3 TRH fan was fixed. I just rebooted it from gully. All looks good.
AFTERNOON UPDATE: All stations looking good from 'gully' but cockpit is making stations looking bad. Gordon is fixing cockpit so it doesn't throw out data points from the future (a clock error).
- All TRHs are fixed now. We have a total five BUILT TRHs in the base trailer and an additional four probes in a bag located in the dorm. Please noticed if a you have to change out a MoteTRH and it's an odd numbered probe, YOU NEED TO REPLACE WITH AN ODD NUMBER PROBE. There is two in the housings in the Base Trailer.
ToDos:
I will walk to A17 to check on 2mTRH. I will head to base to fix the TRHs in there so we have spares.
Took Amperage readings for the East transformer. (readings are rounded up)
C Tower - 0.50A
OSU SODAR - 2.00A
NCAR SODAR - 2.50A
I did not take readings on West transformer due to not easy access to power lines. I may try after the storm (worried about not being able to re-cover good enough).
Weather:
Cloudy and cold today. Very slight winds. Rain came in later in the day and noticed station 6 sonic become unhappy, the first of all the stations now.
Status & Ops:
All stations reporting. Gully is saving data. Cockpit was updated and all plots are working. If anyone is interested in my set-up you can load ChrisCockpit. Be aware that it is Halloween colors. Don't be scared.
To Dos:
Yet some more...clean the base trailer. Will take Amperage readings for all branches of AC power. This is a task Kurt wanted me to do.
I have parked the four wheeler under the base trailer for shelter from rain and snow. I organized some of Cristoph's PVC and things under the front part of the trailer so I could park and lock the ATV up. This may not be a permanent spot for parking the ATV but for now it will work.
October 23, 2012
Afternoon-ish.
I tried to tighten the bolt to the head of the sonic at 10m on Main Tower, but it was snug. I'm presuming this is something similar to A17? If this does not help then I will put on ToDo List that it needs replacing.
Gordon Oct 23
I believe the issue Chris saw this morning on 6 is similar to what I've seen from time to time on other systems. When I've logged into a system and it is slooooow, I've run top and see that ntpd is taking 100% of the cpu. I believe that is because the tee_tty process has died. This message was in the system log for 6:
Oct 23 07:21:35 Ah6 tee_tty: 2012-10-23,07:21:35|NOTICE|received signal Interrupt(2), si_signo=2, si_errno=0, si_code=128
This happens about once every week on 1 or 2 of the 22 systems, so it is hard to diagnose.
tee_tty reads from the physical serial port that the GPS is connected to and sends that data to pseudo-terminals, one read by ntpd and one read by the dsm process. It appears that if tee_tty dies, then ntpd doesn't correctly detect the error on the input port and then likely attempts reads at a high rate.
Previously I tried to change ntpd so that it might catch the error and exit without hanging, without success.
Now I might know why the tee_tty is being sent the SIGINT signal. I turns out the serial port is opened in "cooked" mode, which means that if it somehow receives a ctrl-C on that port then the process is sent a SIGINT. I guess a ctrl-C could also be received on any pseudo-terminal that is opened for reading. A BREAK condition can also result in a SIGINT, but IGNBRK is set on the serial port, and I don't believe BREAKs can be sent over pseudo-terminals.
Today around 3 pm I logged into all the stations and updated the file /etc/gps.conf to change the "c" in GPS_TEE_OPTS to "r", so that the serial port is opened in "raw" mode:
GPS_TEE_OPTS="4800n81lnrxx -p 60 -l pps"
Next I'm rebuilding tee_tty so that if the real serial port is opened in raw mode, the pseudo-terminals are also opened in raw mode. I don't think the ctrl-C could be coming from the dsm process. rserial traps the ctrl-C and terminates.
As of 4:33 pm, the tee_tty app has been updated on all DSMs, and their tee_tty and ntpd processes restarted.
Weather:
Nice crisp and cool morning. Temps hovering around 8C with humidities in the upper 60%.
Status:
All stations are communicating via bluetooth and wifi.
Station 6 down on arrival by Cockpit. Had trouble logging into the dsm but it finally logged me in. I did a 'adn' then 'aup' to see if I could get the system talking again. Extremely slloooooowwwwww communications with Ah6. The 'aup' did not work. I do see sensors alive in Minicom though. Giving Ah6 a reboot to see if things begin to work.
I did see if the head of the M21, 10m sonic was loose but it was not.
Gave a tour to EOL Admins today.
I do want to add that the base trailer has been neglected this project and I am very dissapointed that it got to this condition. We all need to do a better job in getting organized and vacuuming the carpet. This will make the trailer last a lot longer and make attitudes a lot better.
Sensors and Issues:
All sensors look good but Ah6 due to no data coming down the pipeline.
To Dos:
Ah6 needs help. Fixed trh's in base. Try to figure out what probes that are found laying around that are good. Organize base for show-and-tell today and tomorrow.
Weather:
Light winds during the night. A few clouds this morning with calm conditions. High humidity
Status:
TRH at 2m on A17 went down last night. Used TIO command to get it going again.
The cockpit for Lambdasoil shows bad data however cockpit A11rs shows good data. A data_dump confirms the data is coming in. Killed lambdasoil cockpit and started another one. Data coming in.
Also noticed A18 0.5m TRH went out. We tried to use the DIO lines but nothing happened. Went out to tower and rebooted mote. Nothing. Noticed the cable to the TRH was getting pulled out. Fixed tightness of cable and all good.
To Do:
Finish potting TRH units (Done in the AM). Shoot boom angles and clean Kryptons (Done in the PM).
WEATHER & OPS
On arrival weather was sunny and comfortable temperatures(~15C). Very calm winds (compared to last week) and traffic on Hwy 85 light. Can hear prairie dogs singing to the sodars.
Steve S. and Chris G. have taken over for ops these next couple of weeks. We have gotten the new Flux system and plugged everything in with nothing happening. We found out that we need to shut off wifi and tell it to talk ethernet all the time and it all worked. We brought the theodolites down to see if the Main Tower is plumb from tensioning the guy-wires in the winds last week. Half a tube to the West and eighth of a tube to the South. So it wasn't too far off, but we did do some tweeking and it is now a eighth of a tube out to the South. New tensions were took and logged.
SENSOR STATUS AND ISSUES
No issues were found with any of the towers or sensors.
TO DO
Will repair TRHs tomorrow and finishing potting TRH wagon wheels. While doing the potting we will both shoot boom angles. Trying to get issues that require two people since we will try a one person Ops starting Tuesday.
Gordon, Oct 21
We don't have TRH or Handar data from Aph3 last night and today from Oct 20 20:29 to Oct 21 11:51 MDT, due to the old problem of PC104 interrupts stopping.
Installed a new kernel, and rebooted. I don't think the new kernel will solve this issue. Turns out the fix for "pending" interrupts was also in the previous kernel.
Linux Aph3 2.6.35.9-ael1-2-titan #1 PREEMPT Sat Oct 13 12:45:33 MDT 2012 armv5tel unknown