We had a partial network outage in the ISFS trailer that started around 10pm local time on Sunday night. There was a great deal of activity going on in the trailer, that seemed to confuse and complicate the issue, but the initial evidence was that Dave W. lost his wireless connection, and we had to switch him over to a wired connection. Then we noticed that the wired connections in the rear of the trailer that serve the ceilometer, sounding, and data manager computers were not connected to the network... Diagnosis began working from the back to the front, with rebooting the hubs in the rear (linksys) then the middle (d-link), but hesitated at rebooting the white belkin netgear wireless router... Other computers on the d-link (middle) are downstream from the belkin, and DID have outside internet connection. So I was reluctant to risk breaking something that was working, and risk loosing connection to Dave's (critical) laptop during our IOP.
After much diagnosis, I realized there was no other option but to reset the white belkin netgear wireless router, and I finally did about 1:30am, and all seems well after that. I reset the connections to the APG systems in the back, and the backlog of latent connections on Nagios began to clear out. (correction- At 2:15am, still several Nagios services reporting as critical, but are wholeskycamera related to network loss.
Hope I didn't screw things up too much- I blame the lack of sleep!
Thanks, Tim
10/6/13, 6 pm
Sebastian reconnected the krypton connectors, which brought it online. He suspects that it was the "military" connector to the CSAT serializer was not plugged in properly.
10/6/13
Tom is at NEAR to reboot the rad mote, and replace the dessicant in the radiometers.
He arrived at 3:55 pm.
Yellow light in center of radiation and SPN1 mote boards were both on, although both were sampling data (flashing red light). He power cycled both motes, which were not coming in. Rad mote is ID8.
Replacing dessicant at 04:04. Replaced pyranometer dessicants, but did not know how to access pyrgeometers (square dessicant holder).
Left at 4:11.
10/8, twh
Compared 2D sonic 10m wind directions to 3m sonic in order to orient 2D sonics. Only used winds from +/- 20 degrees from CSAT u axis and 10m wind speeds > 3 m/s (at FAR);
2 m/s at FLR. Could recalculate this at the end of the project, but likely will not change much. Used function dir.diff in project S directory.
At FAR, selected the median value of Dir.10m + 230.1 - dir.3m = 3.1 +/- 5.3 degrees. See attached plot. The 2D sonic is aligned with the boom, so need to add the boom direction, 230.1, minus the offset of 3.1 degrees, or 227 degrees to the 2D wind direction data.
10/30, I recalculated the offset using data from Sept 30 through 12:00 Oct 30 and set the minimum speed equal to 10 m/s. The result is Dir.10m - dir.3m = 0.2 +/- 0.8 degree, so need to subtract another 0.2 degrees or add 226.8 degrees to the 2D wind direction data.
. 
10/8, At FLR, the 2D sonic was pointed nominally north, but must be off by 180 degrees. The median value of Dir.10m + 180 - dir.3m = 6.6 +/- 5 deg. Thus need to add 180 - 6.6 = 173.4 deg to 2D wind direction data.
10/30, I recalculated the offset using data from Sept 30 through 12:00 Oct 30 and set the minimum speed equal to 3 m/s. The result is Dir.10m - dir.3m = -0.6 +/- 5.4 degree, so need to add another 0.6 degrees or add 174 degrees to the 2D wind direction data.
10/6/13, twh
Gordon and Tom serviced the RIM tower
Arrived 12:35 pm, left site at 2:00 pm
Taped Gordon hole.
Replaced Verizon modem ethernet cable.
Replaced 20m TRH SN 11 with SN 28 because 20m T was anomolously high. However the new TRH only lowered the anomaly a bit.
Calibrated DataScope; 5m CSAT: 61.9, 61.7, 62.0, 62.1, 61.7, 61.9, 61.7, 62.1 (mean=61.9 magnetic).
10/5/13, twh:
Arrived radiation stand 3:48 pm.
Rad mote missing antenna, replace with antenna from spare; cleaned radiometers. Rad mote has GPS disabled.
Left rad stand at 4:00.
Cleaned kh2o at 4:01.
Calibrated DataScope, CSAT azimuth: 39.6, 38.9, 39.5, 39.1, 39.3, 39.4, 39.2, 39.4 (mean=39.6 magnetic).
2D sonic (Gill Wind Observer) has one path aligned with boom, same as CSAT.
Left site 4:18 pm
10/5/13, twh:
Arrived 3:01 pm. Calibrated DataScope.
3m CSAT azimuth: 39.3, 39.0, 39.3, 39.3, 39.4, 39.3, 39.5, 39.6 (mean = 39.3 magnetic); all sonics visually aligned with 3m sonic
Cleaned kh2o at 3:16, then radiometers. Left radiation stand at 3:22. Note radiometers high to the south ~ 1/2 bubble (Chris said Schwartz screws frozen, which is true),
Left site about 3:30
10/5/13 twh
John was correct. The mote replaced at rad.flr (SN-19) did have the radio disconnected from the socket on the back of the main board. Repaired today.
Gordon, Oct 5
The Gill windsonic at base was installed with no concern about its orientation.
Today Tom calibrated the data scope, then measured the sonic boom azimuth (with declination=0), getting the following readings:
38.2 |
37.5 |
37.1 |
36.6 |
35.1 |
35.8 |
36.4 |
35.4 |
mean=36.5 |
The magnetic declination here is 10.5 degrees. So the boom azimuth is 36.5 + 10.5 + 180 = 227.0 degrees wrt true north.
Then we removed the sonic and boom from the tower to figure out the orientation of the sonic relative to the boom.
By putting a ruler along the transducers and some advanced trig we estimated that the north arrow on the sonic is 2 degrees clockwise from the boom.
So N on the sonic is at 229 degrees true.
Entered this as the offset angle in $ISFF/projects/METCRAXII/ISFF/cal_files/noQC/dir_base_10m.dat.
Yesterday deployed sonics and added 30W solar panels at SSW2 & SSW4.
Measured azimuths of csat sonics in crater.
Fixed P.rim and csat.15m.rim.
Replaced radiation mote and restarted SPN1 mote at FLR.
Sebastian cleaned radiometers and kh2o at FLR.
Review of cockpit around 11am:
T/RH: OK
P: OK, P > 850 mb on crater floor
csat u,v: OK
csat ldiag: OK, flr was at +1 at a previous time
csat w,tc: OK, w > +/- 5 m/s at rim
kh2o: kh2oV.flr = 0; kh2oV.far > 2 V
wetness: rad.flr, far are RIP
radiation: rad.flr, far are RIP
Tsoil: near is RIP
Goil: flr is < -100 W/m^2, far & near ~ -50 W/m^2
Qsoil: far & flr = 0, near OK
TP01: Vheat OK; Vpile on OK, near=1000 uV, others ~ 600 uV; Vpile.off=0; tau OK; lambdasoil are RIP
Spd, Dir: OK
T/RH: OK
10/4/13, twh
Sebastian cleaned the kh2o windows, but found out later that the data still near or at zero. Sebastian will check the wiring on the next trip into the crater.
10/4/13, twh
The radiation mote at FLR was not reporting. It looked like it was sampling the data okay, but John suggested that the radio might have shaken loose from where it plugs into the main mote board. I replaced the mote and it began reporting data okay. I removed the GPS jumper and brought the original mote back to the base for inspection.
Gordon reported that the SPN1 mote was not sending data. It appeared to be hung with a solid yellow light in the middle of the board. I recycled power and that appeared to fix it.
Sebastian cleaned the radiometers and noticed the some require new dessicant. Something to check for all the radiometers.
10/4/13
Tom measured the orientation of the FLR CSAT:
041.6
041.0
041.3
040.8
041.0
mean: 41.1
Calibrated DataScope and reshot CSAT orientation.
38.3
39.3
39.1
39.1
38.6
39.0
mean: 38.9
I don't know whether this difference of 2 degrees is due to calibrating the DataScope or simply standing in a slightly different spot to align myself with the sonic u-path.
Did not have sufficient time to shoot the orientation of the RMYoung 2D sonic at 10m.
10/4/13
Tom, Sebastian, Matt and Eric.
Installed the two sonics at SSW2 and SSW4 today, after much running up and down the crater sidewall to get the correct cables to plug into the motes (and even up to the rim for a delivery of sonic cables to Matt from Gordon). Also installed a second 30W(?) solar panel at both sites. These were scavenged from the solar panel stands at FLR. FLR is now powered by 4 60W(?) solar panels.
SSW4: The battery measured 13.87 V.
After installing the sonic, I tried to calibrate the data scope, but was unsuccessful because of the requirement to rotate through eight 45 degree increments within two minutes while standing still on a steep, treacherous slope. Measured the magnetic orientation of the sonic (no declination applied):
353.4
352.5
350.3
348.2
350.8
348.8
mean: 350.7
SSW2: The battery measured 13.66 V.
Sonic orientation (still no calibration):
337.5
337.0
337.2
336.8
337.2
mean: 337.1
12/4/13
From data files:
ssw2 was SN1121
ssw4 was SN0677
Gordon, Oct 4 04
Typically when the base trailer air conditioner kicks on, the lights brown out and an alarm beeps once on the UPS (APc Back-UPS ES 350).
This afternoon, a brownout caused the ap24 WIFI at the tower to reboot.
Currently these devices are on the UPS:
- /media/backup external UPS drive
- AP24
- 8 port ethernet switch
- netgear router (connected to Hughes modem)
- Hughes satellite modem
May want to get a better UPS. The battery in this one is probably shot.