Blog

Planning:

- Permitting was difficult. Must be a budget line item.

- Site visit was important for comms testing.

- Need a good relationship with UCAR contracts early in project.

Pre-project testing:

- 2+ weeks of staging area testing was critical for start of ops.

- Need to pay more attention to mote comms during testing.

Training and Safety:

- must do tower training, tower rescue training. Bob Wiley has organized tower training in Dec.

- need training on motes, DSMs, base systems, XML

- AED, first aid

- Helo ops are hazardous, buy our own helicopter

- Must notify Bob Wiley on any injury.

Packing:

- In general a very good job was done in organizing equipment for each station, with color ribbons, rubbermaid tubs

- Replace gas can (cheapo Walmart can's spout broke)

Setup

- Setup was slowed each time anchors needed testing.  Would have been nice to install anchors a week or two before-hand.

- Need to get prior permission from dir office when using non-ISFS staff.  Use temp workers?

- should have a pow-wow of on-site staff each morning

- need additional harness, ascender

- need new winch/capstan, new rope

- need portable generator

- can we improve procedures for plumbing tower, guy wire building and tensioning?  Use come-alongs?

- shorter studs on CSAT electronics boxes; suggest using lock nuts (like TRAM) instead of nylocks. Dedicate some mounting plates for sonics that are small enough to ship in the box. Could one plate work for Rohn and pump-up towers?

Sensors:

- sonics were flawless. Will do post-field zero cal in lab

- two serializers became noisy.  Need to replace or get rid of. Will not filter data for PI (Tom)

- kryptons: generally OK

- barometers: problems were all power related? Was NNE on Oct 9 07:00 MST real?

- One TRH fan wired backwards.  Need to check past data from this sensor and double-check all fans. Chris to do bubble tests.

- One TRH had 0.1C bias.  Pretty good for all this stuff.

- Bad TRH data can be fixed by bit shifting (applies to SCP, SOAS, PCAPS?)

- downlooking PYG at far had short in wire from thermopile

- Schwartzeneggers need refurbishing

- need to check/replace radiometer desiccant prior to project

- Rsw.in.flr died for a while? Needs confirmation

- Vpile.off apparently has -1 LSB = -31uV.  .xml limits should allow this.

- SPN1 isn't perfect. Floor diffuse was sometimes zero for a short period at sunrise. Sent note to manufacturer.  They think it would be useful to run as a serial sensor and have the option for "TEST" mode. Needs leveling system.

- Individual levels of Tsoil at near would fail, but would return by a power cycle.

- Power to all sensors on a mote can be remotely controlled.

- Still need to take soil cores: bring oven/scales/soil install kit (buy a second oven and scales for trailer?)

- Qsoil response not normal in this soil.  Need manual calibration.  What should the strategy be here?  I actually liked having a test plot near the base.  In any case, soil quite dry during most IOPs.

- You can see core sampling in the Qsoil data!  The data jump a bit due to nearby vibration while pounding in the core.

- Forgot to use Trime, need to track down its pieces

- Soil installations:  need more consistency. Update guide for single temperature probe installation.

Power:

- new power monitor motes are great

- ISS was happy with the solar trailer

- Still don't understand mote diodes.  Having a diode between charger board and battery caused failures.  Had no failures with diode removed. 

- redesign small charging systems: lithium-ion?, separate charging module?

- Some small charging systems were flooded during setup.

Mote:

- Disabling radio reset was necessary. Xbees had new firmware since PCAPS.

- Xbee radios worked loose, need to be better secured. Oxidation was present.

- Spare mote 15 wouldn't come up plugged in to console cable.  (Something I've done wrong?)  Labelled.

- Power motes feed power into DSM interface panel.....Not True.  See 'power monitor' log entry update.  Problem not reproduced.

DSM:

- One unexplained failure at flr.  Seems to be related to 75mph gust at rim (though etherant later connected).

Network:

- need to test bulgin/rj45 cables that are used with AP24s.

- Verizon modems worked well, new box is good.

- Netgear WiFi needed to be reset many times.  Changing channel and orientation helped, but still needed at least daily resets.  Toss?

- WIFI worked great!  Etherants are old and slow however.  Need U-bolts that work for various diameters.

Computer:

- APS UPS batteries dead.  Buy bigger UPS for base?

- Trailer AC (and even using vacuum cleaner) caused computer issues -- everyone complained that it was warm during the late afternoon. Check trailer power.

- would have been nice to have check_usb.sh/check_ap24.sh feed a Nagios display.

- Need to change cockpit RIP color from background!

- Can we force a browser to auto-reload plots? (found cache killer, but doesn't seem to work). Need html wrappers around plot images.

- xchat was good to have, though died when 4G went out (can/should we configure a local host as the chat server?)

- Could/should we have set up 4G failure to fall-back to HughesNet?  Requires a better router box.

- R is great!  All known bugs are now fixed.

Infrastructure:

- Bird diverters fell off

- Bigger holes in soil/pressure base plates for T-stakes

- How to keep kinks out of hauling line?  (Had same issues using Jack's frame during xHATS.)

- Need more Bulgin cable splices.  (Could these be printed?)

- New bolts/nuts for newer guy attachment brackets

- Check all cables from near (data, network, power).

Tear-down issues:

- unconnected serial cable to 20m.near is bad

- unconnected RJ45 cable on rim is bad.

- let's get GPS positions for flr rad stand and soils

- blown fuse on base ttyS10(?) caused by plugging in bad mote cable.  (Cable labeled.)

10/27/13
Summary:

• IOP7 ending -- pretty good conditions after 11PM last night; will keep lidars going today (despite forecast for strong winds).
• No more IOPs expected
• 0.1 Ci this morning
• JM arrives today

Actions past 24 hours:

• Continued Qsoil.base plot testing
• Cleaned all rads and kryptons
• Reseated kh2o.flr cables
• Reoriented SPN1 back to original -- dip all the way to 0 came back
• Did GPS surveys of all crater P locations.

To dos:

• Continue measuring Qsoil.base
• Take a day off

Sensor status:

T/RH: ok.
P: ok
csat u,v: ok
csat ldiag: a few more than 0.05 about 10PM last night
csat w, tc: ok
kh2o: flr a bit better after cleaning/cable resetting, but noisy (similar to before we changed the serializer board 2 weeks ago)
motes: ok
Wetness: flr had frost yesterday
radiation: Rsw.in.flr died in the middle of last night, but came back by itself.  Other rads fine.  SPN1 reoriented and 0 came back
Tsoil: ok
Gsoil: ok
Qsoil: ok
Cvsoil: ok.
2D sonic: ok

Filled generator 25 Oct 2013 Friday will need filled again on 28 Oct 2013 Monday.

GPS positions both from my phone (in green) and from Sebastion's GPS (in orange).  (black are unit conversion of phone data).

Went in with the IOP7 crew to take care of a few tasks:

- ~14:35 cleaned krypton

- also reseated the analog connectors on the krypton electronics box in an attempt to avoid the mess we saw last night

- 14:52 reoriented SPN1 back to original position (analog cable pointing north)

- ~14:54 cleaned radiometers

Then hiked to NNE, SSW1, S, SSW2, SSW3, SSW4 to take GPS positions (Sebastian did SW, WSW, W at the same time)

From 11:45-11:47 cleaned krypton & rads at far.  kh2oV increased from 0.85->1.05.  Not much change on rads.

Cleaned krypton and rads from 1127-1130 at near.  kh2oV went from 1.03->1.13.  Not much change on Rsw, at least.

10/26/13
Summary:

• IOP7 starts tonight; nice drainage flow last night
• totally clear in Flagstaff; frost
• ISFS staff change: SRS left this morning, JM arrives tomorrow

Actions past 24 hours:

• Found that near USB stick had failed 2 days ago, including last few hours of IOP6.  Lost direct archive for about 29 hours.  WiFi data seem robust.
• Started Qsoil.base plot testing
• SRS is taking last (spare) pocketec home
• Derived fix for TRH data that go into "large value" mode (can be applied to past projects).

To dos:

• Continue measuring Qsoil.base
• Clean all rads/krypton
• Get GPS coords for pressure array (do Sunday?)
• Take a day off

Sensor status:

T/RH: ok.
P: ok
csat u,v: ok
csat ldiag: ok
csat w, tc: ok
kh2o: flr odd all of yesterday.  Will clean today
motes: ok
Wetness: flr had frost yesterday -- may be associated with krypton issues
radiation: ok
Tsoil: ok
Gsoil: ok
Qsoil: ok, doing calibration/test at base (not included in statistics)
Cvsoil: ok.
2D sonic: ok
Labels: 

10:30 - Noticed usbdisk at near went down yesterday at 12:00Z.

SO moved the usbdisk to another usb port and mounted.  Did mount usbdisk by hand.  Took about 2 min, then passed ext3fs check. Disk back up.

Obviously, we have lost any data that may have been dropped by the AP24 during the last 29 hours!  Fortunately, this was not during an IOP...

P.S. Actually, the IOP was still going on for the first 5 hours of this outage.  During this time, 413 samples (20 seconds of data) were dropped from each of the sonics in the period 08:10 - 08:15 MDT.  Just after the IOP ended, essentially the entire 5-minute block of data from 10:50-10:55 MDT was dropped.  I guess losing 20 seconds of data from one DSM during an IOP isn't <too> bad, but it shouldn't have happened.  In our defense, even if we had caught this error as soon as we got up (~06:30 MDT), we likely would not have gotten to the site in time to have prevented this 20-second outage.

10/25/13
Summary:

• No IOP last night
• mostly clear this morning, few clouds to the North
• 40m TRH on rim went out last night

Actions past 24 hours:

• SO added a Qsoil probe to base for testing

To dos:

• SO to water new Qsoil probe at base
• switch pocketec

Sensor status:

T/RH: ok, monitor 40m TRH on rim
P: ok
csat u,v: ok
csat ldiag: ok
csat w, tc: ok
kh2o: ok
motes: ok
Wetness: ok
radiation: ok
Tsoil: ok
Gsoil: ok
Qsoil: ok, monitor Qsoil at base since ground has been watered
Cvsoil: ok.
2D sonic: ok

Sometime between 22:00-23:00 last night, TRH.40m.rim went bad.  I just (04:35) sent it a control-R and got it going again:

PS. HOWEVER, We have found that the "bad" data are correctable!  Traw has 2 extra bits shifted in and RHraw has 4 extra bits.  Dividing by 4 and 16, respectively, and recalculating restore these data!  (We can do this for old projects as well!)

TRH19 157.85 0.00 37 0 5035 2147 117\r\n
TRH19 157.85 0.00 38 0 5035 2148 120\r\n
TRH19 157.85 0.00 39 0 5035 2150 124\r\n
^R\0x80\n
\r Sensor ID19   I2C ADD: 12   data rate: 1 (secs)  fan(44) max current: 80 (ma)\n
\rresolution: 12 bits      1 sec MOTE: off\r\n
calibration coefficients:\r\n
Ta0 = -3.993325E+1\r\n
Ta1 =  4.048482E-2\r\n
Ta2 = -2.386631E-7\r\n
Ha0 = -8.241113E+0\r\n
Ha1 =  5.922000E-1\r\n
Ha2 = -4.567768E-4\r\n
Ha3 =  8.744341E-2\r\n
Ha4 =  1.592101E-3\r\n
Fa0 =  3.222650E-1\r\n
TRH19 10.61 66.10 38 0 1258 134 119\r\n
TRH19 10.61 66.10 38 0 1258 134 118\r\n
TRH19 10.61 66.10 38 0 1258 134 118\r\n
TRH19 10.57 66.58 37 0 1257 135 117\r\n

We've seen these before, but a spike in kh2oV.3m.flr is between 15:00:43 - 15:01:15 today.  I'm calling these bugs, but I don't quite believe this explanation.  If you want stranger ideas, ask Steve Semmer.

P.S. A better explanation is a drop of rain, though none was noticed at the base.

I just added a spare Qsoil to base in a plot that I will water.  My goal is to characterize the behavior of an EC-5 probe in this soil with high iron content.  Hopefully, it will still have a slope of 1, but just an offset.  Although the soil structure changes a bit from site-to-site here, the iron content appears (based on my magnet test) to be about the same at base as at near, so I am optimistic that these test results will be relevant to the other sites.

Probe is ECHO005/PIC015.

The plan is simply to water this plot to field capacity and watch it dry out, while taking gravimetric samples regularly -- at least daily.

15:00 took the first sample (still before wetting; soil was a little bit moist).  I'll just measure the 3-6cm ring, as being close to the location of the EC5 (volume = 3*pi*(5.31/2)^2 )

10/25: 09:35 First watering.  Water ponded.  I'll wait a bit and reapply.

09:45 Second watering.  EC5 still hasn't seen it.

10:00 Water #3.  EC5 still hasn't seen it.

10:15 Water #4.  EC5 noticeably rising.

10:45 Water #5.  EC5 up quite a bit.

11:05 Water #6.

11:15 Water #7.  I'll stop here.

11:40 EC5 beginning to flatten out.

Here is a plot of these data, along with a 1:1 line and a linear regression to all but the first sample (when the EC-5 might not have been settled in).  The regression line has an intercept of 9.9% and a slope of 0.34.   Also shown are the 3 gravimetric samples at far (f), near , and flr (r).   It was my intent that this slope would be applied to all of the METCRAXII Qsoil measurements, and to have the offset computed from the 1 gravimetric sample at each site.

With this philosophy, the adjustments are:

actual = (far + 2.3)/0.34

actual = (near - 4.4)/0.34

actual = (flr + 5.8)/0.34

actual = (base - 9.9)/0.34

The project measurements agree with these numbers reasonably well.  However, near still goes negative at times, far goes negative one night, and flr goes over 61% during one rain event.  Thus, I think that the 0.34 slope model above is not correct.  

Subtracting only 7.6 (rather than 9.6) from flr makes the agreement quite good.  Next, would be to check Cvsoil to see if sensor gains are consistent?

Using the in-field Qsoil adjustment (just a one-point bias correction), I see:

In this plot, I have placed an eyeball fit of a line with a slope equal to Cv_h2o through the data.  Only at near is a systematic variation seen of Cv with Qsoil.  If anything, the near plot has (slightly) too low a slope (Qsoil gain too high).  I would say that this plot does not justify increasing the measured Qsoil values by a factor of 3.

To conclude:

• The test at base suggested increasing the Qsoil gain by a factor of 3.  However, applying this correction to the site values makes negative values, too large flr values, and worse comparisons with Cvsoil from the TP01.
• The one-point in field adjustment of just the bias has none of these problems, though flr then is consistently larger by ~2%.

10/24/13
Summary:

• IOP last night, Dave says it was a good night
• cloudy this morning

Actions past 24 hours:

• SO and SC went to floor to fix P.sw.flr, battery replaced
• SO tried to improve RF comms for P.nne, refer to detail entry
• removed diode from soil battery system at near

To dos:

• SO looking into adding soil sensors to base

Sensor status:

T/RH: ok
P: ok
csat u,v: ok
csat ldiag: ok
csat w, tc: ok
kh2o: ok
motes: ok
Wetness: ok
radiation: ok
Tsoil: ok
Gsoil: ok
Qsoil: ok.  Offsets based on yesterday's gravimetric samples make the data look reasonable (for the entire experiment).
Cvsoil: ok.
2D sonic: ok

At about 3:00 today, SRS visited near and removed the diode from soil.near.  He chose this mote, since its Vmote cycles the most.  We haven't noticed any effect from this change.