Blog

Weather: Warm, scattered cumulus throughout the day

Tasking:

• Addressed 449 amplifier problem
• Checked data streams at SODAR trailer, no new issues
• Ordered three more Deka DOMINATOR deep cycle batteries for station power since no spares remain. They will arrive later this week at a shop in Urbana
• Upon request from PI, removed 6m Dust Track enclosure and associated cabling from Release Tower. They plan to use it during IOPs by placing it at the base of the Release Tower then store it onsite. There is no longer AC power cable tied up the Release Tower
• Grounded the DSMs on the six Aux Towers

Thunderstorms in the forecast for tomorrow so the day will be based around that.

John and Bill informed us yesterday that channel 2 of the 449 profiler went down.  Steve, John, and Bill isolated it to the amplifier/limiter enclosure, and FedEx'd two replacements to me.  I received them this morning, powered down the profiler around 10AM, swapped an amp/limiter.  System was fixed and running again around 12:20PM local time.  Brought system down again around 4PM local to check cable integrity inside and outside the trailer.  No major issues found.  System was back up around 4:30PM local.

Weather: Hot and humid, scattered cumulus throughout the day. Strong wind midday subsiding by late afternoon.

Tasking:

• My first day of Ops. Hendrik and Steve gave me the rundown of the current situation and short/long term punchlists
• Checked data streams and found two sonics inop at the Release Tower:
  
  • 10m CSAT appeared to be streaming but data stream had improper newline characters or something of the sort. Data was being received in large chunks. Power cycling the port did nothing but a hard reboot of its DSM (relm) brought it back
  • 15m CSAT was unresponsive. After troubleshooting found its cable was to blame. Hendrik climbed and added a new cable. Old was left in place and bagged for weather protection to avoid redressing the tower leg. This sonic is now operational
• Added 65 watt solar panel to Release Tower
• Troubleshot battery issue at Release Tower and found one was failing. Replaced with last spare
• Staked and ratchet strapped porta-potty near SODAR trailer due to its problem staying upright
• Shot Lower Convergence angles with theodolite
• Helped Noctis clean area around release tower. Removed 6m Dust Track so it can be shipped back to manufacturer

Relatively calm day.

After checking station statuses, we looked into why the DSMs are so often writing more than two files per day.  The following are some observations, and  a potential fix.

• The file rewrites seem to be caused by events that disconnect and reconnect the USB devices.  These seem to originate from the Ethernet.  Steve can provide the exact error message
• The DSMs that reset the most are lconvm, relm, and relt.  This is based on looking at the files from the past 5 days (10/2 - 10/6).
• DSMs that reset the least are lconv1, lconv2, lconvt, p1-thru-p6, uconv2, and uconvm.
• One possible cause is that high traffic on the DSMs with network switches causes usb reset
• Potential fix:  Steve looked up the problem online, and found a potential fix is to configure the Ethernet switch to work at USB 1 speeds (forget the exact syntax or config file that needs editing, but that's basically what it does).  We've tested the fix out on P6, just to see if it causes any adverse effects.  It doesn't.  Gary, should we try this on the problem DSMs?

We went to the U of IL Atmospheric sciences picnic, invited by Francina who didn't attend.

I took three soil samples for moisture measurement.  

• Release tower cornfield: tare weight = 21.30g, wet soil weight = 134.1g.  tin 4
• Upper Conv grass:  tare weight = 21.45g, wet soil weight = 130.62g.  tin3
• Lower Conv soyfield: Tare = 21.26g , wet soil weight = 134.55g.  tin2

Steve made theodolite measurements of Init and Release tower sensors.

Dinner afterwards with Francina at the Sunsinger.  It isn't cheap, but has a decent wine selection

Started a survey today (our first!) using the multistation.  Here are lots of notes as a memory dump of what I did:

• Didn't orient the multistation (MS).  Assuming that we (I) will be able to stitch together "scenes" from duplicate shoots of the same target from the different MS positions.
• I <think> the MS height is set to the default 1.5m, even though I didn't control the physical position to be at this height.
• All targets were shot with a height of 0m
• Project "SAVANT" was used in internal memory on the theodolite.  Hopefully, I can access it somehow!
• most targets were round (sphere/cylinder) and should have the position adjusted by the target radius in the direction away from the theodolite
• probably should have shot the boom arm of the 2D sonics to get more accuracy on azimuth, but didn't.
• Here is my attempt at a naming convention for points, not always followed
  • first character of i, r, u, l for the 4 towers, p for power line, full names of other objects.
    • when needed, also A for aux1, B for aux2
  • second character t for TRH, s for sonic, top for top of tower, base for base, etc.
  • third character 0 for 0.2m, 1 for 1.5m, 3 for 3m, 4 for 4.5m, 6 for 6m, 8 for 8.5m, a for 10m, f for 15m, aa for 20m (don't ask me why)
  • fourth character t for top, b for bottom, l for left (arbitrary), r for right or rear, f for front
  • fifth character numeric for scan sequence if multiple scans (from multiple positions and/or if an alternate scan for that position was used
• Started near init, from here scanned:
  • used my cell phone to determine the MS position as: 40.211578, -88.410728, 79.7m (not right), 4.00m
  • 3 corners of sodar clutter wall, at the base of where it goes vertical, used s, e, n for the cardinal direction the corner was pointing
  • the center of the luft's radome
  • init's tower base
  • trhs at the top/center of the inlet tube
  • (I'm not sure that I had an "i" as the first character of all of these targets.)
  • 2D sonic along the viewable side of the transducer, mid-height
  • 3D sonics at the center of the two transducer spheres, a point as far forward as I could shoot on the boom, and a point on the boom just forward of the mounting clamp, meant to be similar to those done in Perdigao.
    • square booms were shot about 1mm below the top of the boom, at both the front and back.  Later scans, I remembered to come forward a bit on the back scans to avoid the milled-down part of the boom.
    • I think I inadvertently used "b" for both bottom and back of the 1.5m CSAT, so this point may have been overwritten, since I also wasn't using sequence numbers yet.
  • power poles near uconv (I defined as Pg & Pf), scanned from the point on the upper cross-beam where the insulator hangs down
  • top mounting bump on rel
• Moved to rel, and scanned:
  • cell phone said MS position was: 40.211796, -88.407181, 216m, 4.00m
  • top TRH on init
  • point at which spline ball arm met tower top plate on uconv
  • I think I also did just the top of the north tower member on uconv
  • top TRH on uconv
  • base of splines on spline ball on lconv
  • top TRH on lconv
  • (couldn't see the power lines)
  • (couldn't see the same bump on the top of rel, but shot the hole in the top plate underneath it)
  • TRHs on rel and auxes
  • 2D on rel
  • 3Ds on rel and auxes
  • note that CSAT.aux2 is a CSAT3 with my strange mounting block adaptor.  I scanned the outside of the block (like the other CSATs) with a "0" suffix and the nearest points on the round boom with a "2" suffix.  The "1" suffix was where Hendrik noted that the laser seemed to see both round and square, even though the crosshairs were just on the round.
• Ran out of time, and threat of rain, before getting to uconv and lconv.
• Didn't scan the fiber optic mounting points yet.
• Intended to shoot the lidar trailers/tent, but didn't yet.
  
  

Investigating occasional Pirga spikes, I see a case for 6m.relm on 2 Oct, where at 20:28:16, the signal stops, at 20:28:51 (35s later) the signal resumes, at 20:28:53 (2s later) the sonic resumes being good, and at 20:30:21 (another 88s later) the IRGA signal becomes good.  This is normal behavior after a power cycle, but why was there a power cycle to begin with?  Other sensors stayed up through this time period, though need to recheck.  However, another curiosity is that we don't have local USB data during this time period (there were several breaks in the files that were recorded), even though we have WiFi data.

Some strange things happening out there....

It is only possible to look at high rate date for this project on ustar, so I've wanted to get R going.  Initially, I was getting error messages saying that the packages we had were built for an earlier version of R (barolo is at 3.3), with different "internals" than the current version (3.5 on ustar).  Thus, I had to build packages.

Following the instructions at: https://www.eol.ucar.edu/content/installing-and-using-r, I did:

> install.packages(c("digest", "RNetCDF","splusTimeDate","splusTimeSeries", "quantreg","SparseM","Rcpp","RUnit","lattice","maps", "gWidgets2","gWidgets2tcltk"))

where I had to add "digest" to the list to get the install to work. (P.S., and just updated this in this website)

I tried the described install of the eol packages, but this failed, so I had to go to source.  I saw that Gary had installed R to ~daq/R, so following the instructions at: https://wiki.eol.ucar.edu/sew/R/EOL, I did:

cd ~daq/R

git clone https://github.com/ncareol/R-eol.git

to create a ./R-eol directory here.  The instructions say to:

cd ./R-eol/packages

./build.sh -c -e -i -t

However, this again failed with the same version error.  It turns out that build.sh assumes that the packages are locally installed at /usr/local/lib/R/site-library, but the install.packages() function put everything in

~/R/x86_64-redhat-linux-gnu-library/3.5.

I cheated (as usual for me) and simply linked ..../site-library to ..../3.5 and repeated the ./build.sh command.  Eventually, the build succeeded.

Better would to have figured out the build.sh script to point it to the correct subdirectory, but.....

Now to look at data!

Mostly hanging out at Ken's as rain bands pass over, some hitting us and not the site (and probably vice versa).  We briefly visited the site, to see the waterway with about 2cm of water on top of the ground everywhere.

• we may still want to make adjustments to power, but things are getting closer to where we want 
• Qsoil.b.lconv is going up to 100%, which isn't realistic (even if the soil is saturated, with standing water, as it probably is now.

When the rain abates (tomorrow??), will take more soil cores (our last was near a minimum in soil moisture and now it is maximum at field capacity).  We'll also start theodolite shoots.

Hendrik started a 4-day soil moisture calibration with two extra qsoils (sn 20, 35), using soil from Ken's cornfield (possibly representative of the soil cornfield soil installation).  Measuring readings from air, water, and saturated soil all the way to dry soil.

Rain last night from 01-03AM, according to ISS, and some drizzle just ended.  co2.6m.init now is not reporting, but all else seems to be up.  Some standing water in the fields driving in.  The dusttracks on the ground flooded with apparently flowing water in the gully getting up to 10cm or so (based on grass bent around cables going up the towers).  We'll be working to rearrange these over the weekend.

Did some more power rearrangement after working through the power spreadsheet.  There still are some mysterious loads, but what we did should help for a while.  relm should now be being monitored.  

• Added extra 65W solar panels to UConv.m and Init.m.
• Took a battery from Init.b and added it to Lconv.m.
• Took a battery from Uconv.1, added it to Uconv.m
• Added power monitor cable to Rel.m

Being a cloudy, possibly rainy, and perhaps windy day, the PIs are not planning operations for the next few days.  So, the ISFS team is taking the day off.  As far as we know, everything is operational.

Interesting afternoon: winds at Ken's have been 11m/s (according to ISS1), but at the site have been only 4 m/s (ISS2).  (8m/s at 20m on ISFS towers.)  This over a period of at least an hour.  Must be some exposure at the base and sheltering at the site.  Winds from the SE as I type this at 1518, as a sanity check of wind direction.

Hendrik added some solar panels and power monitor cables.

• LConv.m now has four panels, previously had three.
• Rel.m now has four panels, previously had three.
• Uconv.1, init.b, and init.m now all have power monitor cables.  Steve added these to the XMLs so they're online.
• (observation) LConv.m likely has enough charging capability, but the battery voltage still seems to drop significantly at night.  More so than UConv.m, which has a similar load.  May be dead batteries.  I'll be watching this over the next couple of days and swap out if needed. 

At 1544 noticed that relm was down.  I think it had been down since about 1130 (before we arrived today).  Hendrik reseated the ethernet cable to put it back on the net.  There are some breaks in the raw_data files, but most data appear to be present.  I've seen this behavior before here, and not sure why the dsms are restarting data collection.

Still working on the list.  The overview:

• completed a first round of soil samples
• completed a first round of radiometer and ec150 cleaning
• completed the removal of stuff from the first base location at Ken's
• all that is left is the theodolite scan.  We're thinking of how this can be done.

April and Co. have down days for a while as the weather forecast has higher winds and more precip than they want.  (Though only the lightest of drizzle for a few minutes this morning.)

The details:

Morning round:

(times sometimes approximate):

1040 clean soybean radiometer, wetted the wetness sensor at end of this

1050 soil core taken, jammed corer, brought back later to Ken's

1105 clean ec150.1.5m.lconv

1113 clean ec150.20m.lconv

1117 clean rad.20m.lconv

1126 clean ec150.6m.lconv

1131 clean ec150.1.5m.uconv

1136 clean ec150.6m.uconv

1143 clean ec150.1.5m.rel

1153 clean ec150.20m.rel

1156 clean rad 20m.rel

1202 clean ec150.6m.rel

1210 clean rad.corn

1215 clean rad.grass

1222 clean ec150.1.5m.init

1226 clean ec150.6m

~1530 did a bunch of power tests, involving turning off rel1/rel2/relm networking/various sensors.  Found that the power monitor connected to relt is actually measuring the power to relm+rel1.  Oops.

Afternoon round:

~1630 changed power monitor connected to relt to monitor the power supplying relt+rel2.  Previously (by mistake) it had been monitoring power to relm+rel1.

1638 soil sample at corn

1648 soil sample at grass

Today:

• flagged stuff in the crop
• added a 3rd battery to relt
• finished the cable sleeving project
• fixed an issue with R plotting

We now (2 weeks into the project) have come to the end of our TODO list.  A secondary list:

• get rid of trash/recycle (haven't found a recycle center yet)
• make a theodolite survey of equipment (had hoped to do when corn was harvested, but might be a while)
• move stuff in the other part of Ken's yard to storage
• connect up the remaining power monitors (waiting for cables from Rochell)
• clean radiometers
• take soil samples

In other news, the DTS fiber is now strung between the aux and main towers at uconv, so access may be an issue.

IOP ran last night.  Haven't heard the outcome.  We had relt run out of battery at 0100 and would have lost data from 2 dsms there.  In other news today:

• Swapped out batteries in rel to give it good voltages.  It helps that today was mostly sunny for charging.  We expect things to stay up tonight (finally!)
• Half implemented cable sleeving using PVC pipe.  rel and uconv are now in pretty good shape.  Will work on lconv tomorrow.
• relt's (cabled) ethernet died again today.  Nothing was found wrong with the cable.  Swapped ethernet switch ports with rel2 and both came back.  Strange.  Will keep on monitoring.
• Gary noticed that my naming of the new Ubiquit access point conflicted with a dsm.  He's changed it.
• Did a nids_merge and part of a statsproc_redo to get more complete statistics, but had lots of sample time errors.  Gary and I think that this may be due to the corrupt messages from uconv2's GPS.  (BTW, I note that at least part of the messages are decipherable if bit 7 of each character is forced to 0.) . I've just changed the config on uconv2 to remove the GPS messages from the future data archive.  It still isn't clear how to handle the past...
• Also removed legacy daily plot R code to get more plots going.  Still have one group of plots to work on...

Plan for tomorrow:

• Add USB cables to monitor power on both branches of init and the other branches at rel and uconv.
• Start on the guy-wire flagging project

Several fix-it tasks today:

• found loose screw inside solar panel junction box supplying power to lconvm.  Now Icharge between lconvt and lconvm are the same, though still pretty marginal during today's totally overcast sky.  We may yet want to swap/add batteries.
• added Ubiquiti to lconv.a1 as access point for P4.  Thought we had prepped by telling P4 to use the new SSID (now savant4, rather than savant, for this link), but the Ubiquti firmware wanted us to enter a confirmation command so it didn't take.  Later did the command at P4 and got it back on the network.  Repointed P4 Ubiquiti azimuth a bit.  Note that this link doesn't have line-of-sight, since the AP is only at 10' on lconv.a1.  P4 can only see above 7m on lconv, nevertheless, we got a link of about 67dB; 30Mb/s.  No error messages now display on the ustar console (that has been very distracting).
• added usb cable to dustrack.6m.rel
• reseated usb cable to dustrack.6m.lconv
• replaced TRH housing at 6m.rel, with a very noisy fan
• found chewed-up ethernet cable causing outages on relt, replaced with consumer grade, so this one will get eaten as well.  We are now on the search for better cable sleeving
• various cable bundling and protection at lconv and rel

I note that the PIs have added AC power to rel, though AC would be live only during IOPs when the generator is turned on.  We definitely will NOT CLIMB when AC is live.  The connections are not weather-tight.  I just taped up the ones that I noticed, but this should be improved.