Weather: Cold morning warming up to a pleasantly warm afternoon, scattered cumulus throughout
Tasking:
- Spent part of the morning troubleshooting my VPN issues, which have been resolved (thanks Ted)
- Power cycled P1 & P2 to bring them back on the net so Gary can implement his radio schedule script
- Checked streams at SODAR trailer, all seem nominal
- Hendrik participated in a RELAMPAGO telecon related to upcoming maintenance visits
- Brought accumulated trash and recycling from base trailer to hotel. They are very receptive about large amounts of waste so there is no need to find a landfill/recycling center going forward
Tomorrow we plan to troubleshoot the ISS 449 site DSM and replace a TRH with a failing fan at Upper Convergence Tower.
Weather: Cold and breezy. Low stratus early followed by scattered cumulus through the afternoon.
Tasking:
- Did the rounds to all six pressure sites to perform a power cycle. The pi sleep command was supposed to bring the antennas back up after a period of time, however they failed to do so. Battery voltages were looking acceptable but Hendrik planned to power down the antennas again so there is no risk of a site dying during the IOP tonight
- Gary has been working on a script to periodically power the antennas on and off to limit the draw down of the batteries on site. This is nearing implementation.
- Hendrik continued the soil moisture experiment.
- Hendrik helped ISS troubleshoot 449 issues. One of the main culprits was determined to be water pooling on the radomes.
- I surveyed the 2 University of Wisconsin's temporary met stations they installed south of the swale.
- A collection of students and a professor from an instrumentation class will be around for the IOP this evening to test different instrumentation, including some that will be utilized for the CHEESEHEAD field campaign. As I finished up they survey they were testing a UAV with a thermal camera. They plan to do a flight at dawn tomorrow morning when the maximum gradient in the swale will be measurable (in theory)
Hope all goes well for everyone out there this evening!
Today was a relatively short day. The morning was cold, humid, and cloudy. The sun came out at times in the afternoon.
- Last night, I did a power budget on the pressure sites, which I should've done long ago. We need more solar charging at all the pressure sites. A single 145W panel at each site should be adequate. We will likely be requesting that a total of six 145W panels be shipped from Boulder for this purpose. We'll then swap those in and use the 65W panels to beef up the main towers.
- We checked the sensors using Cockpit. P5 was down due to power.
- We cleaned all radiometers and EC150s, and checked cable tensions
Upper convergence tensions
- see Guy tensions page
- Lower convergence tensions
- see Guy tensions page
Release tower tensions
North low 360
North mid 320
North hi 400
Southwest low 360
Southwest mid 290
Southwest hi 370
Southeast low 370
Southeast mid 300
Southeast hi 370
Init tower trensions
Northwest 300
Southwest 300
East 320
- P5 came back online by 11:30AM, when we were done with sensor cleaning.
- We added an extra (well used) battery to P5. It now has two dying batteries (better than one, right?). We plan to double up batteries at the rest of the pressure sites as soon as we receive the new batteries we ordered, and purchase three additional coolers from Target. We'll add an extra battery back to init.b too.
- April emailed and mentioned the next IOP is the evening of 10/15. Additionally, U of Wisconson students will be arriving tomorrow to set up sensors.
Weather: Low stratus early followed by nimbostratus throughout the afternoon. Rain began around 13:00 and was heavy at times. Transitioned to wintery mix/snow at 14:30.
Tasking:
- There was an IOP last night. We have not yet heard how it went.
- Started the morning with a visit to MUTI-Sabre Industries Telecom Services. A couple of weeks ago, Steve and Hendrik noticed a winch up trailer tower in their yard while on the way to SAVANT site from the hotel. They dropped in but no one was able to meet with them at that time. We decided to give it another shot. Met with their VP of Operations Barry Schlickman. He showed us two towers, an AllTech 150' and and a ITS 100'. Both are about ten years old and have been used as temporary platforms for telecom equipment. The insight he provided into issues and benefits of these types of platforms was quite valuable.
- Hendrik checked the raw data files and found the USB speed change seemed to be working. Performed the change to the remainder of the DSMs.
- Checked streams at SODAR trailer. 0.5m TRH at Init Tower (DSM initb, usb2) was down. The sensor still appeared to be streaming data but it was hex and seemingly much longer than a TRH message. This would repeat about once every two seconds. Switched the SHT probe and power cycled the port but this incorrect message format continued. Finally after a power cycle of the DSM, TRH started transmitting normally.
- Ordered ten more DEKA 8G-24M batteries for station power
- Hendrik continued his soil moisture probe calibration experiment
- Continued with Base Trailer deep clean
- All the corn around the swale has been harvested. Reconnected the release tower mote.
It is still snowing but there is sun in the forecast for tomorrow. We plan on cleaning NR01s and EC150s if conditions permit.
Today was a busy day.
- Much colder today.
- But, thankfully sunny. The batteries need to be charged! P3 and P4 went down due to power (cloudy days, single battery at each site). P6 was on its way out due to battery issues too. So, we spent the morning swapping the three Deka batteries we just bought yesterday at those stations. Then, we went back to the base trailer.
- Dan got on a Safety meeting conference call.
- I edited the config for RELT in an effort to fix the USB reset issues. /boot/cmdline.txt now has a command that reduces the USB speed to 12MBPS, in an effort to reduce resets. Since, it has reset once, and therefore hasn't solved the problem but possibly reduced the frequency. I'm holding off on editing other DSMs until results are more definitive.
- April and Junming have decided there is an IOP tonight.
- On Junming's request, we helped Noctis calibrate the 3m 8530 dustracks on Release and Lower Convergence towers, and 6m 8530 dustrack on Upper Convergence tower.
- After lunch, we just happened to drive by the site and see Ken and crew harvesting the corn! We scrambled to disconnect the release tower mote (will stay DC for IOP, and until corn is harvested). Ken said they plan to harvest all fields to the north of the gulley today (10/11), and will harvest the south side tomorrow. So for this night's IOP, the south still has corn.
- Dan got a ride with Ken in the harvester, which actually an opportunity to talk to Ken on how he could get access to the south side without driving over the power cables to the SODAR, and Sodar trailer. We dug a small trench that the cables would lie in. Ken tested our cable trench by driving the harvester, and trucks over it. Power stayed on for the Sodar and trailer.
- The lack of corn made the ATV more visible from the road, so Dan parked it at a less visible location near the release tower.
Weather: Low stratus early followed by nimbostratus & scattered cumulonimbus through the afternoon. Rain was heavy at times.
Tasking:
- Checked data streams at SODAR trailer. Raw data at Lower Convergence appeared to have a short gap last night though battery voltage was nominal. 6m EC150 at that site was giving erroneous readings upon first check but seemed to self correct
- Started deep clean of Base Trailer, still a long way to go
- Picked up three more Deka DOMINATOR deep cycle batteries for station power from Shaff Implement Co in Urbana. If more are needed they can be ordered through them and picked up within a couple of days
The rain seems to have passed, tomorrow we will inspect the swale for standing and evidence of running water. It sounds like there will be multiple IOPs this weekend due to visiting students, regardless of conditions.
Weather: Warm, scattered cumulus throughout the day
Tasking:
- Addressed 449 amplifier problem
- Checked data streams at SODAR trailer, no new issues
- Ordered three more Deka DOMINATOR deep cycle batteries for station power since no spares remain. They will arrive later this week at a shop in Urbana
- Upon request from PI, removed 6m Dust Track enclosure and associated cabling from Release Tower. They plan to use it during IOPs by placing it at the base of the Release Tower then store it onsite. There is no longer AC power cable tied up the Release Tower
- Grounded the DSMs on the six Aux Towers
Thunderstorms in the forecast for tomorrow so the day will be based around that.
John and Bill informed us yesterday that channel 2 of the 449 profiler went down. Steve, John, and Bill isolated it to the amplifier/limiter enclosure, and FedEx'd two replacements to me. I received them this morning, powered down the profiler around 10AM, swapped an amp/limiter. System was fixed and running again around 12:20PM local time. Brought system down again around 4PM local to check cable integrity inside and outside the trailer. No major issues found. System was back up around 4:30PM local.
Weather: Hot and humid, scattered cumulus throughout the day. Strong wind midday subsiding by late afternoon.
Tasking:
- My first day of Ops. Hendrik and Steve gave me the rundown of the current situation and short/long term punchlists
- Checked data streams and found two sonics inop at the Release Tower:
- 10m CSAT appeared to be streaming but data stream had improper newline characters or something of the sort. Data was being received in large chunks. Power cycling the port did nothing but a hard reboot of its DSM (relm) brought it back
- 15m CSAT was unresponsive. After troubleshooting found its cable was to blame. Hendrik climbed and added a new cable. Old was left in place and bagged for weather protection to avoid redressing the tower leg. This sonic is now operational
- Added 65 watt solar panel to Release Tower
- Troubleshot battery issue at Release Tower and found one was failing. Replaced with last spare
- Staked and ratchet strapped porta-potty near SODAR trailer due to its problem staying upright
- Shot Lower Convergence angles with theodolite
- Helped Noctis clean area around release tower. Removed 6m Dust Track so it can be shipped back to manufacturer
Relatively calm day.
After checking station statuses, we looked into why the DSMs are so often writing more than two files per day. The following are some observations, and a potential fix.
- The file rewrites seem to be caused by events that disconnect and reconnect the USB devices. These seem to originate from the Ethernet. Steve can provide the exact error message
- The DSMs that reset the most are lconvm, relm, and relt. This is based on looking at the files from the past 5 days (10/2 - 10/6).
- DSMs that reset the least are lconv1, lconv2, lconvt, p1-thru-p6, uconv2, and uconvm.
- One possible cause is that high traffic on the DSMs with network switches causes usb reset
- Potential fix: Steve looked up the problem online, and found a potential fix is to configure the Ethernet switch to work at USB 1 speeds (forget the exact syntax or config file that needs editing, but that's basically what it does). We've tested the fix out on P6, just to see if it causes any adverse effects. It doesn't. Gary, should we try this on the problem DSMs?
We went to the U of IL Atmospheric sciences picnic, invited by Francina who didn't attend.
I took three soil samples for moisture measurement.
- Release tower cornfield: tare weight = 21.30g, wet soil weight = 134.1g. tin 4
- Upper Conv grass: tare weight = 21.45g, wet soil weight = 130.62g. tin3
- Lower Conv soyfield: Tare = 21.26g , wet soil weight = 134.55g. tin2
Steve made theodolite measurements of Init and Release tower sensors.
Dinner afterwards with Francina at the Sunsinger. It isn't cheap, but has a decent wine selection
Started a survey today (our first!) using the multistation. Here are lots of notes as a memory dump of what I did:
- Didn't orient the multistation (MS). Assuming that we (I) will be able to stitch together "scenes" from duplicate shoots of the same target from the different MS positions.
- I <think> the MS height is set to the default 1.5m, even though I didn't control the physical position to be at this height.
- All targets were shot with a height of 0m
- Project "SAVANT" was used in internal memory on the theodolite. Hopefully, I can access it somehow!
- most targets were round (sphere/cylinder) and should have the position adjusted by the target radius in the direction away from the theodolite
- probably should have shot the boom arm of the 2D sonics to get more accuracy on azimuth, but didn't.
- Here is my attempt at a naming convention for points, not always followed
- first character of i, r, u, l for the 4 towers, p for power line, full names of other objects.
- when needed, also A for aux1, B for aux2
- second character t for TRH, s for sonic, top for top of tower, base for base, etc.
- third character 0 for 0.2m, 1 for 1.5m, 3 for 3m, 4 for 4.5m, 6 for 6m, 8 for 8.5m, a for 10m, f for 15m, aa for 20m (don't ask me why)
- fourth character t for top, b for bottom, l for left (arbitrary), r for right or rear, f for front
- fifth character numeric for scan sequence if multiple scans (from multiple positions and/or if an alternate scan for that position was used
- first character of i, r, u, l for the 4 towers, p for power line, full names of other objects.
- Started near init, from here scanned:
- used my cell phone to determine the MS position as: 40.211578, -88.410728, 79.7m (not right), 4.00m
- 3 corners of sodar clutter wall, at the base of where it goes vertical, used s, e, n for the cardinal direction the corner was pointing
- the center of the luft's radome
- init's tower base
- trhs at the top/center of the inlet tube
- (I'm not sure that I had an "i" as the first character of all of these targets.)
- 2D sonic along the viewable side of the transducer, mid-height
- 3D sonics at the center of the two transducer spheres, a point as far forward as I could shoot on the boom, and a point on the boom just forward of the mounting clamp, meant to be similar to those done in Perdigao.
- square booms were shot about 1mm below the top of the boom, at both the front and back. Later scans, I remembered to come forward a bit on the back scans to avoid the milled-down part of the boom.
- I think I inadvertently used "b" for both bottom and back of the 1.5m CSAT, so this point may have been overwritten, since I also wasn't using sequence numbers yet.
- power poles near uconv (I defined as Pg & Pf), scanned from the point on the upper cross-beam where the insulator hangs down
- top mounting bump on rel
- Moved to rel, and scanned:
- cell phone said MS position was: 40.211796, -88.407181, 216m, 4.00m
- top TRH on init
- point at which spline ball arm met tower top plate on uconv
- I think I also did just the top of the north tower member on uconv
- top TRH on uconv
- base of splines on spline ball on lconv
- top TRH on lconv
- (couldn't see the power lines)
- (couldn't see the same bump on the top of rel, but shot the hole in the top plate underneath it)
- TRHs on rel and auxes
- 2D on rel
- 3Ds on rel and auxes
- note that CSAT.aux2 is a CSAT3 with my strange mounting block adaptor. I scanned the outside of the block (like the other CSATs) with a "0" suffix and the nearest points on the round boom with a "2" suffix. The "1" suffix was where Hendrik noted that the laser seemed to see both round and square, even though the crosshairs were just on the round.
- Ran out of time, and threat of rain, before getting to uconv and lconv.
- Didn't scan the fiber optic mounting points yet.
- Intended to shoot the lidar trailers/tent, but didn't yet.
Investigating occasional Pirga spikes, I see a case for 6m.relm on 2 Oct, where at 20:28:16, the signal stops, at 20:28:51 (35s later) the signal resumes, at 20:28:53 (2s later) the sonic resumes being good, and at 20:30:21 (another 88s later) the IRGA signal becomes good. This is normal behavior after a power cycle, but why was there a power cycle to begin with? Other sensors stayed up through this time period, though need to recheck. However, another curiosity is that we don't have local USB data during this time period (there were several breaks in the files that were recorded), even though we have WiFi data.
Some strange things happening out there....
It is only possible to look at high rate date for this project on ustar, so I've wanted to get R going. Initially, I was getting error messages saying that the packages we had were built for an earlier version of R (barolo is at 3.3), with different "internals" than the current version (3.5 on ustar). Thus, I had to build packages.
Following the instructions at: https://www.eol.ucar.edu/content/installing-and-using-r, I did:
> install.packages(c("digest", "RNetCDF","splusTimeDate","splusTimeSeries", "quantreg","SparseM","Rcpp","RUnit","lattice","maps", "gWidgets2","gWidgets2tcltk"))
where I had to add "digest" to the list to get the install to work. (P.S., and just updated this in this website)
I tried the described install of the eol packages, but this failed, so I had to go to source. I saw that Gary had installed R to ~daq/R, so following the instructions at: https://wiki.eol.ucar.edu/sew/R/EOL, I did:
cd ~daq/R
git clone https://github.com/ncareol/R-eol.git
to create a ./R-eol directory here. The instructions say to:
cd ./R-eol/packages
./build.sh -c -e -i -t
However, this again failed with the same version error. It turns out that build.sh assumes that the packages are locally installed at /usr/local/lib/R/site-library, but the install.packages() function put everything in
~/R/x86_64-redhat-linux-gnu-library/3.5.
I cheated (as usual for me) and simply linked ..../site-library to ..../3.5 and repeated the ./build.sh command. Eventually, the build succeeded.
Better would to have figured out the build.sh script to point it to the correct subdirectory, but.....
Now to look at data!
Mostly hanging out at Ken's as rain bands pass over, some hitting us and not the site (and probably vice versa). We briefly visited the site, to see the waterway with about 2cm of water on top of the ground everywhere.
- we may still want to make adjustments to power, but things are getting closer to where we want
- Qsoil.b.lconv is going up to 100%, which isn't realistic (even if the soil is saturated, with standing water, as it probably is now.
When the rain abates (tomorrow??), will take more soil cores (our last was near a minimum in soil moisture and now it is maximum at field capacity). We'll also start theodolite shoots.
Hendrik started a 4-day soil moisture calibration with two extra qsoils (sn 20, 35), using soil from Ken's cornfield (possibly representative of the soil cornfield soil installation). Measuring readings from air, water, and saturated soil all the way to dry soil.
Rain last night from 01-03AM, according to ISS, and some drizzle just ended. co2.6m.init now is not reporting, but all else seems to be up. Some standing water in the fields driving in. The dusttracks on the ground flooded with apparently flowing water in the gully getting up to 10cm or so (based on grass bent around cables going up the towers). We'll be working to rearrange these over the weekend.
Did some more power rearrangement after working through the power spreadsheet. There still are some mysterious loads, but what we did should help for a while. relm should now be being monitored.
- Added extra 65W solar panels to UConv.m and Init.m.
- Took a battery from Init.b and added it to Lconv.m.
- Took a battery from Uconv.1, added it to Uconv.m
- Added power monitor cable to Rel.m