Shift change Kate→Hendrik, so spent most time transferring information. Took a break to Allerton this morning.
Outstanding task list (many mentioned a few days ago):
- debug lconvm power shutdowns (high priority)
- redo dustrack.6m.lconv USB connection (high priority: this will be Hendrik's first climb tomorrow)
- flag guy anchor points and radiometers in crop
- still some cable bundling to do
- replace grey PVC cables that are getting chewed up
- sleeve other PVC (USB, etc.) cables on ground
- add ubiquiti link P4→lconv2 (med priority, but just about ready to do)
- software
- add R code for more webplots
- get statsproc computing correct statistics (I think I got this running last night with a new stats_5min.xml and savant/../check_processes..., but haven't checked)
- restart ncharts to use new statistics and new noqc_geo dataset
I don't yet have a good way to view these on qctables, so I'll make my own table:
| dsm | dsmID | Iload | Icharge | Panels | Batteries |
|---|---|---|---|---|---|
| p1 | 41 | 0.6 | 0.8 | 1 | 1 |
| p2 | 42 | 0.7 | 0.8 | 1 | 1 |
| p3 | 43 | 1 | 1 | ||
| p4 | 44 | 1 | 1 | ||
| p5 | 45 | 0.7 | 0.9 | 1 | 1 |
| p6 | 46 | 0.7 | 0.7 | 1 | 1 |
| relt | 14 | 2.5 | 5.0 | 3 | 2 |
| uconv1 | 21 | 2 | 2 | ||
| lconvm | 33 | 2.7 | 1.9 | 3 | 2 |
| lconvt | 34 | 1.6 | 4.9 | 3 | 2 |
Clearly, lconvm is only operating on one solar panel, so the short cable between panels is probably not completely connected.
This morning, a quick trip to the field:
- reinstalled csat3a.1.5m.a1.rel (the same head and box that was removed yesterday, but now after changing AA to 50, as suggested by Larry at CSI). Green Lights!
- swapped in a fresh battery to lconvm/lconv2
- connected power monitor to lconvm
- swapped out Gill.0.2m.uconv1 and its cable that had some blue corrosion and had died the night before last
Quick update before Francina brings her class here for a tour....
lconvm (and thus all lconv networking) died again last night, along with lconv2, at 22:00 and revived just before I arrived this morning. Presumably, this is due to power loss, despite us adding an extra solar panel and swapping in 2 new batteries yesterday, with an entire afternoon of sun coming through scattered/broken skies. lconv1, lconvt, and all rel seem to have stayed up.
Campbell reports that all 5 sonics that we sent to them as dead were working fine as received. This is really annoying. So far, we have 3 dead CSAT3As here – both of our spares and one deployed (now at 1.5m.a1.rel).
Given the guy tensions measured yesterday, every tower will need to be retensioned. <heavy sigh>
Overcast, rain off and on last night and more expected today. No IOP tonight.
- lconv down again, as expected, since there would have been no charging after I reoriented solar panels last evening
- I think relm stayed up (since cockpit is still running), presumably because I swapped in the battery last night
- csat3a.1.5m.rel still bad. We probably should swap in the .a1 head.
Actions done today:
- swapped csat3aw.1.5m.a1.rel with csat3aw.1.5.rel. a1 is now bad, as expected, but a lower priority
- taped up the cable end of the other 3 Gill WindObservers (but upon inspection, they all seemed dry and clean, despite rain overnight)
- added a pmon usb cable to relt (still need to add it to the config)
- added a third solar panel to each of the lconv and rel power systems, i.e 3 panels now go into 1 charger with 2 batteries, then duplicated at each site
- swapped a charged battery into each bank of lconv (though the old lconv batteries seemed to have charged up in the broken-to-scattered skies today). In the process, momentarily shorted the charge controller at lconv.a1, causing one of the ground traces to burn up!. Rewired the controller to make this work again. Drat.
- measured all guy wire tensions. Kate will have updated the wiki table by now.
- found that lconv2's Pi had locked up – perhaps associated with the power work I did earlier. Cycling power brought it back.
FYI, just after we returned to the hotel after dinner (about 9pm) a gust front with short-lived, but intense rain, went through. Radar showed that it went through the field site perhaps 30min earlier, with winds up to 10 m/s.
| Location | Angle | Type |
|---|---|---|
| Init | 38 deg | M (Magnetic) |
| Rel1 | 110 | M |
| Rel | 108 | M |
| Rel2 | 106 | M |
| Uconv1 | 132 | M |
| Uconv | 132 | M |
| Uconv2 | 132 | M |
| Lconv1 | 54 | M |
| Lconv | 54 | M |
| Lconv2 | 54 | M |
I left last night before things really started getting going for the dry run. From co2 data, it seems like conditions were good for the first ~2 hours, and CTEMPS was running (for the bit of it that was deployed). April still doesn't have the Doppler lidar, but they were going to try smoke and 2 aerosol lidars. Kate reports that the one radiosonde worked, but had a very low rise rate – they cut it off after 45min to 700mb.
- visited P5, found battery at 10.33V, so Pi was continuously rebooting (and chrony was not happy). Swapped in a good battery from the base. I <think> the solar in connector to the sunsaver wasn't quite seated properly. In the meantime noticed that the config still didn't have a pmon entry. Back at the sodar trailer, tried to use ansible to reload the config, but got a huge error message, suggesting that I use "--limit @/home/daq/isfs/projects/SAVANT/ISFS/ansible/update-projects.retry", which didn't work. Logged into P5 and added pmon manually. Now seems happy.
- decided to check the other pressure sites, to see if low power was systematic. P4 was at 12.44V. P6 was at 12.59V. So power availablilty should be okay, even though today is overcast and drizzle. BTW. P4 could ping ustar, and ustar can ping P4, but I can't ssh, and noticed (from lsu) that rsync isn't working to P4.
- csat3a.1.5m.rel died about 6pm last night, about when I was playing with 232/422 jumpers inside this DSM to get csat3.1.5m.a2 working. However, the EC150 data are still coming in, so it isn't as simple as power or comms. I note that the co2/h2o statsproc values also aren't being created due to the missing sonic data. Guess I'll be wandering down there soon, in the mosquito-filled mist.
- all of lconv seemed to be down a bit ago, but is up now.
- and still have to do:
- check guy tensions
- grab serial numbers from soybeans?
- shoot boom angles
- clean up several sets of cables
- flag tower anchor points in the field
- and April wants to figure out how to attach the CTEMPS fiber vertically between the tall and aux towers....
- Also noticed that several serial cables are grey PVC. I don't know why green ones weren't used. The PVC is already being chewed on – I taped up the one bit that I noticed.
We have been about 700 mg/m3 during the day, but now are going above 900. I assume that the corn is trapping air from respiring soil/grass (the corn itself is pretty much dead, I think).
More fixes today:
- reseated Bulgin/Bulgin connection (missing a purple ring) for rad.20m.rel
- changed trh.8.5m.rel to RS232 mode (rather than RS422)
- replaced CSAT3A at 6m.init with CSAT3A from 1.5m.a2.rel
- replaced CSAT3A at 1.5m.a2.rel with a CSAT3, requiring a new cable, new mount to be invented, new serial port in RS232 mode. Note that the new mount puts the path about 5cm higher and 3cm further north.
- installed a new sensor for Tsoil.grass.uconv
- also tried to organize cables at rel.a2 and grass.a2, but am not really happy with the job I did. I really should have started by disconnecting the ethernet cables.
I <thought> that fixed all sensor issues, but now see a dead CSAT3A at 1.5m.rel. I can't win and the stupid CSAT3A's keep on dying and Campbell hasn't fixed the batch of 5 or 6 bad ones that I shipped to them a month ago....
I still don't know why pmons at uconv, P3 and P5 aren't working.
gps.uconv2 is a known problem that Gary has worked around.
Dry run IOP just starting now (9pm). The PI team is pretty excited, and so are the mosquitos.
@Gary: I'm not sure I did the best change-over of the sonic.1.5m.rel2. I changed the port, (usb1 → usb5), id (1010 → 1016), and name (.a2 → .a2n), which made ck_xml happy, but now we have another set of variable names. I <thought> there was a way that we could, with one config, process both the data up to now and the data from now, into the same variable names.
This hasn't been working since I've started checking data. I assume that it is the old I2C issue on the Pi.
@Gary: should we replace this DSM? If not, is it set up to use another NTP server?
Heavy dew this morning.
Took most of yesterday off. Still a few sensors issues to deal with today:
- grass Tsoil – probably replace
- init.6m - probably need to replace, but don't have a spare (still at least 6 back at Campbell being repaired) Confirmed a red sonic LED in EC100, and all aspects of installation look fine.
- TRH.8.5m.rel - unknown issue (climbed and replaced SHT sensor, but turned out to be RS422/232 jumpering. Now fixed.)
- mote.20m.rel not sending data (mote console cable Bulgin to standard serial Bulgin connection at 20m doesn't have a purple ring. Retaped and got mote LEDs alive.)
Electric fence is mostly up (to protect CTEMPS fiber).
Dry run IOP is scheduled for tonight: 2200-0000.
Data flow was horrible this morning – cockpit wouldn't connect to streams/nagios showed critical something on almost all dsms. Did an ansible restart_dsm on everything, which got most going (P4 still won't connect, as usual with morning dew).
Installed HF14 to replaced the chewed HFT. i note that the yellow shrink-wrap over the TP01 wisard board also was chewed, so taped it up. All soil cables now have a plastic sleeve.
Took a photo of half-dollar-sized footprints in the mud, directly on top of the soil sensor installation patch, with footpads and sharp points – looks like a small fox footprint to me.
- Pirga not using P_RANGE
- Vbatt not using VIN_RANGE
- Tbatt not using T_RANGE
- need to redefine Ixx_RANGE in mA, not A
- need different dTsoil/dt range
- T.8.5m.rel dead
- relt dead (oops, had done "ddn" when using minicom and hadn't brought back up)
- need Alt range
- co2 offscale (why?)
- lconv - no pmon data (got working, but ansibile didn't update config, and config needed to have a suffix defined to work)
- csat.6m.init dead
- gps.uconv2 dead
- pmon.p3, p4, p5 dead
- Tsoil.grass not right, but others now okay!
20m.rel: mote not communicating at all. Tried fuse and port swapping. Cable seems okay. Most likely a connector issue up top. Will need to climb.
20m.lconvt: working
soybeans.lconv: working
grass.uconv: working. HFT and QS were reporting "na" – just needed to create a cal_file.
corn.rel: HFT eaten. Haven't yet replaced.
Our first critter casualty: HFT cable bitten through, cut in 3 pieces. This time, close to the wisard board, so repairable, but still will need to replace. Insulation on TP01 also abraided. Need to come up with a cable protection scheme quick!