At the site between around 8:45 and 3:00 today. Sunny, calm in the morning and increasingly windy as the day went on.
Started by looking at the 5m csat/ec150 on CM, which was completely unresponsive. Will climbed up to look in the box, which showed red light for gas but green for power and winds. He tried reseating the sonic, ec150, and bulgin cables and power cycling, which didn't bring it back. We decided the next step would be to replace the ec100 box, thinking that if we weren't getting messages from it at all it would likely be a box problem. New box that we swapped in had the same problem–no data at all, lights showing red for gas but green for power and sonic. Tried replacing the bulgin cable too, no change. Tried changing the config to swap the trh on port 1 with the csat/ec150 on port 0, did successfully get trh data through port 0, did not get csat data through port 1. Tried plugging the 5m csat into port 4 on CL (which is set up for 3m csat, which was working), didn't give data. Tried the 3m sonic in port 0 on CM, working fine, so we know that dsm port can ingest RS422. Running out of ideas, so we tried powering off the ec100, unplugging the gas head, and powering on again. Works! Nans for gas, of course, but we're getting data. Since it turns out the problem was in the ec150, we swapped back to the original ec100 box and bulgin cable. Will pivoted the boom up and replaced the ec150 head, now everything is working. We are taking the bad ec150 head back to Boulder so we can figure out why it's killing ec100 boxes, leaving the spare ec100 box (which we tried but ended up not using) in the spares room at RMBL.
Will also looked at the 15m ec150 since we had been concerned about it giving negative values for h2o. He didn't see any residue on the glass, and currently there's no cleaning supplies at the site or at RMBL, so we didn't try cleaning it. Later heard from Steve that it's a bias and didn't need replacing.
Replaced apogee cables with the newer versions at C, UW, D. Swapped the original popsicle stick back in at UE, after we finally got it detached from the old cable last night.
Got Ethan's lidar pis on the network. At UE all I had to do was power cycle pi 1 by unplugging and replugging the cable. At UW I put the SD card from pi 8 that Ethan had given me into the box for pi 4. That didn't seem to do anything. Later tried pulling the pi out of the dsm box and letting it hang below, to see if that would allow it to connect to wifi. That seemed to work, potentially because now the pi was oriented horizontally rather than vertically. Had to close the dsm box before we left, so tried putting the pi in more horizontally (not sure how successful this was, since in the cold the ethernet and power cables to the pi didn't bend well). However it ended up seems to work, and it's still connected now.
Attempted to do a firmware upgrade on snow pillow #3 by plugging directly into the electronics board in the snow pillow box, bypassing the external SDI connector. Opened up the box (voiding the warranty) only to discover that the connector on the dongle they gave us doesn't match the connector on the board inside the box:
So no luck there. Tried once more the original way, but didn't work (same error as Steve got last time). Incidentally, I was completely blocking the apogee on C as I was trying to get into the snow pillow #3 box, between like 1 and 1:30pm.
Left at 3 as we had lost the sun and it was getting cold fast. Stopped in at the salamander room at RMBL and left the spare csat we had taken just in case, as well as the good spare ec100 whose ec150 we used. Left the extra apogee popsicle stick in the user spares box.