Blog

checked main tower tensions today when wind was mostly calm

East vector  180  lowest guy

                     325 middle

                     360 top

Northwest vector   225 lowest

                              310 middle

                              350 top

Southwest vector  180 lowest

                               310 middle

                               350 top

weather: lovely, morning, calm and crystal clear. Wind pick up around noon becoming fairly breezy, about 12 m/s, mostly sunny

status:  all stations up and reporting

cleaned licors at 11:49

cleaned radiometers at 11:51

cleaned EC 150 at 11:55

checked main tower tensions, all looks O.K.

Ap8 EC150 boom angle 114.2 degs

Ap8 csat 111.1 degs

A11 csat 101.5 degs

M21 0.5m csat 96.1

        1.0m csat 97.2

        2.0m csat 99.2

        3.0m csat 98.3

        4.0m csat 99.5

       5.0m csat 98.3

tried to shoot other angles, but winds became to gusty to trust the readings as well as angle of attack for higher sonics introduces too much error......

weather: beautiful day, low winds, mostly clear in the morning, to mostly cloudy late afternoon

all data systems up and reporting

Weather: calm, partly cloudy

Status: all stations reporting

Within 2 seconds of me turning off cockpit for the night last night, this sensor died.  The DIN connector had pulled out in the mote box.  Fixed about 7:20 this morning (15 Nov).

The towers are damp, and what looks like a dusting of snow is near the dorm, where the ground was cooler.  I don't see this event on the wetness sensor (but maybe this is due to the cockpit scale).  Both kryptons were affected and 10m still hasn't recovered.

Larry also noticed wind last night, though it only got to 7 m/s.

Another day of mostly messing with the Picarro:

- Swapped in a new ADM242AN RS232 interface chip into the Picarro's rotary valve controller.  This got it working!  Reinstalled and the Picarro now is able to run a inlet sequence.

- Set final inlet locations and secured them all.  Checked that fittings were tight and that filter packs had the paper filter insert.  Did timing tests using Picarro's clock (via walkie-talkie) by breathing into the inlet.  Since we were running at 2.5 lpm, the flow rate was about 0.5 m/s -- I walk faster!

- Also blocked the sonic at 0.5m.M in an attempt to verify that the Picarro clock is synchronized with our DSMs.  Started blocking with my hand at 02:38:36 and held it there for a bit longer than 10s.  

P.S.  I the data gap appears from 22:38:37.12 -- 22:39:52.26 UTC, indicating about a 1s offset in time (assuming that Jielun's reading was off by an hour???), and that my count to 10 actually took 15s, which I could believe.  I'll check the Picarro clock (to the nearest hour!) now.

P.P.S. I found that the Picarro had been reset to Pacific time.  At 09:06PM, I reset it to Mountain time.  I'm not sure if this change will be picked up by the Picarro's program.

- Somehow in the process, the Picarro stopped sending data out the serial port.  I've restarted (adn/aup) and verified that we are still prompting for the data.

Also about 03:10 started a cleaning sequence: kh2o.5m, li7500.2m, li7500.1m, radiometers.  Didn't bother with kh2o.10m, since voltages have been good.

A fair number of people today:  Larry and I are here, Jessica and Nic left this afternoon, Steve Cohn visited this afternoon.  Mark L. also dropped in.  We expect Chadi back late tonight.

Spent most of the day on the Picarro.

- Serial port testing -- started by looking from data to the DSM from the Picarro and didn't see any.  Thought it was a Picarro config error, but all seemed okay.  Along the way found that Picarro's logmein wasn't working due to an incorrect account setting that I fixed.  Now Jielun can connect from Boulder.  Eventually realized that the Picarro needs prompting, which wasn't happening when I had just the Acer hooked up!  Found that the DSM was prompting as it should and WAY too much later (in the early evening) realized that data were coming in (using rserial).  We just needed to tweak the message parsing and all was good.  Cockpit no longer has RIPs on the Picarro channels, i.e. all sensors are now coming in!

- Also looked at the rotary valve serial communications.  Never saw a message out of the Picarro PC here either, but wasn't surprised since it probably turned off connection to the MPV, since it didn't detect it upon Picarro start-up.  Measured voltages and realized that the MPV didn't have RS232 signal levels (no negative voltages).  Removed the MPV, opened up in the lab (later moved to the dorm), and could find no negative voltages on the 232 driver chip in the (brand) VICI (model) EMTCA valve controller box, even though 5V logic levels seemed fine.  Guessed that this chip was bad, so had Chris order another that should be here tomorrow.  (Saunders didn't have one :( .)  To test further, cobbled together a way to jumper a V2 mote's MAX232 chip in place of the bad valve controller chip (Thanks, John).  With this bench configuration, was able to control the valve from the Acer.  Now I know that replacing this chip tomorrow will get the MPV working.

- Finally, laid out the remaining tubing, which required some adjustment of lengths.  Still need to secure the inlets (and the tubing itself under the fiber optics) and possibly rearrange inlet ports (so the sequencing makes sense).  Also need to do the inlet timing tests (on the list for tomorrow).

The stars are out, so it apparently is quite stable.  Wind directions are wandering all around, though some are smoothly varying as if there is no turbulence.  A great case for Larry.

The wiki was down yesterday (12 Nov) morning, and then we had lots of people and activity here (Larry, Jielun/Alex, Jessica, Nic), so I didn't get a chance to log stuff:

- ~0800?? A5 emerald ports were down.  Rebooted.

- ~0800 TRH somewhere(!) had fan=0, responded to control-R.

- Found wiring problem with the EC150, due to a misleading SW wiki entry.  ~1030 reinstalled EC150 at A8 and updated wiki.  By 1130, Gordon had it configured, logging data, and displaying on cockpit.  (I changed configs.xml to update scales.)

- ~1245 Replaced A3.0.5m TRH (which had stopped) with an entirely new sensor.  Old TRH is in the base with a label suggesting bad wiring.

- Installed Picarro (with Jielun), wired it up, got it to warm up, checked calibration with one cylinder (CO2 was 768, read 766; CH4 was 1.713, read 1.713), "replaced" keyboard, added to internet (gets served an IP address ending .134).  Hung 3 inlets on M at 1,2,5m.  Works on network and can run remote desktop (when in my bedroom's subnet).  Still not sending data over serial port to DSM.  Tried to get sequencer running, but doesn't see the rotary valve.  This is either a configuration issue or else the rotary serial port also is blown!

I spent this afternoon failing to get the EC150/100 going.  The cabling is okay, the instrument is configured properly, the interface panel jumper is set properly, and I'm sure that I am using the correct port configuration command.  Nevertheless, I can't get it going on either A8's DSM or the spare we have in the Base.  The only things I haven't done are try another port on the Base system and check I/F panel jumpers on A8.  Guess I'll try these tomorrow.

P.S. Campbell's nomenclature is odd.  EC100 is the electronics box that handles the signals from an IRGASON or separate EC150 CO2/H2O analyzer plus a CSAT3A head.  I'm going to keep calling the sensor as a whole an EC150.

From the storm yesterday (10 Nov), most of the site is covered with snow.  The coverage ranges from a clear ice glaze (that probably melted and refroze today) to drifts of up to 8cm.  (Larry says that he found drifts of more than 20cm near the crests of the hills.)  Caution for snow-covered holes again!  (Once I found a clear path, I tried to walk along my footprints on the return journey.)

I did look carefully at the radiometers and LiCors.  From the winds today (often 10m/s), all traces of snow had been blown away and optics looked perfect.  Thus, I didn't attempt to clean anything.  Kryptons are both reading > 2V, so they are fine.

Gordon, Nov 10

This TRH has been flakey lately, dropping out, but then coming back on its own.

Here's the result of a data_dump of data late on Nov 9, and then using awk to look for gaps over 2 seconds. There are 10 gaps over 2 seconds in this 6 hours, one of them 355 seconds, and one 5678:

nice data_dump -i 3,40 -A isfs_20121109_175854.dat.bz2 isfs_20121109_200000.dat.bz2 > /tmp/TRH_a3_nov_9.dat
awk '{if($5 > 2) print $0}' /tmp/TRH_a3_nov_9.dat 
|--- date time --------|  deltaT     len bytes
2012 11 09 18:38:20.3220   2.981      37 TRH48 8.93 58.04 35 0 1227 112 110\r\n
2012 11 09 18:39:50.0419   20.94      37 TRH48 8.89 57.53 34 0 1226 111 108\r\n
2012 11 09 18:42:33.5019   38.87      37 TRH48 9.10 55.05 33 0 1231 106 103\r\n
2012 11 09 18:44:11.1819   3.989      37 TRH48 8.77 56.01 33 0 1223 108 103\r\n
2012 11 09 18:45:08.9919    3.99      37 TRH48 8.89 56.03 34 0 1226 108 108\r\n
2012 11 09 19:10:14.8819   5.983      38 TRH48 10.02 53.16 34 0 1254 102 107\r\n
2012 11 09 19:11:29.6224   14.95      37 TRH48 10.50 52.72 31 0 1266 101 99\r\n
2012 11 09 19:18:55.0620   355.8      37 TRH48 10.58 50.67 34 0 1268 97 107\r\n
2012 11 09 19:38:58.6419   187.3      37 TRH48 11.99 47.75 33 0 1303 91 105\r\n
2012 11 09 21:15:08.7021    5678      36 TRH48 12.96 40.98 31 0 1327 78 99\r\n

One can see bad characters just before the 355 second gap above:

2012 11 09 19:12:56.3119  0.9999      38 TRH48 10.34 52.18 33 0 1262 100 104\r\n
2012 11 09 19:12:58.3019    1.99      38 TRH48 10.34 52.69 35 0 1262 101 110\r\n
2012 11 09 19:12:59.2975  0.9957      15 \xd42f\xb3g\x9097 102\r\n
2012 11 09 19:18:55.0620   355.8      37 TRH48 10.58 50.67 34 0 1268 97 107\r\n
2012 11 09 19:18:56.0619  0.9998      37 TRH48 10.62 50.68 35 0 1269 97 110\r\n

Similar bad characters were seen at the other gaps.

Serial port 5 shows 17 fe (framing errors) and 3 brks since the last reboot 2 days ago. The other serial ports look clean:

cat /proc/tty/driver/serial 
serinfo:1.0 driver revision:
0: uart:XScale mmio:0x40100000 irq:38 tx:3079 rx:0 RTS|DTR|CD
1: uart:XScale mmio:0x40200000 irq:37 tx:0 rx:4387536 RTS|DTR
2: uart:XScale mmio:0x40700000 irq:36 tx:15 rx:58501140 RTS|DTR
3: uart:XR16850 mmio:0x10000000 irq:122 tx:0 rx:0
4: uart:XR16850 mmio:0x10800000 irq:123 tx:0 rx:0
5: uart:ST16654 port:F1000100 irq:3 tx:119 rx:7913814 fe:17 brk:3 RTS|DTR
6: uart:ST16654 port:F1000108 irq:3 tx:0 rx:8612447 RTS|DTR
7: uart:ST16654 port:F1000110 irq:3 tx:0 rx:4630167 RTS|DTR
8: uart:ST16654 port:F1000118 irq:3 tx:0 rx:1462530 RTS|DTR
9: uart:ST16654 port:F1000120 irq:3 tx:4871 rx:35077537 RTS|DTR

My best guess is that it's a cable problem.