Blog from July, 2008

upwind now has the Viper board from spare and has just been restarted.  Also the Ethernet cable was changed to a different port on the hub located in downwind2. We'll see whether it stays up tonight!

upwind serial boards crashed again about 9:30pm last night.  I noted that there were long ping times to it, though Gordon believes that this is just a symptom of a generally sick adam.  Gordon actually tried to download a fix about 11pm that ended up crashing the adam.

We drove out about 8am this morning and got it running again by cycling power.

Steve and Gordon are now configuring "spare" in preparation for swapping it with upwind.

I note that winds were too much to the south all last night, so we didn't miss much data. 

From about 1635-1735 I reconfigured pressure yet again.  (See 11b signal for exact times.)

- 11b transducer removed and inserted into the CHATS system (which was buried yet again) to monitor Pref (in parallel with the Vaisala PTB220).

- a leak installed into the far end of the AHATS reservoir (black tube).  About 4m of the 1/16" line is attached, mostly coiled up in the ground, but the open end is exposed.

We'll see what <this> looks like... 

Tom asked that we measure the crosswind spacings between the Rohn towers.  I think these are what he wanted:

5t  1.327m  6t  1.279m 7t

5u 1.325m  6u 1.282m 7b

11b 1.283 12b 1.283m 13b

Steve Cohn and I just shot boom angles for config#2.  A few notes:

- We were standing upwind of the sonics near the berm from about 12:00-13:30 doing this, during a period when the winds were good. A critical investigator may want to remove these data. 

- Where there are 2 values, Cohn's are listed second (no reflection on him or his measurements)

- 3u and 8u are not misprints--they really did look quite different

- 5t was behind 5u's PAM mast, but I think I got a good reading by centering the visible outside of the "claws".

P.S. Today (the following morning) I've entered all of the Oncley angles -131.7 degrees into each of the cal_files in boom_normal.  We'll see if the wind azimuths start falling into place.

A new set of angles shot by Semmer on 7/28.

 cal: entry in calibration file = measured magnetic azimuth - 131.7

The horizontal plane marks were put on the tower structures this morning.

After the 10:30 pm event, Upwind kept running throughout the night. Gordon had a chance to look at the log messages and thinks the problem may be with the seating of the micro-drive on the Viper board. The micro-drive was re-seated this morning.

To evaluate the problem with the 4m TRH, I swapped it with 5.5m between ~0915-0925.  It appears that S/N 10 (the one formerly at 4m, now at 5.5m, that we swapped in as our "spare" a few days ago) has a bias.  We'll leave it in this configuration, since 5.5m RH is not a critical measurement.

Upwind has gone down a few times tonight. It started around 9:30pm. I did notice while monitoring the kernel log that the serial locked up after a string of error messages.

example:   Jul 26 05:01:14 upwind kernel: viper_irq_handler: irq=2 spurious= 101001, #/sec=100000

The last crash occurred about 30 minutes ago. I will hang around the base for another hour or so.

We built a 26-liter volume out of 2 5' lengths of 4" ABS pipe, stuffed it with steel wool and buried it in the ground (after checking that there were no leaks).  We disinterred the N2 cylinder.  After the usual 2 hours, Pref has settled down.  I <do> think the signal is better behaved than for any of the other volumes, however there still are fluctuations that are larger than the static pressure signal (by about an order of magnitude in the power spectra between 0.01 and 1.0 Hz). 

I have attempted to despike the Pref signal (similar spikes don't occur on P for some reason) and "add" it back to get the "actual" pressure.  This procedure undoubtedly alters the high-frequency information in the signal. 

At Tom's suggestion, I also have computed a w'p' cospectrum with the resulting signal.  This cospectrum looks <okay>, with a peak at 0.03 Hz for the last 30-min of data.  There is some change of sign at high frequencies, indicative of a phase lag.  I can mostly get rid of this by (arbitrarily) shifting the pressure data forward by 2 seconds.  Again, my manipulation using Pref may be causing some of this, though there is a hint of a phase lag using the original p' signal as well.

The only explanation I can come up with is that much of the Pref fluctuations are still due to temperature changes of the 1/16" tubing.  (I also am a bit concerned that the "curly" tubing on the "+" side of the ports could be filtering the real signal, but I've removed this from 6b and it agrees quite well with 7b.)  Let's see: if 50 ml of tubing changes temperature by 5 degC connected to a 25 l reservoir at 300 K, the pressure change would be (1000 mb) * (5 K/300 K) * (0.05 l/25 l) = 0.03 mb.  The standard deviation that I'm seeing is 0.015 mb now!

I can see 4 possible actions:

1. Keep the present configuration and hope that the tubing temperature fluctuations are not correlated with w (though other p' statistics might have errors).  It would be best if we could get a better sensor (another 202BG) to monitor Pref, but our spare 202BG is now dead.  (Should we sacrifice one of the 14?)
2. Treat the tubing so that the temperature changes are smaller.  (This was the reason I chose clear tubing originally.  Perhaps simply washing it would reduce temperature changes?)  It isn't clear what else I could do (wrap it all in foil?).
3. Make an even bigger reservoir!  (Home depot has lots of pipe!)
4. Rebuild the entire array with individual reservoirs.  (How would they be made thermally stable?  The reservoirs would have to be on the towers [and cause <more> flow distortion].)

AHATS daily status 7/25/08

Staff: Semmer, Oncley, Nguyen, Tudor with ISS
Temps: generally 99F/59F

I took a break in status reports since we have been working to figure
out how to get good pressure measurements.  Also, ADAMs upwind, profile,
and now pressure2 have died (of course at night, in some cases after we've
gone to bed).  Solutions to these problems are still progressing.

For more details, see the ahats logbook at https://wiki.ucar.edu/display/ahatslogbook

Good wind direction: The standard function reports 36 hours (14
unstable, 22 stable) in this configuration, with small segments the
last 2 days.  However we have had some sensors and data systems down
(and still don't know about pressure measurements) so the actual
numbers would be lower.

Local data storage: (all still low)
------------------
upwind:/dev/sda1 57685532 6106140 51579392 11% /var/tmp/usbdisk
downwind1:/dev/sda1 57685532 8354176 49331356 14% /var/tmp/usbdisk
downwind2:/dev/sda1 57685532 9895484 47790048 17% /var/tmp/usbdisk
profile:/dev/sda1 57685532 5623556 52061976 10% /var/tmp/usbdisk
pressure1:/dev/sda1 3940812 1485784 2455028 38% /var/tmp/usbdisk
pressure2:/dev/sda1 3940812 1376288 2564524 35% /var/tmp/usbdisk

aster:/dev/sdb1 721075720 264374616 456701104 37% /media/isff2
isff:/dev/sdc1 1922890480 390811412 1434401864  22% /media/isff15

We now have a 2Tb drive for the backup!  We'll send the old backup
to Boulder soon -- we'll see if Steve-Cohn-courier-service is available.]

Pressure:
--------
(+/- = ~1 std deviation among variables at the same height)
pressure2 ADAM died last night at ~0030 local.  Also, lots of playing around
with plumbing yesterday.

[redo, smaller time period]

It appears that most of the "signal" is variation of the reference pressure.
We changed its character (for the better?) by changing from the nitrogen
cylinder to the CHATS 1 liter reservoir yesterday, however there still is
too much variation.  We'll try building yet another reservoir today.
p: ok, +/- 0.03 mb
p'p': ok, +/- 0.002 mb^2
w'p': ok, +/- 0.001 m/s mb
t: 6b&6t are outliers by 3degC during day, others +/- 1.5 degC
(may be orientation of sun shields?)
Pref: still steadily drifting low, though a bit of a diurnal cycle now.
(might be cooling of the reservoir)
now 14mb below ambient.

Profile: [,select.p]
--------
Serial boards crashed 3 nights ago at ~1AM and was restarted the next morning.
Okay since then.

T: ok
RH: 4m looks different even after the sensor swap 3 days ago.  We'll try
exchanging sensors between levels next.
P: okay

diag: ok
samples.sonic: ok
spd: ok?, 8m still lower than 7m sometimes.
dir: ok
w: ok
tc: ok, 8m low by ~0.3 degC? (related to the speeds being low as well?)
w'w': ok
u*: ok, lots of imaginary values with light winds
sigma_w/u*: ok (1.3 at night)
w'tc': ok
h2o: ok?, 2 g/m^3 offset from dat("Q"), except in afternoon.
w'h2o': ok, 0.03 m/s g/m^3 at midday
co2: ok, 15-20 mmol/m^3 (Large values at night except for the period
11-21 July.  Does this indicate shallow stable boundary layers sometimes?
We should check ISS data.)
w'co2': ok, -0.013 m/s mmo/m^3 at midday

Upwind (hts=3.3): [,select.u]
----------------
Has crashed each of the last 3 nights at about 10pm.  Didn't fix it until
the next morning on the first night.  Have been able to get it running
by midnight the last 2 nights.

diag: ok
samples.sonic: ok
spd: ok, +/- 10 cm/s
dir: ok, mean offsets of up to 13deg -- need to enter new Vazimuths??
w: ok, +/- 5 cm/s
tc:  ok, +/- 0.2 deg,
w'w': ok, +/- 0.005 m^2/s^2  (30 min avg for second moments)
u*: ok,  +/- 2 cm/s, again some imaginary values
sigma_w/u*: ok (1.3 at night)
w'tc': ok, +/- 0.01 m/s degC
tc'tc': ok, +/- 0.05 degC^2

Downwind Lower (hts=3.3): [,select.b]
------------------------

diag: ok
samples.sonic: ok
spd: ok, +/- 30 cm/s
dir: ok, +/- 3 deg
w: ok, +/- 15 cm/s [need to check tilt angles]
tc:  ok, +/- 0.2 deg
w'w': ok, +/- 0.01 m^2/s^2  (30 min avg for second moments)
u*: ok,  +/- 2 cm/s, some imaginary mid-day
sigma_w/u*: ok (1.3 at night)
w'tc': ok, +/- 0.01 m/s degC
tc'tc': ok, +/- 0.05 degC^2

Downwind Upper (hts=4.3): [,select.t]
------------------------

diag: ok
samples.sonic: ok
spd: ok, +/- 10 cm/s, 3t lower by 20cm/s
dir: ok, +/- 2 deg, 3t now outlier
w: ok, +/- 5 cm/s, 5t&6t outliers
tc:  ok, +/- 0.2 degC, offsets up to 0.6 degC
w'w': ok, +/- 0.01 m^2/s^2  (30 min avg for second moments)
u*: ok,  +/- 2 cm/s, again imaginaries
sigma_w/u*: ok (1.3 at night)
w'tc': ok, +/- 0.005 m/s degC
tc'tc': ok, +/- 0.05 degC^2

----------------------------------------------------------------

Although upwind stayed up after a terrible evening, pressure2 died at midnight last night.  We just restarted it.  Even GPS was down, so it is as if data acquisition just stopped.

For the last two hours, upwind's serial board has been crashing.  I've lost count of the number of adn/aups and reboots I've done.  Last I checked, it stayed up for more than 3 minutes, so I'm going to bed.

If it helps to diagnose this:

1. The last 3 nights, these failures ocurred in the 10pm-12am timeslot.  sonic temps dropped from 29 to 25C during this time.

2. I had rserial running several times during these failures.  It would appear that the data rate slowed down and became erratic (like dropped samples) beginning one second to half second before the crash.

The results of this morning's tests indicate that the "medium frequency" signals we've been seeing in the pressure spectra are from the reference!  This may be caused by convection in the cylinder that Jim Wilczak warned us about.  Since we don't have a perfect alternative at the moment, we've just plumbed all the references to the CHATS reference.  Unfortunately, we had to disinter and rebury this reference in order to connect to it.

6b and 6t are now returned to their original configurations.

This work was done between 1500-1600 local. (6t should go offline and then online [was briefly online before being in the final config]).

I'm hopeful that earlier data can be recovered using Pref, but it will take some work (including despiking Pref) to work this all out.

Gordon suggested that the problems with upwind might be due to power.  Measuring at the power frontpanel (both inside and outside the box) I see 12.93V, which I would say is fine.  We need another explanation...