After the 10:30 pm event, Upwind kept running throughout the night. Gordon had a chance to look at the log messages and thinks the problem may be with the seating of the micro-drive on the Viper board. The micro-drive was re-seated this morning.
To evaluate the problem with the 4m TRH, I swapped it with 5.5m between ~0915-0925. It appears that S/N 10 (the one formerly at 4m, now at 5.5m, that we swapped in as our "spare" a few days ago) has a bias. We'll leave it in this configuration, since 5.5m RH is not a critical measurement.
Upwind has gone down a few times tonight. It started around 9:30pm. I did notice while monitoring the kernel log that the serial locked up after a string of error messages.
example: Jul 26 05:01:14 upwind kernel: viper_irq_handler: irq=2 spurious= 101001, #/sec=100000
The last crash occurred about 30 minutes ago. I will hang around the base for another hour or so.
We built a 26-liter volume out of 2 5' lengths of 4" ABS pipe, stuffed it with steel wool and buried it in the ground (after checking that there were no leaks). We disinterred the N2 cylinder. After the usual 2 hours, Pref has settled down. I <do> think the signal is better behaved than for any of the other volumes, however there still are fluctuations that are larger than the static pressure signal (by about an order of magnitude in the power spectra between 0.01 and 1.0 Hz).
I have attempted to despike the Pref signal (similar spikes don't occur on P for some reason) and "add" it back to get the "actual" pressure. This procedure undoubtedly alters the high-frequency information in the signal.
At Tom's suggestion, I also have computed a w'p' cospectrum with the resulting signal. This cospectrum looks <okay>, with a peak at 0.03 Hz for the last 30-min of data. There is some change of sign at high frequencies, indicative of a phase lag. I can mostly get rid of this by (arbitrarily) shifting the pressure data forward by 2 seconds. Again, my manipulation using Pref may be causing some of this, though there is a hint of a phase lag using the original p' signal as well.
The only explanation I can come up with is that much of the Pref fluctuations are still due to temperature changes of the 1/16" tubing. (I also am a bit concerned that the "curly" tubing on the "+" side of the ports could be filtering the real signal, but I've removed this from 6b and it agrees quite well with 7b.) Let's see: if 50 ml of tubing changes temperature by 5 degC connected to a 25 l reservoir at 300 K, the pressure change would be (1000 mb) * (5 K/300 K) * (0.05 l/25 l) = 0.03 mb. The standard deviation that I'm seeing is 0.015 mb now!
I can see 4 possible actions:
1. Keep the present configuration and hope that the tubing temperature fluctuations are not correlated with w (though other p' statistics might have errors). It would be best if we could get a better sensor (another 202BG) to monitor Pref, but our spare 202BG is now dead. (Should we sacrifice one of the 14?)
2. Treat the tubing so that the temperature changes are smaller. (This was the reason I chose clear tubing originally. Perhaps simply washing it would reduce temperature changes?) It isn't clear what else I could do (wrap it all in foil?).
3. Make an even bigger reservoir! (Home depot has lots of pipe!)
4. Rebuild the entire array with individual reservoirs. (How would they be made thermally stable? The reservoirs would have to be on the towers [and cause <more> flow distortion].)
AHATS daily status 7/25/08
Staff: Semmer, Oncley, Nguyen, Tudor with ISS
Temps: generally 99F/59F
I took a break in status reports since we have been working to figure
out how to get good pressure measurements. Also, ADAMs upwind, profile,
and now pressure2 have died (of course at night, in some cases after we've
gone to bed). Solutions to these problems are still progressing.
For more details, see the ahats logbook at https://wiki.ucar.edu/display/ahatslogbook
Good wind direction: The standard function reports 36 hours (14
unstable, 22 stable) in this configuration, with small segments the
last 2 days. However we have had some sensors and data systems down
(and still don't know about pressure measurements) so the actual
numbers would be lower.
Local data storage: (all still low)
------------------
upwind:/dev/sda1 57685532 6106140 51579392 11% /var/tmp/usbdisk
downwind1:/dev/sda1 57685532 8354176 49331356 14% /var/tmp/usbdisk
downwind2:/dev/sda1 57685532 9895484 47790048 17% /var/tmp/usbdisk
profile:/dev/sda1 57685532 5623556 52061976 10% /var/tmp/usbdisk
pressure1:/dev/sda1 3940812 1485784 2455028 38% /var/tmp/usbdisk
pressure2:/dev/sda1 3940812 1376288 2564524 35% /var/tmp/usbdisk
aster:/dev/sdb1 721075720 264374616 456701104 37% /media/isff2
isff:/dev/sdc1 1922890480 390811412 1434401864 22% /media/isff15
We now have a 2Tb drive for the backup! We'll send the old backup
to Boulder soon -- we'll see if Steve-Cohn-courier-service is available.]
Pressure:
--------
(+/- = ~1 std deviation among variables at the same height)
pressure2 ADAM died last night at ~0030 local. Also, lots of playing around
with plumbing yesterday.
[redo, smaller time period]
It appears that most of the "signal" is variation of the reference pressure.
We changed its character (for the better?) by changing from the nitrogen
cylinder to the CHATS 1 liter reservoir yesterday, however there still is
too much variation. We'll try building yet another reservoir today.
p: ok, +/- 0.03 mb
p'p': ok, +/- 0.002 mb^2
w'p': ok, +/- 0.001 m/s mb
t: 6b&6t are outliers by 3degC during day, others +/- 1.5 degC
(may be orientation of sun shields?)
Pref: still steadily drifting low, though a bit of a diurnal cycle now.
(might be cooling of the reservoir)
now 14mb below ambient.
Profile: [,select.p]
--------
Serial boards crashed 3 nights ago at ~1AM and was restarted the next morning.
Okay since then.
T: ok
RH: 4m looks different even after the sensor swap 3 days ago. We'll try
exchanging sensors between levels next.
P: okay
diag: ok
samples.sonic: ok
spd: ok?, 8m still lower than 7m sometimes.
dir: ok
w: ok
tc: ok, 8m low by ~0.3 degC? (related to the speeds being low as well?)
w'w': ok
u*: ok, lots of imaginary values with light winds
sigma_w/u*: ok (1.3 at night)
w'tc': ok
h2o: ok?, 2 g/m^3 offset from dat("Q"), except in afternoon.
w'h2o': ok, 0.03 m/s g/m^3 at midday
co2: ok, 15-20 mmol/m^3 (Large values at night except for the period
11-21 July. Does this indicate shallow stable boundary layers sometimes?
We should check ISS data.)
w'co2': ok, -0.013 m/s mmo/m^3 at midday
Upwind (hts=3.3): [,select.u]
----------------
Has crashed each of the last 3 nights at about 10pm. Didn't fix it until
the next morning on the first night. Have been able to get it running
by midnight the last 2 nights.
diag: ok
samples.sonic: ok
spd: ok, +/- 10 cm/s
dir: ok, mean offsets of up to 13deg -- need to enter new Vazimuths??
w: ok, +/- 5 cm/s
tc: ok, +/- 0.2 deg,
w'w': ok, +/- 0.005 m^2/s^2 (30 min avg for second moments)
u*: ok, +/- 2 cm/s, again some imaginary values
sigma_w/u*: ok (1.3 at night)
w'tc': ok, +/- 0.01 m/s degC
tc'tc': ok, +/- 0.05 degC^2
Downwind Lower (hts=3.3): [,select.b]
------------------------
diag: ok
samples.sonic: ok
spd: ok, +/- 30 cm/s
dir: ok, +/- 3 deg
w: ok, +/- 15 cm/s [need to check tilt angles]
tc: ok, +/- 0.2 deg
w'w': ok, +/- 0.01 m^2/s^2 (30 min avg for second moments)
u*: ok, +/- 2 cm/s, some imaginary mid-day
sigma_w/u*: ok (1.3 at night)
w'tc': ok, +/- 0.01 m/s degC
tc'tc': ok, +/- 0.05 degC^2
Downwind Upper (hts=4.3): [,select.t]
------------------------
diag: ok
samples.sonic: ok
spd: ok, +/- 10 cm/s, 3t lower by 20cm/s
dir: ok, +/- 2 deg, 3t now outlier
w: ok, +/- 5 cm/s, 5t&6t outliers
tc: ok, +/- 0.2 degC, offsets up to 0.6 degC
w'w': ok, +/- 0.01 m^2/s^2 (30 min avg for second moments)
u*: ok, +/- 2 cm/s, again imaginaries
sigma_w/u*: ok (1.3 at night)
w'tc': ok, +/- 0.005 m/s degC
tc'tc': ok, +/- 0.05 degC^2
----------------------------------------------------------------
Although upwind stayed up after a terrible evening, pressure2 died at midnight last night. We just restarted it. Even GPS was down, so it is as if data acquisition just stopped.
For the last two hours, upwind's serial board has been crashing. I've lost count of the number of adn/aups and reboots I've done. Last I checked, it stayed up for more than 3 minutes, so I'm going to bed.
If it helps to diagnose this:
1. The last 3 nights, these failures ocurred in the 10pm-12am timeslot. sonic temps dropped from 29 to 25C during this time.
2. I had rserial running several times during these failures. It would appear that the data rate slowed down and became erratic (like dropped samples) beginning one second to half second before the crash.
The results of this morning's tests indicate that the "medium frequency" signals we've been seeing in the pressure spectra are from the reference! This may be caused by convection in the cylinder that Jim Wilczak warned us about. Since we don't have a perfect alternative at the moment, we've just plumbed all the references to the CHATS reference. Unfortunately, we had to disinter and rebury this reference in order to connect to it.
6b and 6t are now returned to their original configurations.
This work was done between 1500-1600 local. (6t should go offline and then online [was briefly online before being in the final config]).
I'm hopeful that earlier data can be recovered using Pref, but it will take some work (including despiking Pref) to work this all out.
Gordon suggested that the problems with upwind might be due to power. Measuring at the power frontpanel (both inside and outside the box) I see 12.93V, which I would say is fine. We need another explanation...
The pam towers for configuration 3 are now setup. Tomorrow we will mark them for height measurents.
To diagnose our strange p' spectra problem, I've made 2 changes:
6b - the + side is now plumbed through the CHATS garden hose to the Bedard quad-disk-probe, which I've lashed to the top of the tripod in the middle pitchfork position. Good winds at this location are bit restricted due to the config#3 towers that Steve has erected, but should give us some idea. Changes done 0915-0930.
6t - I've removed this transducer and placed it inside my CHATS sensor tube which is now in buried in the ground near the middle of the horizontal array and covered with a space blanket. The + side is connected to the AHATS reference system and the - side is connected to the reference volume inside this tube. Changes done 0940-1020. Note that the AHATS reference system was open to the world a couple of times during this work.
NOTE THAT THIS DESTROYS THE AHATS p' ARRAY, since no center sensor at either height. I think these tests are worth it, though.
Last night about 10pm, upwind's serial boards crashed again. "adn/aup" worked, but crashed again after ~10min. I "reboot"ed from the apartments and got it working. However, the usbdrive wasn't working this morning, so it probably died in the reboot. About 9:30 this morning, I cycled power on upwind and all seems well at the moment.
We would change this serial board, but it isn't clear to us which board to swap with.
We lost data from the serial boards in both profile and upwind last night.
profile died at 20:20 local and upwind at 03:05. Both were restarted with "adn" and "aup" at 09:43.
Sorry it took so long to fix this.
I guess we now should take seriously the "To Do" task of swapping profile's serial board!
Okay, I've been looking at pressure data and I can't figure out some things:
1. The reference pressure is slowly ramping away from P. My best guess is that the soil surrounding the reservoir is actually cooling now that it has the space blanket on top of it, thus making the pressure go down. I believe that this is consistent with the sequence of operations (from Khuong) when the cylinder was reburied (hole dug Fri AM, old hold dug up Fri AM, cylinder moved to new hole Fri AM, pressure lines sealed & checked Sat AM, hole backfilled and space blanket added Sat AM). Although Pref is now 14mb lower than P, this is well within the 150mb range of the 202BGs, so I'm not excessively concerned. We'll keep on monitoring this signal.
2. I've now looked at selected pressure spectra from two periods, one in config#1 and one in config#2. The two periods have spectra that are quite similar to each other and the 3 sensors I've looked at also are similar to each other. However, these spectra are completely different than those seen in CHATS. CHATS p spectra followed f^-5/3 reasonably well from a peak more-or-less at 0.1Hz until a digitizing white noise floor at ~7Hz. The amplitude of the noise was about 2.5e-6 (mb^2).
Here, we are seeing a nice peak at about 0.02 Hz, but with a roll-off that goes more like f^-3.5 and thus intersects the noise floor (of about 2.5e-7 mb^2) at a frequency of only about 1Hz. It is gratifying that the noise floor is lower (given all the hard work on our counter boards), though I thought we would gain more than this factor of 3 (the square root of 2.5e-6/2.5e-7). The amplitude of the signal at 0.01 Hz is actually about the same as the CHATS cases but, with this sharper roll-off at high frequencies, there is a lot less energy at, say, 1 Hz.
I don't understand the sharper roll-off. At first, I thought that the low frequencies might be contaminated by tower wakes in our configuration #2, but I would have expected energy at high frequencies as well and I see the same roll-off in configuration #1 data. However, I can't see how filtering like f^2 (the approximate difference between -5/3 and -3.5) could be caused by a physical process (say, too small tubing). My best guess is that it is a signal processing artifact, though I can't imagine what. The cut-off frequency of such a filter would be about 0.1 Hz (10 s period).
I don't know at the moment what the calibration code is doing. Since the variables "pper" and "p" are almost perfectly correlated, I'm assuming that the temperature correction, if it is being applied, is quite small (at least at these frequencies). In any case, since this "filtering" appears in "pper", I'm assuming that it hasn't been introduced by post-processing.
Did anyone else look at pressure spectra before I arrived? If so, did you see similar behavior??
I might try changing the sample-side tubing on one of the transducers, in a long-shot attempt to figure this out, but otherwise I'm stumped.
I got the following replies from Tom and Gordon:
Steve:
Perhaps you have already considered the averaging of the counter board output. It is
counting for approximately 0.1 second, which is applying a low-pass filter equal to
sin
/x where x = 2/pi*f*dt/2 and dt=0.1 sec. This does not nearly correspond to the
attenuation you are seeing at high frequencies. At the Nyquist frequency, 5 Hz, the
attenuation is 0.637 (0.405 for the power spectrum: a rough anti-aliasing filter) and at
1 Hz the attenuation is 0.984 (0.968).
Still not much help. It is perhaps worth noting this in the logbook for future users of
the pressure data.
Tom
We can alter that averaging period. It is the NumPulses parameter in the XML,
currently set to 3000 for pressures and 10,000 for temperatures.
Gordon
My status reports have noted issues with 4u and 8m.
8m: By just playing with offsets, I'm able to get better looking profiles by adding 0.3 m/s to dat("u"). As yet, I have no justification for doing this. Tom wondered about flow distortion around the etherant, but this seems only a remote possiblity after looking at it today. The next step would be to move the etherant.
4u: At 09:10 on 20 July, tc on this jumps by 0.25 degC. I can see no evidence of a similar step in any of the wind components. Tom confirmed that an upwind sensor might have been "bumped" by a ladder at this time, but his recollection was that it was one at the southwest end, whereas 4u would be at the northeast end. I don't see any reason for further action with 4u.