Here are the things that I recall that need to be dealt with:

- Remove radiometer and open-path data (li7500, krypton, ec150) during cleaning events

- Try to recover odd Tsoil.grass data for about a week in early Oct. when scrambled.  (Already have code to do this, but might be improved.)

- Remove TRH data when iFan is zero

- Adjust some early TRH data from probes that were swapped based on postcals

- Determine which set of boom angles to use

- Swap Cactus/Grass data from first day

- Determine what to do with Qsoil.c, Qsoil.c2 not matching. Qsoil.c data (when available) are self-consistent from the initial data, even through the lightning.  Qsoil.c2 is always lower, but data were reasonable compared to one gravimetric sample.  We will measure probe depths to see if this helps explain the differences.  (Fix Qsoil probe that dies when cold.)

Weather

Partly cloudy with high cirrus, light winds, <2 m/s. Temperatures a round 0C to 2C.

During the night the winds stayed below 5 m/s.

Status

morning (7:30):  All systems running. This is the last day of ops. Teardown will start

tomorrow. I plan to do the final boom angle check today.

afternoon (14:45):  Completed shooting angles. Per Steve O. request the boom height of

the EC150 is 2.502m.

Station 17 was showing a few Idiag spikes. I believe this was due to the teardown of Christoph's

equipment.

Cores taken at cactus and then grass at about 1445 today (29 Nov).  I brought them back to Boulder and did the first weighing at about 1715, so hopefully they hadn't dried too much.  Both samples did not completely fill up the corer (despite being pounded in all the way on the outside).  Thus I would estimate that the first core is 0-2.5cm and the second from 2.5-5.5cm. 

From the PCAPS entry: The coring tool was set up with two 3cm rings on the top (used) followed by two 1cm rings on the bottom (ignored).  Thus, each 3cm sample volume was c(2.5,3)pi(5.31/2)^2=c(55.4,66.4)cm^3.

S+ plot:

dpar(start="2012 nov 28 00:00",lenday=2,stns=0)

plot(dat("Qsoil"),ylim=c(0,25))

points(nts(9.06,utime("2012 nov 29 14:45")),col=3,pch="+")

points(nts(c(4.86,12.57),utime(rep("2012 nov 29 14:45",2))),col=3,pch="-")

points(nts(16.21,utime("2012 nov 29 15:00")),col=4,pch="+")

points(nts(c(10.91,20.63),utime(rep("2012 nov 29 15:00",2))),col=4,pch="-")

Resulting plot is below.  Both Qsoils are between the values computed for the entire layer (0-5.5cm) and those from the bottom layer (2.5-5.5cm).  Note that better agreement might be expected with the bottom layer, since the probe is installed at about 5cm.  This is true for Grass, but not quite for Cactus.  Also note that the original cactus Qsoil.c probe compares poorer to the gravimetric measurements than the new Qsoil.c2 probe.  Overall, these gravimetric data do not suggest that modifying the Qsoil observations is needed.

Gordon, Nov 29

ssh'd into the systems, and checked the interrupt rates With the "intcount" command.

On vipers and titans, ttyS1, 2 and 3 are the serial ports on the CPU board. On both systems ttyS1 is served by interrupt 37, ttyS2 by interrupt 36. On vipers, ttyS3 is interrupt 116. On titans, ttyS3 is interrupt 122

The Emerald serial expansion board(s), serving serial ports ttyS5 to tty12 (and ttyS13-ttyS20 on m21) are configured to use ISA interrupt 3. The PC104 interrupts (in this case just IRQ 3) get multiplexed by a CPLD on the CPU board into a GPIO interrupt. On vipers, the PC104/GPIO interrupt is number 25, GPIO line 01. On titans, the PC104/GPIO interrupt is number 129, on GPIO 17.

Interrupts/second

Notes:
1: These vipers have large anomalous pc104 interrupt rates, which occur with both Sep 14 (a1,a8) and the Oct 4 (a5) kernels.
2: Not completely sure why on m21 the IRQ3 rate is less than the pc104 load. I may have installed a kernel on that system with a pc104 irq routine that exits if pending bits are 0, instead of attempting to serve he unmasked interrupts.
3: High USB interrupts are seen on vipers with bluetooth radios interfaced via USB. High USB interrupts are not seen on Titans, even though on "A" site titans, USB serves both the bluetooth radios and flash drives. On "A" site Vipers, USB has only bluetooth radios. Viper USB driver is isp116x_hcd. Titan USB driver is ohci_hcd.
4: On c20,m21, m22, USB is used only for flash drives

The big question is why the huge PC104 interrupt rates on vipers at a1, a5, a8 and m21. Previously traced this to a floating CTS/RTS line, but I'm not sure that is the current reason. Saw high PC104 interrupt rate on m21 even with the ribbon cables disconnected from the Emerald headers! Will have to investigate those systems back in the lab.

The only titans with PC104 serial are a3 and m22. They're both OK.

PC104 interrupts for vipers at a2, a6 and c20 are OK.

USB Interrupts

The viper kernel has a patch to use an assembler delay function for the isp116x, and not the kernel ndelay() function. It is not known whether this is related to the high interrupt rate. An incorrect delay may result in the USB interface completely failing.

The driver code that sets up the interrupt, uses two configuration values, int_act_high and int_edge_triggered, which are used to configure the interface:

if (board->int_act_high)
                val |= HCHWCFG_INT_POL;
        if (board->int_edge_triggered)
                val |= HCHWCFG_INT_TRIGGER;

The current isp116x driver is configured for level triggered (int_edge_triggered=0, default) and active high (int_act-high=1) interrupts. However, the setting for the corresponding GPIO interrupt is IORESOURCE_IRQ_HIGHEDGE. This might mean the interrupt is not being acknowledged correctly.

Planned Testing

• Figure out what might be causing the high PC104 interrupt load on some vipers. By swapping cards and looking at IRQ3 level on a scope, figure out whether the problem is associated with the Emerald serial cards, or some of the Vipers, or is related to the RS232 connections.
• Try various USB interrupt configurations on Vipers and see if we can get the USB interrupt rate to drop, while sending data through a USB Bluetooth radio at data rates that were used at SCP. Could this be related to the issue of data corruption on USB flash drives that we see on Vipers, when two or more USB devices are connected?

Weather

The morning gives us partly cloudy skies, mostly high cirrus. The temperatures throughout the night range from about

-8C to 2C. The winds stayed below 5 m/s and are presently at about 1 to 2 m/s.

Status

morning (7:30)   All stations are up. No issues with the sensors.

Christoph and his crew started teardown today.

afternoon (13:30)  For the afternoon entertainment, Steve O. conducted a tour of the site for Kate, Rudy, and Ned.

During that time Steve O. took a soil sample at the two soils sites and we determined the elevation between sites

16, 18, and 19.

evening (20:45) Still running like the Energizer Bunny! WInds are at ~2 m/s and are dropping.

Weather:

Overcast with light winds. Winds throughout the night peaked around 6 m/s. Temperatures pushed down to 8C.

Status:

11:00:  John left yesterday and Steve S. is now on site. All stations reporting.

21:00:  Everything looking good. I did notice A14 and A17 both went down for a short period, ~5 minutes. I could

not ping them.

Gordon, Nov 27

Chris wants the CSAT3 serial numbers to simplify boxing up the units during teardown. From the log files (/var/log/isfs/*_SCP_csat3.log) on each of the DSMs, here are the reported serial numbers:

csat3: Ah1:/dev/ttyS2, id=1,20, SN0923, 3.0f
csat3: Ah2:/dev/ttyS2, id=2,20, SN0833, 3.0f
csat3: Aph3:/dev/ttyS2, id=3,20, SN0743, 3.0f
csat3: A4:/dev/ttyS2, id=4,20, SN1120, 3.0f
csat3: Ah5:/dev/ttyS2, id=5,20, SN0732, 3.0f
csat3: Ah6:/dev/ttyS2, id=6,20, SN0800, 3.0f
csat3: A7:/dev/ttyS2, id=7,20, SN0673, 3.0f
csat3: Ap8:/dev/ttyS2, id=8,20, SN0176, 3.0f
csat3: Ap9:/dev/ttyS2, id=9,20, SN1121, 3.0f
csat3: Ap10:/dev/ttyS2, id=10,20, SN0677, 3.0f
csat3: Ars11:/dev/ttyS2, id=11,20, SN0674, 3.0f
csat3: Ap12:/dev/ttyS2, id=12,20, SN0855, 3.0f
csat3: A13:/dev/ttyS2, id=13,20, SN0745, 3.0f
csat3: Ap14:/dev/ttyS2, id=14,20, SN1124, 3.0f
csat3: A15:/dev/ttyS2, id=15,20, SN1122, 3.0f
csat3: A16:/dev/ttyS2, id=16,20, SN0740, 3.0f
csat3: A17:/dev/ttyS2, id=17,20, SN0856, 3.0f
csat3: A18:/dev/ttyS2, id=18,20, SN0744, 3.0f
csat3: A19:/dev/ttyS2, id=19,20, SN0178, 3.0f

csat3: C20:/dev/ttyS5, id=20,100, SN0200, 3.0f (1 m)
csat3: C20:/dev/ttyS8, id=20,200, SN0197, 3.0f (2 m)

csat3: M21:/dev/ttyS1, id=21,50, SN1455, 3.0f
csat3: M21:/dev/ttyS5, id=21,100, SN1117, 3.0f
csat3: M21:/dev/ttyS9, id=21,200, SN0536, 3.0f
csat3: M21:/dev/ttyS12, id=21,300, SN0540, 3.0f
csat3: M21:/dev/ttyS14, id=21,400, SN0738, 3.0f
csat3: M21:/dev/ttyS16, id=21,500, SN0537, 3.0f

csat3: Mu22:/dev/ttyS7, id=22,1000, SN1119, 3.0f
csat3: Mu22:/dev/ttyS10, id=22,2000, SN0741, 3.0f

Chris G. and Larry present

Weather:

Mostly sunny skies w/ cirrus.  Temps just above 10C and calm winds.  Looks like a good day to play frisbee golf, using the towers as holes.

Status:

Nothing new in this category.  Things are running great and smooth.  We will probably run around to some towers to check on configurations to see if we can add barometers.  We have the new nano barometers and EC150 up here now.  

To Do:

Work on EC150 mounting

Clean KH20, Licor and Radiometers

Status:

All stations reporting in am

On-Site: John

Weather: Clearing, fairly calm overnight.   Roughly 40 cirrus cover in am.   Chilly, temps dropped to -8 to -10 since roughly 10-11pm last night.   Winds fairly light roughly <2 from west.    No evidence of fog, dew, but very light frost with humidity rather high, 80%ish.   

Gully:  Pocketec local storage swapped.   Out #11 with data from 11/21 through 26th.   In #8

Spare DSM gray box (unlabeled) returning to Boulder for misc. testing support there.

Sensors:

Sensors:  Looking good.   A few diag hits on ec150 sonic "low signal" but data ok.

Radiometers, Licors Cleaned: 8:55-9:05 MST,  with methanol.   Frost was on uplooking Rsw but didn't appear to be any on Rlw.    Rsw.both desiccant was shot: replaced 9:40-9:50MST.

Status

morning (7:30):  All stations reporting. Temperatures are dropping and the humidity is up.    Overcast and maybe some fog in area in am.   Winds were up since yesterday afternoon and still 2-5ms this morning shifting to e/ne.   Not a good last night for science. 

Remainder of day, overcast continued, winds remained low in afternoon and temps through the day were cooler hovering around freezing+/-.   All stations remain up.

Boom angles:  Yet another set was shot.   Data scope was calibrated using camera tripod before hand, it reported "good = 8".   Didn't complete all shots: getting dark while doing main.   Some of my Handar reading were 'off' because of sighting the wrong angle, so they should be repeated.  jm

On-Site: John.

STATUS

Things still looking good. Wind speeds picked back up for awhile before midnight shifting south (frontal) and back to west with velocities 1-3 -ish rest of the night.   Temps to -3 to -5 -ish overnight

No one on site today.

Local store pocketec at 55% full.

STATUS

Things look good. Wind speed throughout the night stayed below 5 m/s.

John may visit the site today.

Afternoon (13:30) : John payed a visit to the site. Said all was well with winds pushing 10m/s.

~14:10-14:20 5m,10m Kryptons cleaned, JM.   The voltages didn't improve and if anything the 10m went down.   I cleaned twice with q-tips/h2o and wiped off as well but perhaps there was something on the cotton or perhaps i touched the cotton with my glove but i don't think so.   The concentration went up on the 10m and is tracking that of the 5m since.   Otherwise things still looking good.

Local store pocketec at 55% full.

Evening (20:00): Winds topped off at about 12 - 14 m/s. They are now down around 2 m/s. All stations reporting ok.

Daily status

We have entered the remote monitoring stage of SCP due to the Thanksgiving Holiday period.

John and Steve S. will monitor the data remotely until Monday, Nov. 26 when John will go to the site.

Everything looking good this morning. I did notice there is no T.baro from the new nano baro sites; #5, #10, #21.

This is expected since the sensor is not putting out that parameter.

M21 down:  Noticed after entering this log that m21 went down. I was able to ping to the station. The data collection process

was not running. An 'aup'  got it going.

AFTERNOON (16:30): All stations running. All sensors look good. Very light winds, <2m/s.

weather: morning was clear and calm -beautiful day

afternoon about  12:30 wind picked up a bit for awhile, but only a few m/s, clouds rolled in and is mostly overcast about 2:00pm

all stations are up and reporting

replaced pocket tek to bring back to Boulder

cleaned EC 100 at 1:37

cleaned licors at 1:39

cleaned radiometers at 1:42

shot all angles for sonics - see earlier entery for updated angles

rebooted Christophs system at about 12:15 - all looks ok per Chadi

weather:  clear,light winds

all stations up and reporting