Gordon, Nov 29

ssh'd into the systems, and checked the interrupt rates With the "intcount" command.

On vipers and titans, ttyS1, 2 and 3 are the serial ports on the CPU board. On both systems ttyS1 is served by interrupt 37, ttyS2 by interrupt 36. On vipers, ttyS3 is interrupt 116. On titans, ttyS3 is interrupt 122

The Emerald serial expansion board(s), serving serial ports ttyS5 to tty12 (and ttyS13-ttyS20 on m21) are configured to use ISA interrupt 3. The PC104 interrupts (in this case just IRQ 3) get multiplexed by a CPLD on the CPU board into a GPIO interrupt. On vipers, the PC104/GPIO interrupt is number 25, GPIO line 01. On titans, the PC104/GPIO interrupt is number 129, on GPIO 17.

Interrupts/second

Notes:
1: These vipers have large anomalous pc104 interrupt rates, which occur with both Sep 14 (a1,a8) and the Oct 4 (a5) kernels.
2: Not completely sure why on m21 the IRQ3 rate is less than the pc104 load. I may have installed a kernel on that system with a pc104 irq routine that exits if pending bits are 0, instead of attempting to serve he unmasked interrupts.
3: High USB interrupts are seen on vipers with bluetooth radios interfaced via USB. High USB interrupts are not seen on Titans, even though on "A" site titans, USB serves both the bluetooth radios and flash drives. On "A" site Vipers, USB has only bluetooth radios. Viper USB driver is isp116x_hcd. Titan USB driver is ohci_hcd.
4: On c20,m21, m22, USB is used only for flash drives

The big question is why the huge PC104 interrupt rates on vipers at a1, a5, a8 and m21. Previously traced this to a floating CTS/RTS line, but I'm not sure that is the current reason. Saw high PC104 interrupt rate on m21 even with the ribbon cables disconnected from the Emerald headers! Will have to investigate those systems back in the lab.

The only titans with PC104 serial are a3 and m22. They're both OK.

PC104 interrupts for vipers at a2, a6 and c20 are OK.

USB Interrupts

The viper kernel has a patch to use an assembler delay function for the isp116x, and not the kernel ndelay() function. It is not known whether this is related to the high interrupt rate. An incorrect delay may result in the USB interface completely failing.

The driver code that sets up the interrupt, uses two configuration values, int_act_high and int_edge_triggered, which are used to configure the interface:

if (board->int_act_high)
                val |= HCHWCFG_INT_POL;
        if (board->int_edge_triggered)
                val |= HCHWCFG_INT_TRIGGER;

The current isp116x driver is configured for level triggered (int_edge_triggered=0, default) and active high (int_act-high=1) interrupts. However, the setting for the corresponding GPIO interrupt is IORESOURCE_IRQ_HIGHEDGE. This might mean the interrupt is not being acknowledged correctly.

Planned Testing

• Figure out what might be causing the high PC104 interrupt load on some vipers. By swapping cards and looking at IRQ3 level on a scope, figure out whether the problem is associated with the Emerald serial cards, or some of the Vipers, or is related to the RS232 connections.
• Try various USB interrupt configurations on Vipers and see if we can get the USB interrupt rate to drop, while sending data through a USB Bluetooth radio at data rates that were used at SCP. Could this be related to the issue of data corruption on USB flash drives that we see on Vipers, when two or more USB devices are connected?

Weather

The morning gives us partly cloudy skies, mostly high cirrus. The temperatures throughout the night range from about

-8C to 2C. The winds stayed below 5 m/s and are presently at about 1 to 2 m/s.

Status

morning (7:30)   All stations are up. No issues with the sensors.

Christoph and his crew started teardown today.

afternoon (13:30)  For the afternoon entertainment, Steve O. conducted a tour of the site for Kate, Rudy, and Ned.

During that time Steve O. took a soil sample at the two soils sites and we determined the elevation between sites

16, 18, and 19.

evening (20:45) Still running like the Energizer Bunny! WInds are at ~2 m/s and are dropping.

Weather:

Overcast with light winds. Winds throughout the night peaked around 6 m/s. Temperatures pushed down to 8C.

Status:

11:00:  John left yesterday and Steve S. is now on site. All stations reporting.

21:00:  Everything looking good. I did notice A14 and A17 both went down for a short period, ~5 minutes. I could

not ping them.

Gordon, Nov 27

Chris wants the CSAT3 serial numbers to simplify boxing up the units during teardown. From the log files (/var/log/isfs/*_SCP_csat3.log) on each of the DSMs, here are the reported serial numbers:

csat3: Ah1:/dev/ttyS2, id=1,20, SN0923, 3.0f
csat3: Ah2:/dev/ttyS2, id=2,20, SN0833, 3.0f
csat3: Aph3:/dev/ttyS2, id=3,20, SN0743, 3.0f
csat3: A4:/dev/ttyS2, id=4,20, SN1120, 3.0f
csat3: Ah5:/dev/ttyS2, id=5,20, SN0732, 3.0f
csat3: Ah6:/dev/ttyS2, id=6,20, SN0800, 3.0f
csat3: A7:/dev/ttyS2, id=7,20, SN0673, 3.0f
csat3: Ap8:/dev/ttyS2, id=8,20, SN0176, 3.0f
csat3: Ap9:/dev/ttyS2, id=9,20, SN1121, 3.0f
csat3: Ap10:/dev/ttyS2, id=10,20, SN0677, 3.0f
csat3: Ars11:/dev/ttyS2, id=11,20, SN0674, 3.0f
csat3: Ap12:/dev/ttyS2, id=12,20, SN0855, 3.0f
csat3: A13:/dev/ttyS2, id=13,20, SN0745, 3.0f
csat3: Ap14:/dev/ttyS2, id=14,20, SN1124, 3.0f
csat3: A15:/dev/ttyS2, id=15,20, SN1122, 3.0f
csat3: A16:/dev/ttyS2, id=16,20, SN0740, 3.0f
csat3: A17:/dev/ttyS2, id=17,20, SN0856, 3.0f
csat3: A18:/dev/ttyS2, id=18,20, SN0744, 3.0f
csat3: A19:/dev/ttyS2, id=19,20, SN0178, 3.0f

csat3: C20:/dev/ttyS5, id=20,100, SN0200, 3.0f (1 m)
csat3: C20:/dev/ttyS8, id=20,200, SN0197, 3.0f (2 m)

csat3: M21:/dev/ttyS1, id=21,50, SN1455, 3.0f
csat3: M21:/dev/ttyS5, id=21,100, SN1117, 3.0f
csat3: M21:/dev/ttyS9, id=21,200, SN0536, 3.0f
csat3: M21:/dev/ttyS12, id=21,300, SN0540, 3.0f
csat3: M21:/dev/ttyS14, id=21,400, SN0738, 3.0f
csat3: M21:/dev/ttyS16, id=21,500, SN0537, 3.0f

csat3: Mu22:/dev/ttyS7, id=22,1000, SN1119, 3.0f
csat3: Mu22:/dev/ttyS10, id=22,2000, SN0741, 3.0f

Chris G. and Larry present

Weather:

Mostly sunny skies w/ cirrus.  Temps just above 10C and calm winds.  Looks like a good day to play frisbee golf, using the towers as holes.

Status:

Nothing new in this category.  Things are running great and smooth.  We will probably run around to some towers to check on configurations to see if we can add barometers.  We have the new nano barometers and EC150 up here now.  

To Do:

Work on EC150 mounting

Clean KH20, Licor and Radiometers

Status:

All stations reporting in am

On-Site: John

Weather: Clearing, fairly calm overnight.   Roughly 40 cirrus cover in am.   Chilly, temps dropped to -8 to -10 since roughly 10-11pm last night.   Winds fairly light roughly <2 from west.    No evidence of fog, dew, but very light frost with humidity rather high, 80%ish.   

Gully:  Pocketec local storage swapped.   Out #11 with data from 11/21 through 26th.   In #8

Spare DSM gray box (unlabeled) returning to Boulder for misc. testing support there.

Sensors:

Sensors:  Looking good.   A few diag hits on ec150 sonic "low signal" but data ok.

Radiometers, Licors Cleaned: 8:55-9:05 MST,  with methanol.   Frost was on uplooking Rsw but didn't appear to be any on Rlw.    Rsw.both desiccant was shot: replaced 9:40-9:50MST.

Status

morning (7:30):  All stations reporting. Temperatures are dropping and the humidity is up.    Overcast and maybe some fog in area in am.   Winds were up since yesterday afternoon and still 2-5ms this morning shifting to e/ne.   Not a good last night for science. 

Remainder of day, overcast continued, winds remained low in afternoon and temps through the day were cooler hovering around freezing+/-.   All stations remain up.

Boom angles:  Yet another set was shot.   Data scope was calibrated using camera tripod before hand, it reported "good = 8".   Didn't complete all shots: getting dark while doing main.   Some of my Handar reading were 'off' because of sighting the wrong angle, so they should be repeated.  jm

On-Site: John.

STATUS

Things still looking good. Wind speeds picked back up for awhile before midnight shifting south (frontal) and back to west with velocities 1-3 -ish rest of the night.   Temps to -3 to -5 -ish overnight

No one on site today.

Local store pocketec at 55% full.

STATUS

Things look good. Wind speed throughout the night stayed below 5 m/s.

John may visit the site today.

Afternoon (13:30) : John payed a visit to the site. Said all was well with winds pushing 10m/s.

~14:10-14:20 5m,10m Kryptons cleaned, JM.   The voltages didn't improve and if anything the 10m went down.   I cleaned twice with q-tips/h2o and wiped off as well but perhaps there was something on the cotton or perhaps i touched the cotton with my glove but i don't think so.   The concentration went up on the 10m and is tracking that of the 5m since.   Otherwise things still looking good.

Local store pocketec at 55% full.

Evening (20:00): Winds topped off at about 12 - 14 m/s. They are now down around 2 m/s. All stations reporting ok.

Daily status

We have entered the remote monitoring stage of SCP due to the Thanksgiving Holiday period.

John and Steve S. will monitor the data remotely until Monday, Nov. 26 when John will go to the site.

Everything looking good this morning. I did notice there is no T.baro from the new nano baro sites; #5, #10, #21.

This is expected since the sensor is not putting out that parameter.

M21 down:  Noticed after entering this log that m21 went down. I was able to ping to the station. The data collection process

was not running. An 'aup'  got it going.

AFTERNOON (16:30): All stations running. All sensors look good. Very light winds, <2m/s.

weather: morning was clear and calm -beautiful day

afternoon about  12:30 wind picked up a bit for awhile, but only a few m/s, clouds rolled in and is mostly overcast about 2:00pm

all stations are up and reporting

replaced pocket tek to bring back to Boulder

cleaned EC 100 at 1:37

cleaned licors at 1:39

cleaned radiometers at 1:42

shot all angles for sonics - see earlier entery for updated angles

rebooted Christophs system at about 12:15 - all looks ok per Chadi

weather:  clear,light winds

all stations up and reporting

weather: overcast all morning, calm winds

all stations up and reporting

Mark came by today to observe tower safety, and I was able to clean the KH2O's (10 & 5m) at 11:30 local time

I've gotten the impression by looking at cockpit during the last hour that the winds were West, then switched to SE, then back to W.

However, since this shift also occurred at 20m, it probably isn't related to local gully topography.