DSM interrupt loads

Gordon, Nov 29

ssh'd into the systems, and checked the interrupt rates With the "intcount" command.

On vipers and titans, ttyS1, 2 and 3 are the serial ports on the CPU board. On both systems ttyS1 is served by interrupt 37, ttyS2 by interrupt 36. On vipers, ttyS3 is interrupt 116. On titans, ttyS3 is interrupt 122

The Emerald serial expansion board(s), serving serial ports ttyS5 to tty12 (and ttyS13-ttyS20 on m21) are configured to use ISA interrupt 3. The PC104 interrupts (in this case just IRQ 3) get multiplexed by a CPLD on the CPU board into a GPIO interrupt. On vipers, the PC104/GPIO interrupt is number 25, GPIO line 01. On titans, the PC104/GPIO interrupt is number 129, on GPIO 17.

Interrupts/second

Notes:
1: These vipers have large anomalous pc104 interrupt rates, which occur with both Sep 14 (a1,a8) and the Oct 4 (a5) kernels.
2: Not completely sure why on m21 the IRQ3 rate is less than the pc104 load. I may have installed a kernel on that system with a pc104 irq routine that exits if pending bits are 0, instead of attempting to serve he unmasked interrupts.
3: High USB interrupts are seen on vipers with bluetooth radios interfaced via USB. High USB interrupts are not seen on Titans, even though on "A" site titans, USB serves both the bluetooth radios and flash drives. On "A" site Vipers, USB has only bluetooth radios. Viper USB driver is isp116x_hcd. Titan USB driver is ohci_hcd.
4: On c20,m21, m22, USB is used only for flash drives

The big question is why the huge PC104 interrupt rates on vipers at a1, a5, a8 and m21. Previously traced this to a floating CTS/RTS line, but I'm not sure that is the current reason. Saw high PC104 interrupt rate on m21 even with the ribbon cables disconnected from the Emerald headers! Will have to investigate those systems back in the lab.

The only titans with PC104 serial are a3 and m22. They're both OK.

PC104 interrupts for vipers at a2, a6 and c20 are OK.

USB Interrupts

The viper kernel has a patch to use an assembler delay function for the isp116x, and not the kernel ndelay() function. It is not known whether this is related to the high interrupt rate. An incorrect delay may result in the USB interface completely failing.

The driver code that sets up the interrupt, uses two configuration values, int_act_high and int_edge_triggered, which are used to configure the interface:

       if (board->int_act_high)
                val |= HCHWCFG_INT_POL;
        if (board->int_edge_triggered)
                val |= HCHWCFG_INT_TRIGGER;

The current isp116x driver is configured for level triggered (int_edge_triggered=0, default) and active high (int_act-high=1) interrupts. However, the setting for the corresponding GPIO interrupt is IORESOURCE_IRQ_HIGHEDGE. This might mean the interrupt is not being acknowledged correctly.

Planned Testing

• Figure out what might be causing the high PC104 interrupt load on some vipers. By swapping cards and looking at IRQ3 level on a scope, figure out whether the problem is associated with the Emerald serial cards, or some of the Vipers, or is related to the RS232 connections.
• Try various USB interrupt configurations on Vipers and see if we can get the USB interrupt rate to drop, while sending data through a USB Bluetooth radio at data rates that were used at SCP. Could this be related to the issue of data corruption on USB flash drives that we see on Vipers, when two or more USB devices are connected?