4m TRH at Marshall started generating intermittent data starting around 00:00 LT, May 30, and appears to have quit altogether by 04:00

If you let it run long enough, data_stats shows a record length of 8191 bytes which indicates that the data system is not finding a terminator (newline) in the data:

marshall:/dev/ttyS11 2 410 1 2009 06 01 17:18:06.062 06 01 17:18:06.062 nan 0.000 0.000 8191 8191

data_dump shows that the data is all NULLs (0x00):

cd /data/projects/ASP09/raw_data:
data_dump -i 2,410 -A isff_20090601_160000.dat

The serial driver is reporting a bazillion "break" events for that port:

ssh root@marshall
more /proc/tty/driver/serial
...
11: uart:ST16654 port:F1000170 irq:104 tx:13 rx:129438833 fe:32 brk:109632291 RTS|DTR

I recommend temporarily connecting a working TRH to port 11 and see if things look good.

At May 30 09:32 UTC (03:32 MDT) this TRH died until 03:49 MDT, then
looks like it reset itself, and worked until 03:56 MDT, and then died
completely. It did this same thing several times between 00:00
and 03:32 MDT.

2009 05 30 09:32:20.9023 0.8324 31 TRH013 16.73 37.69 5626 1173\r\n
2009 05 30 09:32:21.7418 0.8395 31 TRH013 16.72 37.73 5625 1174\r\n
2009 05 30 09:32:22.5737 0.8319 33 TRH013 16.71 37.73 5624 11\x07\x00\x00\x00\x00\n
2009 05 30 09:49:48.7838 1046 17 \r Sensor TRH013\n
2009 05 30 09:49:48.8031 0.01928 29 \rcalibration coefficients:\r\n
2009 05 30 09:49:48.8528 0.04968 21 Ta0 = -4.021784E+1\r\n
2009 05 30 09:49:48.8914 0.03862 21 Ta1 = 1.025356E-2\r\n
2009 05 30 09:49:48.9344 0.04306 21 Ta2 = -2.321813E-8\r\n
2009 05 30 09:49:48.9712 0.03673 21 Ha0 = -9.861796E+0\r\n
2009 05 30 09:49:49.0133 0.0421 21 Ha1 = 4.218116E-2\r\n
2009 05 30 09:49:49.0527 0.03948 21 Ha2 = -3.237849E-6\r\n
2009 05 30 09:49:49.0916 0.03889 21 Ha3 = 8.039271E-2\r\n
2009 05 30 09:49:49.1396 0.04793 21 Ha4 = 6.041142E-5\r\n
2009 05 30 09:49:49.4596 0.3201 31 TRH013 16.26 39.07 5579 1214\r\n
2009 05 30 09:49:50.2896 0.83 31 TRH013 16.25 39.14 5578 1216\r\n