The plot of GPSdiff_max for May 22, 2011 shows a maximum value of around 25 seconds between 11:05 and 17:00 UTC, indicating something went haywire with the data collection during those times.
Looking at the downloaded data, it appears that data stopped coming in on the Diamond serial ports at 11:05, and resumed at 17:00. There is not a large gap in data from the 3 sensors that are sampled on the Viper serial ports, but they show large time tag delta-Ts, followed by many small delta-Ts, symptomatic of the system getting behind and having to do major buffering. Perhaps a "storm" of interrupts from the Diamond that finally cleared up? Moisture somewhere?
I don't have time to look into it at this time, but here are some dumps of data around the problem.
data_stats /scr/isfs/projects/BEACHON_SRM/raw_data/manitou_20110522_000000.dat
2011-05-24,16:56:30|INFO|opening: /scr/isfs/projects/BEACHON_SRM/raw_data/manitou_20110522_000000.dat
EOFException: /scr/isfs/projects/BEACHON_SRM/raw_data/manitou_20110522_000000.dat: open: EOF
sensor dsm sampid nsamps |------- start -------| |------ end -----| rate minMaxDT(sec) minMaxLen
1 20 39901 2011 05 22 00:00:00.998 05 22 11:05:01.461 1.00 0.336 1.643 19 19
1 30 84632 2011 05 22 00:00:00.571 05 22 11:59:59.719 1.96 0.042 26.076 51 73
1 100 843525 2011 05 22 00:00:00.160 05 22 11:59:59.944 19.53 -2.052 27.951 12 16
1 110 415176 2011 05 22 00:00:00.113 05 22 11:59:59.959 9.61 -2.016 28.058 39 87
1 120 41042 2011 05 22 00:00:00.185 05 22 11:05:01.388 1.03 0.294 1.643 29 31
1 200 798028 2011 05 22 00:00:00.166 05 22 11:05:01.873 20.00 0.005 0.596 12 12
1 220 40687 2011 05 22 00:00:00.791 05 22 11:05:01.158 1.02 0.357 1.604 28 29
1 300 798048 2011 05 22 00:00:00.168 05 22 11:05:01.884 20.00 0.002 0.585 12 12
1 310 399023 2011 05 22 00:00:00.000 05 22 11:05:01.841 10.00 0.042 0.454 44 49
1 320 40488 2011 05 22 00:00:00.233 05 22 11:05:01.428 1.01 0.356 1.613 28 29
1 330 7981 2011 05 21 23:59:55.213 05 22 11:04:55.250 0.20 4.415 5.586 49 60
1 400 798041 2011 05 22 00:00:00.005 05 22 11:05:01.883 20.00 0.001 0.591 12 12
1 420 40772 2011 05 22 00:00:00.328 05 22 11:05:01.192 1.02 0.344 1.614 29 30
1 500 798042 2011 05 22 00:00:00.171 05 22 11:05:01.913 20.00 0.002 0.603 12 12
1 510 399020 2011 05 22 00:00:00.007 05 22 11:05:01.814 10.00 0.037 0.436 44 49
1 520 40747 2011 05 22 00:00:00.342 05 22 11:05:01.149 1.02 0.372 1.588 27 29
[maclean@porter ~]$ data_stats /scr/isfs/projects/BEACHON_SRM/raw_data/manitou_20110522_120000.dat
2011-05-24,16:57:04|INFO|opening: /scr/isfs/projects/BEACHON_SRM/raw_data/manitou_20110522_120000.dat
EOFException: /scr/isfs/projects/BEACHON_SRM/raw_data/manitou_20110522_120000.dat: open: EOF
sensor dsm sampid nsamps |------- start -------| |------ end -----| rate minMaxDT(sec) minMaxLen
1 20 25156 2011 05 22 17:00:47.215 05 22 23:59:59.960 1.00 0.019 1.371 19 29
1 30 73288 2011 05 22 12:00:25.745 05 22 23:59:59.741 1.70 0.104 26.116 51 73
1 100 750227 2011 05 22 12:00:28.065 05 22 23:59:59.959 17.38 -2.052 27.951 12 16
1 110 337790 2011 05 22 12:00:28.120 05 22 23:59:59.922 7.82 -2.016 28.058 17 69
1 120 25895 2011 05 22 17:00:47.289 05 22 23:59:59.553 1.03 0.029 1.756 29 38
1 200 502634 2011 05 22 17:01:05.737 05 22 23:59:59.999 20.00 0.000 2.518 2 466
1 220 25715 2011 05 22 17:00:47.290 05 22 23:59:59.392 1.02 0.028 1.594 28 39
1 300 502351 2011 05 22 17:01:20.419 05 22 23:59:59.968 20.00 0.000 2.538 2 465
1 310 251531 2011 05 22 17:00:47.284 05 22 23:59:59.926 10.00 0.050 0.154 44 66
1 320 25590 2011 05 22 17:00:47.291 05 22 23:59:59.734 1.02 0.028 1.315 28 39
1 330 5032 2011 05 22 11:05:00.251 05 22 23:59:50.264 0.11 0.061 21347.049 6 61
1 400 502055 2011 05 22 17:01:35.059 05 22 23:59:59.987 20.00 0.001 2.508 3 466
1 420 25769 2011 05 22 17:00:47.292 05 22 23:59:59.864 1.02 0.029 1.523 29 37
1 500 502936 2011 05 22 17:00:45.341 05 22 23:59:59.998 19.99 0.003 4.123 12 466
1 510 251530 2011 05 22 11:05:01.916 05 22 23:59:59.963 5.41 0.027 21345.401 43 50
1 520 25730 2011 05 22 17:00:47.294 05 22 23:59:59.423 1.02 0.028 1.631 26 39
|
Data kept coming in on the Viper serial ports (ids=30,100,110, GPS, sonic and licor at 2m), but had time tag issues, apparently because buffers filled up. Here's a snippet of the 2m sonic data in hex, showing a 28 second time tag jump, but no skip in the sonic sequence (byte 9: f8,f9,fa,fb,fc, etc)
2011 05 22 11:05:53.8243 0.05004 12 20 06 0b 03 39 ff 6e e8 f5 0f 55 aa 2011 05 22 11:05:53.8743 0.04997 12 6a 06 0f 03 1b ff 6a e8 f6 0f 55 aa 2011 05 22 11:05:53.9243 0.05002 12 bf 05 b1 02 de fe 69 e8 f7 0f 55 aa 2011 05 22 11:05:53.9743 0.04998 12 4d 05 cb 02 d0 fe 6e e8 f8 0f 55 aa 2011 05 22 11:06:21.9147 27.94 12 35 05 d6 02 df fe 71 e8 f9 0f 55 aa 2011 05 22 11:06:21.9272 0.0125 12 03 06 7f 03 30 ff 6f e8 fa 0f 55 aa 2011 05 22 11:06:21.9397 0.0125 12 32 06 7f 03 98 ff 65 e8 fb 0f 55 aa 2011 05 22 11:06:21.9522 0.0125 12 46 05 59 02 35 ff 6d e8 fc 0f 55 aa 2011 05 22 11:06:21.9647 0.0125 12 83 05 1a 03 c3 fe 75 e8 fd 0f 55 aa |
After the gap, the 0.0125 dTs result from unpacking multiple samples from a buffer, where the code computes a dT = 10 bits/byte / (9600 bits/sec) * 12 bytes/sample = 0.0125 sec/sample
I see no indication of a problem in the system logs, or any weird clock adjustments in the NTP loopstats,peerstats.