Blog

re-sorted 2 merged archive files
Gordon Maclean posted on Jan 02, 2013

Two merged archive files have been found with fairly big blocks of backwards samples.

The statsproc process logs the following after every 100 backwards samples, and many were seen in the logs of processing the PCAPS data.

2013-01-01,22:45:17|WARNING|NearestResampler: sample id 2,102 backwards by 2325.2 sec at 2011 01 26 22:21:05.469201

I re-sorted the data with a 1 hour sorter in nidsmerge:

nidsmerge -s "2011 01 29 16:00" -e "2011 01 30 00:00" -i isfs_20110129_160000.dat.bz2 -r 3600 -o $s/isfs_%Y%m%d_%H%M%S.dat.bz2 -l $((8*3600))

The files have been replaced on /scr/isfs and on the HPSS.

Here are data_stats listing of the old and new files.

data_stats isfs_20110126_160000.dat.bz2
2013-01-02,09:30:18\|NOTICE\|parsing: /net/isf/isff/projects/PCAPS/ISFF/config/pcaps.xml
Exception: EOFException: isfs_20110126_160000.dat.bz2: open: EOF
sensor                   dsm sampid    nsamps \|------\- start \-----\--\|  \|-\----\- end \-----\|    rate    minMaxDT(sec) minMaxLen
isfs1:/dev/ttyS2           1     20     30083 2011 01 26 16:00:00.138  01 26 23:59:59.726    1.04-2332.729 890.893   19   19
isfs1:/var/tmp/gps_pty0    1     30     60658 2011 01 26 16:00:00.500  01 26 23:59:59.708    2.11-2333.150 890.859   51   73
isfs1:/dev/ttyS4           1     40     30604 2011 01 26 16:00:00.559  01 26 23:59:59.606    1.06-2332.339 890.471   18   18
isfs1:/dev/ttyS5           1     50     27235 2011 01 26 16:00:00.201  01 26 23:59:59.538    0.95-2332.128 891.041   21   39
isfs1:/dev/ttyS1           1    100   1804701 2011 01 26 16:00:00.004  01 26 23:59:59.995   62.66-2333.789 890.018   14   14
isfs1:/dev/ttyS6           1  32768     18054 2011 01 26 16:00:02.016  01 26 23:59:59.688    0.63-2332.560 892.475   21   80
isfs2:/dev/ttyS2           2     20     30082 2011 01 26 16:00:00.403  01 26 23:59:59.191    1.04-2333.743 890.905   20   20
isfs2:/var/tmp/gps_pty0    2     30     60157 2011 01 26 16:00:00.505  01 26 23:59:59.742    2.09-2334.145 890.857   72   73
isfs2:/dev/ttyS5           2     50     30729 2011 01 26 16:00:00.105  01 26 23:59:59.569    1.07-2333.530 891.714   29   31
isfs2:/dev/ttyS1           2    100   1804650 2011 01 26 16:00:00.006  01 26 23:59:59.989   62.66-2334.522 890.388   14   14
isfs2:/dev/ttyS6           2  32768     18062 2011 01 26 16:00:01.207  01 26 23:59:58.369    0.63-2330.950 894.940    6  116
isfs3:/dev/ttyS2           3     20     30084 2011 01 26 16:00:00.059  01 26 23:59:59.855    1.04-2333.907 890.009   20   20
isfs3:/var/tmp/gps_pty0    3     30     60160 2011 01 26 16:00:00.501  01 26 23:59:59.730    2.09-2333.186 890.829   72   73
isfs3:/dev/ttyS5           3     50     30730 2011 01 26 16:00:00.602  01 26 23:59:59.258    1.07-2333.224 890.619   25   30
isfs3:/dev/ttyS8           3     80      6011 2011 01 26 16:00:03.430  01 26 23:59:55.246    0.21-2330.504 895.250    9    9
isfs3:/dev/ttyS1           3    100   1804820 2011 01 26 16:00:00.004  01 26 23:59:59.994   62.67-2334.070 889.825   14   14
isfs3:/dev/ttyS6           3  32768     18062 2011 01 26 16:00:03.144  01 26 23:59:58.299    0.63-2330.892 894.098    7  106
isfs4:/dev/ttyS2           4     20     30088 2011 01 26 16:00:00.189  01 26 23:59:59.753    1.04-2333.458 891.704   20   20
isfs4:/var/tmp/gps_pty0    4     30     60656 2011 01 26 16:00:00.497  01 26 23:59:59.752    2.11-2333.119 890.881   51   73
isfs4:/dev/ttyS5           4     50     30744 2011 01 26 16:00:00.379  01 26 23:59:59.972    1.07-2332.118 891.174   29   30
isfs4:/dev/ttyS8           4     80      5998 2011 01 26 16:00:04.099  01 26 23:59:59.640    0.21-2329.348 896.573    9    9
isfs4:/dev/ttyS1           4    100   1804616 2011 01 26 16:00:00.014  01 26 23:59:59.987   62.66-2333.565 890.692   14   14
isfs4:/dev/ttyS6           4  32768     18170 2011 01 26 16:00:00.547  01 26 23:59:59.591    0.63-2330.193 894.345   13   80
isfs5:/dev/ttyS2           5     20     30082 2011 01 26 16:00:00.317  01 26 23:59:59.481    1.04-2332.861 890.942   19   19
isfs5:/var/tmp/gps_pty0    5     30     60659 2011 01 26 16:00:00.486  01 26 23:59:59.700    2.11-2333.165 890.869   51   73
isfs5:/dev/ttyS5           5     50     30858 2011 01 26 16:00:00.008  01 26 23:59:59.130    1.07-2332.543 891.871   27   30
isfs5:/dev/ttyS1           5    100   1804680 2011 01 26 16:00:00.011  01 26 23:59:59.995   62.66-2333.643 890.550   14   14
isfs5:/dev/ttyS6           5  32768     24082 2011 01 26 16:00:00.057  01 26 23:59:59.848    0.84-2331.946 891.964   22   80
isfs6:/dev/ttyS2           6     20     30077 2011 01 26 16:00:00.794  01 26 23:59:59.165    1.04-2331.884 891.950   19   19
isfs6:/var/tmp/gps_pty0    6     30     60657 2011 01 26 16:00:00.512  01 26 23:59:59.755    2.11-2333.168 890.929   51   73
isfs6:/dev/ttyS4           6     40     30601 2011 01 26 16:00:00.442  01 26 23:59:59.043    1.06-2332.396 891.479   18   18
isfs6:/dev/ttyS5           6     50     30813 2011 01 26 16:00:00.794  01 26 23:59:59.555    1.07-2332.146 891.269   29   30
isfs6:/dev/ttyS8           6     80      6023 2011 01 26 16:00:01.303  01 26 23:59:58.299    0.21-2326.959 893.833    9    9
isfs6:/dev/ttyS1           6    100   1804581 2011 01 26 16:00:00.004  01 26 23:59:59.985   62.66-2333.460 890.460   14   14
isfs6:/dev/ttyS6           6  32768     24069 2011 01 26 16:00:00.322  01 26 23:59:59.868    0.84-2333.189 891.923   22   80
isfs7:/dev/ttyS2           7     20     30081 2011 01 26 16:00:00.874  01 26 23:59:59.581    1.04-2332.898 890.961   19   19
isfs7:/var/tmp/gps_pty0    7     30     60660 2011 01 26 16:00:00.513  01 26 23:59:59.724    2.11-2333.152 890.841   51   73
isfs7:/dev/ttyS5           7     50     30870 2011 01 26 16:00:00.224  01 26 23:59:59.083    1.07-2332.989 890.713   29   30
isfs7:/dev/ttyS1           7    100   1804759 2011 01 26 16:00:00.012  01 26 23:59:59.988   62.67-2333.731 890.255   14   14
isfs7:/dev/ttyS6           7  32768     18060 2011 01 26 16:00:01.147  01 26 23:59:57.571    0.63-2330.000 891.367    7  112

data_stats /scr/tmp/maclean/isfs_20110126_160000.dat.bz2 
2013-01-02,10:37:32|NOTICE|parsing: /net/isf/isff/projects/PCAPS/ISFF/config/pcaps.xml
Exception: EOFException: /scr/tmp/maclean/isfs_20110126_160000.dat.bz2: open: EOF
sensor                   dsm sampid    nsamps |------- start -------|  |------ end -----|    rate  minMaxDT(sec) minMaxLen
isfs1:/dev/ttyS2           1     20     28804 2011 01 26 16:00:00.138  01 26 23:59:59.726    1.00  0.724   1.277   19   19
isfs1:/var/tmp/gps_pty0    1     30     58080 2011 01 26 16:00:00.500  01 26 23:59:59.708    2.02  0.033   1.299   51   73
isfs1:/dev/ttyS4           1     40     29302 2011 01 26 16:00:00.559  01 26 23:59:59.606    1.02  0.431   1.529   18   18
isfs1:/dev/ttyS5           1     50     25927 2011 01 26 16:00:00.201  01 26 23:59:59.538    0.90  0.029 103.568   21   39
isfs1:/dev/ttyS1           1    100   1727994 2011 01 26 16:00:00.004  01 26 23:59:59.995   60.00  0.000   0.134   14   14
isfs1:/dev/ttyS6           1  32768     17287 2011 01 26 16:00:02.016  01 26 23:59:59.688    0.60  0.007   3.473   21   80
isfs2:/dev/ttyS2           2     20     28803 2011 01 26 16:00:00.403  01 26 23:59:59.191    1.00  0.020   2.335   20   20
isfs2:/var/tmp/gps_pty0    2     30     57600 2011 01 26 16:00:00.505  01 26 23:59:59.742    2.00  0.014   2.254   72   73
isfs2:/dev/ttyS5           2     50     29424 2011 01 26 16:00:00.105  01 26 23:59:59.569    1.02  0.029   2.104   29   31
isfs2:/dev/ttyS1           2    100   1728020 2011 01 26 16:00:00.006  01 26 23:59:59.989   60.00  0.000   0.277   14   14
isfs2:/dev/ttyS6           2  32768     17292 2011 01 26 16:00:01.207  01 26 23:59:58.369    0.60  0.004   4.133    6  116
isfs3:/dev/ttyS2           3     20     28802 2011 01 26 16:00:00.059  01 26 23:59:59.855    1.00  0.003   1.979   20   20
isfs3:/var/tmp/gps_pty0    3     30     57600 2011 01 26 16:00:00.501  01 26 23:59:59.730    2.00  0.029   1.627   72   73
isfs3:/dev/ttyS5           3     50     29422 2011 01 26 16:00:00.602  01 26 23:59:59.258    1.02  0.494   1.466   25   30
isfs3:/dev/ttyS8           3     80      5755 2011 01 26 16:00:03.430  01 26 23:59:55.246    0.20  4.041   5.959    9    9
isfs3:/dev/ttyS1           3    100   1728005 2011 01 26 16:00:00.004  01 26 23:59:59.994   60.00  0.000   0.164   14   14
isfs3:/dev/ttyS6           3  32768     17290 2011 01 26 16:00:03.144  01 26 23:59:58.299    0.60  0.001   4.864    7  106
isfs4:/dev/ttyS2           4     20     28809 2011 01 26 16:00:00.189  01 26 23:59:59.753    1.00  0.020   2.763   20   20
isfs4:/var/tmp/gps_pty0    4     30     58080 2011 01 26 16:00:00.497  01 26 23:59:59.752    2.02  0.067   2.303   51   73
isfs4:/dev/ttyS5           4     50     29438 2011 01 26 16:00:00.379  01 26 23:59:59.972    1.02  0.030   2.823   29   30
isfs4:/dev/ttyS8           4     80      5745 2011 01 26 16:00:04.099  01 26 23:59:59.640    0.20  4.989   5.033    9    9
isfs4:/dev/ttyS1           4    100   1727993 2011 01 26 16:00:00.014  01 26 23:59:59.987   60.00  0.000   0.269   14   14
isfs4:/dev/ttyS6           4  32768     17403 2011 01 26 16:00:00.547  01 26 23:59:59.591    0.60  0.003   5.149   13   80
isfs5:/dev/ttyS2           5     20     28802 2011 01 26 16:00:00.317  01 26 23:59:59.481    1.00  0.049   1.952   19   19
isfs5:/var/tmp/gps_pty0    5     30     58080 2011 01 26 16:00:00.486  01 26 23:59:59.700    2.02  0.132   1.531   51   73
isfs5:/dev/ttyS5           5     50     29547 2011 01 26 16:00:00.008  01 26 23:59:59.130    1.03  0.360   1.591   27   30
isfs5:/dev/ttyS1           5    100   1727995 2011 01 26 16:00:00.011  01 26 23:59:59.995   60.00  0.000   0.153   14   14
isfs5:/dev/ttyS6           5  32768     23061 2011 01 26 16:00:00.057  01 26 23:59:59.848    0.80  0.004   3.022   22   80
isfs6:/dev/ttyS2           6     20     28801 2011 01 26 16:00:00.794  01 26 23:59:59.165    1.00  0.019   2.175   19   19
isfs6:/var/tmp/gps_pty0    6     30     58080 2011 01 26 16:00:00.512  01 26 23:59:59.755    2.02  0.029   1.625   51   73
isfs6:/dev/ttyS4           6     40     29301 2011 01 26 16:00:00.442  01 26 23:59:59.043    1.02  0.018   2.367   18   18
isfs6:/dev/ttyS5           6     50     29505 2011 01 26 16:00:00.794  01 26 23:59:59.555    1.02  0.002   2.223   29   30
isfs6:/dev/ttyS8           6     80      5768 2011 01 26 16:00:01.303  01 26 23:59:58.299    0.20  4.381   5.605    9    9
isfs6:/dev/ttyS1           6    100   1727971 2011 01 26 16:00:00.004  01 26 23:59:59.985   60.00  0.000   0.272   14   14
isfs6:/dev/ttyS6           6  32768     23046 2011 01 26 16:00:00.322  01 26 23:59:59.868    0.80  0.001   3.838   22   80
isfs7:/dev/ttyS2           7     20     28801 2011 01 26 16:00:00.874  01 26 23:59:59.581    1.00  0.452   1.548   19   19
isfs7:/var/tmp/gps_pty0    7     30     58080 2011 01 26 16:00:00.513  01 26 23:59:59.724    2.02  0.023   1.666   51   73
isfs7:/dev/ttyS5           7     50     29555 2011 01 26 16:00:00.224  01 26 23:59:59.083    1.03  0.225   1.716   29   30
isfs7:/dev/ttyS1           7    100   1727986 2011 01 26 16:00:00.012  01 26 23:59:59.988   60.00  0.000   0.157   14   14
isfs7:/dev/ttyS6           7  32768     17292 2011 01 26 16:00:01.147  01 26 23:59:57.571    0.60  0.000   4.963    7  112

data_stats isfs_20110129_160000.dat.bz2 
2013-01-02,10:42:50|NOTICE|parsing: /net/isf/isff/projects/PCAPS/ISFF/config/pcaps.xml
Exception: EOFException: isfs_20110129_160000.dat.bz2: open: EOF
sensor                   dsm sampid    nsamps |------- start -------|  |------ end -----|    rate     minMaxDT(sec) minMaxLen
isfs1:/dev/ttyS2           1     20     29000 2011 01 29 16:00:00.073  01 29 23:59:59.629    1.01-1741.788 1547.804   19   19
isfs1:/var/tmp/gps_pty0    1     30     58475 2011 01 29 16:00:00.503  01 29 23:59:59.719    2.03-1743.099 1548.802   51   73
isfs1:/dev/ttyS4           1     40     29501 2011 01 29 16:00:00.473  01 29 23:59:59.487    1.02-1741.648 1548.014   18   18
isfs1:/dev/ttyS5           1     50     29685 2011 01 29 16:00:00.033  01 29 23:59:59.150    1.03-1742.440 1549.053   29   30
isfs1:/dev/ttyS1           1    100   1739733 2011 01 29 16:00:00.011  01 29 23:59:59.990   60.41-1743.284 1547.603   14   14
isfs1:/dev/ttyS6           1  32768     17405 2011 01 29 16:00:03.399  01 29 23:59:59.859    0.60-1741.478 1553.678    7  123
isfs2:/dev/ttyS2           2     20     28999 2011 01 29 16:00:00.451  01 29 23:59:59.251    1.01-1742.803 1548.830   20   20
isfs2:/var/tmp/gps_pty0    2     30     57992 2011 01 29 16:00:00.520  01 29 23:59:59.721    2.01-1743.136 1548.758   72   73
isfs2:/dev/ttyS5           2     50     29624 2011 01 29 16:00:00.833  01 29 23:59:59.512    1.03-1742.302 1548.496   29   30
isfs2:/dev/ttyS1           2    100   1739768 2011 01 29 16:00:00.010  01 29 23:59:59.993   60.41-1743.354 1547.614   14   14
isfs2:/dev/ttyS6           2  32768     17410 2011 01 29 16:00:01.412  01 29 23:59:59.279    0.60-1739.993 1549.991   21   79
isfs3:/dev/ttyS2           3     20     28997 2011 01 29 16:00:00.393  01 29 23:59:59.236    1.01-1742.872 1548.937   20   20
isfs3:/var/tmp/gps_pty0    3     30     57992 2011 01 29 16:00:00.525  01 29 23:59:59.715    2.01-1743.127 1548.869   72   73
isfs3:/dev/ttyS5           3     50     29624 2011 01 29 16:00:00.176  01 29 23:59:59.515    1.03-1742.966 1549.202   29   30
isfs3:/dev/ttyS8           3     80      5796 2011 01 29 16:00:03.156  01 29 23:59:56.543    0.20-1734.665 1549.710    9    9
isfs3:/dev/ttyS1           3    100   1739768 2011 01 29 16:00:00.003  01 29 23:59:59.995   60.41-1743.641 1547.626   14   14
isfs3:/dev/ttyS6           3  32768     17406 2011 01 29 16:00:00.915  01 29 23:59:58.465    0.60-1739.996 1549.990    7  106
isfs4:/dev/ttyS2           4     20     29003 2011 01 29 16:00:00.931  01 29 23:59:59.669    1.01-1742.759 1548.489   20   20
isfs4:/var/tmp/gps_pty0    4     30     58471 2011 01 29 16:00:00.512  01 29 23:59:59.739    2.03-1742.107 1548.797   51   73
isfs4:/dev/ttyS5           4     50     29639 2011 01 29 16:00:00.615  01 29 23:59:59.469    1.03-1742.215 1548.529   29   30
isfs4:/dev/ttyS8           4     80      5787 2011 01 29 16:00:00.624  01 29 23:59:55.437    0.20-1737.845 1552.578    9    9
isfs4:/dev/ttyS1           4    100   1739651 2011 01 29 16:00:00.014  01 29 23:59:59.990   60.40-1742.863 1547.598   14   14
isfs4:/dev/ttyS6           4  32768     17524 2011 01 29 16:00:03.872  01 29 23:59:59.529    0.61-1740.727 1554.277   13   82
isfs5:/dev/ttyS2           5     20     28997 2011 01 29 16:00:00.397  01 29 23:59:59.554    1.01-1741.895 1548.918   19   19
isfs5:/var/tmp/gps_pty0    5     30     58473 2011 01 29 16:00:00.508  01 29 23:59:59.704    2.03-1742.166 1548.915   51   73
isfs5:/dev/ttyS5           5     50     29533 2011 01 29 16:00:00.740  01 29 23:59:59.526    1.03-1742.235 1548.008   28   29
isfs5:/dev/ttyS1           5    100   1739719 2011 01 29 16:00:00.011  01 29 23:59:59.984   60.41-1742.993 1547.616   14   14
isfs5:/dev/ttyS6           5  32768     23212 2011 01 29 16:00:01.708  01 29 23:59:58.034    0.81-1739.995 1549.525   22   80
isfs6:/dev/ttyS2           6     20     28997 2011 01 29 16:00:00.089  01 29 23:59:59.448    1.01-1741.898 1548.919   19   19
isfs6:/var/tmp/gps_pty0    6     30     58473 2011 01 29 16:00:00.518  01 29 23:59:59.698    2.03-1742.108 1548.810   51   73
isfs6:/dev/ttyS4           6     40     29499 2011 01 29 16:00:00.497  01 29 23:59:59.106    1.02-1741.667 1549.023   18   18
isfs6:/dev/ttyS5           6     50     29704 2011 01 29 16:00:00.109  01 29 23:59:59.117    1.03-1741.878 1548.562   29   30
isfs6:/dev/ttyS8           6     80      5806 2011 01 29 16:00:04.617  01 29 23:59:55.740    0.20-1737.703 1552.949    9    9
isfs6:/dev/ttyS1           6    100   1739682 2011 01 29 16:00:00.011  01 29 23:59:59.982   60.41-1742.753 1547.620   14   14
isfs6:/dev/ttyS6           6  32768     23199 2011 01 29 16:00:00.698  01 29 23:59:58.787    0.81-1739.997 1549.902   22   80
isfs7:/dev/ttyS2           7     20     28997 2011 01 29 16:00:00.321  01 29 23:59:59.024    1.01-1741.924 1547.932   19   19
isfs7:/var/tmp/gps_pty0    7     30     58475 2011 01 29 16:00:00.522  01 29 23:59:59.713    2.03-1743.147 1548.820   51   73
isfs7:/dev/ttyS5           7     50     29757 2011 01 29 16:00:00.533  01 29 23:59:59.975    1.03-1742.586 1548.651   29   30
isfs7:/dev/ttyS1           7    100   1739746 2011 01 29 16:00:00.007  01 29 23:59:59.998   60.41-1743.534 1547.602   14   14
isfs7:/dev/ttyS6           7  32768     17405 2011 01 29 16:00:02.423  01 29 23:59:58.715    0.60-1740.449 1554.782   22   87

data_stats $s/isfs_20110129_160000.dat.bz2
2013-01-02,10:55:26|NOTICE|parsing: /net/isf/isff/projects/PCAPS/ISFF/config/pcaps.xml
Exception: EOFException: /scr/tmp/maclean/isfs_20110129_160000.dat.bz2: open: EOF
sensor                   dsm sampid    nsamps |------- start -------|  |------ end -----|    rate minMaxDT(sec) minMaxLen
isfs1:/dev/ttyS2           1     20     28804 2011 01 29 16:00:00.073  01 29 23:59:59.629    1.00  0.577  1.403   19   19
isfs1:/var/tmp/gps_pty0    1     30     58080 2011 01 29 16:00:00.503  01 29 23:59:59.719    2.02  0.106  1.111   51   73
isfs1:/dev/ttyS4           1     40     29302 2011 01 29 16:00:00.473  01 29 23:59:59.487    1.02  0.873  1.087   18   18
isfs1:/dev/ttyS5           1     50     29485 2011 01 29 16:00:00.033  01 29 23:59:59.150    1.02  0.603  1.348   29   30
isfs1:/dev/ttyS1           1    100   1727990 2011 01 29 16:00:00.011  01 29 23:59:59.990   60.00  0.000  0.145   14   14
isfs1:/dev/ttyS6           1  32768     17288 2011 01 29 16:00:03.399  01 29 23:59:59.859    0.60  0.000  4.197    7  123
isfs2:/dev/ttyS2           2     20     28803 2011 01 29 16:00:00.451  01 29 23:59:59.251    1.00  0.020  2.555   20   20
isfs2:/var/tmp/gps_pty0    2     30     57600 2011 01 29 16:00:00.520  01 29 23:59:59.721    2.00  0.132  2.021   72   73
isfs2:/dev/ttyS5           2     50     29424 2011 01 29 16:00:00.833  01 29 23:59:59.512    1.02  0.029  2.303   29   30
isfs2:/dev/ttyS1           2    100   1728021 2011 01 29 16:00:00.010  01 29 23:59:59.993   60.00  0.000  0.278   14   14
isfs2:/dev/ttyS6           2  32768     17293 2011 01 29 16:00:01.412  01 29 23:59:59.279    0.60  0.005  4.471   21   79
isfs3:/dev/ttyS2           3     20     28801 2011 01 29 16:00:00.393  01 29 23:59:59.236    1.00  0.568  1.413   20   20
isfs3:/var/tmp/gps_pty0    3     30     57600 2011 01 29 16:00:00.525  01 29 23:59:59.715    2.00  0.059  1.448   72   73
isfs3:/dev/ttyS5           3     50     29424 2011 01 29 16:00:00.176  01 29 23:59:59.515    1.02  0.116  1.843   29   30
isfs3:/dev/ttyS8           3     80      5757 2011 01 29 16:00:03.156  01 29 23:59:56.543    0.20  4.746  5.265    9    9
isfs3:/dev/ttyS1           3    100   1728005 2011 01 29 16:00:00.003  01 29 23:59:59.995   60.00  0.000  0.146   14   14
isfs3:/dev/ttyS6           3  32768     17289 2011 01 29 16:00:00.915  01 29 23:59:58.465    0.60  0.005  3.229    7  106
isfs4:/dev/ttyS2           4     20     28808 2011 01 29 16:00:00.931  01 29 23:59:59.669    1.00  0.020  3.039   20   20
isfs4:/var/tmp/gps_pty0    4     30     58080 2011 01 29 16:00:00.512  01 29 23:59:59.739    2.02  0.022  2.280   51   73
isfs4:/dev/ttyS5           4     50     29440 2011 01 29 16:00:00.615  01 29 23:59:59.469    1.02  0.029  2.211   29   30
isfs4:/dev/ttyS8           4     80      5748 2011 01 29 16:00:00.624  01 29 23:59:55.437    0.20  4.197  5.804    9    9
isfs4:/dev/ttyS1           4    100   1727993 2011 01 29 16:00:00.014  01 29 23:59:59.990   60.00  0.000  0.285   14   14
isfs4:/dev/ttyS6           4  32768     17406 2011 01 29 16:00:03.872  01 29 23:59:59.529    0.60  0.003  4.496   13   82
isfs5:/dev/ttyS2           5     20     28802 2011 01 29 16:00:00.397  01 29 23:59:59.554    1.00  0.617  1.383   19   19
isfs5:/var/tmp/gps_pty0    5     30     58080 2011 01 29 16:00:00.508  01 29 23:59:59.704    2.02  0.138  1.184   51   73
isfs5:/dev/ttyS5           5     50     29333 2011 01 29 16:00:00.740  01 29 23:59:59.526    1.02  0.146  1.824   28   29
isfs5:/dev/ttyS1           5    100   1727994 2011 01 29 16:00:00.011  01 29 23:59:59.984   60.00  0.001  0.155   14   14
isfs5:/dev/ttyS6           5  32768     23056 2011 01 29 16:00:01.708  01 29 23:59:58.034    0.80  0.001  4.276   22   80
isfs6:/dev/ttyS2           6     20     28802 2011 01 29 16:00:00.089  01 29 23:59:59.448    1.00  0.365  1.635   19   19
isfs6:/var/tmp/gps_pty0    6     30     58080 2011 01 29 16:00:00.518  01 29 23:59:59.698    2.02  0.129  1.574   51   73
isfs6:/dev/ttyS4           6     40     29301 2011 01 29 16:00:00.497  01 29 23:59:59.106    1.02  0.001  1.969   18   18
isfs6:/dev/ttyS5           6     50     29504 2011 01 29 16:00:00.109  01 29 23:59:59.117    1.02  0.154  1.796   29   30
isfs6:/dev/ttyS8           6     80      5767 2011 01 29 16:00:04.617  01 29 23:59:55.740    0.20  4.426  5.561    9    9
isfs6:/dev/ttyS1           6    100   1727972 2011 01 29 16:00:00.011  01 29 23:59:59.982   60.00  0.000  0.165   14   14
isfs6:/dev/ttyS6           6  32768     23043 2011 01 29 16:00:00.698  01 29 23:59:58.787    0.80  0.005  4.236   22   80
isfs7:/dev/ttyS2           7     20     28801 2011 01 29 16:00:00.321  01 29 23:59:59.024    1.00  0.006  2.144   19   19
isfs7:/var/tmp/gps_pty0    7     30     58080 2011 01 29 16:00:00.522  01 29 23:59:59.713    2.02  0.015  1.349   51   73
isfs7:/dev/ttyS5           7     50     29556 2011 01 29 16:00:00.533  01 29 23:59:59.975    1.03  0.029  2.034   29   30
isfs7:/dev/ttyS1           7    100   1727988 2011 01 29 16:00:00.007  01 29 23:59:59.998   60.00  0.000  0.195   14   14
isfs7:/dev/ttyS6           7  32768     17288 2011 01 29 16:00:02.423  01 29 23:59:58.715    0.60  0.000  5.747   22   87
CG4 and PIR correction factors
Tom Horst posted on Nov 20, 2012

We did not have the opportunity to calibrate the pyrgeometers prior to the start of PCAPS, and therefore used old thermopile calibration factors for the Kipp & Zonen CG4 and Eppley PIR pyrgeometers.  These were subsequently recalibrated in the NOAA black body on March 8 & 10, 2011.  The following table lists multiplicative correction factors for Rpile outputs during the PCAPS field program.

Sensor

Location

Position

Pre-ops

Post-ops

Correction

k*

 

 

 

uV-m^2/W

uV-m^2/W

pre/post

 

PIR 031980

Playa (1)

in

3.58

3.58

1.000

4.0

PIR 031976

Playa

out

3.93

3.74

1.051

3.5

PIR 031975

ABC (2)

in

3.97

3.98

0.997

3.5

PIR 031978

ABC

out

3.50

3.53

0.992

4.0

CG4 050824

Highland (3)

in

14.09

13.81

1.020

NA

CG4 040740

Highland

out

7.85

7.76

1.012

NA

CG4 030675

West Valley (4)

in

9.97

9.93

1.004

NA

CG4 040741

West Valley

out

8.82

8.76

1.007

NA

PIR 031979

East Slope (5)

in

3.99

3.81

1.047

4.5

PIR 031981

East Slope

out

3.69

3.66

1.008

3.5

CG4 100225

West Slope (6)

in

11.93

11.28

1.058

NA

CG4 100226

West Slope

out

11.51

10.81

1.065

NA

CG4 050823

Riverton (7)

in

9.32

9.21

1.012

NA

CG4 030676

Riverton

out

11.27

11.21

1.005

NA

*per Ellsworth Dutton, blackbody values for k have been increased by 0.3 and then rounded up to nearest 0.5

editing TP01 data
Steve Oncley posted on Apr 17, 2012

When the TP01 is power cycled, the values for the first 300? seconds (while it is taking its first reading) are not valid.  In the present raw_data, Vheat and Tau63 are set to missing values during this period, but not Lambdasoil (which reads as 0).  These spikes to zero are averaged into the 5min means of Lambdasoil, which introduce spikes into our Cvsoil values and eventually Gsfc.  Ordinarily, this <shouldn't> be much of a problem, but it appears that there were periods during PCAPS when these systems were power cycled quite often (e.g. every 5 hours on 21 Jan).  Presumably these were mote or power issues -- I haven't yet checked.

Regardless, it would be best to set all TP01 values to NA during these start-up periods.

1 Comment  · 
TP01 analysis
Steve Oncley posted on Sep 14, 2011

UPDATE to the below discussion:

After discussions with Tom, I've realized that my entire philosophy of deriving soil moisture in frozen conditions from Cv is flawed.  In mixed conditions, Cv = Cv_liq * Qsoil_liq + Cv_ice * Qsoil_ice + Cv_dry, where Cv_liq = 4.81 and Cv_ice = 2.11.  Supposedly, from the plots below, I can compute Cv_dry by looking at the relation between Cv and Qsoil_liq when Qsoil_ice is zero (Tsoil well above freezing).  IF it was known that Qsoil_liq was zero (Tsoil well below freezing), this Cv relation could be used.  However, in mixed conditions, I don't know how much moisture is liquid or ice, so I can't independently compute soil water content (in any phase or combined).  Thus, I have 3 situations:

1. Soil not frozen: TP01 gives redundant Qsoil_liq to EC-5, once calibrated (using the EC-5).  Thus, not really independent.

2. Soil totally frozen: TP01 gives Qsoil_ice, once calibrated in totally warm conditions using the EC-5.  This is a unique measurement, but requires a long-term project.

3. Mixed soil conditions: TP01, combined with EC-5, gives Qsoil_ice, once calibrated in totally warm conditions using the EC-5.  As this procedure has several steps, there would be a large uncertainty in the result!

-------------------------------------

I've been trying to figure out why the Qsoil/Csoil comparison has looked lousy at most sites.  What I've found:

1. I now have better values for the soil densities than my first cut at looking at the data (which only used the first set of gravimetric measurements).  I'm now using the mean of all measured values at 3-6cm.  (There sometimes is a lot of variation between these measurements.)

2. The soil motes apparently had a "watchdog timeout" that reset power on the soil sensors rather frequently.  I believe that this occurred at all sites sometime, and, when bad, happens every ~4hours:50minutes.  Each time, the TP01 reported values of 0 until it was able to take another reading.  These 0 values were averaged in to each TP01 variable, generating lots of spikes.  A range check on the TP01 Vheat and Tau63 values has now been added to eliminate these zeros.  The covars are being rerun now.

(Note that gaps of one sample (5 s) every 5-minutes are normal, as John's code uses this time to sample the mote's power voltage.)

3. Even with all this, the comparison is good at only about 1/2 of the sites.  ("Good" means that the change of Cvsoil with Qsoil follows what I would expect.  "Bad" means that Cvsoil changes more than I would expect.)  I can trace the bad values to Vpile.on being too low at many of the sites.  (asoil values seem reasonable at all sites and Vpile.off and Vheat -- the only other variables in the Lambdasoil calculation -- are okay.)  This is true at most of these sites even during the last half of January, when there was a warm period that might have thawed the soil.  Possible reasons for Vpile.on being low are:

a. Inadequate contact between the probe and soil (the most likely culprite, despite my best efforts)

b. Voltage drop on Vheat (unlikely, since the 4-wire probe should measure this)

c. Electrical contact problem on Vpile (unlikely, but should be checked)

d. Heat going into latent heat of fusion (melting mixed-phase moisture), rather than sensible heat (can partly check by looking at warm periods -- a new idea for me).  [The results of this check are not very convincing -- see below plot.]

4. Horizontal lines on the plot below are caused by rapid changes of measured qsoil (which is really liquid qsoil) that, in the one case I examined in detail, was caused by rapid warming (and thus melting) of the soil.  The portion of this change that occurs in the 3 hours that the TP01 is reporting replicate values appears as a horizontal line.  I'll update this plot with data values averaged to 3hours to eliminate this artifact, but there still will be the question of whether the qsoil value is valid.

By the way, the warming case I found is 23 Jan, near noon, at ABC.

This is closer than I've been before with this comparison, but I still have a long way to go.

Attached is my current comparison plot for each of the 9 soil sites after filtering the statistics for:

- good Tau63 and Vheat values

- rerun of covars over first half of the data

- removing qsoil values when tsoil<0C

Points in blue are for the first month of the project, when probes may have been settling (actually, mostly not too bad).  Arguably, these values may be ignored.  Points in green are for tsoil>5C, when soil should not have ice.  The black line is the expected variation of Cvsoil with Qsoil (4.184 J/kgK for 100% soil moisture by mass) with no dry soil heat capacity.  Actual values should fall along the green line, which is parallel to the black line and has been fitted by assuming dry soil heat capacities of  1.3,1.0,0.9,0.2,0.6,0.2,0.6,1.8,0.2 J/kg K, respectively, for each of the sites.

In the "even newer" version, I've manually eliminated periods where I think there might be a chance of frozen soil moisture (that would give me a false qsoil measurement).  This eliminates about 3/4 of the data, but greatly reduces the scatter.

Even newer plot:

Updated plot:

Old plot:

Site coordinates
Tom Horst posted on May 16, 2011

Here are the site coordinates as recorded by Gordon in the netcdf files:

"We looked at the GPS output from each station, when there was satellite lock
and the lat and long were constant, and put them as non-time series variables
in the netcdf."

Site

Latitude

Longitude

Playa(1)

40.7786

-112.1031

ABC(2)

40.7377

-111.9132

Highland(3)

40.6969

-111.839

W Valley(4)

40.681

-111.9949

E Slope(5)

40.6135

-111.8039

W Slope(6)

40.5938

-112.0621

Riverton(7)

40.5136

-111.9737

More TP01 testing
Steve Oncley posted on Apr 15, 2011

From the Sept/Oct 2010 tests, I found that new, sensor-specific, calibrations of E.lambda and Tau63_ref from the agar test produced results with less scatter.  Thus, the plan is to use these for data processing.  Since I will need to rely on these values, it seems like a good idea to do agar post-cals on these probes.  In addition, I might as well double-check the probe resistances, so Chris is now making a test cable that I'll use with the HP5006 plug-in 4-wire meter.  With these revised calibrations, I'll go back and look at the PCAPS data when the soil wasn't frozen.  Finally, we'll rerun some tests in the FLAB yard.  I'm not planning a new set of keg tests.

Resistance values measured 15 Apr 2011 using 4-wire meter:

ID

Sensor S/N

R_measured (Ohms)

R_manufacturer (Ohms)

Difference (Ohms)

1

200589

16.14

16.1

0.04

2

200590

15.40

15.4

0.00

3

200236

14.79

14.8

-0.01

4

200591

16.70

16.7

0.00

5

200239

16.37

16.2

0.17

6

200592

15.53

15.5

0.03

7

200241

16.27

16.2

0.07

8

200242

16.10

15.9

0.20

9

200243

15.76

15.6

0.16

10

200234

14.72

14.7

0.02

11

200238

15.50

15.4

0.10

 

200235

sliced

14.8

 

 

200240

lightning fried

15.0

 

All are within 0.2 ohms of manufacturer values.  Except for one at 0.01 ohms, our values are higher.  <Might> be connectors, but 4-wire measurement should compensate.

Agar results (first row Sep 2010, second row Apr 2011).  Lambda values use original Hukseflux coefficients in this table:

ID

Sensor

Vheat (uV)

Vpile.on (uV)

Vpile.off (uV)

V63 (uV)

Tau63 (0.01s)

Lambda (mW/m K)

 

Expected

1200000

 

0

 

1900

600

1

200589

1236927

375

0

139

1943

666

 

 

1236416

375

0

139

1921

665

2

200590

1194264

314

0

116

1940

668

 

 

1194652

314

0

116

1985

669

3

200236

1238958

313

0

116

1877

578

 

 

1237230

313

0

116

1892

576

4

200391

1240328

313

0

116

2073

686

 

 

1239126

313

0

115

2042

684

5

200239

1239719

470

0

174

1965

591

 

 

1238975

470

0

173

1972

590

6

200592

1237441

407

0

150

2125

655

 

 

1237288

407

0

150

2170

655

7

200241

1241515

376

0

139

1892

596

 

 

1239874

376

0

139

1867

595

8

200242

1236806

407

0

150

2089

600

 

 

1236332

407

0

150

2105

600

9

200243

1236372

438

0

162

1746

598

 

 

1236324

438

0

162

1755

598

10

200234

1237511

281

0

104

1679

603

 

 

1237304

281

0

104

1673

603

11

200238

1238329

439

0

162

1773

586

 

 

1236740

439

0

162

1792

584

All of these have very close agreement between the Sept 2010 and Apr 2011 values.  Tau63 differences are 0-0.5s, Lambda differences are 0-2.  Thus, best approach is to take the average of these values to generate new Tau63_ref and E.lambda values.

Agar recipe
Steve Oncley posted on Apr 15, 2011

Thought I'd start by scribbling down the Agar recipe:

Add 7g agar flakes to 1.75l water in 2l beaker.

Place on 1000W burner of 2-burner hotplate.

Rig up a shroud and lid around beaker to reduce heat loss.  I borrowed a chunk of pipe and PVC from the shop this time (see photo).  The spot I used in the cage had a draft, so I used the blue foam to block it.

It took about 20min on high to get to a boil.  I stir every couple of minutes using a glass rod Chris found to the left of the cal lab sink.

Reduce heat to medium-high and simmer with lid off (packet instructions said for 5min, TP01 said 20min, I did 15min this time).  I noted that the volume reduced to 1.6l after all this time, which is still fine.

Let cool until milky white (several hours).

Tsoil check
Chris Golubieski posted on Mar 15, 2011

March 14, 2011

TESTED GOOD:  TS016, TS017, TS015, TS013(cleaned connector), TS005, TS011.

BAD:  TS002, TS004, TS003.

TS012 tested good but had #4 sensor out.

Post Krypton cal
Steven Semmer posted on Mar 01, 2011

The kryptons were checked in the chamber at 10C. The density range went from 1 to 9 g/m^3.

sensor

error range (g/m^3)

1525

   1.52 to 3.00

1389

   1.01 to 1.56

1390

   0.65 to 0.81

1393

   -0.21 to 0.02

1394

  2.49 to 3.43

1133

   1.88 to 2.47

1258

   1.95 to 2.32
CM21 PSP correction factors
Steven Semmer posted on Feb 25, 2011

At the start of the project we did not have the new calibration coefficients for the KZ CM21 and EPPLEY PSP shortwave radiation sensors.

Therefore a correction factor must be applied to the data.

sensor

location

pistion

cal factor

EPP 16945

Playa

in

1.040632

EPP 17877

Playa

out

1.033604

EPP 17414

ABC

in

1.009423

EPP 15108

ABC

out

1.024915

KZ 940185

Highland

in

0.993432

KZ 970378

Highland

out

0.987761

KZ 970380

West Valley

in

1.002629

KZ 940181

West Valley

out

0.993238

EPP 16107

East Slope

in

1.023545

EPP 16714

East Slope

out

1.033453

EPP 26224*

East Slope

out

1.013483

KZ 970379

West Slope

in

0.988474

KZ 940187

West Slope

out

0.993795

KZ 970377

Riverton

in

0.992930

KZ 940186

Riverton

out

0.999155

* removed Rsw.out at East Slope ~18:30 Dec 4; replaced 14:20 Dec 6

Post cal TRH
Steven Semmer posted on Feb 24, 2011

Post cal results of temperature, temperature range 15C to -20C

Sensor

mean (error)

sd (error)   

min to max

017

.019

.014

-.036 to .001

023

.017

.012

-.004 to .030

019

.016

.014

-.001 to .032

013

.000

.006

-.007 to .010

025

-.008

.011

-.034 to .002

024

.002

.008

-.011 to .013

009

-.035

.007

-.044 to -.022

028

-.018

.007

-.028 to -.004

027

-.011

.016

-.027 to .02

Post cal of RH, two runs from 10% to 90% at 10C and -4C, Gortex still on

Sensor

mean (error)

sd (Error)

min to max

017

-3.3

0.9

-4.8 to -1.4

023

-1.7

1.2

-3.8 to -0.4

019

-8.6

1.2

-11.1 to -6.3

013

0.1

0.5

-0.7 to 0.8

025

0.7

1.44

-2.2 to 2.2

024

1.1

1.7

-1.1 to 3.8

009

-0.6

0.6

-1.1 to 0.9

028

0.6

1.9

-3.6 to 2.6

027

1.2

1.6

-2.1 to 2.6

Location of SHT sensors through out the project. Some sensors replaced during the project were sent back to Boulder for repairs then the sensor was deployed again

                                              DATE of CHANGE

STATION

11/28

12/07

12/12

12/13

1/15

11/16

1/17

1/18

1/27

1/28

2/6

2/7

1

13

 

9

26

 

 

21

 

13

 

 

13

2

15

21

 

 

 

 

24

 

 

 

 

24

3

20

 

 

 

 

 

19

23

 

 

 

23

4

19

 

 

 

 

25

 

 

 

 

 

25

5

18

 

 

 

 

 

 

 

 

9

17

17

6

23

24

 

 

 

 

27

 

 

 

 

27

7

17

 

 

 

28

 

 

 

 

 

 

28
Post Barometer cal check
Steven Semmer posted on Feb 24, 2011

A post cal of the barometers was completed today. Each barometer was run over the range of 800 to 900 mb.

The average error was .16 mb, error span was -.21 to .26 mb, sd was .06

Post Serial Numbers
Chris Golubieski posted on Feb 23, 2011

Sensor

1
PLAYA

2
ABC

3
HIGHLAND

4
WEST VALLEY

5
EAST
SLOPE

6
WEST
SLOPE

7
RIVERTON

CSAT3

0537

0800

0855

853

0671

1119

0539

Krypton

1395 / 1390 (1/2/11)

1393

1525

1258

1133

1389

1394

Barometer

B8
U4110003

B7
U4110001

B3
NCAR003

B10
491125

B4
S0610001

B5
S0610002

B9
U411004

TRH

 

 

 

 

 

 

 

Propvane

00318

----------------

----------------

---------------------

--------------------

???

-------------------

Rad. IN (pyn/pyg)

16945 /
31980

17414 /
31975

940185 /
050824

970380 /
030675

16107 /
31979

970379 /
100225

970377 /
050823

Rad. OUT (pyn/pyg)

17877 /
31976

15108 /
31978

970378 /
040740

940181 /
040741

26224 /
31981

940187 /
100226

940186 /
030676

LiCor

PY7121(DIR)
PY99291(DIFF)

----------------

----------------

---------------------

-------------------

---------------

PY10835(DIR)
PY10836(DIFF)

Tsoil

TS017

TS016

SING(026)

TS002/TS015

TS023
TS004
TS012

TS013
SING(022)
TS003
TS011

TS005

Qsoil

ECHO007(017)

ECHO010(020)

ECHO005(015)

ECHO009(019)

ECHO004(014)
ECHO006(016)

  MISSING
ECHO001(011)

ECHO002(012)

Gsoil

H993563(007)

H013390(009)

H983211(005)

H023009(002)

H013391(004)
87034(010)

H993561(003)
H993560(006)

H983213(008)

TPO1

TP025(200590)

TP006(200592)

TP005(200239)

TP007(200241)

TP009(200243)
TP008(200242)

TP010(200234)
TP004(200391) [200591?  spo 4/18/11]

TP001(200589)

Rain Gauge

---------------------

---------------------

???????????

????????????

-------------------

????????????

--------------------

Motes

B,7,19

F,4,5

D,22,6

15,15,8

13,12,11,20

11,10,9,21

17,16,18

Vbatt

31

32

38

34

35

36

37

Viper
Sunshine
Modem/
Router

16


M-8(09105212383)/
R-1(MM090194300014)

1


M-2(09105211885)/
R-2(MM090168701586)

14


M-3(09105211890)/
R-3(MM090168700567)

12


M-4(09105211882)/
R-4(MM090139201501)



M-5(09105212400)/
R-5(MM091468703640)

6


M-6(09105211884)/
R-6(MM090139202886)

3
SPN1-A465


M-7(09105212393)/
R-7(MM090139202713)


Soil sensor settling time
Steve Oncley posted on Feb 18, 2011

I've just plotted Cvsoil from the entire project to see if I could determine a time when the TP01 (as a surrogate for all soil sensors) had "healed" into the soil.  I see a definite step at about 7 Dec at 1, 5, 6, and to some extent 6.aux and 7.  This was a few days into the first PCAPS warming period with temperatures above freezing.  Given what comes later (other steps in Jan), it isn't obvious that the TP01 had completely settled in (perhaps ever).  Thus, the most definite conclusion I can make is that at least half of the sensors took at least 5 weeks to settle in.

Rain gauges, Sites 3 and 4
Tom Horst posted on Feb 07, 2011

February 7

~13:30 removed rain gauge, W Valley

14:30 removed rain gauge, Hiland

Precip started 15:00