Page tree
Skip to end of metadata
Go to start of metadata

This is to get an overall sense of where data are not being reported or are bad/suspicious. Green is good or N/A until it isn't. 





































t (thermo)





3D Windsprsdcsccuppcdcmwmhshsplc




















































  1. NChart and QC tables are not reporting the data (NCharts as of 2022-01-04 23:27:30) so I looked at snapshots of data from the DSM Dashboard (thanks Gary). The exceptions are the supersites which I cannot find on the Dashboard. The default is green until I or anyone can see the data.

    • All Rpile at DCS measure near zero values.
    • Qsoil.dcs is twice as high as the rest of the sites.
    • Sonics running at all sites now except at SH
    • h2o/co2 
      • DCS stopped reporting as of 21 Jan 21:12:30
      • SH recording on the Dashboard, but not on NCharts
      • Noting negative spikes of h2o at several sites already - notably at MH, SP, and PRS. 
    • is not like the others.
    • Depth is negative except at MH, UP
    • RH.cs.1m = -99 at all sites that measure them.


    • Rfan.17m.dcs is super duper high.
    • Vcharge.up is not like the others
    • Vcharge.dcs is flatlined.
  2. Radiometer

    • All Rpile at DCS measure near zero values.
    • Rpile at 7m and 32m at PRS look like they are reversed, i.e. reads like Rpile.out and vice versa

    Tsoil.pc - measuring in the negative hundreds.  Added na to cal file in QC/ folder until fixed.


    • SH recording on the Dashboard, but not on NCharts
    • Noting negative spikes of h2o at several sites - notably at MH, SP, and PRS. is not like the others.


    • Vcharge at UP and CC are not like the others
    • Vcharge.dcs is flatlined.
    • All Rlw measurements (Rlw, Rpile, Tcase) at DCS come through Sebastian's data logger.  I'm still not sure which sensors are which in his data message – I think we'll have to manually test them the next time the tower is lowered.  Also, his Rpile is actually the voltage, not W/m2, which is why the values are low.  I'm not sure how to solve this.  Furthermore, he says his Tcase is resistance, not degrees. 
    • At PRS, we log all the Rlw data, so we should know which is which.  It is possible that .in data are labeled .out and vice versa, but I thought I checked that we got it right.  Definitely worth checking manually.
    • Thanks for picking up on Tsoil.dc.  It was okay last I checked.
    • Statsproc (and thus qctables and ncharts) calculates statistics of h2o and co2 along with the sonic variables, so these become NA anytime the sonic is bad.  In the case of SH, the sonic had been removed for repair.  We just replaced this this afternoon and all should be okay now.
    • I'm assuming that h2o spikes are real due to frost/snow melt and such.
    • Agree about  It is mostly due to Vpile_On, making it appear that this probe may have come dislodged about 20 Dec.  We can try installing a spare, though this is under 10cm of crusty snow by now, so an installation nearby could be disruptive.
    • UP and CC are on solar power, everything else is line power, thus Vcharge would be expected to be different.
    • DCS line power became disconnected, but should have been okay yesterday afternoon.
  3. Thanks Steve.


    • Made note of different radiometer units at DCS in the data tracker. Keeping it orange since we will have to apply a conversion factor and re-examine these data at post-processing.
    • Removed red from MH - assuming no radiometers were installed.


    • Tsoil.pc - weird spikes are gone. Updated the QC file. 
    • Removed red from MH - assuming no soil sensors were installed - measurements have settled since ~ 2022 01 07 13:57:30


    • h2o.17m.prs is not like the others - measures two to three times higher than the rest.
    • NCharts recording data at SH as of Jan 8th.


    • Do data at NCharts for CC and SP sites.
    • MH and UP behave opposite and different from the others.
    • Not sure how to interpret these data - but marking them suspicious until otherwise.

  4. Radiometer

      • DCS switched from Rpile to Rlw as of Jan 09 22:27:30
      • Why is there a 1m folder in the PRS and DCS noQC/ folders?
      • and Rpile.out @ 7m, 32m at PRS switched at Jan 09 22:17:30. Now correct. 
        • Post-processing - Need to switch the time series before Jan 09 22:17:30
      • Filtered spikes in QC files for PRS, LC, CC

    h2o.17m.prs - still not like the others. 3-4 times higher than the others.


    • MH, DCS are measuring in the positive direction.
    • CC - no data
    • UP - highly variable - not like the others
  5. The HRXL at UP occasionally reports messages of R5000 which are clearly not measurements but some kind of status or error message.  I set a minValue for Depth to -0.5m, so that will filter out all the 5000 values which get converted to -3 by the default Snow.dat calfile.  dc, cc, and up have all exhibited this problem, so those depth measurements should look better now.  I think positive values are not a problem; they indicate either snow accumulation or else the sensor was mounted less than 2m above the ground, since all sites have a default offset of 2m.

    I have generated a merged dataset manually in /scr/isfs/projects/CFACT/merge and am running statsproc on the merge for noqc_geo and noqc_instrument.  When that finishes, the data in ncharts should contain all the data collected so far processed with the latest configuration.

    I thought we needed to measure the height above ground of each HRXL to create a per-site calfile with a custom offset, but I don't know if anyone has done that yet.

  6. Lowered/Raised DCS tt: Jan 10 15:12:30 - Jan 10 16:22:30

    • Used the Rlw time series. (Note: realized we can’t use P since there is no barometer at the tower top!) - refer to Status 1/10

    Lowered/Raised DCS tt: Jan 07 10:07:30 - Jan 08 04:27:30

    • This is a guestimate based on Rpile and TRH at 32m. There is a large data gap - refer to Status 1/7 - I used the end time when data resumed.

    Updated the tracker - FYI - there is a tab for Raise/Lower towers.

    Do we have to manually add these filters to all the DCS QC files? 

  7. Soils

    • Strange behavior in 
      • Gsoil.dcs since ~ 2022 Jan 10 05:52:30
      • Gsoil.sp since ~ 2022 Jan 10 00:12:30
      • Qsoil.sp 
      • Qsoil.dcs still not like the others.



    • Refer to Gary’s comments yesterday/above
    • DCS, PRS, CC, SP - no data - still not like the others - see QC tables

  8. Looks like h2o.17m.prs is roughly 2-3 times that of h2o.17m.dcs.

    Multiple by scale factor ~ 0.36?

    • Every EC100 has a Pirga that can be used (albeit, with a manual bias correction) to determine if the tower is up or down.
    • Today, I replaced the TP01 at SH, though it was a terrible installation in mostly frozen soil.  Expect Lambda to look different.
    • h2o can have significant biases, that I'm not at all concerned about.  A gain error would be significant.  The best way to look for gain differences is to compare the standard deviation (or variance h2o'h2o').  It seems that h2o'h2o' is correctly ordered with height – with largest variations near the source of moisture (at the surface).  Still, it wouldn't hurt to clean this sensor.
    • Gary is completely correct about snow depth.  The heights need to be measured.  I forgot to add that to the site documentation tasks.
    • Today, we rewired csat.2m.prs into another DSM in an attempt to get it going.
  9. Soils

    • - good since ~ Jan 11 15:27:30 - Steve reburied it yesterday. Updated the QC file. 
    • Lamda at DCS and MW are not like the others. Coloring them suspicious until otherwise. 
    • Qsoil and Gsoil at DCS, SP - looks different


    • PRS back online
    • MH needs a reference - see Steve’s comment yesterday/above
    • DCS, CC, SP - no data - refer to Jira report


    • PRS @ 2m - no data 
    • h2o.17m.prs - high relative to DCS
    • t.dc - measures quite high - not sure if that is of concern.

    Lowered/Raised PCS tt: Jan 11 09:12:30 - Jan 11 10:12:30

    1. Until further notice, none of the satellite sites have thermocouples installed, so these t.xx data can be ignored.  We're hoping that the PIs will decide to install these later in the project when the survivability rate of thermocouples in the field is better known.

  10. The sample at 2022 01 13 06:23:05.1063 MST missed sending a "." (actually two characters "6." or ".6" in Tcase, so Tcase, and the resulting and Rlw.out values were huge.  As far as I can tell, this was only one (5 second) sample, and would be trivially filtered in the xml by putting min/maxValues for Tcase.  I think I've seen a similar error a week or so ago, though I don't know if it was on this sensor. 

    Of course, if one "." can be skipped, it begs the question of whether other, more subtle, data transmission errors are occurring.  I'm not (yet?) using checksums in V2.8 code.

  11. t.7m.dcs - spiked, then no data - no data


    • DCS, MH - positive values - reference?
    • SP - no data


    • Gsoil - a number of sites are showing very high gradients, especially around nighttime. Tsoil temperatures aren’t varying all that much so I’m not such what is causing these large changes. Is it melting snow percolating into the ground then freezing? Phase changes between liquid water and ice releases thermal energy. There is correspondence to changes in Qsoil which I think is moisture content. It’s slowly decreasing so I imagine there is gradual water loss via evaporation of the soil water during the day - which would cool the soil locally. Tsoil.4.4cm at some sites show a slight cooling tendency.  I’m just throwing that out there. I’ve been staring at time series all morning, my mind starting wandering and wondering. I’m just going to make it all green since more sites are starting to exhibit similar diurnal behavior. 

    Everything else is the same - see Data Matrix above.

  12. Site SP

    • NCharts/DSM Dashboard: show a large number of sensors are not reporting data.
    1. SP was off the net until Gary’s modem script brought it back, but for some reason nidas didn’t start after reboot. I started it manually and we should be getting data now. 

  13. MW

    • IRGA - no data (on NCharts as of Jan 14 22:42:30)

    Depth - no data at DCS, CC and SP 

  14. Soils @ DCS - no data since 12:57:30. 

  15. SP - no data anywhere since Jan 18 13:47:30. Its Dashboard site cannot be reached. 


    • PC - no data since Jan 18 10:32:30

    Lowered/Raised PCS tt: 2022 Jan 17 10:22:30 - 14:42:30 

    • Used Radiometer at 0.5m, 2m and Pirga.1m.prs  (Pirga and R* at all other levels were invariant)
    • See Wiki Ops 1/16
  16. sonic/irga at PRS.2m - data are back!! (as of Jan 17 10:12:30) 


    • CC, DCS - no data


    • DCS, SP, MW - still suspicious

    1. 2m sonic at PRS wasn't showing up in netcdfs for a while because it was plugged in to prsg instead of prsr due to port problems: It was collecting data, but the variable name had to be different to avoid conflicts so I added a .g suffix. I haven't updated the stats_5min xml to get the new variable name so it wasn't showing up in ncharts and qctables.

      That's also what's up with the ott at PRS - it's currently plugged in to prsg, not prsr, and so I had to add a .g suffix to the variable name, so it's not showing up in the netcdfs.

      Rebooted cc and dcsr, HRXLs are now reporting.

  17. Data checks through ~09:07:30 this morning - what Ncharts is showing so far ...

    PRS @2m for sonic/irga all green since these data are actually not missing, just renamed but not integrated into the visualizations. See Isabel’s comment yesterday (above). Thank you Isabel! 

    • Tracking Note: Double check these data in post-processing. - Thermopile time series is not like the others. Started 06 Jan 20:02:30. See QC tables. - no data


    • DCS, SP, MW - still suspicious. 


    • MH - still positive - still not like the others
    • CC, DC, SP - no data

  18. Are netcdfs being generated on barolo? Ncharts stops at ~ 09:07:30 this morning. I’m not sure how often they get updated. 

    1. I just restarted dsm_server – almost a daily occurrence these days.  "dsp" now works again, so hopefully statistics will start flowing.

      FYI, the steps are:

      • log in as user isfs on barolo
      • ps -ef | grep dsm; look for the ID of the dsm_server process
      • kill -9 whatever_the_ID_was
      • cd scripts
      • ./
  19. Noticed

    h2o/co2 - no data coming in on NCharts or Dashboard at: MW, UP, PC, MH

  20. h2o/co2 - back online on the Dashboard. - recording 0’s or near 0’s. - behaving like Rpile.out between Jan 20 23:17:30 - Jan 21 10:42:30. Suspicious? Made note in tracker. - thermopile time series is not like the others. Started 06 Jan 20:02:30. Behaves similar to - no data  

    1. Rpile.up is now back to non-zero. shows a similar decrease during this period and RH.2m.up is essentially saturated during this period.  Interestingly, Wetness.up just seems to wander.  I conclude that this is a real fog event where the "sky" and "ground" radiative temperatures are essentially the same as Tcase, so that Rpile, which represents the difference between these radiative temperatures and Tcase, is near zero.  All is okay.

    2. " - thermopile is not like other" thermocouple?

      There are currently no thermocouples installed at satellite sites, so the values reported to nidas are meaningless.

      1. Still, I don't know why mh, and differently, dc, report differently, since they are all the same circuit.  I'm partly expecting some unpleasant surprises when/if we finally populate the satellite thermocouples.

        Perhaps, it would be a good exercise on a nice day to plug thermocouples in at mh, dc, and one other location for a few hours....

  21. PRS for all sensors at 7m, 17m, 32m -  no data since 10:42:30 

    (I don’t know if I just have to wait longer for it to appear on NCharts)

    1. 7/17/32m sensors all go to the prst dsm, which as been off the net since 10:45. Hopefully it'll reboot and come back after 4 hours, or we might go visit PRS this afternoon.

  22. Seems all I do these days is report no data on Ncharts ... 

    A few times-series remain suspicious - see Data Matrix. 

    • Note on Tracker:  re-examine Thermopile measurements at MH and DC.
    • Rfan at DC - noisy measurements ~ Jan 21 that corrects itself by today. 

    Thanks Isabel for the quick responses. 

  23. Radiometer at DCS, PRS

    • Rsw.out.2m >
    • All other sites show the reverse, i.e Rsw.out.2m at PRS and DCS behaves like<site>, and vice versa.
    • This is true since the start of both time series. 
    • Is Rsw.out.2m.dcs, Rsw.out.2m.pcs ↔, ? 


    • Strange daily peaks forming from ~1200-1700 since Jan 12. The peaks are increasing and spreading in time. Is this real? Exposed sensor? 

    h2o/co2 - no data at 

    • CC since Jan 24 00:32:30
    • 7m.dcs & 17m.dcs since Jan 23 18:32:30 

    Depth - no data at DCS, CC (again)

    Lamda - still suspicious behavior at DCS, SP, MW

  24. Rfan @ DC

    • Looking back - values dip ~ Jan 14 04:47:30. Doesn’t look like the others -but I’m not sure how critical this behavior is. Just noting an observed difference.
  25. Tsoil.0.6cm.sp has started exhibiting similar behavior as  Tsoil.0.6cm.prs (refer to yesterday’s comment)

    • Strange daily peaks in the mid afternoon > 1400 since Jan 22 .
    • The peaks are increasing and spreading in time. Is this real? Exposed sensor? 

    Sonic - - Notice values are an order of magnitude higher than the rest. Reaching ~+1.5 m/s, which is high! 

    Can someone tell me if RH.cs.1m is supposed to be -99’s for a number of sites? 


    • An interesting step change noticed at MH, PC, SH, UP between Jan 23 16:57:30 - Jan 24 16:07:30
    • Not sure if this is real, a data blip, or what.
    • Noting this in the tracker, just in case.
  26. Lamda (TP01) @ DCS, CC, SP - all green

    • Initially marked as suspicious, but now having read the sensor brochure I learned that the measurement range is 0.3 to 4 W/(m•K) and all these sites are within that range, except for a spike in DCS which is taken care of in the cal_files. 
    • I notice that the units on NCharts is W/m (without the K). Typo?

    Depth - no data at DCS, CC, SP … just saying - Really high, unlike the rest. From Jan 25 07:22:30 - 21:02:20. Making note in the tracker - something interesting to review. 

    Didn’t look at the thermopiles since I didn’t keep track of what is active and what is false. Oh well. 

    Everything else is the same - see Data Matrix.

    1. So, the tilt plot for mh is quite strange, as expected.  Winds coming into the sonic (pointing east) would be deflected to cause an upward vertical velocity, as the plot shows.  Winds from behind are in the wake of the parking lot/flagpole bases/trees and show as widely-varying downdrafts.

      In other words, odd values of mean vertical velocity are completely to expected at this site.

  27. Lowered/Raised Towers: See Ops 1/26

    • DCS -  2022 Jan 26 13:47:30 - 15:22:30 using Pirga@32m
    • PRS - 2022 Jan 26 10:57:30 - 16:42:30  using Pirga@32m

    So far, nothing new to report. That’s good.

  28. After a talk with Jacquie, I swapped which supersite needed to use 2compSswap (to account for which Rsw was in which position).  I've restarted the statsprocs, so hopefully we see reasonable Rsw values tomorrow...

    1. We don't have too much data yet this morning, but it does appear that all .ins and .outs look similar now.

  29. No data on NCharts since yesterday morning. However, the Dashboard shows data at the non-supersites, except for MW where it hangs. I’m sure folks already know about this and are working on it. 

    In discussions with Steve I learned:

    • There are no RS.cs measurements. This may extend to ignoring T.cs time series. Have a discussion which CS125 variables to include in the QC netCDF’s (noted in tracker). 
    • The PI’s are enthusiastic about continuing collecting soil samples.

    Rsw values have been swapped as of Jan 28. Thanks Steve!

    1. I've noticed the lack of data on NCharts also. This seems to be an issue only with NCharts, as everything is mostly working on this end. I am emailing Gary about it. 

  30. Sonic/IRGA at LC are showing spotty data on NCharts. The Dashboard is showing a lot of invalid measurements, particularly for the Sonic.  

    NOTE: dsm_*.dat and isfs_*.dat.bz2 files have been manually backed up to Campaign Storage through 20220204_120000.

    1. I've been noticing that too. It seems to be primarily at night, so I've been thinking that it's ice/frost/moisture issue of some kind. We haven't been by to check it out, but we need to take a soil sample and Leica measurement, so will swing by one of the next few days. I'm open to any ideas from Steve if he thinks something else could be causing this issue.

  31. Lowered/Raised PRS (yesterday)

    2022 Feb 07 13:37:40 - 14:27:30 - using Pirga@32m

  32. I have rsync'd all the ISFS raw data onto campaign storage through about 1500 MST on 2/18/22.