CFACT Data Matrix

NChart and QC tables are not reporting the data (NCharts as of 2022-01-04 23:27:30) so I looked at snapshots of data from the DSM Dashboard (thanks Gary). The exceptions are the supersites which I cannot find on the Dashboard. The default is green until I or anyone can see the data.

• All Rpile at DCS measure near zero values.
• Qsoil.dcs is twice as high as the rest of the sites.
• Sonics running at all sites now except at SH
• h2o/co2 
• DCS stopped reporting as of 21 Jan 21:12:30
• SH recording on the Dashboard, but not on NCharts
• Noting negative spikes of h2o at several sites already - notably at MH, SP, and PRS. 
• Lamda.sh is not like the others.
• Depth is negative except at MH, UP
• RH.cs.1m = -99 at all sites that measure them.

Other

• Rfan.17m.dcs is super duper high.
• Vcharge.up is not like the others
• Vcharge.dcs is flatlined.

Radiometer

• All Rpile at DCS measure near zero values.
• Rpile at 7m and 32m at PRS look like they are reversed, i.e. Rpile.in reads like Rpile.out and vice versa

Tsoil.pc - measuring in the negative hundreds.  Added na to cal file in QC/ folder until fixed.

h2o/co2

• SH recording on the Dashboard, but not on NCharts
• Noting negative spikes of h2o at several sites - notably at MH, SP, and PRS. 

Lamda.sh is not like the others.

Other

• Vcharge at UP and CC are not like the others
• Vcharge.dcs is flatlined.

• All Rlw measurements (Rlw, Rpile, Tcase) at DCS come through Sebastian's data logger.  I'm still not sure which sensors are which in his data message – I think we'll have to manually test them the next time the tower is lowered.  Also, his Rpile is actually the voltage, not W/m2, which is why the values are low.  I'm not sure how to solve this.  Furthermore, he says his Tcase is resistance, not degrees. 
• At PRS, we log all the Rlw data, so we should know which is which.  It is possible that .in data are labeled .out and vice versa, but I thought I checked that we got it right.  Definitely worth checking manually.
• Thanks for picking up on Tsoil.dc.  It was okay last I checked.
• Statsproc (and thus qctables and ncharts) calculates statistics of h2o and co2 along with the sonic variables, so these become NA anytime the sonic is bad.  In the case of SH, the sonic had been removed for repair.  We just replaced this this afternoon and all should be okay now.
• I'm assuming that h2o spikes are real due to frost/snow melt and such.
• Agree about Lambda.sh.  It is mostly due to Vpile_On, making it appear that this probe may have come dislodged about 20 Dec.  We can try installing a spare, though this is under 10cm of crusty snow by now, so an installation nearby could be disruptive.
• UP and CC are on solar power, everything else is line power, thus Vcharge would be expected to be different.
• DCS line power became disconnected, but should have been okay yesterday afternoon.

Thanks Steve.

Radiometers

• Made note of different radiometer units at DCS in the data tracker. Keeping it orange since we will have to apply a conversion factor and re-examine these data at post-processing.
• Removed red from MH - assuming no radiometers were installed.

Soils

• Tsoil.pc - weird spikes are gone. Updated the QC file. 
• Removed red from MH - assuming no soil sensors were installed

Lamda.sh - measurements have settled since ~ 2022 01 07 13:57:30

h2o/co2

• h2o.17m.prs is not like the others - measures two to three times higher than the rest.
• NCharts recording data at SH as of Jan 8th.

Depth

• Do data at NCharts for CC and SP sites.
• MH and UP behave opposite and different from the others.
• Not sure how to interpret these data - but marking them suspicious until otherwise.

Radiometer

• • DCS switched from Rpile to Rlw as of Jan 09 22:27:30
  • Why is there a 1m folder in the PRS and DCS noQC/ folders?
  • Rpile.in and Rpile.out @ 7m, 32m at PRS switched at Jan 09 22:17:30. Now correct. 
    • Post-processing - Need to switch the time series before Jan 09 22:17:30
  • Filtered spikes in QC files for PRS, LC, CC

h2o.17m.prs - still not like the others. 3-4 times higher than the others.

Depth

• MH, DCS are measuring in the positive direction.
• CC - no data
• UP - highly variable - not like the others

The HRXL at UP occasionally reports messages of R5000 which are clearly not measurements but some kind of status or error message.  I set a minValue for Depth to -0.5m, so that will filter out all the 5000 values which get converted to -3 by the default Snow.dat calfile.  dc, cc, and up have all exhibited this problem, so those depth measurements should look better now.  I think positive values are not a problem; they indicate either snow accumulation or else the sensor was mounted less than 2m above the ground, since all sites have a default offset of 2m.

I have generated a merged dataset manually in /scr/isfs/projects/CFACT/merge and am running statsproc on the merge for noqc_geo and noqc_instrument.  When that finishes, the data in ncharts should contain all the data collected so far processed with the latest configuration.

I thought we needed to measure the height above ground of each HRXL to create a per-site calfile with a custom offset, but I don't know if anyone has done that yet.

Lowered/Raised DCS tt: Jan 10 15:12:30 - Jan 10 16:22:30

• Used the Rlw time series. (Note: realized we can’t use P since there is no barometer at the tower top!) - refer to Status 1/10

Lowered/Raised DCS tt: Jan 07 10:07:30 - Jan 08 04:27:30

• This is a guestimate based on Rpile and TRH at 32m. There is a large data gap - refer to Status 1/7 - I used the end time when data resumed.

Updated the tracker - FYI - there is a tab for Raise/Lower towers.

Do we have to manually add these filters to all the DCS QC files? 

Soils

• Strange behavior in 
• Gsoil.dcs since ~ 2022 Jan 10 05:52:30
• Gsoil.sp since ~ 2022 Jan 10 00:12:30
• Qsoil.sp 
• Qsoil.dcs still not like the others.

Sonics/irga

• PRS.2m - No data reporting - refer to Jira report
• h2o.17m.prs - much higher than the others

Depth

• Refer to Gary’s comments yesterday/above
• DCS, PRS, CC, SP - no data

Lamda.sh - still not like the others - see QC tables

Looks like h2o.17m.prs is roughly 2-3 times that of h2o.17m.dcs.

Multiple by scale factor ~ 0.36?

• Every EC100 has a Pirga that can be used (albeit, with a manual bias correction) to determine if the tower is up or down.
• Today, I replaced the TP01 at SH, though it was a terrible installation in mostly frozen soil.  Expect Lambda to look different.
• h2o can have significant biases, that I'm not at all concerned about.  A gain error would be significant.  The best way to look for gain differences is to compare the standard deviation (or variance h2o'h2o').  It seems that h2o'h2o' is correctly ordered with height – with largest variations near the source of moisture (at the surface).  Still, it wouldn't hurt to clean this sensor.
• Gary is completely correct about snow depth.  The heights need to be measured.  I forgot to add that to the site documentation tasks.
• Today, we rewired csat.2m.prs into another DSM in an attempt to get it going.

Soils

• Lamda.sh - good since ~ Jan 11 15:27:30 - Steve reburied it yesterday. Updated the QC file. 
• Lamda at DCS and MW are not like the others. Coloring them suspicious until otherwise. 
• Qsoil and Gsoil at DCS, SP - looks different

Depth

• PRS back online
• MH needs a reference - see Steve’s comment yesterday/above
• DCS, CC, SP - no data - refer to Jira report

Sonics/Irga

• PRS @ 2m - no data 
• h2o.17m.prs - high relative to DCS
• t.dc - measures quite high - not sure if that is of concern.

Lowered/Raised PCS tt: Jan 11 09:12:30 - Jan 11 10:12:30

• Refer to Ops 1/11
• Used Pirga.32m.prs - Thanks Steve!
• Added to tracker

The NR01.mw sample at 2022 01 13 06:23:05.1063 MST missed sending a "." (actually two characters "6." or ".6" in Tcase, so Tcase, and the resulting Rlw.in and Rlw.out values were huge.  As far as I can tell, this was only one (5 second) sample, and would be trivially filtered in the xml by putting min/maxValues for Tcase.  I think I've seen a similar error a week or so ago, though I don't know if it was on this sensor. 

Of course, if one "." can be skipped, it begs the question of whether other, more subtle, data transmission errors are occurring.  I'm not (yet?) using checksums in V2.8 code.

Site SP

• NCharts/DSM Dashboard: show a large number of sensors are not reporting data.

MW

• IRGA - no data (on NCharts as of Jan 14 22:42:30)

Depth - no data at DCS, CC and SP 

Soils @ DCS - no data since 12:57:30. 

SP - no data anywhere since Jan 18 13:47:30. Its Dashboard site cannot be reached. 

Soils

• PC - no data since Jan 18 10:32:30

Lowered/Raised PCS tt: 2022 Jan 17 10:22:30 - 14:42:30 

• Used Radiometer at 0.5m, 2m and Pirga.1m.prs  (Pirga and R* at all other levels were invariant)
• See Wiki Ops 1/16

Thanks Isabel for getting data reporting! 

See Wiki Blog Ops 1/18 and Jira link

sonic/irga at PRS.2m - data are back!! (as of Jan 17 10:12:30) 

Depth

• CC, DCS - no data

Lamda

• DCS, SP, MW - still suspicious

Data checks through ~09:07:30 this morning - what Ncharts is showing so far ...

PRS @2m for sonic/irga all green since these data are actually not missing, just renamed but not integrated into the visualizations. See Isabel’s comment yesterday (above). Thank you Isabel! 

• Tracking Note: Double check these data in post-processing.

t.mh - Thermopile time series is not like the others. Started 06 Jan 20:02:30. See QC tables.

Vis.ott.1m.prs - no data

Lamda

• DCS, SP, MW - still suspicious. 

Depth

• MH - still positive - still not like the others
• CC, DC, SP - no data

Are netcdfs being generated on barolo? Ncharts stops at ~ 09:07:30 this morning. I’m not sure how often they get updated. 

Noticed

h2o/co2 - no data coming in on NCharts or Dashboard at: MW, UP, PC, MH

h2o/co2 - back online on the Dashboard.

Rpile.in.up - recording 0’s or near 0’s. - behaving like Rpile.out between Jan 20 23:17:30 - Jan 21 10:42:30. Suspicious? Made note in tracker. 

t.mh - thermopile time series is not like the others. Started 06 Jan 20:02:30. Behaves similar to tc.mh

Depth.cc - no data  

PRS for all sensors at 7m, 17m, 32m -  no data since 10:42:30 

(I don’t know if I just have to wait longer for it to appear on NCharts)

Seems all I do these days is report no data on Ncharts ... 

A few times-series remain suspicious - see Data Matrix. 

• Note on Tracker:  re-examine Thermopile measurements at MH and DC.
• Rfan at DC - noisy measurements ~ Jan 21 that corrects itself by today. 

Thanks Isabel for the quick responses. 

Radiometer at DCS, PRS

• Rsw.out.2m > Rsw.in.2m
• All other sites show the reverse, i.e Rsw.out.2m at PRS and DCS behaves like Rsw.in.<site>, and vice versa.
• This is true since the start of both time series. 
• Is Rsw.out.2m.dcs, Rsw.out.2m.pcs ↔ Rsw.in.2m.dcs,  Rsw.in.2m.prs ? 

Tsoil.0.6cm.prs 

• Strange daily peaks forming from ~1200-1700 since Jan 12. The peaks are increasing and spreading in time. Is this real? Exposed sensor? 

h2o/co2 - no data at 

• CC since Jan 24 00:32:30
• 7m.dcs & 17m.dcs since Jan 23 18:32:30 

Depth - no data at DCS, CC (again)

Lamda - still suspicious behavior at DCS, SP, MW

Rfan @ DC

• Looking back - values dip ~ Jan 14 04:47:30. Doesn’t look like the others -but I’m not sure how critical this behavior is. Just noting an observed difference.

Tsoil.0.6cm.sp has started exhibiting similar behavior as  Tsoil.0.6cm.prs (refer to yesterday’s comment)

• Strange daily peaks in the mid afternoon > 1400 since Jan 22 .
• The peaks are increasing and spreading in time. Is this real? Exposed sensor? 

Sonic - w.mh - Notice values are an order of magnitude higher than the rest. Reaching ~+1.5 m/s, which is high! 

Can someone tell me if RH.cs.1m is supposed to be -99’s for a number of sites? 

Depth

• An interesting step change noticed at MH, PC, SH, UP between Jan 23 16:57:30 - Jan 24 16:07:30
• Not sure if this is real, a data blip, or what.
• Noting this in the tracker, just in case.

Lamda (TP01) @ DCS, CC, SP - all green

• Initially marked as suspicious, but now having read the sensor brochure I learned that the measurement range is 0.3 to 4 W/(m•K) and all these sites are within that range, except for a spike in DCS which is taken care of in the cal_files. 
• I notice that the units on NCharts is W/m (without the K). Typo?

Depth - no data at DCS, CC, SP … just saying

w.mh - Really high, unlike the rest. From Jan 25 07:22:30 - 21:02:20. Making note in the tracker - something interesting to review. 

Didn’t look at the thermopiles since I didn’t keep track of what is active and what is false. Oh well. 

Everything else is the same - see Data Matrix.

Lowered/Raised Towers: See Ops 1/26

• DCS -  2022 Jan 26 13:47:30 - 15:22:30 using Pirga@32m
• PRS - 2022 Jan 26 10:57:30 - 16:42:30  using Pirga@32m

So far, nothing new to report. That’s good.

After a talk with Jacquie, I swapped which supersite needed to use 2compSswap (to account for which Rsw was in which position).  I've restarted the statsprocs, so hopefully we see reasonable Rsw values tomorrow...

No data on NCharts since yesterday morning. However, the Dashboard shows data at the non-supersites, except for MW where it hangs. I’m sure folks already know about this and are working on it. 

In discussions with Steve I learned:

• There are no RS.cs measurements. This may extend to ignoring T.cs time series. Have a discussion which CS125 variables to include in the QC netCDF’s (noted in tracker). 
• The PI’s are enthusiastic about continuing collecting soil samples.

Rsw values have been swapped as of Jan 28. Thanks Steve!

Sonic/IRGA at LC are showing spotty data on NCharts. The Dashboard is showing a lot of invalid measurements, particularly for the Sonic.  

NOTE: dsm_*.dat and isfs_*.dat.bz2 files have been manually backed up to Campaign Storage through 20220204_120000.

Lowered/Raised PRS (yesterday)

2022 Feb 07 13:37:40 - 14:27:30 - using Pirga@32m

I have rsync'd all the ISFS raw data onto campaign storage through about 1500 MST on 2/18/22.