This is to get an overall sense of where data are not being reported or are bad/suspicious. Green is good or N/A until it isn't.
Met | prs | dcs | cc | up | pc | dc | mw | mh | sh | sp | lc | Rad | prs | dcs | cc | up | pc | dc | mw | mh | sh | sp | lc | |
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
P.2m | Rpile.in | |||||||||||||||||||||||
RH.0.5m | Rpile.out | |||||||||||||||||||||||
RH.1m | Tcase.out | |||||||||||||||||||||||
RH.2m | Rsw.in | |||||||||||||||||||||||
RH.3m | Rsw.out | |||||||||||||||||||||||
RH.7m | Tcase.out | |||||||||||||||||||||||
RH.17m | Rpile.in.0.5m | Rlw | ||||||||||||||||||||||
RH.32m | Rpile.in.2m | Rlw | ||||||||||||||||||||||
RH.cs.1m | Rpile.in.7m | Rlw | ||||||||||||||||||||||
T.0.5m | Rpile.in.32m | Rlw | ||||||||||||||||||||||
T.1m | Rpile.out.0.5m | Rlw | ||||||||||||||||||||||
T.2m | Rpile.out.2m | Rlw | ||||||||||||||||||||||
T.3m | Rpile.out.7m | Rlw | ||||||||||||||||||||||
T.7m | Rpile.out.32m | Rlw | ||||||||||||||||||||||
T.17m | Rsw.in.0.5m | |||||||||||||||||||||||
T.32m | Rsw.in.2m | |||||||||||||||||||||||
T.cs.1m | Rsw.in.7m | |||||||||||||||||||||||
Rainr.cs.1m | Rsw.in.32m | |||||||||||||||||||||||
Rainr.ott.1m | Rsw.out.0.5m | |||||||||||||||||||||||
Vis.cs.1m | Rsw.out.2m | |||||||||||||||||||||||
Vis.ott.1m | Rsw.out.7m | |||||||||||||||||||||||
Depth | Rsw.out.32m | |||||||||||||||||||||||
t (thermo) | Tcase.out.0.5m | |||||||||||||||||||||||
Tcase.out.2m | ||||||||||||||||||||||||
Tcase.out.7m | ||||||||||||||||||||||||
Tcase.out.32m |
3D Winds | prs | dcs | cc | up | pc | dc | mw | mh | sh | sp | lc | IRGA | prs | dcs | cc | up | pc | dc | mw | mh | sh | sp | lc | |
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
spd | h2o | |||||||||||||||||||||||
u | co2 | |||||||||||||||||||||||
v | h2o.1m | |||||||||||||||||||||||
w | h2o.2m | |||||||||||||||||||||||
spd.1m | h2o.3m | |||||||||||||||||||||||
spd.2m | h2o.7m | |||||||||||||||||||||||
spd.3m | h2o.17m | |||||||||||||||||||||||
spd.7m | h2o.32m | |||||||||||||||||||||||
spd.17m | co2.1m | |||||||||||||||||||||||
spd.32m | co2.2m | |||||||||||||||||||||||
u.1m | co2.3m | |||||||||||||||||||||||
u.2m | co2.7m | |||||||||||||||||||||||
u.3m | co2.17m | |||||||||||||||||||||||
u.7m | co2.32m | |||||||||||||||||||||||
u.17m | Soils | prs | dcs | cc | up | pc | dc | mw | mh | sh | sp | lc | ||||||||||||
u.32m | Gsoil | |||||||||||||||||||||||
v.1m | Qsoil | |||||||||||||||||||||||
v.2m | Tau63 | |||||||||||||||||||||||
v.3m | Lamda | |||||||||||||||||||||||
v.7m | Tsoil.0.6cm | |||||||||||||||||||||||
v.17m | Tsoil.1.9cm | |||||||||||||||||||||||
v.32m | Tsoil.3.1cm | |||||||||||||||||||||||
w.1m | Tsoil.4.4cm | |||||||||||||||||||||||
w.2m | Vheat | |||||||||||||||||||||||
w.3m | ||||||||||||||||||||||||
w.7m | ||||||||||||||||||||||||
w.17m | ||||||||||||||||||||||||
w.32m |
49 Comments
Jacquelyn Witte
Jan 06, 2022NChart and QC tables are not reporting the data (NCharts as of 2022-01-04 23:27:30) so I looked at snapshots of data from the DSM Dashboard (thanks Gary). The exceptions are the supersites which I cannot find on the Dashboard. The default is green until I or anyone can see the data.
Jacquelyn Witte
Jan 07, 2022Other
Jacquelyn Witte
Jan 08, 2022Radiometer
Tsoil.pc - measuring in the negative hundreds. Added na to cal file in QC/ folder until fixed.
h2o/co2
Lamda.sh is not like the others.
Other
Steve Oncley
Jan 08, 2022Jacquelyn Witte
Jan 09, 2022Thanks Steve.
Radiometers
Soils
Lamda.sh - measurements have settled since ~ 2022 01 07 13:57:30
h2o/co2
Depth
Jacquelyn Witte
Jan 10, 2022Radiometer
h2o.17m.prs - still not like the others. 3-4 times higher than the others.
Depth
Gary Granger
Jan 10, 2022The HRXL at UP occasionally reports messages of R5000 which are clearly not measurements but some kind of status or error message. I set a minValue for Depth to -0.5m, so that will filter out all the 5000 values which get converted to -3 by the default Snow.dat calfile. dc, cc, and up have all exhibited this problem, so those depth measurements should look better now. I think positive values are not a problem; they indicate either snow accumulation or else the sensor was mounted less than 2m above the ground, since all sites have a default offset of 2m.
I have generated a merged dataset manually in /scr/isfs/projects/CFACT/merge and am running statsproc on the merge for noqc_geo and noqc_instrument. When that finishes, the data in ncharts should contain all the data collected so far processed with the latest configuration.
I thought we needed to measure the height above ground of each HRXL to create a per-site calfile with a custom offset, but I don't know if anyone has done that yet.
Jacquelyn Witte
Jan 11, 2022Lowered/Raised DCS tt: Jan 10 15:12:30 - Jan 10 16:22:30
Lowered/Raised DCS tt: Jan 07 10:07:30 - Jan 08 04:27:30
Updated the tracker - FYI - there is a tab for Raise/Lower towers.
Do we have to manually add these filters to all the DCS QC files?
Jacquelyn Witte
Jan 11, 2022Soils
Sonics/irga
Depth
Lamda.sh - still not like the others - see QC tables
Jacquelyn Witte
Jan 11, 2022Looks like h2o.17m.prs is roughly 2-3 times that of h2o.17m.dcs.
Multiple by scale factor ~ 0.36?
Steve Oncley
Jan 11, 2022Jacquelyn Witte
Jan 12, 2022Soils
Depth
Sonics/Irga
Lowered/Raised PCS tt: Jan 11 09:12:30 - Jan 11 10:12:30
Steve Oncley
Jan 12, 2022Until further notice, none of the satellite sites have thermocouples installed, so these t.xx data can be ignored. We're hoping that the PIs will decide to install these later in the project when the survivability rate of thermocouples in the field is better known.
Steve Oncley
Jan 13, 2022The NR01.mw sample at 2022 01 13 06:23:05.1063 MST missed sending a "." (actually two characters "6." or ".6" in Tcase, so Tcase, and the resulting Rlw.in and Rlw.out values were huge. As far as I can tell, this was only one (5 second) sample, and would be trivially filtered in the xml by putting min/maxValues for Tcase. I think I've seen a similar error a week or so ago, though I don't know if it was on this sensor.
Of course, if one "." can be skipped, it begs the question of whether other, more subtle, data transmission errors are occurring. I'm not (yet?) using checksums in V2.8 code.
Jacquelyn Witte
Jan 13, 2022t.7m.dcs - spiked, then no data
Vis.ott.1m.prs - no data
Depth
Soils
Everything else is the same - see Data Matrix above.
Jacquelyn Witte
Jan 14, 2022Site SP
Isabel Suhr
Jan 14, 2022SP was off the net until Gary’s modem script brought it back, but for some reason nidas didn’t start after reboot. I started it manually and we should be getting data now.
Jacquelyn Witte
Jan 15, 2022MW
Depth - no data at DCS, CC and SP
Jacquelyn Witte
Jan 17, 2022Soils @ DCS - no data since 12:57:30.
Isabel Suhr
Jan 17, 2022long story short, they're back: https://jira.ucar.edu/browse/ISFS-535
Jacquelyn Witte
Jan 18, 2022SP - no data anywhere since Jan 18 13:47:30. Its Dashboard site cannot be reached.
Soils
Lowered/Raised PCS tt: 2022 Jan 17 10:22:30 - 14:42:30
Jacquelyn Witte
Jan 18, 2022Thanks Isabel for getting data reporting!
See Wiki Blog Ops 1/18 and Jira link
Jacquelyn Witte
Jan 19, 2022sonic/irga at PRS.2m - data are back!! (as of Jan 17 10:12:30)
Depth
Lamda
Isabel Suhr
Jan 19, 20222m sonic at PRS wasn't showing up in netcdfs for a while because it was plugged in to prsg instead of prsr due to port problems: https://jira.ucar.edu/browse/ISFS-529. It was collecting data, but the variable name had to be different to avoid conflicts so I added a .g suffix. I haven't updated the stats_5min xml to get the new variable name so it wasn't showing up in ncharts and qctables.
That's also what's up with the ott at PRS - it's currently plugged in to prsg, not prsr, and so I had to add a .g suffix to the variable name, so it's not showing up in the netcdfs.
Rebooted cc and dcsr, HRXLs are now reporting.
Jacquelyn Witte
Jan 20, 2022Data checks through ~09:07:30 this morning - what Ncharts is showing so far ...
PRS @2m for sonic/irga all green since these data are actually not missing, just renamed but not integrated into the visualizations. See Isabel’s comment yesterday (above). Thank you Isabel!
t.mh - Thermopile time series is not like the others. Started 06 Jan 20:02:30. See QC tables.
Vis.ott.1m.prs - no data
Lamda
Depth
Jacquelyn Witte
Jan 20, 2022Are netcdfs being generated on barolo? Ncharts stops at ~ 09:07:30 this morning. I’m not sure how often they get updated.
Steve Oncley
Jan 20, 2022I just restarted dsm_server – almost a daily occurrence these days. "dsp" now works again, so hopefully statistics will start flowing.
FYI, the steps are:
Jacquelyn Witte
Jan 21, 2022Noticed
h2o/co2 - no data coming in on NCharts or Dashboard at: MW, UP, PC, MH
Jacquelyn Witte
Jan 21, 2022h2o/co2 - back online on the Dashboard.
Rpile.in.up - recording 0’s or near 0’s. - behaving like Rpile.out between Jan 20 23:17:30 - Jan 21 10:42:30. Suspicious? Made note in tracker.
t.mh - thermopile time series is not like the others. Started 06 Jan 20:02:30. Behaves similar to tc.mh
Depth.cc - no data
Steve Oncley
Jan 21, 2022Rpile.up is now back to non-zero. Rpile.mw shows a similar decrease during this period and RH.2m.up is essentially saturated during this period. Interestingly, Wetness.up just seems to wander. I conclude that this is a real fog event where the "sky" and "ground" radiative temperatures are essentially the same as Tcase, so that Rpile, which represents the difference between these radiative temperatures and Tcase, is near zero. All is okay.
Isabel Suhr
Jan 21, 2022"t.mh - thermopile is not like other" thermocouple?
There are currently no thermocouples installed at satellite sites, so the values reported to nidas are meaningless.
Steve Oncley
Jan 21, 2022Still, I don't know why mh, and differently, dc, report differently, since they are all the same circuit. I'm partly expecting some unpleasant surprises when/if we finally populate the satellite thermocouples.
Perhaps, it would be a good exercise on a nice day to plug thermocouples in at mh, dc, and one other location for a few hours....
Jacquelyn Witte
Jan 22, 2022PRS for all sensors at 7m, 17m, 32m - no data since 10:42:30
(I don’t know if I just have to wait longer for it to appear on NCharts)
Isabel Suhr
Jan 22, 20227/17/32m sensors all go to the prst dsm, which as been off the net since 10:45. Hopefully it'll reboot and come back after 4 hours, or we might go visit PRS this afternoon.
Jacquelyn Witte
Jan 22, 2022Seems all I do these days is report no data on Ncharts ...
A few times-series remain suspicious - see Data Matrix.
Thanks Isabel for the quick responses.
Jacquelyn Witte
Jan 24, 2022Radiometer at DCS, PRS
Tsoil.0.6cm.prs
h2o/co2 - no data at
Depth - no data at DCS, CC (again)
Lamda - still suspicious behavior at DCS, SP, MW
Jacquelyn Witte
Jan 24, 2022Rfan @ DC
Jacquelyn Witte
Jan 25, 2022Tsoil.0.6cm.sp has started exhibiting similar behavior as Tsoil.0.6cm.prs (refer to yesterday’s comment)
Sonic - w.mh - Notice values are an order of magnitude higher than the rest. Reaching ~+1.5 m/s, which is high!
Can someone tell me if RH.cs.1m is supposed to be -99’s for a number of sites?
Depth
Jacquelyn Witte
Jan 26, 2022Lamda (TP01) @ DCS, CC, SP - all green
Depth - no data at DCS, CC, SP … just saying
w.mh - Really high, unlike the rest. From Jan 25 07:22:30 - 21:02:20. Making note in the tracker - something interesting to review.
Didn’t look at the thermopiles since I didn’t keep track of what is active and what is false. Oh well.
Everything else is the same - see Data Matrix.
Steve Oncley
Jan 28, 2022So, the tilt plot for mh is quite strange, as expected. Winds coming into the sonic (pointing east) would be deflected to cause an upward vertical velocity, as the plot shows. Winds from behind are in the wake of the parking lot/flagpole bases/trees and show as widely-varying downdrafts.
In other words, odd values of mean vertical velocity are completely to expected at this site.
Jacquelyn Witte
Jan 27, 2022Lowered/Raised Towers: See Ops 1/26
So far, nothing new to report. That’s good.
Steve Oncley
Jan 27, 2022After a talk with Jacquie, I swapped which supersite needed to use 2compSswap (to account for which Rsw was in which position). I've restarted the statsprocs, so hopefully we see reasonable Rsw values tomorrow...
Steve Oncley
Jan 28, 2022We don't have too much data yet this morning, but it does appear that all .ins and .outs look similar now.
Jacquelyn Witte
Feb 01, 2022No data on NCharts since yesterday morning. However, the Dashboard shows data at the non-supersites, except for MW where it hangs. I’m sure folks already know about this and are working on it.
In discussions with Steve I learned:
Rsw values have been swapped as of Jan 28. Thanks Steve!
Matthew Paulus
Feb 01, 2022I've noticed the lack of data on NCharts also. This seems to be an issue only with NCharts, as everything is mostly working on this end. I am emailing Gary about it.
Jacquelyn Witte
Feb 04, 2022Sonic/IRGA at LC are showing spotty data on NCharts. The Dashboard is showing a lot of invalid measurements, particularly for the Sonic.
NOTE: dsm_*.dat and isfs_*.dat.bz2 files have been manually backed up to Campaign Storage through 20220204_120000.
Matthew Paulus
Feb 04, 2022I've been noticing that too. It seems to be primarily at night, so I've been thinking that it's ice/frost/moisture issue of some kind. We haven't been by to check it out, but we need to take a soil sample and Leica measurement, so will swing by one of the next few days. I'm open to any ideas from Steve if he thinks something else could be causing this issue.
Jacquelyn Witte
Feb 08, 2022Lowered/Raised PRS (yesterday)
2022 Feb 07 13:37:40 - 14:27:30 - using Pirga@32m
Matthew Paulus
Feb 18, 2022I have rsync'd all the ISFS raw data onto campaign storage through about 1500 MST on 2/18/22.