Blog

Blog

NR01 cleaning test

I just used the opportunity of staging for ARTSE to evaluate the effect of not cleaning our radiometers during Perdigao.  All of the Perdigao radiometers were packed without being cleaned, to enable this test to be done.  However, it is possible that the stretchy film that is part of the NR01 packing may have wiped some stuff off of the domes.  Two of these NR01s were brought back (by me) by hand – the rest are still in the seatainer.

Round 1 (27 Jul 2017)

Today (27 July), which is partly cloudy with boundary-layer Cu, we installed NR01 #7 at approximately 13:00 (local).  At approximately 14:43, I cleaned it during a period when the sun disk was clear of clouds.  The order was: wetting Rsw.in, Rlw.in, Rlw.out, Rsw.out, then wiping in the same order.  At the end of this cleaning, I added water to the wetness sensor, left the water on for about 10s, and wiped it clean, just as an indicator flag.  I ran rserial during this cleaning on a laptop.

From the rserial output, I see Rsw.in change from about 926 W/m2 before cleaning to 912 W/m2 after.  Thus, the effect of the dirt/pollen/smoke/oil/etc. was an enhancement of incoming solar radiation by about 1.4%.  Obviously, the primary effect of the dirt was to enlarge the image of the solar disk.  This effect will be difficult to model and thus correct the data.  It would have been a good idea to measure the direct and diffuse radiation separately.  When I measure the other NR01, I'll also take data using the shadowing paddle.

Round 2 (1 Aug 2017)

Continuing the ARTSE piggy-back, also tested NR01 #12 at about 11:20 on 1 Aug 2017 with nearly clear skies.  This time the procedure was:

  • make sure data were being recorded on USB stick
  • nevertheless, also logged data through minicom, with rserial running
  • ran in clear skies for a while
  • shaded Rsw.in for 20s with a paddle.  (forgot this time that I should crouch down to prevent my head from being visible to radiometers)
  • shaded Rlw.in for 20s with paddle
  • added water to wetness sensor to indicate cleaning
  • wetted and wiped Rsw.in
  • same to Rlw.in (was visibly dirty)
  • same to Rlw.out
  • same to Rsw.out
  • dry-wiped Rsw.in
  • same to Rlw.in
  • same to Rlw.out
  • same to Rsw.out
  • wiped dry the wetness sensor (showed total time cleaning was 175s)
  • shaded Rsw.in for 20s with paddle
  • shaded Rlw.in for 20s with paddle

All of this is to try to get the effect of cleaning separately for direct and diffuse radiation (though we don't expect much change on Rlw), since previous data showed that a correction model might need to treat direct and diffuse separately.

In this case, the results were:

  • Rsw.in total before/after: 887/887 (no change)
  • Rsw.in diffuse before/after: 101.5/102.5
  • Rlw.in total before/after: 371.8/370.4
  • Rlw.in diffuse before/after: 374/372

All of these changes are quite small, <1%.  In the case of Rlw.in, my head in the field of view may have affected the results.  Thus, this radiometer did not require this full procedure and we will not have to correct its data.  Nevertheless, we should repeat this procedure on all the other radiometers when they return.

Also notable is that the diffuse effect on Rlw.in was small – under 2 W/m2 for a change in Rsw of 785 W/m2, or a "swcor" value of < 0.25%.  The Epply PIRs we tested in 2003 had values from 0.2–1.9%.  Thus, the NR01 Rlw is pretty good.

Round 3 (18 Sep 2017)

Approximately same procedure as Round 2, now with the remaining NR01s from the seatainer.  All but 2 Rsw.in values were lower after cleaning, however most data were taken during the afternoon when light was decreasing.  Time series were logged through minicom capture.  I used yet another new paddle, this one with aluminum tape (shiny) towards the sun, electricians tape (black) towards the sensor.

The final table with the analysis so far:

Sensor IDRsw.in beforeRsw.in afterExtrapolated change(before-change)/after (%)Rsw diff beforeRsw diff afterRlw beforeRlw afterbefore/after (%)Rlw diff beforeRlw diff afterdiff/normal before (%)diff/normal after (%)
1837.6834.7-2.00.669.565.3365.3360.91.2362.5359.0-0.8-0.5
2750.0736.525.0-1.668.957.3334.7337.9-0.9336.8347.30.62.8
3861.2858.5-0.80.478.671.7334.1336.5-0.7340.0340.41.81.2
4688.5669.214.90.768.258.4355.4352.50.8353.6349.7-0.5-0.8
5834.4835.0-11.0

1.2

72.567.4334.1335.4-0.4336.0339.70.61.3
6850.1844.0-6.81.572.164.6345.1343.30.5346.1345.90.30.8
7926912-1.5---------
8638.5633.38.1-0.562.654.3347.8347.50.1349.7348.80.50.4
9721.0713.911.7-0.667.364.1335.8336.8-0.3341.8343.21.81.9
10728.5715.411.40.2
65.458.3346.4347.5-0.3349.6351.90.91.3
11623.0606.01.02.661.653.7340.3341.7-0.4339.8338.6-0.1-0.9
12887887-0.0101.5102.5371.8370.40.4374

372

0.60.4
13681.0662.213.70.868.463.0342.6343.7-0.3344.9346.00.7

0.7

14854.6858.3-20.7

2.0

74.069.5338.2339.1

-0.3

336.9339.9-0.40.2

These data support the following conclusions:

  1. The maximum change in Rsw.in due to cleaning was 2.6%.
  2. The sense of all but 3 of the Rsw.in errors was less radiation after cleaning.  Thus, the "sparkle" effect seems to have dominated over shading.
  3. The maximum change in Rlw.in due to cleaning was 1.2%.  Most changes were quite small and were both positive and negative, i.e. not systematic.
  4. The maximum change of Rlw.in due to direct shading (the "f" correction) was 2.8%, but all but 2 sensors showed an increase of Rlw with decreasing Rsw. This suggests that the difference was due to the measurement method (paddle being too close?).  I don't understand this result.
  5. Cleaning did not have a <major> change on the f-correction values.

Overall, not terrible results, though radiation to 2% certainly isn't up to our usual standards. My time trend analysis could be improved, but probably won't change the general trend here.

Next step – repeat these tests on the 4-component sensors!



network work

Jakob had pointed out that most of the network was down.  (I've been on vacation and hadn't been monitoring.)  Apparently stations went down in the last lightning storm (again!), either Fri or Sat.  I just:

  • found a dead power supply in rne01 and replaced it with the one from rne02.  Of course, rne02 is a <repeater> so much of the network is still down.
  • reset differential protection on tse13
  • reset differential protection on tse11
  • reset differential protection on tse09

Of course, this is now <tear down>, so we shouldn't be working to <maintain> the network.  Nevertheless, several groups are still running, using this period for post-cals, etc.  Eventually, DTU will rework the network for operation after we are gone.

 

tse04 TRH.2m

While touring my family yesterday, stopped by tse04 and noticed that TRH.2m fan wasn't working.  Cycling power (unplug/plug at DSM) brought it back.  This done about 1230 (17 Jun).  Not that we need the data now, but hopefully this will be a check on whether Ifan monitoring was working.

Status up to 2017-06-14

No changes in instruments, everything seems to have been working as usual for the last few days.

I had to cycle the power on rsw02 twice in the last 24 hours, after it got hung up and stopped responding to pings, so we lost a few hours of data for P.2m.rsw02.

The substitute Ubiquiti radio at tnw05 eventually stopped working for some reason.  When I visited the site on Sunday to fix it, it came back after cycling the power.  Then it went out again by the next day.  So I visited the site again on Monday and swapped the original radio back into operation.  Since then that radio has not had any problems.  Go figure.

I leave today, Kurt and Dan are on their way here with another truck.

 

NR01 installed at v04

Tcase.in from the KZRAD has been missing since 2017-06-02.  It started showing problems on 2017-05-26.  So this afternoon I finally installed the last spare NR01 as a backup, from about 13:45 to 14:45 WEST on 2017-06-10.

It is only level by eye, and there was some disturbance to the ground to wash the radiometers. (I only washed the NR01, I don't know why I didn't think to wash the KZ also...)

I assumed it was better to mount it further from the dark horse beam, but now I see that the legs are probably in view of the NR01 downlooking radiometers.  Let me know if I need to change it.

 

Status as of 2017-06-09

Sonics and gas analyzers

I think all the sonics are working.

v01 10m IRGA

The v01 10m irgadiag has regular stretches of non-zero values, maybe 40% of samples are good.  I don't know if that's worth trying to fix, given the chance a new head could make it worse.  Maybe it's just a spider web, so if we get a chance to climb we should take a look.

v07 20m LiCOR

The LiCOR at v07 20m (port 3 on v07t) was not reporting, fixed by cycling power.

Other issues

tnw05u

I replaced the Ubiquiti radio at tnw05.  It was able to connect to its AP without climbing the tower to mount it.  Since the original had already been upgraded to firmware 8.1.4, the replacement was also upgraded to 8.1.4, and then I just saved the config from the original and restored it to the replacement.  Traffic to and from tnw05b and tnw05t has been normal, including rsyncs, however for some reason now tnw05u does not respond to ping, ssh, or https access on the WLAN interface.

 

Status as of 2017-06-08

Sonics and gas analyzers

All sonics appear to be working. The new METEKs needed cal files, so the netcdf data prior to the cal file additions are probably missing values. I think we also fixed an inconsistency in the cal file paths, so sites with more than one dsm may not have been applying the most accurate site-specific boom bearings.

TRH sensors

No change.

Other issues

tnw05u

The radio continued to hang up rsync connections today, even after a few configuration changes to match it exactly with a working radio, rsw06. The latest attempt was to upgrade to the latest firmware, 8.1.4, but even that did not work. We may try adding another radio in its place.  This is being tracked in ISFS-152.

Generating stats_5min.xml

Isabel has written a python script to generate the stats xml file from the sensor list, to reduce the chance for human error in generating in manually. She found several missing TRH sensors and one sonic at the wrong height. Once we've compared the netcdf output using the generated xml file, the manual file will be replaced.

 

Status as of 2017-06-07

Sonics and gas analyzers

No known problems at this time. Unlike the RMYoungs they replaced, the METEKs at tnw07b 4m and v07 8m do not report ldiag, so those are all missing in the QC tables, but I don't know at the moment how to fix that.

TRH Sensors

The only sensor known to be down at this point is the 60m on tse11.

Other issues

v05

We visited v05 to investigate why it was offline, even though the Ubiquiti was still reachable. Eventually we rebooted the Ubiquiti and the connection came back. The DSM had been up the whole time so no data were lost.

tnw05 rsync

The tnw05u radio continues to be a problem, hanging up rsync connections. No explanation or fix yet.

v04

Tcase.in is still down and nothing has been done about it. We still need to work on mounting a NR01 on the dark horse with the KZ.

 

Status as of 2017-06-06

Sonics and gas analyzers

v07 8m

Replaced RMYoung with a METEK, using the same port since all are filled. The arrow is pointing towards the tower in direction of boom. We discovered that the fuse to the serial interface boards would blow eventually once the METEK was running, so we replaced the 1A fuse with a 3A. The configurations have been updated and all appears to be working now. Port 1 is jumpered for RS485.

tnw07b 4m

Replaced RMYoung with a METEK, using port 5, now jumpered for RS485.

TRH sensors

No change.

Other issues

rsync

We have added rsync monitoring to nagios, and discovered that a few sites are days behind on rsync.  It turned out the problem was the Ubiquiti not allowing certain traffic, same symptoms as documented in ISFS-152. Two of the four systems were tnw05t and tnw05b, the same ones as reported in the original issue.  The problem keeps happening within several hours after rebooting the Ubiquiti, so there must be something particularly wrong with the radio at that site.

 

Status as of 2017-06-05

Sonics and gas analyzers

tnw09 10m

Replaced CSAT3A sonic head at 10m on tnw09. At first it reported all bad samples. So we plugged in the original again, thinking maybe some spider webbing had been interfering, but it still reported bad samples also. So we left the replacement installed, and lo and behold, a few hours later the diagnostic bits turned to zero and remained zero the next day. Go figure.

v07 8m

Had a METEK tested and ready to go, did not have time to install it.

tnw07b 4m

Had a METEK tested and ready to go, did not have time to install it.

Other issues

ARTSE

I have one full EC150 tested and set aside.

Status as of 2017-06-03

Sonics and gas analyzers

v07 8m

RMYoung on port 1 is still bad, but now we know that it is actually at 8m.

tnw09 10m

CSAT3 IRGA winds on port 2 still about 50% flagged.

tnw07b 4m

RMYoung on port 1 keeps going in and out. It looks like the bad samples peak in the afternoons, similar to v07 8m.

tse01 10m

No sign of problems since going out for several hours on 2017-06-02.

TRH

tse11 60m

No change.

Other issues

There seems to be a problem with the winds in the high-rate dataset not being oriented correctly to geographic coordinates, so I need to investigate that.

tse05 finally stayed up overnight.

v04 Tcase.in is still out. I will figure out how to mount our last NR01 on the dark horse and just record it as an additional radiation sensor.

Around 2017-06-03,14:00 UTC, Isabel and I visited v07 to attempt to replace the flaky RMYoung in port 1 with a less flaky RMYoung. We were able to use a ladder to reach the 4m boom and swap in the new sonic, but that did not improve anything on port 1. We discovered a fuse was not quite plugged in the whole way on port 3, so plugged it in. And at some point we "lost" all the serial ports. Probing the bulgin pins only showed 5V on some pins and no pins with 12V. So just to be sure, we powered down the DSM and replaced all the fuses leading up to port 1, including the 7.5A and 3A fuses on the power panel. After that all the ports resumed working, but I don't know if that means we really did have a partially failing fuse somewhere (seems doubtful), or something else got into a funky state.

We still saw almost all flagged samples from the RMYoung on port 1. So then we discovered that the configuration was incorrect. Port 1 is at 8m and not 4m, so we had replaced the wrong sonic. We unclipped all the cable loops hung on the tower and mapped them to their sensors, here's what we found:

PortHeightSensor
02mCSAT
18mRMYoung
26mRMYoung
34mRMYoung
62mPTB
70mSoil mote

So ports 1 and 3 were swapped, and I've fixed the configuration to match.

This means the sonic that really has been failing during the day is at 8m, so we cannot replace it without climbing. During our visit, I thought we determined eventually that the replacement did not work any better, but now I'm not so sure that we were looking at the right sonic, so maybe we still have one RMYoung with 2% flagged samples which could replace the 8m. Otherwise we have a METEK we could install, whenever we're able to climb.

Note that we disturbed the flow for the 2m CSAT sonic during our visit because we were working right next to it.

 

Status as of 2017-06-02

Sonics and gas analyzers

v07 4m

The plan is to replace it with the last RMYoung. 2% bad samples all the time would be better than 2% good samples during the day.

tse01 10m

CSAT3A on port 2 is reporting all bad samples as of 2017-06-02,08 UTC. So we only have to replace whichever of head or box is bad, or both, but we have spares. Cycling power did not help.

tnw09 10m

Lots of flagged samples still. This is a CSAT3 IRGA. Up to %30 of the winds are flagged in qctables, need to check on the gas diagnostic.

tnw07b 4m

Diagnostic still mostly bad.

TRH Sensors

tse11 60m

Still needs to be replaced.

Other issues

tse05 power

Isabel and I installed a second solar panel at tse05 around 16z. The panel is more horizontal and is aimed more west, since the first panel is aimed more east. It Looks like voltage jumped up a tenth, but maybe there will not be enough charging left today to get it through tonight. Maybe tomorrow night.

Status as of 2017-06-01

Sonics and gas analyzers

rsw02 20m

Replaced around 16z with help from Orson (ND).

v07 4m

Since this is a RMYoung, the problem could be related to voltage, but we have no more 14V adapters. The easiest replacements are the last working RMYoung, which reports about 2% bad samples in the ops center, and a Metek, since the Meteks mount onto the Samortechnica booms.

We visited the site around 17z, and measured 12.45V at the port 2 bulgin pin, and 12.85V at the power panel input pin. So I'd guess the RMYoung is getting enough volts since it is only 4m up. Vdsm is always more than 12V, even though it is measured at a soil mote 5m from the DSM.

I forgot to bring a ladder to look at the adapter in the junction box, so at the moment I'm assuming it's a 14V and it's performing well enough that the voltage should be high enough for the RMYoung.

tnw09 10m

Diagnostic flipping back and forth.

tnw07b 4m

Diagnostic mostly bad.

tse10 10m

Some spiking.

TRH Sensors

tse04 60m

INEGI replaced it, working now.

tse11 60m

Still needs to be replaced.

Status as of 2017-05-31

Sonics and gas analyzers

tse07 10m sonic

Replaced and working.

rsw02 20m

Still down, still needs to be replaced.

v07 4m

Still suspicious. A plot of ldiag.4m.v07 over the last several days is below, and the bad periods look diurnal. Maybe a temperature problem? This is an RMYoung, so I can look for a working RMYoung spare with which to replace it.

v06 20m LiCOR

Still missing samples later in the day, but it doesn't look that bad judging from a plot of the last several days. I think lidiag is usually non-zero, so the only diagnostic that matters is how many of co2.20m.v06 and h2o.20m.v06 are missing.  It does look like it's getting worse, but I don't know of anything we can do about it since we have no spares.

TRH Sensors

tse04 and tse11 60m

INEGI will replace at least tse04 tomorrow.

Other issues

tnw08 Vdsm

Still not there.  Steve says we have no more voltage motes, so it will never be there.

LiCORs for rsw01 and rsw07

The ARL shipment did not arrive today. Chris says ARL is going to plan as if they won't arrive and make do with the instruments they have.