Weather: windy and rainy this morning. May clear up in the afternoon.
Sensor Status Overview
A number of sonics, ~7, were down this morning. It appears rain was the reason. Most are back up.
Three Licors were down but I was able to get running again via PIO commands.
TRH 10m and 60m on rsw06 were down due to fan current going to high. I was able to get them back
up via PIO. The max current cutoff has been increased, will monitor.
The nano-barometer at tse06 reports at a rate of 30Hz, according to data_stats and my own crude timing of the messages, unlike the other nano-barometers which report at 14.83 Hz, such as at tse08. Steve suggested there might be a configuration problem, so I tried to fix the configuration. Here's what I found out.
For reference, the configuration the nano should be using is described here: http://wiki.eol.ucar.edu/sew/ISFS/SensorInformation/NanoParo. I also compared the nano on tse06 with the one on tse08.
The settings all match: OI=0, TI=67, IA=7, XM=1, TS=0, MD=2.
Both are model 6000-15A, both have firmware version: R5.23.
I also checked serial numbers and crystal correction coefficients for each:
tse06: SN=123996 TC=.6781674
tse08: SN=123997 TC=.6781677
Both running at 9600 baud.
As a test, I set tse06 PI=134, cycled power on it, and then verified that PI was still set to 134, but the messages are still 30Hz. I did the same thing except setting OI=1 instead. That still does not change the rate from 30Hz, though the manual says that should cut the rate by half.
So I think there's something flaky going on here, and at the moment I am unable to do anything about it. The sensor continues to report at 30Hz, the others at 14.83Hz, and all will be converted to 20Hz for the high-rate netcdf output.
we instrumented V01 today, and it is on the network.
all sensors worked fine in the ops center this morning, but EC150 is not happy on the tower???
will work on servicing problem sensors tomorrow.
No sign of Samortechnica today, tomorrow is a holiday, so I don't expect them here then either.
applied power to TrNW01, Vall03, and fixed dc power cable to TrSE12 (a vehicle, probably the electric company tore it literally out of the electric box. We raised a new cable by about 1 meter over the old location and flagged the part overhanging the road.
Replaced soil mote cable which had been also yanked out of the mote box, and had bent pins.
Multiple TRH's were replaced at various stations - looks like that will be the lions share of work during the IOP, replace, repair, repeat.(Gnats are gumming up the fans)
There are now 37 stations collecting data and on the network.
went to RSW 03 to replace TRH's that were reading bad at 40 m (missing) and 10 m
- found that ports were not working (2 and 6) so moved to 8 and 10 (2 -8, 6 -10) called Gary to change the configs
- turns out that both TRH's were pulling to much current and blowing the fuses - will check out the sensors tomorrow to determine the problem.
- visitied RSW 05 for a problematic Metek sonic - also suspect a bad DSM because a replacement sonic and cable showed the same problem. will investigate the Metek parameters tomorrow as concerned with Viper DSM.
- Added TRH's at TNW03 at 10m (port 3) and 2 meters (port 1)
- noticed looking across the ridge that RNE 04 tower is now up. Will attempt to instrument that tower as time allows. Priority is to make sure existing network is operational.
- searched for other towers being erected, but found no signs of activity at TrNW04 or TrSE 15 and 16. Not getting any updates from Inegi about tower progress, or them installing batteries at 2 sites still waiting for power (TrNW01, or Va03)
- TrSe08 now has power according to the tower update spread sheet but is not operational - will investigate tomorrow?
- assisted water vapor dial with manpower unloading equipment.
instrumented TrNW06 today. last of the standing towers that I know of.
My apologies for redundant entries - I could not see my earlier posts for some reason, and wasn't sure if they had actually been posted.
Samortechnica was in the area today, working on ??? no idea where.
Enigi will be here tomorrow to install batteries and solar systems hopefully at TNw01, Va03, and TrSE 5?.
we will start replacing bad TRh's at multiple sites, and wearing down the "to do" list
already having people asking to borrow the truck and a driver to assist in various user deployments,
will help when I can, but our work must come first, and not sure how people will take that. Is NCAR running this circus or are we just another participant?
set up most of TNW 5,
(Upper part was done on the 13th)
CSAT3A 2047 and EC100 1798 at 10m
CSAT3A 2049 EC100 1696, EC150 1384 at 20 m
NR01-T2 s/n 2540 20 m
power mote #37
leaf wetness LW 013
soil moe # 7
Tsoil #09
Echo ERC5 001
TP01 200669
Flux Plate HO 23009
12m RMY 81000 ND/ARL # 650
8m RMY ND/ARL #1377
top DSM #253
mote # 37
6m RMY ND/ARL 1360
4m RMY ND/ARL 498
2m RMY ND/ARL 495
all sonics 12 meters and lower were pointed to NW because of trees (had to trim many branches just to mount the brackets)
RSW 2 visit - 10 m sonic ok, 20 m sonic not good, will replace later
RSW 3 40 m TRH fallty, will replace later.
RSW 4 leaf wetness sensor - added pic processor
20 m licor RTS/DTS? no jumpers in Viper box to address.
RSW.in - RSW.out - config change? not sure how to accommodate.
RSW 5 added POE cable (ops center made for Ubiquity Ant, now have telecoms but realized that was not needed because
sonic port and Ubiquity ports were switched.
sentra pwr was not plugged in.
now all is O.K.
not sure what happened to my blog note last 3 days, I noticed they were missing...
anyway friday 4/14 V04 hooked up pwr and soil mote
S/n for RNE 03 : DSM # 240
Pwr mote # 33
Csat3a P1005
EC100 #1807
TSe 13 replaced 2m TRH - 40 and 100 M trh's still bad - no safety equipment to go that high with us.
Grounded RNE 07, TNW 11, RNE 6,TNW 10, TNW 9, RNE 1, RNE 2, RNE 3. TSE 11 DSM's
Replaced RSW 3 20 m TRH and recorded S/N's
finished TNW 07 today
added 4 levels of RMY sonics to tower to finish the array.
also visited TNW 05 to configure the Ubiquity and upper DSM with proper ISP address to put it on the network.
Cleared the sea container of surface group items before the horde arrives tomorrow.
Only one more tower to outfit (TNW 06) until Samortechnica builds more! (we were told today that they will have 6 -7 more towers by the end of the week....)
purchased more ground rods and grounding wire to accommodate.
we set up TNW05 today (and yesterday)
all sonics below 20 m are pointing to the NW because of trees. Had to remove many limbs from a tree that was leaning into the tower just to be able to hang the booms.
lower DSM has wrong IP address, need to bring a laptop up to the site to remedy.
visited many sites yesterday to work on "to do" list, see Gary's posts from earlier today.
The latest cockpit has been installed on eddy and is ready for testing on the large display. At the moment, we do not know the location of the stand or the long 12' HDMI cable, nor do we know where it should end up. Once those are figured out (or a new cable purchased), we should be able to get it working with eddy same as at VERTEX. eddy has a wire ethernet connection to the ASUS wireless bridge, so it should be possible to move eddy anywhere to be near the location of the display. I think it would also work to set the eddy wireless interface to the same static IP as the wired interface, so long as it is on the tower network (SSID perdigao and not CSCR).
Details on the VERTEX installation, such as desktop configuration for the second display, are in ISFS-113.
We visited rsw05 yesterday to get the DSM on the network. This is a viper v1. I logged in on the console, verified still could not ping radio or anything else.
The config says the METEK is in port 5, but actually found it in port 1, even though port 5 still had the cap with the leash. The mote was correct in port 2. USB at 54%.
vio reports ports are off, even though they clearly are on...?
daq@rsw05:/media/usbdisk/projects/Perdigao/raw_data$ vio 5
0
daq@rsw05:/media/usbdisk/projects/Perdigao/raw_data$ vio 2
0
After Dan climbed the tower to replace the bulgin ethernet cable with a straight ethernet cable, and came down dressing the cables, we finally realized that the ethernet and metek cables had been swapped from the beginning. That makes a lot of sense, since it explains why there was no signal at all on ethernet rather than just bad signal, and why Dan could hear the Metek at the beginning, very loud, but couldn't after I disconnected what I assumed was the ethernet bulgin.
The Setra pressure sensor was not plugged into power, now the a2d pressure measurements look much better.
We plugged the METEK into port 5 and verified that the messages looked good. The Viper-radio link now works over the new ethernet cable, but probably it would have worked over the bulgin ethernet also. Later, from ustar, I ran update, rsync, tinyproxy, and clean-usb playbooks successfully on rsw05. rsw05 has been up since 2017-04-12, probably the last power outage.
Today, there is no METEK data. Looking back for data on port 5 at rsw05, this is the first data file where messages appear and look good, at least at first:
[daq@ustar raw_data]$ data_stats rsw05_20170415_150215.dat
2017-04-16,11:47:44|NOTICE|parsing: /home/daq/isfs/projects/Perdigao/ISFS/config/perdigao.xml
Exception: EOFException: rsw05_20170415_150215.dat: open: EOF
sensor dsm sampid nsamps |------- start -------| |------ end -----| rate minMaxDT(sec) minMaxLen
rsw05:/dev/gps_pty0 36 10 6986 2017 04 15 15:02:15.645 04 15 15:59:59.645 2.02 0.135 1.148 51 73
rsw05:/var/log/chrony/tracking.log 36 15 237 2017 04 15 15:02:26.327 04 15 15:59:52.910 0.07 0.000 16.199 100 100
rsw05:/dev/ttyS5 36 100 174737 2017 04 15 15:02:15.771 04 15 15:55:16.049 54.94 -0.005 0.359 2 89
rsw05:/dev/dmmat_a2d0 36 208 69288 2017 04 15 15:02:15.597 04 15 15:59:59.999 20.00 0.048 0.052 4 4
rsw05:/dev/ttyS2 36 32768 693 2017 04 15 15:02:20.421 04 15 15:59:55.402 0.20 0.195 5.695 11 52But a few minutes later in this file problems appear:
[daq@ustar raw_data]$ data_dump -i 36,100 rsw05_20170415_150215.dat
|--- date time --------| deltaT len data...
2017 04 15 15:02:15.7711 0 44 M:x = 124 y = 519 z = -38 t = 2218\r\n
2017 04 15 15:02:15.9941 0.223 44 M:x = 125 y = 523 z = -4 t = 2257\r\n
2017 04 15 15:02:16.0165 0.0224 44 M:x = 87 y = 514 z = -2 t = 2248\r\n
2017 04 15 15:02:16.0389 0.0224 44 M:x = 68 y = 535 z = -11 t = 2236\r\n
2017 04 15 15:02:16.0613 0.0224 44 M:x = 122 y = 528 z = -8 t = 2203\r\n
2017 04 15 15:02:16.0837 0.0224 44 M:x = 158 y = 520 z = -7 t = 2191\r\n
2017 04 15 15:02:16.1061 0.0224 44 M:x = 112 y = 511 z = -3 t = 2195\r\n
...
2017 04 15 15:05:47.0883 0.05001 44 M:x = 170 y = 141 z = -46 t = 2119\r\n
2017 04 15 15:05:47.1386 0.05031 44 D:x = 32767 y =-32768 z = 32767 t = 0\r\n
2017 04 15 15:05:47.1611 0.02245 18 E:quality < 10%\r\n
2017 04 15 15:05:47.1698 0.008694 20 E:data lost 40403\r\n
2017 04 15 15:05:47.1883 0.01849 44 M:x = 167 y = 150 z = -43 t = 2114\r\n
2017 04 15 15:05:47.2384 0.05013 44 M:x = 176 y = 154 z = -50 t = 2121\r\n
2017 04 15 15:05:47.2884 0.04997 44 M:x = 170 y = 154 z = -51 t = 2126\r\n
2017 04 15 15:05:47.3450 0.05668 44 M:x = 171 y = 149 z = -48 t = 2116\r\n
...Then finally it goes berzerk:
2017 04 15 15:06:26.2324 0.05 44 M:x = 55 y = 175 z = 15 t = 2126\r\n
2017 04 15 15:06:26.2823 0.04994 44 M:x = 60 y = 177 z = 14 t = 2129\r\n
2017 04 15 15:06:26.3391 0.05681 44 M:x = 64 y = 181 z = 16 t = 2132\r\n
2017 04 15 15:06:26.3547 0.01561 89 C:\xbf\xa5\xbf\xb7\xbf\x9d\x95\x95\xbfK\xbf\xa5\xbf\xbf\xbf\xbf\xbf\x95\xbf\x17\xbf\xc5\xbf\xbf\x9b\x9d\xd9\x9f\xe5k~\xc0~\xee\xe8+\xfb\xfb\xfb\xfb\xfb~\xfb~\xa5\xbf\xbb\xbf\xbf\xbf\x9f\xbfK\xbf\xa5\xbf\xbf\xbf\xbf\xbf\x9f\xbfW\xbf\xa5\xbf\xbf\xbf\xbf\xbf\xdb\xe5\xebu\xcb]\x95\xbd\xb7-\x97M\xbf\xa7\r\n
2017 04 15 15:06:26.4005 0.04579 19 E:unknown symbol\r\n
2017 04 15 15:06:26.4099 0.009346 44 D:x = 67 y = 182 z = 19 t = 2133\r\n
2017 04 15 15:06:26.4323 0.02237 89 C:\xbf\xa5\xbf\xbf\xbf\x95\xaf\x9d\xbf\x0b\xbf\xa5\xbf\xbf\xbf\xbf\x9d\x91\xbf\x17\xbf\xa5\xbf\xbf\x9b\x9d\xd9\xdb\xe5\xeby\x8b~ 155 Z"- 5 t!\x1d !\xb210MJ\xa2~~~($"0 Z"- 0 T"- \r\n
2017 04 15 15:06:26.4786 0.04634 19 E:unknown symbol\r\n
2017 04 15 15:06:26.4880 0.009367 44 M:x = 68 y = 183 z = 19 t = 2126\r\nOccasionally there are still good messages. It stops entirely at 15:55:16. Given the theory that the Metek was plugged into the 24V ethernet port for a few weeks, maybe it burned up? I have no other explanation for this behavior.
I suppose we could try a different port.
Added a second DSM to this site to sample all but the analog Setra sensor. Fixed noisy messages from 10m RMYoung by moving it to a different port. RMYoung at 20m is reporting non-zero diagnostic flag, and nothing we did fixed it. The voltage at the DSM mote is 12.7V, but maybe there's too much of a drop up the tower, or maybe the RMYoung is beyond it's useful life.
- Replace the RMYoung at 20m.
- Rearrange the DSMs so that both have a clear view of the sky. Putting rsw02x directly above rsw02 seems to have prevented GPS lock in rsw02. Oops.
The JIRA issue on the crashing TItan problem has been updated: ISFS-149.
System updates to ustar and eddy:
- dnf update
- timedatectl set-timezone UTC
- set desktop on eddy to KDE (aka Plasma)
- reboot