tnw07b is responding to pings but ssh connections are reset. Oops, I made a mistake, nmap does look like a DSM. (Port 22 is ssh, 8888 is tinyproxy, 30000 is dsm. Xinetd check-mk 6556 is not listed I suspect because nmap does not scan it by default.)
[daq@ustar raw_data]$ nmap tnw07b
Starting Nmap 7.40 ( https://nmap.org ) at 2017-03-10 17:40 WET
Nmap scan report for tnw07b (192.168.1.146)
Host is up (0.055s latency).
Not shown: 997 closed ports
PORT STATE SERVICE
22/tcp open ssh
8888/tcp open sun-answerbook
30000/tcp open ndmps
And data are coming in:
tnw07b:/dev/gps_pty0 7 2 32 2017 03 10 17:41:25.874 03 10 17:41:41.036 2.04 0.142 0.925 69 80
tnw07b:/dev/ttyUSB0 7 22 16 2017 03 10 17:41:25.828 03 10 17:41:40.958 0.99 1.008 1.009 39 39
tnw07b:/dev/ttyUSB4 7 102 16 2017 03 10 17:41:26.368 03 10 17:41:41.434 1.00 1.004 1.005 38 38
tnw07b:/dev/ttyUSB7 7 32768 4 2017 03 10 17:41:26.049 03 10 17:41:41.049 0.20 4.772 5.228 17 30
The data connection to dsm_server is in fact from the right IP address, so tnw07b appears to be configured correctly:
[root@ustar daq]# netstat -ap | grep tnw07b
tcp 0 0 ustar:51968 tnw07b:43666 ESTABLISHED 9440/dsm_server
According to nagios, tnw07b was responding to check-mk requests until 2017-03-09 10:52 UTC, so something happened then which now causes network connections to be reset. Probably this system needs to be rebooted.
Steve noticed that ports ttyS11 and ttyS12 are no longer reporting any data on rsw04. After getting rsw04 updated and clearing off the USB yesterday and restarting DSM, those ports are still not reporting. They were working until Feb 25. ttyS10 was out for a while also, but it came back this morning at 2017 03 10 12:16:47.339, before the reboot.
[daq@ustar raw_data]$ data_stats rsw04_20170*.dat
Exception: EOFException: rsw04_20170310_155028.dat: open: EOF
sensor dsm sampid nsamps |------- start -------| |------ end -----| rate minMaxDT(sec) minMaxLen
rsw04:/dev/gps_pty0 35 10 3944084 2017 02 03 09:44:08.995 03 10 15:58:31.569 1.29 0.015 1090606.000 51 73
rsw04:/var/log/chrony/tracking.log 35 15 133438 2017 02 03 09:44:53.133 03 10 15:58:25.024 0.04 0.000 1090616.750 100 100
rsw04:/dev/ttyS11 35 100 38021353 2017 02 03 09:44:08.517 02 25 09:48:48.136 20.00 -0.016 0.992 60 77
rsw04:/dev/ttyS12 35 102 38021782 2017 02 03 09:44:12.831 02 25 09:48:48.206 20.00 -0.107 1.390 40 125
rsw04:/dev/dmmat_a2d0 35 208 39114544 2017 02 03 09:44:08.570 03 10 15:58:31.363 12.84 0.034 1090604.875 4 4
rsw04:/dev/ttyS10 35 32768 767733 2017 02 03 09:44:13.130 03 10 15:58:31.137 0.25 -0.031 1132080.875 12 104
Steve tried connecting to the ports directly yesterday and did not see anything. After the reboot, I still don't see anything either. This is a viper, so I'm thinking ports 11 and 12 are on the second emerald serial card, and these log messages are relevant:
[ 41.641774] emerald: NOTICE: version: v1.2-522
[ 41.842945] emerald: INFO: /dev/emerald0 at ioport 0x200 is an EMM=8
[ 41.871947] emerald: WARNING: /dev/emerald1: Emerald not responding at ioports=0x240, val=0x8f
[ 41.881346] emerald: WARNING: /dev/emerald1: Emerald not responding at ioports=0x2c0, val=0x8f
[ 41.890757] emerald: WARNING: /dev/emerald1: Emerald not responding at ioports=0x300, val=0x8f
(I assume Preban also was there...)
Working very late (until at least 10:30pm), the DTU crew got all of our installed masts on the network, though a few DSMs didn't come up. We're very grateful!
From several Per emails:
The fiber for the internet is still not working, but José C. has promised that someone will come on Tuesday to have a look at it. I can see that the media converter reports an error on one the fibers.
We have brought a Litebeam 5 AC 23dBi with us and we have placed it on the antenna pole of the ops center. That has helped significantly on the performance and stability of link to the ridges. So I don’t think It’ll be necessary for you to manufacture any special brackets.
We have then placed the “old” litebeam from the ops center according to Teds plan at rNE_06. We have also placed the 19 dBi spare NanoBeam on RiNE_07 and reconfigured Tower 10 to match the new NanoBeam. So now we’re only lacking to replace the last of the 3 Prisms which I noticed was now mounted in tower 37. The Litebeam that Ted has ordered could maybe then replace that one?
We have gained some more bandwidth from the ops center to tower 29 by moving the frequencies further away from the ones being used by the two sector antennas at tower 29. It seemed like these three antennas close by each other were interfering.
As you already has discovered the fiber was fixed to day. It turned out that we had two issues with the connection out of here. Rx fiber was broken close to the first junction box they have. Aparently a couple of kilometers from here. The Tx fiber also had a problem with too sharp a bent in the very first electricity pole outside the building. The latter could explain the changing performance we were seeing on the line performance.
The last 100m tower was successfully instrumented today, and your DSM’s should with a little luck be visible on the network.
We have changed the Ubiquiti config in the 4 army alu towers behind riNE07. They should now be online.
A few of the ubiquities on the towers were not set up with the proper wireless security rules, some were locked on the MAC address of the old AP we replaced (the Prism) and the last one was set in the wrong network mode.
We have moved a few towers from the planned accesspoint to another were the signal quality was higher. I still miss to correct it on the spreadsheet, I’ll do that asap.
The ARL ubiquities were all having the wrong PSK. José C. forwarded me a mail from a Sean, where he says there’s an IP conflict in one of his units, but they all seemed to have the IP address stated to the far right in the spreadsheet. And not the .110 to .113 stated in the mail. I were not able to access the web config page as described in his mail either, but since the IP’s matched Ted’s spreadsheet I put them on the network.
This was reporting all NA. pio got it to work. I'm actually surprised, since I thought we had seen this problem in Jan and had even sent people up the tower to check the sonic head connection, with no success then...
Now that the network is up and we can look at things, I'm finding lots of TRHs with ifan=0:
tse06.2m: #67, no response to ^r, responded to pio (after power cycle, responds to ^r)
tse06.10m: #43, no response to ^r, pio didn't restart fan (after power cycle, responds to ^r)
tse06.60m: #8, responds to ^r and pio, but didn't restart fan
tse09b.2m: #103, ^r worked
tse11.2m: #120, no response to ^r, responded to pio
tse11.20m: #116, responds to ^r and pio, but didn't restart fan
tse11.40m: #110, responds to ^r and pio, but didn't restart fan
tse11.60m: #121, was in weird cal land, no response to ^r, responded to pio
tse13.2m: #119, no response to ^r, pio didn't restart fan (after power cycle, responds to ^r)
tse13.100m: #111, no response to ^r, pio didn't restart fan, reset CUR 200, now running at 167mA (and T dropped by 0.2 C). WATCH THIS! (has been running all day today)
tnw07.10m: #42, no response to ^r, responded to pio
tnw07.60m: #125, ^r killed totally! pio doesn't bring back. dead.
Josés Carlos and Palma today adjusted the PITCH of the ops center antenna, which immediately brought tse13 and rne06 online, and improved the connection to rne07. So, for the first time, we now have all links to the ne ridge in place and working. With these links up, we are also able to get to many of the sites in the valley and sw ridge – now 17 total stations. There are another 4 sites that should be on the network, but aren't connected. An additional 3 sites are on the network, but the data system won't connect. So...progress, but still some work to do.
We think we've figured out our problem with the RMYoung sonics – our power supplies are not producing high enough voltage. The RMY has a minimum voltage requirement of 12V and our power supplies are producing 12V. However, the voltage drop through as much as 55m of DC cable (from the fuse boxes on the power drop through the DSM to the sensor) has been 0.5V or more. tnw11, for example, was reporting Vdsm=11.2V. We note that the minimum voltage for all other sensors is below 10.5V, so this is a problem specific to the RMYs. This explanation is consistent with these sensors working in the ops center (and earlier, at ARL), but not on the tower.
The solution is to use higher-voltage power supplies for the (7) masts that have RMYs and AC power. (Solar-powered stations should have batteries providing nominally 12.7V.) We've ordered 14 V supplies that we will bring with us on the next trip. (They need to have our DC power connector added to them before use, which is easiest to do in the lab.)
Noticed that some Ifans were 0:
tse13.80m: wouldn't respond to control-R, but came back up with a power cycle (pio)
tse13.2m: responds to control-R, but immediately fan is 0. I tried increasing the current limit to 200mA, and still is immediately 0. I suspect this is the VERTEX fan-monitoring-board-dead issue.
tse06.60m: responds to control-R, but immediately fan is 0. I tried increasing the current limit to 200mA, and still is immediately 0. I suspect this is the VERTEX fan-monitoring-board-dead issue.
tse09.100m: wouldn't respond to control-R, but came back up with a power cycle (pio)
Last (part) day in the field – I am writing this from Lisbon...
All but 2 masts that have been constructed are now powered up and collecting data. There are 7 masts remaining to be prepped in the ops center. There are 2? masts that have DTU sensors but have not yet been configured to send their data to our system (easy for them to do once they are on the network). We also have a handful of sensor issues (mostly RMYoung sonics) that are reporting continuously bad values. Also, standard maintenance, such as cleaning the radiometers, has not been done (though the rain is performing much of this task!). We had staged equipment at the two remaining constructed towers, but pulled this equipment off again for safekeeping until we next arrive.
Much of the network is functioning. HOWEVER, the primary mast that we have set up to get data out of the valley (rne07) failed to pair with the ops center. Thus we are receiving less than half of the masts that are working. (rne07 did power up after the quick work by the electricians to resolve a power issue at this site this morning.) The best guess is that the antenna is not pointed correctly, but time didn't allow us to fix it. Unfortunately, this means that ISS also is unable to connect with the network, so Gary reinstalled a cell modem at that site.
We also consolidated our mess at the ops center.
The German/Danish group also is leaving today (having placed many of the lidars), so no one will be on site for a while.
(Wifi wasn't working in my hotel last night, so this is late...)
A mixed day:
- rsw05, rsw07, and tnw03 all instrumented. Just need to hook up power (that we weren't supplied to do – will happen today)
- added access point for west slope on rsw06
- half instrumented v07
So...we've decided to remove tse02 prep (which I did this morning), which will leave 2 towers built here, but not instrumented, and fly home on our original schedule.
Measure and record tower guy wire tensions when appropriate. Two tension meters are available, will be stored in ops center. Bring a sharpie if planning on measuring tension to label tower leg. Living document available to anyone with access to the Perdiago Project Management Google Doc.It is located in the ISFS folder, or by the link below.
Site staged for future install. Power is available and verified at tower.
- Soil install
- Install and verify tower instrumentation
Valley site staged for future install. Power is available and verified at tower.
- Soil install
- Install and verify instrumentation
Slow progress today. Split into same 3 crews, except two crews also had 2 each INEGI climbers. Each INEGI crew installed 2 towers:
- tnw09: up, with solar power. Connects to network, but connection not yet built to get out of the valley.
- tnw11: up, installed, but all 3 RMYs report continuous bad data. This is very odd, since they worked in the ops center and (some) other RMYs work elsewhere in the experiment, so we don't think we're doing something systematically wrong.
- rne06: up, installed, on network, but it appears that the 20m sonic anemometer head was not plugged in – an easy fix.
- rne07: installed (including soils), but DC power connection is not live (even when a power supply was swapped with tnw11). Not clear what is going on.
The 3rd crew prepped v06 and v07 at the base of the tower and measured remaining guy wire tensions. We also installed soil sensors at tse02.
More prep work also was done at the ops center.
The question is where to go from here...