Josés Carlos and Palma today adjusted the PITCH of the ops center antenna, which immediately brought tse13 and rne06 online, and improved the connection to rne07. So, for the first time, we now have all links to the ne ridge in place and working. With these links up, we are also able to get to many of the sites in the valley and sw ridge – now 17 total stations. There are another 4 sites that should be on the network, but aren't connected. An additional 3 sites are on the network, but the data system won't connect. So...progress, but still some work to do.
We think we've figured out our problem with the RMYoung sonics – our power supplies are not producing high enough voltage. The RMY has a minimum voltage requirement of 12V and our power supplies are producing 12V. However, the voltage drop through as much as 55m of DC cable (from the fuse boxes on the power drop through the DSM to the sensor) has been 0.5V or more. tnw11, for example, was reporting Vdsm=11.2V. We note that the minimum voltage for all other sensors is below 10.5V, so this is a problem specific to the RMYs. This explanation is consistent with these sensors working in the ops center (and earlier, at ARL), but not on the tower.
The solution is to use higher-voltage power supplies for the (7) masts that have RMYs and AC power. (Solar-powered stations should have batteries providing nominally 12.7V.) We've ordered 14 V supplies that we will bring with us on the next trip. (They need to have our DC power connector added to them before use, which is easiest to do in the lab.)
Hi Alec and Julian,
The Visual Basic program was completely blank (just a blank grey window) when I logged into the system and the wheel of death was rolling when I rolled the cursor over the VB windows. Had to use the task manager to close the windows since there weren’t even the controls in the upper right corner.
Shut down the O2 system. The program was locked up when I logged in remotely. This is the second time in the past few days that it has done this. We were running the system with the door open to the Analyzer box, and the ambient temp (on the omega meter) was only 30C so I don’t think the system was overheating. We will do our best to swap out the Licor for you, fortunately we will have some extra staff for this port call so the odds are good we can find or make the time to do the swap. I assume we should ship the old unit back to you via our cargo system. Thanks for the support and understanding of our limitations down here.
So the 02 system has been on the fritz for the past several days. Despite our reboots and cursing, the Licor seems to have finally quit.
Question – do you want the system to continue to run even though the Licor isn’t working? Is it collecting other data that’s worthwhile to you?
Here’s the weekly maintenance sheet. Still having some issues with the Licor locking up, reboots seem to be working but often it takes multiple attempts to get it going. Also, Working Tank 2 is pretty low (~250 psi), at what level should we switch to the other tank? I also adjusted the low pressure side to 3.5, since it was at 4.5.
Logsheet attached. No problems to report today!
Performed your changes to both the Omega and the program. So far the system is running as advertised, Tomega is in the 34.4°C area. After running for a couple hours the red temp light hasn’t turned off, we will keep an eye on it.
We will keep you advised on how the system is progressing.
So after I opened up the cabinet and let it cool, things seem to be running smoothly.
I propped the door open and let it run and we haven’t had an issue since for a full 24 hours.
Have you considered added better ventilation to that box? Even with the door open, the Licor is still about 105 degrees and warm to the touch.
When it locks up, there is no option to debug – it’s just totally hosed up and we have to use the task manager to kill the process.
The system locked up yet again. I opened the box to restart the Licor again, and I noticed it seemed a bit warm in there. I used a laser thermometer and measured the outside of the Licor to be about 45C and the ambient inside the box above 35C. Could the Licor perhaps be overheating? I turned it off and propped open the case to let it cool down. I will try again in an hour or so to turn it back on.
Thanks Alec! Just to clarify "CO2 A pegged at 400" is the correct behavior and not a sign of a problem. We force it to think the reference is 400 as it really is just a differential sensor so needs some known to compare to. Cheers,
Yes, the VM is showing data. That said, we are seeing what you say about the CO2A and CO2B (picture attached), any suggestions? We’ve had pretty good weather this cruise, temps haven’t dropped below freezing yet and today was sunny and clear, so I don’t suspect clogs in the intakes. My DAS display is showing O2(CO2) = 389 uAtm.
Nice work! Are you seeing something other than NR's in the Li-7000 box on the Visual Basic window?
Sorry the phone call today was so crappy, it’s a harsh continent J. Did as you suggested below, it took a couple reboots of the analyzer box but the Li-7000 finally came up. The front panel on the Li-7000 came up but without data, after the second reboot and a remote log into the instrument, it came up correctly. BTW it lives on Com1. I have attached pics of both states.
Hi Julian - thanks for all the help. Slight correction to what Andy said: just cycling power on the Licor will not fix the problem as it receives configuration information when the VB program starts up, so you would need to restart the VB program too. The best thing to do if you have time is a full shut down and restart. Sorry for the hassle, and thanks again,
The problem continues. Today during my checks I found the red watchdog lights on
Logging in to the computer I found the software was locked up again:
After using the task manager to kill the process, I was able to restart the software. No USB errors present themselves, and it seems to run fine.
We are somewhat at a loss as to what else we can do to troubleshoot the problems with your system.
Please advise how you’d like us to proceed. Until then, we’ll continue to restart the software as it crashes.
Last night the software on the O2 system locked up and required a reboot of the system. We are getting a green light on our RVDAS but the value for the O2(CO2) is 0 uAtm. Julian and I are not sure if this is valid data or if there is still something quirky with your system. The software seems to be running fine, but just want to make sure everything is good from your end. Let us know if the data is good and/or if we need to do anything on our end.
Happy New Year from the LMG. We just passed 68W and booted everything up.
Here’s what your system is showing. Not sure what ‘good’ Licor values look like, so please have a look and let us know if you’d like us to check on anything.
Noticed that some Ifans were 0:
tse13.80m: wouldn't respond to control-R, but came back up with a power cycle (pio)
tse13.2m: responds to control-R, but immediately fan is 0. I tried increasing the current limit to 200mA, and still is immediately 0. I suspect this is the VERTEX fan-monitoring-board-dead issue.
tse06.60m: responds to control-R, but immediately fan is 0. I tried increasing the current limit to 200mA, and still is immediately 0. I suspect this is the VERTEX fan-monitoring-board-dead issue.
tse09.100m: wouldn't respond to control-R, but came back up with a power cycle (pio)
Last (part) day in the field – I am writing this from Lisbon...
All but 2 masts that have been constructed are now powered up and collecting data. There are 7 masts remaining to be prepped in the ops center. There are 2? masts that have DTU sensors but have not yet been configured to send their data to our system (easy for them to do once they are on the network). We also have a handful of sensor issues (mostly RMYoung sonics) that are reporting continuously bad values. Also, standard maintenance, such as cleaning the radiometers, has not been done (though the rain is performing much of this task!). We had staged equipment at the two remaining constructed towers, but pulled this equipment off again for safekeeping until we next arrive.
Much of the network is functioning. HOWEVER, the primary mast that we have set up to get data out of the valley (rne07) failed to pair with the ops center. Thus we are receiving less than half of the masts that are working. (rne07 did power up after the quick work by the electricians to resolve a power issue at this site this morning.) The best guess is that the antenna is not pointed correctly, but time didn't allow us to fix it. Unfortunately, this means that ISS also is unable to connect with the network, so Gary reinstalled a cell modem at that site.
We also consolidated our mess at the ops center.
The German/Danish group also is leaving today (having placed many of the lidars), so no one will be on site for a while.
(Wifi wasn't working in my hotel last night, so this is late...)
A mixed day:
- rsw05, rsw07, and tnw03 all instrumented. Just need to hook up power (that we weren't supplied to do – will happen today)
- added access point for west slope on rsw06
- half instrumented v07
So...we've decided to remove tse02 prep (which I did this morning), which will leave 2 towers built here, but not instrumented, and fly home on our original schedule.
Measure and record tower guy wire tensions when appropriate. Two tension meters are available, will be stored in ops center. Bring a sharpie if planning on measuring tension to label tower leg. Living document available to anyone with access to the Perdiago Project Management Google Doc.It is located in the ISFS folder, or by the link below.
Site staged for future install. Power is available and verified at tower.
- Soil install
- Install and verify tower instrumentation
Valley site staged for future install. Power is available and verified at tower.
- Soil install
- Install and verify instrumentation
Slow progress today. Split into same 3 crews, except two crews also had 2 each INEGI climbers. Each INEGI crew installed 2 towers:
- tnw09: up, with solar power. Connects to network, but connection not yet built to get out of the valley.
- tnw11: up, installed, but all 3 RMYs report continuous bad data. This is very odd, since they worked in the ops center and (some) other RMYs work elsewhere in the experiment, so we don't think we're doing something systematically wrong.
- rne06: up, installed, on network, but it appears that the 20m sonic anemometer head was not plugged in – an easy fix.
- rne07: installed (including soils), but DC power connection is not live (even when a power supply was swapped with tnw11). Not clear what is going on.
The 3rd crew prepped v06 and v07 at the base of the tower and measured remaining guy wire tensions. We also installed soil sensors at tse02.
More prep work also was done at the ops center.
The question is where to go from here...
Station is up and sensors reporting, EXCEPT CSAT3A at 20m. Since it doesn't show a sonic head connected (using enter, D shows "SN0000" for the sonic), the fix should be just plugging in the head to the EC100.
Ted and Gary staged the sensors yesterday, all except for the Licor since we were missing the boom clamp. Here are the serial numbers:
- Licor 1166
- Nr01 2537
- EC100 1821
- Csat3 1120
- Csat3a 2033
Lessons learned: We should have demonstrated how to connect the Licor and sonics on the ground, before the climbers went up.
After the install Gary checked the data:
- Licor working.
- GPS working.
- CSAT3 at 10m wind and temperature measurements looked good.
- 20m CSAT3A winds are nan, as noted above by Steve.
- NR01 reporting.
- Attached soil mote, I think all the measurements look ok. The mote voltage is only 9-10V.
- The Ubiquiti nanobeam ethernet cable was too short, so Ted decided to move it down 2m from the top to reach the DSM at the usual height.
- The soil bulgin cable barely reaches, so it might be a good idea to string some cord between tower and tree next to mote and hang the cable from that.
Since the ops center power outage noted in the blog ops center power was off, the power has gone out three times two evenings ago and at least twice last evening. Each time it's the main breaker that has tripped and must be reset, and (afaik) always in the evening with the overhead fluorescent lights on. Power will stay on for around 30 minutes before tripping again. Yesterday I think the overhead lights were on all day, but the power didn't go out until the evening. Someone please correct me where this report is wrong.
- Seems like we should report this problem to the person responsible for the ops center and see if they can get an electrician to investigate the problem. Maybe the main breaker needs to be replaced or upgraded.