Blog from February, 2017

Network status 15 Feb

Josés Carlos and Palma today adjusted the PITCH of the ops center antenna, which immediately brought tse13 and rne06 online, and improved the connection to rne07.  So, for the first time, we now have all links to the ne ridge in place and working.  With these links up, we are also able to get to many of the sites in the valley and sw ridge – now 17 total stations.  There are another 4 sites that should be on the network, but aren't connected.  An additional 3 sites are on the network, but the data system won't connect.  So...progress, but still some work to do.

 

RMYoung issue

We think we've figured out our problem with the RMYoung sonics – our power supplies are not producing high enough voltage.  The RMY has a minimum voltage requirement of 12V and our power supplies are producing 12V.  However, the voltage drop through as much as 55m of DC cable (from the fuse boxes on the power drop through the DSM to the sensor) has been 0.5V or more.  tnw11, for example, was reporting Vdsm=11.2V.  We note that the minimum voltage for all other sensors is below 10.5V, so this is a problem specific to the RMYs.  This explanation is consistent with these sensors working in the ops center (and earlier, at ARL), but not on the tower.

The solution is to use higher-voltage power supplies for the (7) masts that have RMYs and AC power.  (Solar-powered stations should have batteries providing nominally 12.7V.)  We've ordered 14 V supplies that we will bring with us on the next trip.  (They need to have our DC power connector added to them before use, which is easiest to do in the lab.)

 

LMG17-01

2/7/2017

Hi Alec and Julian, 

It looks like you guys are finally getting back to PA after almost a month and a half. Phew! That's a long haul! 
I wanted to remind you that we had also talked about getting the spare HS and LS loaded during the port call (in addition to swapping out the LI7000 all in three days!). Since the system has been down for a while, we haven't used much gas (HS last read at 500 psi, LS at 550 psi). Britt's getting back into town soon, and I will confirm with him that we want the cylinders swapped. Getting the LI7000 swapped should take priority obviously, since that leaves us dead in the water, so to speak. It would be nice to at least get the spare cylinders on-board since the next cruise also looks like a long one. Do you have to tell Octavio about wanting these crates staged for loading? The TCN's I have on record for these two cylinders are 404-00034182 and 404-00034184.
Remember that its fine to call or email over the weekend if you run into any troubles swapping the LI7000 -- 303-710-5598 or 303-258-7231
Good luck, 
Andy

2/2/2017

A-

The Visual Basic program was completely blank (just a blank grey window) when I logged into the system and the wheel of death was rolling when I rolled the cursor over the VB windows.  Had to use the task manager to close the windows since there weren’t even the controls in the upper right corner.

-A

2/2/2017

Thanks for putting her out of her misery. One question: when you say the program was locked up, do you mean Visual Basic or just the Licor?
Glad you'll have help with the Licor swap. Be sure to call and/or email with questions. 
Andy

2/1/2017

Andy,

Shut down the O2 system. The program was locked up when I logged in remotely.  This is the second time in the past few days that it has done this.  We were running the system with the door open to the Analyzer box, and the ambient temp (on the omega meter) was only 30C so I don’t think the system was overheating.  We will do our best to swap out the Licor for you, fortunately we will have some extra staff for this port call so the odds are good we can find or make the time to do the swap.  I assume we should ship the old unit back to you via our cargo system.  Thanks for the support and understanding of our limitations down here.

Cheers,

Alec

2/1/2017

Hi guys, 

Britt thinks that we might as well shut the unit off per instructions for the end of a cruise. It would be great to get it going for LMG17-02, but I know there's a lot to do at the end of an LTER cruise. It looks like you get back to PA next Friday, Feb. 10 and leave the following Tuesday. Call my cell phone if you're trying to swap out the failed Licor for the newer (still used!) one over the weekend and have questions:  303-710-5598. I think we should keep the temperature controller at 30 C with the new one in. When I called Licor to try to figure out what was happening, the tech there said running at elevated ambient temperatures can cause problems like we were seeing. I looked in the manual at the spec's and it says Temperature Range: 0-50 °C ambient, 0-55 °C bench, so we should be OK. I will get in touch with Licor and ask them what is going on. 
Thanks for trying so diligently to make it work. If you get the failed unit out and have time to pack it up in the materials the newer one came down it, we could see if Licor can figure out what happened to it. It worked for quite a while at the elevated temperature. 
Andy

1/30/2017

Hi Andy

So the 02 system has been on the fritz for the past several days.  Despite our reboots and cursing, the Licor seems to have finally quit.

Question – do you want the system to continue to run even though the Licor isn’t working?  Is it collecting other data that’s worthwhile to you?

Thanks

-Julian

1/28/2017

Andy,

Here’s the weekly maintenance sheet.  Still having some issues with the Licor locking up, reboots seem to be working but often it takes multiple attempts to get it going.  Also, Working Tank 2 is pretty low (~250 psi), at what level should we switch to the other tank?  I also adjusted the low pressure side to 3.5, since it was at 4.5.

Cheers,

Alec

1/17/2017

Hi Julian,

I know everyone on the ship is really busy, so hopefully the O2 system will be less of a burden the rest of the cruise. Today's data was almost perfect, but it did look like room and box T steadily climbed throughout the day and the Licor flat-lined for about an hour when it hit 45 C. Feel free to prop the door open a few inches if it continues to give you problems - with the set point down to 30 the heaters are less likely to run continuously and burn out. The only other thing I noticed is that the Fridge pressure ramped up rapidly over the past 3 days - potentially a sign that the trap that went in on Saturday may not have been completely dry - I'll keep an eye on it but if you notice flows dropping out, you could preemptively swap the other trap back in. Of course, you should feel free to ignore the whole thing for awhile too, as PALTER activities should take priority. Thanks again,
Britt

1/14/2017

Hi Guys

Logsheet attached.  No problems to report today!

-Julian

1/13/2017

Britt,

Performed your changes to both the Omega and the program.  So far the system is running as advertised, Tomega is in the 34.4°C area.  After running for a couple hours the red temp light hasn’t turned off, we will keep an eye on it.

We will keep you advised on how the system is progressing.

Cheers,

Alec

1/13/2017

Hi Julian,
1) Please see pages 14 and 17 of the attached manual for the Omega temperature controller that controls the Analyzer Box temperature, and is mounted in the upper left of the box, display facing out.  Please try to turn the set point down from ~38 to 30 C (it still has to be above ambient or it can't control by heating).  To do this, I believe you push "Menu" (circular arrow) repeatedly until you see "SP1", then push "Enter" (right angle arrow) to get into the change setting mode, then push "down" (downward triangle) until it reads "30.0", then push "Enter" and it should say "Strd". At this point I think you can just leave it and it will time out and go back to reading the actual temperature.
2) Then please close the door and hopefully it will find this control point (after possibly some big swings over 30 min to several hours) - you can watch on the Tomega box in the software - and the Licor will be happy.
3) The red light will show on the outside of the box, because this new setting is outside of our warn detection range. To fix this, please search for this chuck of code in the program:
If tomega > 36 And tomega < 39 And tfcrtd > 36 And tfcrtd < 39 And tfridge > 0 And tfridge < 4 Then
            templgt(0).Value = True
Else
            templgt(1).Value = True
End If
and change the "tomega > 36" to be "tomega > 25"
Please let me know if you have any questions and thanks a lot!
Britt

1/13/2017

Hi Julian,

That box is actually actively temperature controlled (heated) to 38 C. And the O2 sensor box in it is also actively temperature controlled (heated and cooled) to maintain a constant temperature of 38 (to within a few hundredths of a degree). Running with the door open has the risk of burning out our heating elements as they will be on full bore all the time.  Also, the Licor readings are sensitive to T variations and our O2 sensor will not control as well. This discovery of a temperature problem is some great detective work, but I would suggest trying to turn down the set point on the Omega controller for the box, rather than leaving the door open. I will send instructions on how to do this in a few minutes. 
Britt

1/13/2017

So after I opened up the cabinet and let it cool, things seem to be running smoothly.

I propped the door open and let it run and we haven’t had an issue since for a full 24 hours.

Have you considered added better ventilation to that box?  Even with the door open, the Licor is still about 105 degrees and warm to the touch.

When it locks up, there is no option to debug – it’s just totally hosed up and we have to use the task manager to kill the process.

-Julian

1/12/2017

Shoot sorry Julian. When it locks up, does Visual Basic give an option to "debug"?  If so please click it and let us know what line of code gets highlighted. Thanks,
Britt

1/12/2017

Hi Brit

The system locked up yet again.  I opened the box to restart the Licor again, and I noticed it seemed a bit warm in there.  I used a laser thermometer and measured the outside of the Licor to be about 45C and the ambient inside the box above 35C.   Could the Licor perhaps be overheating?  I turned it off and propped open the case to let it cool down.  I will try again in an hour or so to turn it back on.

Thanks

 

-Julian

1/12/2017

Thanks Alec!  Just to clarify "CO2 A pegged at 400" is the correct behavior and not a sign of a problem. We force it to think the reference is 400 as it really is just a differential sensor so needs some known to compare to. Cheers,

Britt

1/11/2017

Yes, the VM is showing data.  That said, we are seeing what you say about the CO2A and CO2B (picture attached), any suggestions?  We’ve had pretty good weather this cruise, temps haven’t dropped below freezing yet and today was sunny and clear, so I don’t suspect clogs in the intakes.  My DAS display is showing O2(CO2) = 389 uAtm.

 -Alec

Displaying O2.JPG

1/11/2017

Nice work! Are you seeing something other than NR's in the Li-7000 box on the Visual Basic window?

Not sure if the bad Licor comms were causing the crashes, but let's hope it works as long as the CO2 is reading out. CO2 A is pegged to 400 ppm and CO2 B is the real outside air measurement, usually around 390 ppm. 

1/11/2017

Andy,

Sorry the phone call today was so crappy, it’s a harsh continent J. Did as you suggested below, it took a couple reboots of the analyzer box but the Li-7000 finally came up.  The front panel on the Li-7000 came up but without data, after the second reboot and a remote log into the instrument, it came up correctly.  BTW it lives on Com1.  I have attached pics of both states.

Cheers,

Alec

Displaying Li-7000_Front_Panel (Medium).jpg

Displaying Li-7000_Front_Panel_11_Jan_2017 (Medium).jpg

1/11/2017

please check in with Julian by phone if he doesn't reply in a few hours - I'd like to get this fixed as soon as possible given the big gap and the location of the ship - thanks

1/11/2017

Hi Julian - thanks for all the help. Slight correction to what Andy said: just cycling power on the Licor will not fix the problem as it receives configuration information when the VB program starts up, so you would need to restart the VB program too. The best thing to do if you have time is a full shut down and restart. Sorry for the hassle, and thanks again,

Britt

1/11/2017

Hi Julian,
Thanks for the update. I assume the screenshot you sent was after rebooting. You can see that all the values in the Li-7000 box are NR. This is the Licor CO2 analyzer, which uses serial communications to talk to the main computer. This happened briefly during the last cruise, but Kevin got it to go away. You could check the front read out on the Li-7000. It's the big green box mounted tilted at the top of the Analyzer Box. You could try just cycling its power using the rocker switch on is front panel. It'd be nice to know if it's the Li-7000 itself or the serial comm's that have failed. 
In the meantime, I'll work on getting a replacement sent down. Sorry this had to happen on your watch. Thanks for your help,
Andy

 

1/11/2017

The problem continues.  Today during my checks I found the red watchdog lights on

Logging in to the computer I found the software was locked up again:

 Displaying image001.jpg

 

After using the task manager to kill the process, I was able to restart the software.  No USB errors present themselves, and it seems to run fine.

 Displaying image002.jpg

 

We are somewhat at a loss as to what else we can do to troubleshoot the problems with your system.

Please advise how you’d like us to proceed.  Until then, we’ll continue to restart the software as it crashes.

 -Julian Race

 

1/9/2017

Andy,

Last night the software on the O2 system locked up and required a reboot of the system.  We are getting a green light on our RVDAS but the value for the O2(CO2) is 0 uAtm. Julian and I are not sure if this is valid data or if there is still something quirky with your system.  The software seems to be running fine, but just want to make sure everything is good from your end.  Let us know if the data is good and/or if we need to do anything on our end.

Cheers,

Alec

12/30/2016

Hi Andy

Happy New Year from the LMG.  We just passed 68W and booted everything up.

Here’s what your system is showing.  Not sure what ‘good’ Licor values look like, so please have a look and let us know if you’d like us to check on anything.

Cheers

 

-Julian Race

 

 Displaying image001.png

 

12/30/2016
Hello ET, 
I'm not sure who is baby sitting the Oxygen instrument for this cruise, but welcome aboard the LTER cruise. I know these can be hectic, so hopefully, things on our side will run smoothly. Since we only had two or three days on a dry trap, I think we can skip Saturday maintenance this week (if its even turned on tomorrow). But please try to start things as soon as possible to make sure everything is running right for the northern end of the crossing. If the chiller was turned off, please turn that on immediately so it has time to cool down before starting gas flow. Other than that, we should be good just to turn on all the boxes (once the chiler is been on for three or more hours) and start collecting data. We did have some issues with our Licor, so if you can make sure to take note of the Licor outputs, that would be great. 
Happy New Year!
Andy

20m.tse11

 

TRH issues

Noticed that some Ifans were 0:

tse13.80m: wouldn't respond to control-R, but came back up with a power cycle (pio)

tse13.2m: responds to control-R, but immediately fan is 0.  I tried increasing the current limit to 200mA, and still is immediately 0.  I suspect this is the VERTEX fan-monitoring-board-dead issue.

tse06.60m: responds to control-R, but immediately fan is 0.  I tried increasing the current limit to 200mA, and still is immediately 0.  I suspect this is the VERTEX fan-monitoring-board-dead issue.

tse09.100m: wouldn't respond to control-R, but came back up with a power cycle (pio)

 

Daily status 3 Feb

Last (part) day in the field – I am writing this from Lisbon...

All but 2 masts that have been constructed are now powered up and collecting data.  There are 7 masts remaining to be prepped in the ops center.  There are 2? masts that have DTU sensors but have not yet been configured to send their data to our system (easy for them to do once they are on the network).  We also have a handful of sensor issues (mostly RMYoung sonics) that are reporting continuously bad values.  Also, standard maintenance, such as cleaning the radiometers, has not been done (though the rain is performing much of this task!).  We had staged equipment at the two remaining constructed towers, but pulled this equipment off again for safekeeping until we next arrive.

Much of the network is functioning.  HOWEVER, the primary mast that we have set up to get data out of the valley (rne07) failed to pair with the ops center.  Thus we are receiving less than half of the masts that are working.  (rne07 did power up after the quick work by the electricians to resolve a power issue at this site this morning.)  The best guess is that the antenna is not pointed correctly, but time didn't allow us to fix it.  Unfortunately, this means that ISS also is unable to connect with the network, so Gary reinstalled a cell modem at that site.

We also consolidated our mess at the ops center.

The German/Danish group also is leaving today (having placed many of the lidars), so no one will be on site for a while.

 

 

 

 

Daily status 2 Feb

(Wifi wasn't working in my hotel last night, so this is late...)

A mixed day:

  • rsw05, rsw07, and tnw03 all instrumented.  Just need to hook up power (that we weren't supplied to do – will happen today)
  • added access point for west slope on rsw06
  • half instrumented v07

So...we've decided to remove tse02 prep (which I did this morning), which will leave 2 towers built here, but not instrumented, and fly home on our original schedule.

 

 

Tower Tensions

Measure and record tower guy wire tensions when appropriate. Two tension meters are available, will be stored in ops center. Bring a sharpie if planning on measuring tension to label tower leg. Living document available to anyone with access to the Perdiago Project Management Google Doc.It is located in the ISFS folder, or by the link below.

 

tower guy wire tensions

v07 Install

Site staged for future install. Power is available and verified at tower.

To do:

  • Soil install
  • Install and verify tower instrumentation
v06 Staged

Valley site staged for future install. Power is available and verified at tower.

To do:

  • Soil install
  • Install and verify instrumentation
Daily status 1 Feb

Slow progress today.  Split into same 3 crews, except two crews also had 2 each INEGI climbers.  Each INEGI crew installed 2 towers:

  • tnw09: up, with solar power.  Connects to network, but connection not yet built to get out of the valley.
  • tnw11: up, installed, but all 3 RMYs report continuous bad data.  This is very odd, since they worked in the ops center and (some) other RMYs work elsewhere in the experiment, so we don't think we're doing something systematically wrong.
  • rne06: up, installed, on network, but it appears that the 20m sonic anemometer head was not plugged in – an easy fix.
  • rne07: installed (including soils), but DC power connection is not live (even when a power supply was swapped with tnw11).  Not clear what is going on.

The 3rd crew prepped v06 and v07 at the base of the tower and measured remaining guy wire tensions.  We also installed soil sensors at tse02.

More prep work also was done at the ops center.

The question is where to go from here...

 

rne06

Station is up and sensors reporting, EXCEPT CSAT3A at 20m.  Since it doesn't show a sonic head connected (using enter, D shows "SN0000" for the sonic), the fix should be just plugging in the head to the EC100.

Ted and Gary staged the sensors yesterday, all except for the Licor since we were missing the boom clamp.  Here are the serial numbers:

  • Licor 1166
  • Nr01 2537
  • EC100 1821
  • Csat3 1120
  • Csat3a 2033

Lessons learned: We should have demonstrated how to connect the Licor and sonics on the ground, before the climbers went up.

After the install Gary checked the data:

  • Licor working.
  • GPS working.
  • CSAT3 at 10m wind and temperature measurements looked good.
  • 20m CSAT3A winds are nan, as noted above by Steve.
  • NR01 reporting.
  • Attached soil mote, I think all the measurements look ok.  The mote voltage is only 9-10V.
  • The Ubiquiti nanobeam ethernet cable was too short, so Ted decided to move it down 2m from the top to reach the DSM at the usual height.
  • The soil bulgin cable barely reaches, so it might be a good idea to string some cord between tower and tree next to mote and hang the cable from that.

Since the ops center power outage noted in the blog ops center power was off, the power has gone out three times two evenings ago and at least twice last evening.  Each time it's the main breaker that has tripped and must be reset, and (afaik) always in the evening with the overhead fluorescent lights on. Power will stay on for around 30 minutes before tripping again.  Yesterday I think the overhead lights were on all day, but the power didn't go out until the evening.  Someone please correct me where this report is wrong.

  • Seems like we should report this problem to the person responsible for the ops center and see if they can get an electrician to investigate the problem.  Maybe the main breaker needs to be replaced or upgraded.