I heard via Bill that Jose Carlos wanted me to meet someone at the 60m valley tower to test something at noon. When I finally found the tower (after getting some directions from Jose over the phone), I discovered it was the INEGI tower crew setting up tnw07. They had brought the two tnw07 DSMs and and 5 TRHs and wanted to install them on the tower along with the DTU box and Gill sonics. Even though I had not heard this would be happening, they have installed our equipment before, and it looked like they had found what they needed from the ops center and knew what they were doing. After some time watching them prep, they told me we did not have the right cable lengths and some other problems, so they were not going to install our equipment. So they asked me to take it all back.
I left the tub with the DSMs, booms, and TRH rack are all together under one of the DSM setup tables. I don't know if this site was ever staged by us in the ops center.
Mote 12 was the one that stopped working when initially installed at rsw02, but it got installed on the radiation stand at rsw04 to cap the soil sensor cables. I installed mote 8 in its place to bring it back to EOL for testing.
No progress on the rsw04 tower. The tower sections are still laying on the ground.
Mostly clean-up today, then Dan and I left to go home. Gary is still on site, nominally assisting ISS. We heard yesterday that our air shipment has been received in Porto. INEGI will deliver the two boxes to the ops center by the time we return in Jan.
rsw01: added Bulgin caps, but realized we needed Binder caps as well. Stole from motes we intended to use elsewhere.
rsw02: fired up and had 3 issues. Can't see GPS (on an Titan DSM – shouldn't "rs G" still work?). Seems to be plugged in with daughter board to front panel properly. RMY at 20m is spiking – about 1 in 10 samples is bad in the brisk winds we had at the time. They are detected (diagnostic flag is "A", not "0"), so can be removed, but still not what we want. This sensor had spikes when tested in the ops center as well, though they had gone away when I took it out of the shipping box. Likely this is a weak transducer. We could replace it, but eventually won't have any spares. Finally, mote#12 would start up, then die (no blinking LEDs). Replacing with a different mote worked fine. We think that this is the second mote this has happened to.
rsw03: brief stop to read Setra serial number (and get berries from the tree)
rsw04: installed 4-component radiometers and soil sensors at darkhorse, though tower is not yet up. This will require a 15m cable to the soil mote.
tse12: back to finish up from last night. Installed soil sensors. Raised the Ubiquiti to have better line of sight to rne02. Found that EC100 again was in ASCII mode, so had to drop the box to reprogram through the USB port. In general, we need to be more rigorous in our prep. (I think this will be my job in Jan.)
I've created a table of the sites in the EOL Perdigao web page, with some comments.
We left the ops center with 4? towers prepped, primarily the tnw07 60m tower for when INEGI returns.
A 2.8-tower day (all 20m), with us not quite finishing the third tower due to darkness. Completed rsw01 and rsw02 in the morning/early afternoon, then back to the ops center to reload and work on tse12. There are no more short towers built that we are aware of. We will return to rsw01 and rsw02 tomorrow to add some missing caps and install the radiometers and soil sensors at rsw04. (The darkhorse is already placed.) We also will return to tse12 to install its soil sensors and a few other odds-and-ends. Thus, there are now 6 tall towers and 6 short towers fully installed.
We'll also do some clean-up in the seatainer and ops center before Dan and I return to Lisbon for our flights home on Sunday. Gary will stay a bit longer to help out ISS.
Two more towers up today. Would have done 3, but forgot to completely prep rne02, which took some time (including breaking down two pallets of stuff in the ARL seatainer). Staged booms and sonics to rne01 and booms to rne02. Have now deployed the CSAT3A with and without an EC150 and RMY with the Samortecnica booms, and figured out systems for both.
We're finding different versions of RMY from the ARL shipment – some need "5BE" and others "178" to configure the output. There are 2 or 3 versions of interface circuit board, though all seem to need to be wired the same. Have found one RMY whose received (DSM to sensor) signal seems flakey, though it is working now.
We finally have our first short tower instrumented! Samortecnica "topped off" several towers on the ne ridge, so we started on them. They made a lot of progress today, so we now have from 4–9 short towers to work on during the next 3 days, and hopefully more will be built while we're working. In the meantime, we've been preparing data systems and sensors, so we have most of these ready to go. Now, we just have to get busy throwing them onto towers...
A few random details:
- Found that one of the old DSMs that had been prepared from old pieces had the ribbon cable going to Emerald ports 1-4 flipped. Now DSM ports 5–8 (Emerald ports 1-4) work.
- This same DSM (rsw04) has an old Emerald board that needed to have jumpers changed to configure it to RS422 for the EC100 box.
- Found that we need an extra short 1/4-20 screw on the CSAT3A booms to connect the ground straps (that we finally found in a tub today).
- We're finding lots of EC100 boxes that are not set to binary mode. This can be done easily by using minicom to connect to the EC100 and typing "+", "Y", "X", "Y". We found one that we thought we configured, but had reverted. Perhaps we forgot the "X" step.
- I finally started cooking the first soil sample that I took 2 days ago.
- Since we don't yet have AC power at the towers, we are using DTU's portable generator to power up the towers to check that things are installed correctly.
- We're still waiting for the ops center router to be reconfigured to allow us to attach our network. (Both the ops center and the towers use the 192.168.1.X subnet. In hindsight, it would have been better to change our subnet to something more unique. It would be a lot of work at this point for us to change every DSM and Ubiquiti device.
- We will change the configuration .xml to default to simple cal_files so that we'll be able to look at data on the DSMs without having to create every cal_file now.
- Four people from IMPA (the Portuguese meteorological service) visited the site and ops center today.
- Found that we can buy fruit from the food truck that comes to the Alvaiade square periodically!
A group from IMPA visited today and asked whether our WiFi network would interfere with the two national weather radars. It is true that their frequencies of 5.63 and 5.64 GHz are within the "DFS" hopping table of the Ubiquitis. However, Bill Brown judges that with the directed beams of the Ubiquitis, the lower power levels, the distance to the nearest radar being 120km, the scanning of the radar, and the frequency hopping of the Ubiquitis (that will hop off of the radar frequency if it detects the interference), the chance of interference is quite low.
If necessary, we could deselect the radar frequencies from our hopping table, but this would be a lot of work, probably requiring an in-person visit to each site.
Thus, we are going to proceed by leaving the Ubiquitis in "auto" frequency selection mode for now.
This taken during the initial installation hole at rne01 on Monday about 4pm. (However, we aren't taking data at this site yet, so it doesn't really matter when it was.)
I didn't get around to weighing until just now (2 days later) so the Qsoil should be lower than the real value. However, again, we don't have measurements to compare to so this doesn't really matter. Nevertheless, it is useful to process this sample to get a bulk density measurement...
Tray (tare): 8.2g
Wet+ring+tray: 172.2g
Great weather, and did get out a little, but still limited by the availability of completed short towers...
- Samortecnica did drop off about 10 booms to allow us to work. They didn't have backplates for the clamps, but INEGI thought we could borrow from the DTU stockpile for the moment...
- The municipality truck came and was perfect for moving the darkhorses. Nevertheless, their time was limited, so we just had them drop them along the valley road. Gary and I transported them to their proper locations (rne01, v04, and rsw04) using the pickup for the up-the-hill run and set them up.
- At v04, we couldn't locate the tower, so this darkhorse will have to be moved.
- At rne02, the darkhorse is south of the tower at the maximum cable distance. It is above low shrubbery, but there are higher pine trees all around. I <think> the direct sun will clear many of these trees, but there undoubtedly will be times that the radiometers are shaded. This is typical of the local environment. I judged that the immediate plot under the darkhorse was similar to the one I selected for the soil sensors that are about 10m away.
- At rsw04, I chose a location NORTH of the tower, which of course puts the tower shadow across the sensors for some part of the day. The alternatives were deploying down the slope, or adjacent to a power pole, which I thought would have an even larger effect on the radiation. Close to the tower, the land cover is greatly disturbed by tower construction operations. Although we raised it to the maximum possible height, this darkhorse is only just above the tops of shrubs and has a view of large rocks on the surface. I actually thought that a feature of this location was that rocks blocked the radiative footprint of the nearby road. All of this seems to me typical of the ridgetop environment.
- In the meantime, Dan worked with the INEGI crew to instrument rsw03 (60m), which is now done.
- After lunch, we participated in the monthly telecon, then had a good discussion with João from Porto about data organization and access. He now understands our data products and how these will flow from the sensors.
- We learned that the rsw short towers will soon be ready at their final heights (12 and 21m), so we began prepping them. Among other things, the Li7500 3A fuse problem bit us again, so the prep is still ongoing. At this point, instrumentation and data systems for rne01, rne02, rne03, tse10, and tse12 are ready to go, and we expect rsw01 and rsw02 to be ready shortly.
- With a bit of spare time, we again picked up the pyrgeometer problem. Steve Semmer pointed us to a few tests that eventually resulted in him sending us new code to load into the pyrgeometer wisard board. This worked for the one sensor we tried, so Steve will prepare new code for the remaining 5 sensors.
Lots of activity today, in no particular order:
- Telecom guy came and (after more than an hour) got our bandwidth to 48 Mbps up; 48 Mbps down. This is close enough to the 50/50 we paid for. Yea!
- We requested from the telecom guy a set of port forwards, but this has to be done back at the office. A workorder has been submitted, but for now we can't get our router on the network.
- We tried a bunch to pair tse13 to ops with no success. Dan climbed and reoriented 2 of the antennae. Later testing in the ops center showed that we need to set the station adaptors to SA PTMP, rather than SA PTP. All of the SAs programmed so far will have to be changed, but this isn't a big deal. (P.S.: By the end of the day, had reprogrammed all except the uncommitted tseXX and rswXX (and still have some vXX and tnwXX to program completely))
- The Metek SODAR was delivered to the ops center today. (This should be in an ISS logbook entry!)
- 10 Gill sonics purchased by U. Porto were delivered today. I presume that these will be used on the 60m towers connected to the DTU data system.
- We prepped for most of the towers expected to be ready tomorrow.
- We installed a set of soil sensors at rne01. The ground was easy to work with, going only to 5cm.
- And last, but not least, we now have access to the circuit breaker panel in the ops center where we can turn on the ballroom lights – that makes a HUGE difference in nighttime lighting! (It is interesting that cold is much easier to bear if it isn't dark and cold.)
For tomorrow, we are expecting:
- Samortecnica will deliver our booms – needed to add the DTU and ARL sonics to the short masts.
- Samortecnica will add sections to the ridge towers to get them to 12 and 21m. Then we can start instrumenting them!
- Samortecnica will pour concrete for the valley short towers. The idea is to have some of the solar-powered towers up before we go.
- Samortecnica will also work on the remaining 60m towers.
- The municipality will provide a longer-bed truck that ISS will use to move profiler stuff to the profiler site in the morning. We will use it to transport one darkhorse to each ridge and valley.
- The INEGI tower crew will return to instrument at least one more 60m tower.
A somewhat frustrating day. We started with the plan to instrument the short towers that have been erected. (Today, we confirmed that tse12 is built and tse10 has two (unguyed) sections in the air.) Already, though, two of those towers need booms from Samortecnica that we do not yet have, so we wouldn't have been able to complete this job. Nevertheless, I gathered what we thought we needed from the seatainer (requiring two pickup trips) and started kitting these 5 towers. Then, we ran into problems:
- The EC100 boxes (for both the CSAT3As and EC150s), were not set to binary mode. It took quite a while to figure out that this can be done by connecting the USB port and, via minicom, setting "+", then "X" (with some yesses to confirm) to establish binary mode.
- Even then, the CSAT3As show errors that require the grounding solution we had found in FLAB. Unfortunately, we can't seem to locate the straps that Rick made. We have some braid to make up more, but it would be nice to locate the prebuilt ones!
- There were lots of nan's in the data. These mostly are due to missing cal_files, so we had to make about 8 so far.
- 4-component radiometer values were coming in on the wrong address. Changed sensor_catalog to set the correct range of I2C addresses.
- Even so, we never got Rlw data – there are no Wisard messages getting through the motes from the K&Z pyrgeometers. This result was consistent through a change of pygs and motes. Semmer suggested looking at the Binder wiring, but it seemed to be the same as for the psp's. We can try to build a test cable, based on the Pi hat USB console module since there apparently are TTL level signals out of the radiometers.
- We noticed that we forgot to bring single-channel wisard boards for the wetness sensors. We only need 3, for the 3 darkhorses.
- We also will have to figure out how to transport darkhorses with a short-bed pickup!
In the middle of all this diagnosis, moderate rain started up with low cloud covering the ridges, so it didn't seem to be a nice day for driving on the ridge or climbing anyway. Rain is forecast to continue tomorrow.
We also slipped in a slight reorientation of the base WiFi access point. I've calculated that it needs to point up by 230m across 2800m, or 4.7 degrees, and the mount only allows pitching up by 3 degrees. However, by swapping the top and bottom brackets, we can tilt up up to 10 degrees. This angle is now set to ~4.5 degrees.
We've decided to take our first (and probably only) day off tomorrow. We assume that Monday will be busy preparing rsw02 for INEGI to instrument, while starting with rne01+ ourselves. We also need to revisit tse13 to replace the bad POE cable (I verified that it had a short) that Per removed.
- 3x single-channel boards for 4-component wetness
- Do we have short binder-bulgin connectors for NR01s? (We have 4-pin cables for motes.)
- Binder sensor test cable(s)
Ops->Foz via Sarnadinha: 10min
Ops->Foz via Vale do Cobrao: 7min
DTU team left the site last night. INEGI is home for the weekend. Gary arrived this afternoon, so we now have our 3-person NCAR team. Dan is moving to Castelo Branco to have more exciting night life (than hanging around with me...)
Work today:
- Finished mote fuse replacement
- Finished configuration of all WiFi routers and configuration of WiFi station adaptors for tneXX. Also assembled the 60-degree antennae with their AirPrisms. Now just plowing through the rest of the station adaptors. Have updated Ted's networking page with color codes to indicate the status of each adaptor
- Residual clean up/organization of the ballroom
- Visited tne09 and Dan climbed to turn on tne09t's power switch. Everything immediately came up, including the Ubiquiti. Increased Bluetooth console power and can now pair from the ground (though you have to stand about 20m away from the base of the tower). Dan got a great photo of Vale do Cobrão. Measured the black rope as 200'.
- At José Carlos' request, drove to ne ridge and confirmed that rne01, rne02, rne03 are all in place, though they are 9, 18, and 9m high, respectively. We really would have liked the full 10, 20, 10m heights...
P.S. Back at the hotel, prepped all rneXX and rswXX adaptors as well. Now just have vXX and tnwXX to do.
It was a national holiday today, so INEGI was back in Portugal and won't work tomorrow or the weekend as well. We assume that Samortecnica also was not working today, but don't know their Friday/weekend plans.
Dan helped DTU instrument rnw06 (60m). Off-and-on rain all day made this an annoying task for them, but they were motivated by wanting to go home tomorrow. (They said that, living in Denmark, they are accustomed to working in rain.)
I started the day with a drive out to the east energy balance/profiler candidate site. Access at the site is actually easier than I thought. However, with trees around (estimated at ca. 11m high), it would be better to have a taller flux tower. (We realize that ARL just packed 10m towers for these sites.)
I next got a lesson in using the MultiScanner, that we tested at tse09 (the valley 100m tower). They had a simple method for georeferencing the MS, but used GPS and remote-connect equipment that we don't have. (Hmmm) Afterwards, DTU decided that the most time efficient procedure is to scan just a few points on each boom, rather than autoscanning either the entire tower, or even just a measurement level.
While at tse09, we powered it up with the generator. Per enabled the UDP ports and the DSM saw all of these data. Unfortunately, it does appear that the top DSM is switched off, since I couldn't see the blue light on the power switch when the power was supposed to be on. This is on our task list.
Back at ops, (mostly Dan) and I continued to replace fuses – getting close. I got a lesson on configuring Ubiquitis. I tested the METEK connection, using the cabling that DTU made. Found that our DSMs aren't preconfigured for RS422 on these ports, but this is easy to do here as we prep these boxes. We still don't have the Samortecnica booms for the METEKS or RMYoungs, that I'll have to ask José Carlos about.
The ops center trailer is mostly set up. It has power (of sorts), and cabled connections to both the ops center ethernet connection (via fiber) and the WiFi access point looking towards the ridge. There is a desk&chair, Ted's router, a 220V power strip, DTU's data logging computer, and DTU's 13Tb backup disk (unconnected) in the trailer. As mentioned yesterday, we are awaiting a network technician to arrive Monday (I think I said Fri previously) to diagnose low bandwidth on the fiber. Also, Per only saw one LED (out of 4) on the Ubiquiti station adaptor when tse13 was powered up, and suspects that the access point antenna will need to be moved. Finally, Per noted that port 1 on Ted's router appeared to be dead, so he plugged the Ubiquiti WiFi antenna into the port 2. Strange.
Per has configured all of the access point Ubiquitis and upgraded their firmware to 7.2.4 (since some of the devices were old and he wanted everything to run the same version. He left most of the 16db station adaptors for us to configure.
rsw03 is mostly prepped, for when INEGI staff return on Monday.