Weather: Sunny, warmer than yesterday, and intermittently breezy.
ISS on Nagios showed a healthy network. The webplots have been updating.
The only issue was the the Lidar Leosphere visualizer froze and I had to exit the software. I still have not been able to bring it back online. However, it looks like data is being saved and webplots are being updated.
Nothing on the Leosphere, even after re-start
Output from the 449 MHz profiler at ISS1 at the end of the day.
Spent much of the day with Matt at S5 (Exxon) who's tower had completely blown down. Refer to Matt's blog for today.
Downed S5 tower.
Today is my last day. Liz arrived to take over ISS ops. Thanks Liz!
Until teardown ...
Weather: Sunny, clear skies all day. Calm-ish winds in the AM transitioning to westerly winds > 10 mph in the PM.
IOP3 was successful. All ISS radiosonde profiles within the 24 hr cycle made it beyond the tropopause (~200 hPa) before bursting.
A rather quiet ISS day. Bill went to ISS1 in the morning to check a few things before departing for a well-deserved break. Thank you Bill.
Nagios is mostly green except for a few orange warnings for ISS1 lidar cfradial/man netcdfs (I was told to ignore) and ISS3 allsky and webcam jpgs.
However, we can see the hourly allsky and webcam images on the webplots so we know these jpegs are being saved.
ISS3 webcam taking a photo of the 2000 UTC launch.
The afternoon was spent accompanying Matt on ISFS site visits. Refer to his blog post for today.
At the end of the day, we dropped off soil samples to cook back at ISFS base and I checked the Lidar data streaming on the DM at the ISS1 trailer.
Output from the 449 MHz profiler.
At the ISS2 Rancho Alegre site, Bill and Lou launched the 10am and 1pm soundings, while Matt and I launched the 4pm and 7pm soundings.
Relatively calm winds at launch switching to northerlies then finally westerlies above ~700 hPa. We saw several dry layers and temperature inversions up to 600 hpa in the afternoon soundings. Winds were very high in the free troposphere > 50 knots (58mph). At the tropopause, winds were ~100 knots (115 mph). Check-out the latest Skew-T plots for ISS2. Sedgewick ISS3 radiosonde profiles also showed similar patterns.
We saw the Twin Otter flying overhead after the 7pm launch and waved at Mack.
Matt launching the 4pm sounding.
Cool and sunny, breezy at times.
At ISS3 Sedgwick: more windy at Sedgwick than Rancho Alegre making the sounding balloon a little difficult to control. Currently the students are inflating the balloons inside the storage pod which works well for northerly winds, however if the winds steer to other directions then is could get more difficult so showed the students how to use a balloon bag. The students emptied their first Helium cylinder this afternoon meaning we have gotten 12 soundings from this cylinder. The Rancho Alegre site also got at least 12 soundings from their first cylinder.
Also at ISS3 fixed a bug in the RASS plots that affected the colour scales. Found that the IDL license wasn't starting this afternoon (reporting a "failure to acquire license" message). It seems to fix itself after a while, it's not clear to Gary or I why, perhaps it needs to refresh occasionally. Nagios had been reporting an error with the Allysky camera images but checking the ceilometer PC (which hosts it) couldn't find any problems. Have been backing up the ds (data source) directories at each site's DM as well as Modular Profiler correlation data to a portable disk to take back to Boulder.
A windy sounding at Sedgwick this afternoon
A sunny but cooler day, breezy at times. Sundowner winds are forecast and an IOP will be held tomorrow starting with soundings at 10am PDT.
Today Lou and Jacquie went to Sedgwick to diagnose problems with the ISS3 wind profiler. It had stopped working yesterday and as expected it appeared to be a problem with the UPS. Lou recabled the systems so that the profiler is now on the older "best" branded UPS and the sounding system is now on the Cyberpower UPS. We think that this older UPS will be more stable and cope with the power fluctuations that appear to be present at the site. They also had to restart the ceilometer PC so there was a brief interruption in those measurements.
Back in Santa Barbara at the Fire HQ site, I did some work on data backs and tidying up at ISS1. This afternoon we had an Education and Outreach event with the administrative staff of the County Education office. They have been very supportive of our operation at SWEX, the power for the site comes from their building and they are also supplying our high speed internet. I gave a talk to the group (around 40 staff) in their auditorium and then we took them out to the site and gave them a tour of the equipment and the trailer.
Later in the afternoon, Stephan de Wekker (University of Virginia) arrived with his mobile lidar trailer and we set up an intercomparison experiment between his Halo Streamline lidar and our Vaisala/Leosphere Windcube lidar. We set both to similar scan strategys with approximately 5 minute vertical stares, followed by VAD scans at 75 deg (his preferred scan) and 35 deg (our preferred scan) with an 8 minute cycle. We will let these run overnight, he is planning to leave his system run there until about midday tomorrow after which he will go off to do his usual IOP mobile runs with the lidar along the coast.
Stephen with his mobile trailer and Halo lidar at the ISS1 site with the NCAR windcube lidar
The Santa Barbara County Education department administrators tour of the ISS site
The day started off warm and mostly sunny, but the wind picked up, more clouds arrived and the temperature dropped as a cold front moved in. Sundowner winds are forecast for the next few days, but the IOP that had been planned for tomorrow was delayed another day in the hopes of catching some more interesting wind patterns. Currently the IOP is proposed for Wednesday. Steve left today and Lou will be starting tomorrow.
Jacquie and Matt went off to ISS2 and some of the ISFS sites in the valley. I had meetings and worked at ISS1 today where I did some tidying up, further work on the lidar and set up a couple of backup disks for data. Jacquie and Matt came back to ISS1 for a class in running the lidar and we did a few more hard target scans to check the orientation. A couple of problems cropped up later in the day:
The ISS3 profiler computer appeared to go down, presumably because of the same UPS problem we had seen previously. The computer isn't responding to pings from the ISS3 DM so must have powered off. We will go out to Sedgwick tomorrow to investigate.
The ISS1 data manager froze late afternoon just web apparently while browsing on the field catalog and was unresponsive on the main terminal. I could get to it by sshing from the profiler computer, but killing the browser processes didn't fix it so we did a reboot. Unfortunately after rebooting, a network configuration problem occurred and it couldn't connect to the LAN or outside internet. Eventually we got Gary on the case and he figured out that the two network port were swapped after rebooting for some reason and talked me through getting it back online (see his blog entry for more details). Late in the day the wind picked up further and started blowing loose ISFS items around so had to secure those.
The iss1 data manager display froze this afternoon, but Bill was able to ssh from thatcher and reboot it. However, on reboot the networking was broken, no Internet and no ping to lan hosts. It turned out that the networking configuration had been lost. Worse, it looked almost correct because one of the interfaces (looking at ifconfig output) showed the correct lan address, 192.168.0.56.
It looks like NetworkManager assigned the LAN connection configuration to the WAN ethernet device, and then left the LAN connection disconnected. So ping to the profiler would show "network unreachable".
For reference, here is the ifconfig output when the ports are configured correctly:
[root@iss1 network-scripts]# ifconfig
enp3s0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 192.168.155.56 netmask 255.255.255.0 broadcast 192.168.155.255
ether d4:ae:52:a7:f3:d5 txqueuelen 1000 (Ethernet)
RX packets 11233 bytes 2713246 (2.5 MiB)
RX errors 0 dropped 30 overruns 0 frame 0
TX packets 15283 bytes 11810144 (11.2 MiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
device interrupt 16
enp4s0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 192.168.0.56 netmask 255.255.255.0 broadcast 192.168.0.255
ether d4:ae:52:a7:f3:d4 txqueuelen 1000 (Ethernet)
RX packets 265383 bytes 370343046 (353.1 MiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 66966 bytes 6893951 (6.5 MiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
device interrupt 17
Interface enp3s0 is the WAN port, while enp4s0 is the LAN port.
The fix was to use the system network configuration to change the port configurations. Click on the network icon in the lower right of the screen, and try to match up the correct network connection on the left with the correct configuration on the right. This particular time, we clicked the top connection on the left and assigned it the "WAN Static" configuration on the right. Then click on the bottom connection on the left, and assign it the remaining configuration on the right "ISS enp4s0".
I have seen NetworkManager confuse connection configurations before, but I don't know what causes it. Perhaps there is some configuration setting that is missing which would force the right configuration to each port. NM should be using the "ifcfg-rh" storage backend, so it is using the ifcfg network scripts in /etc/sysconfig/network-scripts . Looking at those scripts, I noticed that only the ifcfg-WAN_DHCP script has a HWADDR setting, associating that configuration with the WAN device, enp3s0, with hardware address as shown above in the ifconfig output, d4:ae:52:a7:f3:d5. (Note the ifcfg-WAN_DHCP script unfortunately has a name leftover from past configurations, but it does in fact contain the configuration with the "WAN Static" description.)
I'm guessing that NM can initialize the ports in different orders, and in this case it configured the enp3s0 port first, but assigned it the ifcfg-enp4s0 config because that one did not have a HWADDR setting. I have since fixed that file to contain a HWADDR setting for the LAN port, and for good measure I added the DEVICE setting also. I don't know why that was missing either...
[root@iss1 network-scripts]# cat ifcfg-enp4s0 DEVICE=enp4s0 ONBOOT=yes BOOTPROTO=none NM_CONTROLLED="yes" DEFROUTE=no NAME="ISS enp4s0" IPADDR=192.168.0.56 NETMASK=255.255.255.0 TYPE=Ethernet NETWORK=192.168.0.0 BROADCAST=192.168.0.255 DOMAIN=field.eol.ucar.edu DNS1=127.0.0.1 PROXY_METHOD=none BROWSER_ONLY=no PREFIX=24 IPADDR1=192.168.0.1 PREFIX1=24 NETMASK1=255.255.255.0 IPV4_FAILURE_FATAL=no IPV6INIT=no UUID=b325fd44-30b3-c744-3fc9-e154b78e8c82 HWADDR=D4:AE:52:A7:F3:D4
I added the device setting to the WAN config also.
[root@iss1 network-scripts]# cat ifcfg-WAN_DHCP DEVICE=enp3s0 ETHTOOL_OPTS="autoneg on" TYPE=Ethernet PROXY_METHOD=none BROWSER_ONLY=no BOOTPROTO=none DEFROUTE=yes IPV4_FAILURE_FATAL=yes IPV6INIT=no NAME="WAN static" UUID=2d18da56-d5b6-4e8e-9873-21e35ace2439 ONBOOT=yes HWADDR=D4:AE:52:A7:F3:D5 IPADDR=192.168.155.56 PREFIX=24 GATEWAY=192.168.155.1 DNS1=192.168.155.1
Hopefully that will prevent future mix-ups on reboots.
A little cooler today, still mostly sunny with some scattered cloud, very hazy. Sundowner winds are still forecast with a front tomorrow, however the IOP has been delayed a day and is now proposed for Tuesday and Wednesday.
We mostly took a break today, although I did spend a few hours this morning at ISS1 mainly working on the lidar. I rechecked the pointing by scanning the nearby cell phone tower. That showed a peak at around 221.4 - 221.5 deg, very close to the Google Earth estimated orientation os 221.56 deg. I also experimented with the lowest viewing angles by doing PPI scans at 0, 2, 3, and 5 deg. There are lots of trees to the north and east, but there are some narrow gaps so I'm trying to determine the precise angles of those in order to choose the optimal azimuths for the RHI scans. After various tests I selected 18 deg azimuth for the North-South RHI scan, and 95 deg for the East-West scan, not quite orthogonal but this gives the best views between the trees.
The sequence of lidar scans are a vertical stare for 5 minutes to get good statistics on the vertical motion and to get boundary layer depth estimates, followed by a PPI scan at 35 deg for VAD winds and turbulence (1.5 minutes), followed by the North-South RHI azimuth 18 deg and the East-West RHI at azimuth 95 deg (both also 1.5 minutes). The full sequence takes approximately 10 minutes. All scans are at 50 meters resolution.
Views across the top of the lidar head to the north and east showing the gaps in the trees that were selected to provide the lowest RHI scans in each direction.
Another very warm day with light winds and no clouds. Sundowner winds are forecast for tonight and the next few days. An IOP is planned Monday and Tuesday. Matt arrived last evening and spent the day with Steve at ISFS sites.
Jacquie and I went to Sedgwick to investigate a problem with the ISS3 ceilometer computer and give her an introductory tour of the site. The problem turned to be a windows issue with the computer running but appearing to be almost frozen. The computer clock said the time was 0720 UTC, but the actual time was 18 UTC. Clicking on anything produced error dialogs "Insufficient resources to complete the requested service". It appears that some kind of operating system error was occurring. We did a hard reboot on the computer and it came back to life. Note for this system, the ceilometer processes (the CL-View and MLH software), require manual restarts after reboots. The Allsky camera software does restart automatically.
This afternoon we went to Rancho Alegre where we met Isabel and where a Scout Camporee was taking place. This is a large gathering of scouts where they take part in all kinds of activities and camp overnight. Around 150 scouts were in attendance. We talked to a large group of them (at least 50), gave a tour of the ISS and launched a radiosonde. They were an enthusiastic group and asked lots of questions.
Isabel got test plots of the for the wind lidar vertical stare mode working and also got the sounding temp and bufr messages uploading to EOL for eventual transfer to the GTS. She left this evening but will be back in a couple of weeks (thank you very much Isabel!!!).
The Rancho Alegre Scout Camporee sounding
Another hot day with light winds, sunny with a little scattered thin high level cirrus cloud. Sundowner winds are forecast for the next few days, with a possible IOP on Monday.
Today we worked at ISS1 at the county offices. All systems at the 3 sites appear to be working well, except that the Modular Profiler stopped recording data this morning for some unknown reason. It wasn't obvious at first because it was still drawing current and the profiler display looked normal with no noticeable error messages. Eventually I saw that the data files weren't updating and did a restart to get it running again. No data was recorded from 15 to 18 UTC. I did some cleaning up of data and around the trailer, and also installed a Microseven web camera. It is mounted on the met tower looking towards the north to monitor clouds over the mountains, and also to monitor the wind lidar which is helpful while controlling it remotely. It will take an image every minute saving those to an SD card. Isabel continued working on various software tasks including developing a time-height plot for wind lidar vertical stares.
Sample image from the Microseven camera on the met tower
A very warm day with light winds and no clouds.
We started off at ISS1 this morning, then moved out to Sedgwick where the ISS3 wind profiler had shutdown yesterday. It appears that the UPS powering it had turned off, but it isn't clear if there were any interruptions to power. No other systems went down although they are all on separate UPS units. The profiler UPS display claims that the draw is nearly 400 W and should run for 50 minutes. The draw on the other UPS is less so they should last longer. However a digital clock not on UPS did not reset implying that the power didn;t go out. It is possible there is a problem with that UPS so if there are further problems we plan to switch the profiler UPS with the ceilometer UPS. After reseting, everything is working well.
The NPS Twin Otter did a series of overpasses over all three of our wind profiler sites as part of a training and calibration exercise this afternoon. The aircraft includes a Doppler lidar and their scientists are interested in comparing their measurements with the profilers. They flew a cross pattern at around 11 thousand feet. We also went up Figueroa mountain about 4 miles northeast of Sedgwick to carry out maintenance on ISFS site number 15.
Liz departed today (many thanks for all your work out here). She will return to Santa Barbara late next week. Jacquie arrived this evening and will be here for 10 days.
Searching for the Twin Otter (indicated by the red arrow), the flightradar track over the three ISS sites, and ISFS 15 on Figueroa mountain
The TimeHeightPlot web-visualization tool now displays NIMA-processed data for the ISS2 and ISS3 915MHz profilers.
- http://datavis.eol.ucar.edu/time-height-plot/SWEX/iss2-prof915/nima
- http://datavis.eol.ucar.edu/time-height-plot/SWEX/iss3-prof915/nima
A sunny warm day with light winds. There were patches of fog early, but they disappeared very quickly and almost no other clouds were seen.
Today we mainly worked at ISS1 on a range of miscellaneous tasks. Liz built another Helium board (with flow meter, regulator and valve) to use as a backup in case either of the ones at Rancho Alegre or Sedgwick fails. She also worked on hard target pointing on the wind lidar to ensure it's pointing in the correct direction. Fortunately there is a prominent cell phone tower not far across the valley to serve as a target. It is at 569 meters range and azimuth bearing 221.56 degrees according to Google Earth) or 223 degrees (according to the Leosphere scanning spreadsheet). The lidar registered it at 572 meters range and 221.15 deg. We are still clarifying which is the correct bearing to use, but these measurements imply the uncertainty is within 1 - 2 degrees in orientation and about 3 meters in range.
Hard target pointing for orientation calibration: the cell tower target, Google Earth image and the PPI reflectivity display at the target
We also worked on plotting issues, for example Isabel is working on time - height plots for the vertical stares on the lidar, and I fixed a labelling bug on the soap 4-hour plots for the ISS2 site (they were mislabelled as Sedgwick) and regenerated those plots from March 24 onwards. We also did some tidying up, worked on cameras and installed an SD card in the ISS1 PurpleAir. We briefly visited the botanic gardens to see the Halo lidar, and in the evening we met Mack on the beach for pizza from the pizza oven in the back of his van, thank you Mack!
Warm and breezy today. The winds appear to have been a little less than forecast last night, although there were some strong gusts. A strong Sundowner is also forecast tonight, with gusts up to 70 mph on the eastern ridges. The first IOP was carried out last night and a second shorter IOP started late this afternoon.
All systems are operating normally. The soundings went well last night, although the students reported having to restart the systems a couple of times to get good communications with the ground check units. Also the surface observations are not showing up on the MW41 sounding screen. It does appear they are being ingested, just not displayed for some reason. Therefore for the sounding log spreadsheet we are using surface data from the dsm dashboard as viewed by a web browser on the dm computers browsing directly to the dsm ip addresses.
There were waves reported by the Twin Otter and waves (and perhaps rotors) can be seen in the ISS measurements, particularly the Modular Profiler and Wind Lidar at ISS1. A good example is in the 4-hour Modular Profiler plot from 5 - 9 UTC where the vertical velocity shows an apparent downward standing wave around 2km and more rapid up and down motions with periods of about 5 minutes up to about 1 km. The wind lidar also saw complicated wind structure as can be seen in the PPI scan at 0837 UTC.
The second IOP started with soundings at 4pm (23 UTC) at ISS2 Rancho Alegre and the university sites in Santa Barbara. We carried out the 4pm sounding, students will do the following soundings finishing with a 1 am sounding. The ISS2 sounding appeared to hit a strong downdraft soon after launch. At about 6 minutes into the flight the radiosonde pressure suddenly rose about 15 hPa, suggesting a drop of around 150 meters. Afterwards the balloon continued ascending however the MW41 registered the drop as the balloon bursting. This means that the skew-t and profile plots are truncated at just 780 hPa when in fact the ballon rose to a record 28 hPa before actually bursting. The ascending data is still in the MWX archive file but will have to be accessed in post-project processing.
The MW41 display for the 23 UTC sounding. Not the oscillating blue line which is the pressure trace and indicates that the radiosonde dropped about 6 minutes into the flight.
Foggy along the mountains early, turning in to warm breezy day with just a few mid and upper level clouds. A few lenticular wave clouds towards the south.
IOP1 started today with 10am (17 UTC) soundings from two ISS sites (and two university sites). Soundings will continue every 3 hours ending with 7am (PDT) soundings tomorrow morning. Another limited IOP has been proposed for tomorrow evening with overnight soundings starting at 7pm (PDT) just at two sites (the ISS site at Rancho Alegre and a university site in Santa Barbara).
All ISS systems are working well. The ISS crew started at ISS1 where I turned on tms (time series IQ) data on the Modular Profiler and checked on the lidar (which I switched to 50 meter resolution from 100 meter resolution last night). The lidar is seeing out to 2 to 3 km. Liz and Isabel went to Rancho Alegre to do the daytime soundings at ISS2, while I went to Sedgwick to check on the students doing the soundings from ISS3. The soundings mostly went well, although the students did have to reboot the sounding computer and MW41 a couple of times to get the ground check to work properly.
First student soundings for IOP1 at Sedgwick and Rancho Alegre and the first night-time sounding from Sedgwick