I refueled the upper and lower belly tanks on the generator today at 6pm. I'll plan to refuel again on Sunday evening while training for Clayton and Phil.
Damien Ceus, Field Engineer for Leosphere arrived in Flagstaff late Wednesday night, with plans to replace the troubled computer in the Lidar. The computer arrived via FedEx around 10am, and we departed to the site. As soon as we stopped at the 449 trailer, the problems with the 449 became apparent, and it appeared the profiler PCM had rebooted, and no causes could be identified. The Display manager PC was running just as I had left it on Wednesday night, so power had not been interrupted. After restarting the dashboard, all systems appeared to return to normal, but the backlog of Nagios errors would take time to clear.
We proceeded to the Lidar to commence repairs, and removed the PC after saving the configuration and calibration data to the portable HD. The bulk of the work was done at the ISFS trailer, where we were able to scavenge and utilize two 12v batteries and power supplies to create a 24vdc power source for the pc. We have installed both hard drives- the new and the old in the same PC, but they will only be bootable one at a time, no master or slave jumper settings could be identified. Contact me for further details, but rest assured, we retain the entire HD from the previous PC.
Reinstallation took some time, and many complications were uncovered, but the Lidar is operational - to a degree- there is a problem with the RHI scans, but hopefully the Leosphere team in France will be working on it overnight tonight. PPI scans, and DBS scans appear normal.
The PC is online for the Leosphere team in France tonight, but let them work on it first. Contact me for login info, it's all changed from the previous PC.
more info to come tomorrow.
tim
Weather today is cold and partly cloudy. We've been having light snow off and on all day, but there has been no accumulation.
Power amp is at .143 mW
Nagios screen shows critical error with the profiler_any_profiler (checks 3 of 5 have failed). After checking again at 0500 LST, problem corrected itself and no flag appeared.
Profiler appears to be working and power spectral plots look good. Spoke with Bill and he suspects the error has to do possibly with some work that was done earlier. I'll keep an eye on it.
Nagios also show critical errors with the Lidar. System Tech arrived today and is working with Tim on replacing the computer. I imagine that is why the lidar is flagged red.
In looking the profiler display the dashboard buttons are all green, and the dwell counter is working properly.
An IOP has been called for tomorrow evening.
The generator is half full. I was going to refill, but couldn't figure out where how to hook up the tank to my rental car engine. Will ask Tim for help with this when I see him.
Internet connection is out again at main site, but just for the GAUS and ISS computers (same problem as we saw on Sunday night, cause unknown but data is all coming in.
Windy Day!!!
High gusty windstorm from the South has this trailer rocking like I was going down the interstate.
Critical error and stopped 449 was identified by Bill, and I came to investigate. The Dashboard showed the DRX program had stopped, and all it took was a stop and restart of that program to resolve the error. See 449 stopped for reference, including copied output text of normal and abnormal output.
Additional problems were found in the display of the surface data graphing. The graph as shown by the ISS Icon bar ceased to update around 09:00UTC 09oct, even though current time was around 19:00UTC 09oct. Blanking and restarting the display did not fix the error, but stopping and restarting the ISS Icon bar did result in an updated graph. The data continues to appear updated. Apparently the webplots were not affected, according to Bill.
NAGIOS service monitor continues to show 4 critical errors at this time:
metcraxii/profiler_anyfail_surface
metcraxii/profiler_latency_raw/nidas_1_10
metcraxii/profiler_latency_raw/nidas_1_11
metcraxii/profiler_latency_surface/wxt/standard/nc
Nagios shows WXT ping OK, however..... I'll not be going up to the rim to investigate at this time. The winds are howling at 22 M/Sec, and I don't think they would let me outside, nor do I think I could do much good in this wind!
All other systems appear ok, dust is settling on everything, I vacuumed everything I could, including the dust atop the 449 amplifier electronics.
(wow- motion sickness/queasiness from trailer rocking becoming an issue!- I am going outside!)
The 449 profiler had a critical error on Nagios this morning, and the onsite investigation revealed the DRX program had stopped. The Dashboard showed the DRX program had stopped (red "STOP") but showed the products and status as "running".
The pertinent text in the output of the DRX is as follows.
This first block of output is NORMAL. This was the last section of normal data prior to the error, ending at 07:55:00.
Scroll down below to find ABNORMAL output block, This was the first section of abnormal data, starting at 07:55:06.
sd3c revision: 10007
data rate: 0.805 MB/s
delay: 5.94e-06s
hpa delay: 1.04e-06s
hpa width: 5.73e-06s
blanker delay: 0s
blanker width: 7.81e-06s
nsum: 60
comp code len: 4
schedule name: Metcrax
paramset name: Mecrax_RIM
gates: 40
RIM: yes
channels: 3
sample clock: 4.8e+07
nco frequency: 1.2e+07
coarse mixer mode:9
fifo decimation: 48
*** processing enabled on channel 0
*** processing enabled on channel 1
*** processing enabled on channel 2
Channel 1 waiting for data...
Channel 0 waiting for data...
Channel 2 waiting for data...
384 DAC memory locations being used
Signals blocked in main are:
TTLIn(1:0):11 2013-Oct-09 07:45:22 (37,48 C) 0.2 MB/s nopub:0 tags:0 sync:0 0.2 MB/s nopub:0 tags:0 sync:0 0.2 MB/s nopub:0 tags:0 sync:0
TTLIn(1:0):11 2013-Oct-09 07:45:32 (37,48 C) 0.198 MB/s nopub:0 tags:0 sync:0 0.198 MB/s nopub:0 tags:0 sync:0 0.198 MB/s nopub:0 tags:0 sync:0
TTLIn(1:0):11 2013-Oct-09 07:45:42 (37,48 C) 0.203 MB/s nopub:0 tags:0 sync:0 0.203 MB/s nopub:0 tags:0 sync:0 0.203 MB/s nopub:0 tags:0 sync:0
TTLIn(1:0):11 2013-Oct-09 07:45:52 (37,48 C) 0.203 MB/s nopub:0 tags:0 sync:0 0.203 MB/s nopub:0 tags:0 sync:0 0.203 MB/s nopub:0 tags:0 sync:0
TTLIn(1:0):11 2013-Oct-09 07:46:03 (37,47 C) 0.203 MB/s nopub:0 tags:0 sync:0 0.203 MB/s nopub:0 tags:0 sync:0 0.203 MB/s nopub:0 tags:0 sync:0
TTLIn(1:0):11 2013-Oct-09 07:46:13 (37,47 C) 0.203 MB/s nopub:0 tags:0 sync:0 0.203 MB/s nopub:0 tags:0 sync:0 0.203 MB/s nopub:0 tags:0 sync:0
TTLIn(1:0):11 2013-Oct-09 07:46:23 (37,48 C) 0.203 MB/s nopub:0 tags:0 sync:0 0.203 MB/s nopub:0 tags:0 sync:0 0.203 MB/s nopub:0 tags:0 sync:0
TTLIn(1:0):11 2013-Oct-09 07:46:33 (37,47 C) 0.203 MB/s nopub:0 tags:0 sync:0 0.203 MB/s nopub:0 tags:0 sync:0 0.203 MB/s nopub:0 tags:0 sync:0
TTLIn(1:0):11 2013-Oct-09 07:46:43 (37,47 C) 0.203 MB/s nopub:0 tags:0 sync:0 0.203 MB/s nopub:0 tags:0 sync:0 0.203 MB/s nopub:0 tags:0 sync:0
TTLIn(1:0):11 2013-Oct-09 07:46:53 (37,47 C) 0.203 MB/s nopub:0 tags:0 sync:0 0.203 MB/s nopub:0 tags:0 sync:0 0.203 MB/s nopub:0 tags:0 sync:0
TTLIn(1:0):11 2013-Oct-09 07:47:03 (37,47 C) 0.203 MB/s nopub:0 tags:0 sync:0 0.203 MB/s nopub:0 tags:0 sync:0 0.203 MB/s nopub:0 tags:0 sync:0
TTLIn(1:0):11 2013-Oct-09 07:47:13 (37,47 C) 0.203 MB/s nopub:0 tags:0 sync:0 0.203 MB/s nopub:0 tags:0 sync:0 0.203 MB/s nopub:0 tags:0 sync:0
TTLIn(1:0):11 2013-Oct-09 07:47:23 (37,47 C) 0.203 MB/s nopub:0 tags:0 sync:0 0.203 MB/s nopub:0 tags:0 sync:0 0.203 MB/s nopub:0 tags:0 sync:0
TTLIn(1:0):11 2013-Oct-09 07:47:33 (37,47 C) 0.203 MB/s nopub:0 tags:0 sync:0 0.203 MB/s nopub:0 tags:0 sync:0 0.203 MB/s nopub:0 tags:0 sync:0
TTLIn(1:0):11 2013-Oct-09 07:47:43 (37,47 C) 0.202 MB/s nopub:0 tags:0 sync:0 0.202 MB/s nopub:0 tags:0 sync:0 0.202 MB/s nopub:0 tags:0 sync:0
TTLIn(1:0):11 2013-Oct-09 07:47:53 (38,46 C) 0.201 MB/s nopub:0 tags:0 sync:0 0.187 MB/s nopub:0 tags:0 sync:0 0.184 MB/s nopub:0 tags:0 sync:0
TTLIn(1:0):11 2013-Oct-09 07:48:03 (37,47 C) 0.201 MB/s nopub:0 tags:0 sync:0 0.215 MB/s nopub:0 tags:0 sync:0 0.218 MB/s nopub:0 tags:0 sync:0
TTLIn(1:0):11 2013-Oct-09 07:48:13 (38,47 C) 0.201 MB/s nopub:0 tags:0 sync:0 0.201 MB/s nopub:0 tags:0 sync:0 0.201 MB/s nopub:0 tags:0 sync:0
TTLIn(1:0):11 2013-Oct-09 07:48:23 (37,47 C) 0.202 MB/s nopub:0 tags:0 sync:0 0.202 MB/s nopub:0 tags:0 sync:0 0.202 MB/s nopub:0 tags:0 sync:0
TTLIn(1:0):11 2013-Oct-09 07:48:33 (37,47 C) 0.203 MB/s nopub:0 tags:0 sync:0 0.203 MB/s nopub:0 tags:0 sync:0 0.2 MB/s nopub:0 tags:0 sync:0
TTLIn(1:0):11 2013-Oct-09 07:48:43 (38,48 C) 0.203 MB/s nopub:0 tags:0 sync:0 0.203 MB/s nopub:0 tags:0 sync:0 0.205 MB/s nopub:0 tags:0 sync:0
TTLIn(1:0):11 2013-Oct-09 07:48:53 (37,47 C) 0.203 MB/s nopub:0 tags:0 sync:0 0.203 MB/s nopub:0 tags:0 sync:0 0.203 MB/s nopub:0 tags:0 sync:0
TTLIn(1:0):11 2013-Oct-09 07:49:03 (38,47 C) 0.203 MB/s nopub:0 tags:0 sync:0 0.203 MB/s nopub:0 tags:0 sync:0 0.203 MB/s nopub:0 tags:0 sync:0
TTLIn(1:0):11 2013-Oct-09 07:49:13 (38,48 C) 0.203 MB/s nopub:0 tags:0 sync:0 0.203 MB/s nopub:0 tags:0 sync:0 0.203 MB/s nopub:0 tags:0 sync:0
TTLIn(1:0):11 2013-Oct-09 07:49:23 (37,48 C) 0.203 MB/s nopub:0 tags:0 sync:0 0.203 MB/s nopub:0 tags:0 sync:0 0.203 MB/s nopub:0 tags:0 sync:0
TTLIn(1:0):11 2013-Oct-09 07:49:33 (38,48 C) 0.203 MB/s nopub:0 tags:0 sync:0 0.203 MB/s nopub:0 tags:0 sync:0 0.203 MB/s nopub:0 tags:0 sync:0
TTLIn(1:0):11 2013-Oct-09 07:49:43 (37,48 C) 0.203 MB/s nopub:0 tags:0 sync:0 0.203 MB/s nopub:0 tags:0 sync:0 0.203 MB/s nopub:0 tags:0 sync:0
TTLIn(1:0):11 2013-Oct-09 07:49:53 (37,49 C) 0.203 MB/s nopub:0 tags:0 sync:0 0.203 MB/s nopub:0 tags:0 sync:0 0.203 MB/s nopub:0 tags:0 sync:0
TTLIn(1:0):11 2013-Oct-09 07:50:03 (37,48 C) 0.203 MB/s nopub:0 tags:0 sync:0 0.203 MB/s nopub:0 tags:0 sync:0 0.203 MB/s nopub:0 tags:0 sync:0
TTLIn(1:0):11 2013-Oct-09 07:50:13 (37,49 C) 0.203 MB/s nopub:0 tags:0 sync:0 0.203 MB/s nopub:0 tags:0 sync:0 0.203 MB/s nopub:0 tags:0 sync:0
TTLIn(1:0):11 2013-Oct-09 07:50:23 (38,49 C) 0.203 MB/s nopub:0 tags:0 sync:0 0.203 MB/s nopub:0 tags:0 sync:0 0.203 MB/s nopub:0 tags:0 sync:0
TTLIn(1:0):11 2013-Oct-09 07:50:33 (37,49 C) 0.203 MB/s nopub:0 tags:0 sync:0 0.198 MB/s nopub:0 tags:0 sync:0 0.203 MB/s nopub:0 tags:0 sync:0
TTLIn(1:0):11 2013-Oct-09 07:50:43 (38,49 C) 0.203 MB/s nopub:0 tags:0 sync:0 0.208 MB/s nopub:0 tags:0 sync:0 0.203 MB/s nopub:0 tags:0 sync:0
TTLIn(1:0):11 2013-Oct-09 07:50:53 (38,49 C) 0.203 MB/s nopub:0 tags:0 sync:0 0.203 MB/s nopub:0 tags:0 sync:0 0.203 MB/s nopub:0 tags:0 sync:0
TTLIn(1:0):11 2013-Oct-09 07:51:03 (38,50 C) 0.203 MB/s nopub:0 tags:0 sync:0 0.203 MB/s nopub:0 tags:0 sync:0 0.203 MB/s nopub:0 tags:0 sync:0
TTLIn(1:0):11 2013-Oct-09 07:51:13 (38,49 C) 0.203 MB/s nopub:0 tags:0 sync:0 0.203 MB/s nopub:0 tags:0 sync:0 0.203 MB/s nopub:0 tags:0 sync:0
TTLIn(1:0):11 2013-Oct-09 07:51:23 (38,49 C) 0.196 MB/s nopub:0 tags:0 sync:0 0.196 MB/s nopub:0 tags:0 sync:0 0.196 MB/s nopub:0 tags:0 sync:0
TTLIn(1:0):11 2013-Oct-09 07:51:33 (38,50 C) 0.203 MB/s nopub:0 tags:0 sync:0 0.203 MB/s nopub:0 tags:0 sync:0 0.203 MB/s nopub:0 tags:0 sync:0
TTLIn(1:0):11 2013-Oct-09 07:51:43 (38,50 C) 0.203 MB/s nopub:0 tags:0 sync:0 0.203 MB/s nopub:0 tags:0 sync:0 0.203 MB/s nopub:0 tags:0 sync:0
TTLIn(1:0):11 2013-Oct-09 07:51:53 (38,49 C) 0.203 MB/s nopub:0 tags:0 sync:0 0.203 MB/s nopub:0 tags:0 sync:0 0.203 MB/s nopub:0 tags:0 sync:0
TTLIn(1:0):11 2013-Oct-09 07:52:03 (38,49 C) 0.203 MB/s nopub:0 tags:0 sync:0 0.203 MB/s nopub:0 tags:0 sync:0 0.203 MB/s nopub:0 tags:0 sync:0
TTLIn(1:0):11 2013-Oct-09 07:52:13 (39,50 C) 0.203 MB/s nopub:0 tags:0 sync:0 0.203 MB/s nopub:0 tags:0 sync:0 0.203 MB/s nopub:0 tags:0 sync:0
TTLIn(1:0):11 2013-Oct-09 07:52:23 (38,50 C) 0.203 MB/s nopub:0 tags:0 sync:0 0.203 MB/s nopub:0 tags:0 sync:0 0.203 MB/s nopub:0 tags:0 sync:0
TTLIn(1:0):11 2013-Oct-09 07:52:33 (38,49 C) 0.203 MB/s nopub:0 tags:0 sync:0 0.203 MB/s nopub:0 tags:0 sync:0 0.203 MB/s nopub:0 tags:0 sync:0
TTLIn(1:0):11 2013-Oct-09 07:52:43 (39,50 C) 0.203 MB/s nopub:0 tags:0 sync:0 0.203 MB/s nopub:0 tags:0 sync:0 0.203 MB/s nopub:0 tags:0 sync:0
TTLIn(1:0):11 2013-Oct-09 07:52:53 (38,51 C) 0.203 MB/s nopub:0 tags:0 sync:0 0.203 MB/s nopub:0 tags:0 sync:0 0.203 MB/s nopub:0 tags:0 sync:0
TTLIn(1:0):11 2013-Oct-09 07:53:03 (38,50 C) 0.203 MB/s nopub:0 tags:0 sync:0 0.203 MB/s nopub:0 tags:0 sync:0 0.203 MB/s nopub:0 tags:0 sync:0
TTLIn(1:0):11 2013-Oct-09 07:53:13 (38,50 C) 0.203 MB/s nopub:0 tags:0 sync:0 0.203 MB/s nopub:0 tags:0 sync:0 0.203 MB/s nopub:0 tags:0 sync:0
TTLIn(1:0):11 2013-Oct-09 07:53:23 (38,51 C) 0.203 MB/s nopub:0 tags:0 sync:0 0.203 MB/s nopub:0 tags:0 sync:0 0.203 MB/s nopub:0 tags:0 sync:0
TTLIn(1:0):11 2013-Oct-09 07:53:33 (38,50 C) 0.203 MB/s nopub:0 tags:0 sync:0 0.203 MB/s nopub:0 tags:0 sync:0 0.203 MB/s nopub:0 tags:0 sync:0
TTLIn(1:0):11 2013-Oct-09 07:53:43 (38,49 C) 0.203 MB/s nopub:0 tags:0 sync:0 0.203 MB/s nopub:0 tags:0 sync:0 0.203 MB/s nopub:0 tags:0 sync:0
TTLIn(1:0):11 2013-Oct-09 07:53:53 (38,49 C) 0.203 MB/s nopub:0 tags:0 sync:0 0.203 MB/s nopub:0 tags:0 sync:0 0.203 MB/s nopub:0 tags:0 sync:0
TTLIn(1:0):11 2013-Oct-09 07:54:03 (38,50 C) 0.203 MB/s nopub:0 tags:0 sync:0 0.203 MB/s nopub:0 tags:0 sync:0 0.203 MB/s nopub:0 tags:0 sync:0
TTLIn(1:0):11 2013-Oct-09 07:54:13 (38,50 C) 0.203 MB/s nopub:0 tags:0 sync:0 0.203 MB/s nopub:0 tags:0 sync:0 0.203 MB/s nopub:0 tags:0 sync:0
TTLIn(1:0):11 2013-Oct-09 07:54:23 (38,50 C) 0.203 MB/s nopub:0 tags:0 sync:0 0.203 MB/s nopub:0 tags:0 sync:0 0.203 MB/s nopub:0 tags:0 sync:0
TTLIn(1:0):11 2013-Oct-09 07:54:33 (38,50 C) 0.196 MB/s nopub:0 tags:0 sync:0 0.196 MB/s nopub:0 tags:0 sync:0 0.196 MB/s nopub:0 tags:0 sync:0
TTLIn(1:0):11 2013-Oct-09 07:54:43 (39,50 C) 0.203 MB/s nopub:0 tags:0 sync:0 0.203 MB/s nopub:0 tags:0 sync:0 0.203 MB/s nopub:0 tags:0 sync:0
TTLIn(1:0):11 2013-Oct-09 07:54:53 (38,49 C) 0.203 MB/s nopub:0 tags:0 sync:0 0.203 MB/s nopub:0 tags:0 sync:0 0.203 MB/s nopub:0 tags:0 sync:0
TTLIn(1:0):11 2013-Oct-09 07:55:00 (37,48 C) 0.211 MB/s nopub:0 tags:0 sync:0 0.211 MB/s nopub:0 tags:0 sync:0 0.211 MB/s nopub:0 tags:0 sync:0
Shutting down...
Stopping thread for channel 0
Stopping thread for channel 1
Stopping thread for channel 2
This is the next output with the first set of ABNORMAL output of the DRX program.
sd3c revision: 10007
data rate: 7.17 MB/s
delay: 5.94e-06s
hpa delay: 1.04e-06s
hpa width: 2.92e-06s
blanker delay: 0s
blanker width: 4.58e-06s
nsum: 4
comp code len: 1
schedule name: Metcrax
paramset name: rass
gates: 10
RIM: yes
channels: 3
sample clock: 4.8e+07
nco frequency: 1.2e+07
coarse mixer mode:9
fifo decimation: 48
*** processing enabled on channel 0
*** processing enabled on channel 1
*** processing enabled on channel 2
Channel 1 waiting for data...
Channel 0 waiting for data...
Channel 2 waiting for data...
96 DAC memory locations being used
Signals blocked in main are:
2013-10-09 07:55:06|ERROR[p7142Dn] Dropping data on channel 2, no free buffers available!
2013-10-09 07:55:06|ERROR[p7142Dn] Dropping data on channel 2, no free buffers available!
2013-10-09 07:55:06|ERROR[p7142Dn] Dropping data on channel 2, no free buffers available!
2013-10-09 07:55:06|ERROR[p7142Dn] ERROR! Channel 2 wants to read descriptor 0 while DMA is in progress there! Likely overrun!
2013-10-09 07:55:06|ERROR[p7142Dn] ERROR! Channel 2 wants to read descriptor 0 while DMA is in progress there! Likely overrun!
Dropping data on channel 2, no free buffers available!
2013-10-09 07:55:06|ERROR[p7142Dn] ERROR! Channel 2 wants to read descriptor 1 while DMA is in progress there! Likely overrun!
2013-10-09 07:55:06|ERROR[p7142Dn] ERROR! Channel 2 wants to read descriptor 1 while DMA is in progress there! Likely overrun!
Dropping data on channel 2, no free buffers available!
2013-10-09 07:55:06|ERROR[p7142Dn] Dropping data on channel 0, no free buffers available!
2013-10-09 07:55:06|ERROR[p7142Dn] ERROR! Channel 2 wants to read descriptor 2 while DMA is in progress there! Likely overrun!
2013-10-09 07:55:06|ERROR[p7142Dn] ERROR! Channel 2 wants to read descriptor 2 while DMA is in progress there! Likely overrun!
Dropping data on channel 2, no free buffers available!
2013-10-09 07:55:06|ERROR[p7142Dn] Dropping data on channel 0, no free buffers available!
2013-10-09 07:55:06|ERROR[p7142Dn] ERROR! Channel 2 wants to read descriptor 3 while DMA is in progress there! Likely overrun!
2013-10-09 07:55:06|ERROR[p7142Dn] ERROR! Channel 2 wants to read descriptor 3 while DMA is in progress there! Likely overrun!
Dropping data on channel 2, no free buffers available!
2013-10-09 07:55:06|ERROR[p7142Dn] Dropping data on channel 0, no free buffers available!
2013-10-09 07:55:06|ERROR[p7142Dn] ERROR! Channel 2 wants to read descriptor 0 while DMA is in progress there! Likely overrun!
2013-10-09 07:55:06|ERROR[p7142Dn] ERROR! Channel 2 wants to read descriptor 0 while DMA is in progress there! Likely overrun!
Dropping data on channel 2, no free buffers available!
2013-10-09 07:55:06|ERROR[p7142Dn] ERROR! Channel 0 wants to read descriptor 1 while DMA is in progress there! Likely overrun!
2013-10-09 07:55:06|ERROR[p7142Dn] ERROR! Channel 0 wants to read descriptor 1 while DMA is in progress there! Likely overrun!
Dropping data on channel 0, no free buffers available!
2013-10-09 07:55:06|ERROR[p7142Dn] ERROR! Channel 2 wants to read descriptor 1 while DMA is in progress there! Likely overrun!
2013-10-09 07:55:06|ERROR[p7142Dn] ERROR! Channel 2 wants to read descriptor 1 while DMA is in progress there! Likely overrun!
Dropping data on channel 2, no free buffers available!
2013-10-09 07:55:06|ERROR[p7142Dn] ERROR! Channel 0 wants to read descriptor 2 while DMA is in progress there! Likely overrun!
2013-10-09 07:55:06|ERROR[p7142Dn] ERROR! Channel 0 wants to read descriptor 2 while DMA is in progress there! Likely overrun!
Dropping data on channel 0, no free buffers available!
2013-10-09 07:55:06|ERROR[p7142Dn] ERROR! Channel 2 wants to read descriptor 2 while DMA is in progress there! Likely overrun!
2013-10-09 07:55:06|ERROR[p7142Dn] ERROR! Channel 2 wants to read descriptor 2 while DMA is in progress there! Likely overrun!
Dropping data on channel 2, no free buffers available!
2013-10-09 07:55:06|ERROR[p7142Dn] ERROR! Channel 0 wants to read descriptor 3 while DMA is in progress there! Likely overrun!
2013-10-09 07:55:06|ERROR[p7142Dn] ERROR! Channel 0 wants to read descriptor 3 while DMA is in progress there! Likely overrun!
Dropping data on channel 0, no free buffers available!
This output continued for quite some time before I stopped and restarted the DRX program at roughly 19:00 UTC.
Hope this is useful in debug.
Thanks,
Tim
Gary has reported that the WXT520 installed at the LIDAR on North Rim was down, and not reporting. I investigated, at found no obvious issues. The unit has power, and is constantly making it's wind measurements, and the cable was inspected for (potential) rodent gnawing, none found. I checked and reinstalled the RS232 connector to the digiport, finding no break or abnormality in the connection. I unplugged and repowered the digiport to see if that fixes the problem. There is no Nagios status for the WXT to check, and I cannot find the location of the WXT520 files in the file structures as I know them, so I believe Gary or Bill will have to point me in the right direction to verify status. Hope it's resolved, tell me if it isn't.
I also spent much of the afternoon installing a solar power charging system for the German research group on the rim scintillometer. It should be working now. Installed outdoor lighting at GAUS trailer, and prepared for next potential IOP, could be Thursday night or Friday.
449 running in Metcrax_RIM mode, ok
All other systems, ok
nagios status reports, ok
Gnite, Tim
Daily Report for Monday October 7
Fine today, southerly breeze last night turning to light easterlies, then southerly again by evening. Scattered high cloud, cirrus. Apparently conditions are unfavorable for another IOP for the next few days.
IOP-1 last night went well. Seven soundings were launched, and a tethersonde was operated at the ISFS base, and two other tethers were in the crater. The NCAR windcube scanned into the crater (operated by Nihanth) and the UU Halo lidar was in the crater doing coordinated scans. There are a very sharp inversion seen by the soundings and by RASS, 5C near the ground with the southerly, then 15C up a few hundred meters where there were easterlies (seen by the lidar and 449 wind profiler). Soundings at ISFS were carried out by Tim, Dave and Joanas.
449 Profiler running well. Using "Metcrax" mode which is now 10 minute winds followed by 5 minutes RASS. RASS sound now appears to be stable. Amplifier power is 141W in winds, 199W in RASS.
Did a bunch of 449 tests this evening to check the bandwidth (for freqs see data recorded in the files). Started around 01:54 UT (Oct 8) and continued various tests until 4:06.
From 4;30 to 4:59 used delay line to calibrate range - in standard mode appears to be correct (to better than 50m). Will analyze later to check RIM. Back to regular Metcrax schedule from 5UT.
Training Kate on ISS operations, visiting ISS, lidar, sodar and ISFS base (soundings and ceilometer).
Various data backup activities: DVD on ISS DM; completed disk SHUTTLE449_2 (has data from Oct 1 to Oct 8 5UT) and installed fresh disk (SHUTTLE449_3); installed USB drive on lidar and backed up all data so far.
We met Judy Prossier yesterday and she said that cattle are being moved into our area, so installed an electric fence around the 449 profiler. Tested and no apparent affect on 449 data, but turned off for now until the cattle actually appear.
Generator was serviced today around 21UT. Contrary to expectations, the generator wasn't switched out, but instead was shutdown and serviced in place (apparently they had a problem with their replacement units). Fortunately we were here so we shut down the computers and Kate brought them back up (as a training exercise). The shutdown lasted around 15 minutes so we could have remained operating on the large UPS in the trailer.
I am headed back to Boulder. Kate and Tim are now the operators. Good luck for the rest of the project.
We had a partial network outage in the ISFS trailer that started around 10pm local time on Sunday night. There was a great deal of activity going on in the trailer, that seemed to confuse and complicate the issue, but the initial evidence was that Dave W. lost his wireless connection, and we had to switch him over to a wired connection. Then we noticed that the wired connections in the rear of the trailer that serve the ceilometer, sounding, and data manager computers were not connected to the network... Diagnosis began working from the back to the front, with rebooting the hubs in the rear (linksys) then the middle (d-link), but hesitated at rebooting the white belkin netgear wireless router... Other computers on the d-link (middle) are downstream from the belkin, and DID have outside internet connection. So I was reluctant to risk breaking something that was working, and risk loosing connection to Dave's (critical) laptop during our IOP.
After much diagnosis, I realized there was no other option but to reset the white belkin netgear wireless router, and I finally did about 1:30am, and all seems well after that. I reset the connections to the APG systems in the back, and the backlog of latent connections on Nagios began to clear out. (correction- At 2:15am, still several Nagios services reporting as critical, but are wholeskycamera related to network loss.
Hope I didn't screw things up too much- I blame the lack of sleep!
Thanks, Tim
Daily Report for Sunday October 6
IOP1 - finally good conditions for an IOP. Fine and cool with light northerlies during the day, becoming calm this evening.
IOP started with a sounding at 4pm (22UT), and soundings approx every 3 hours afterwards. Rapid cooling at ground level (5C late evening) but warm aloft (wind profiler RASS showing 15C at 500m. Winds switched to south-westerly at ground level, but aloft the wind profiler and lidar both show easterlies.
Tethersondes operating in crater and at ISFS base. Nihanth is operating our lidar in mainly RHI scans across crater with occasional PPIs. Sebastian is in crater operating UU lidar in coordinated scans.
Some network difficulties at ISFS is affecting sounding operations because of limited connectivity between DSM based surface met there and GAUS DM and sounding laptop, so some soundings may not have surface met so may be delayed in posting to web.
Generator down to 1/4 tank so partly refilled this evening (to 1/3 tank) to ensure we get through to tomorrow afternoon (when generator due to be replaced).
Wind profiler up & down this afternoon as adjustments were made to parameters (eg, blanking, HPA, etc) to get as low measurements as possible. Both winds and RASS now often (although not always) reporting down to 200 meters in standard mode. RASS frequency also increased to a 5-minute run every quarter hour. Charlie made adjustments to sound generator; now resetting every 30 seconds.
Kate arrived this afternoon to assist ISS operations.
Daily Report for Saturday October 5
Fine and cool with northerly breeze. It should become calmer so an IOP still planned for tomorrow evening.
All systems running well. 449 seeing winds to 2km, occasionally 3km, and RASS to 1 - 1.5km. Power amp running at about 140 W in winds and 190W in RASS. Sodar seeing winds to 120 - 150 m and RASS to 120 - 200m.
The air is very clear so the lidar wasn't seeing very far, perhaps 1 - 1.5km so I increased the range gates from 25m to 50m, and appeared to get a few hundred meters extra range. Nihanth and Sebastien were doing tests for tomorrows IOP around 0:40 UT, 3 UT, and 5:30 UT this evening (Oct 6). They managed to spot the ISFS south rim tower at bearing 195.1 deg azimuth, which agrees very well with an estimate from GPS coords and Google Earth of 195.15 deg.
Switched out shuttle disk drive 1 for disk 2 to carry data back to Boulder. The log for disk 1 is at ds/logs/disk_shuttle449_1.log It covers the period Sep 25 - Oct 3 and has nearly 1TB of mainly 449 data, but also some lidar data.
At late afternoon the generator diesel was just over 1/2 full, oil press & temp 60 psi and 180F.
Daily Report for Friday October 4
Sunny, but cool with northerly breeze as a dry cold front moved through. Expecting conditions to improve Sunday with possible IOP Sunday night.
All systems working well. Profiler seeing winds to 2 - 3 km and RASS to 1 - 1.2 km, the sodar seeing winds to 100 - 150 m and RASS to 100 - 200 m, and the lidar seeing out to 1.5 - 2 km. Profiler amp power is 200W for RASS and 145W for winds.
Set up more of the web site today, and now wind profiler plots are appearing http://www.eol.ucar.edu/content/apg-metcraxii
We adjusted the lidar heading and level today, which required moving the platform a little. Previously the lidar had been pointing at about 353 deg, and from 22 UTC it was pointed directly at true north. Note that the internal compass seems to be about 2 deg off, and claims that it is pointing at 347 deg (mag) or 358 deg true. We also corrected the level.
Generator at late afternoon is about 3/4 on the dipstick, oil press & temp is 60 psi and 180 F. Set up generator servicing with Sunstate Equip for Monday afternoon at 2pm.
Daily Report for Thursday Oct 3
Fine, warm and brisk southwesterly breeze since this morning. Forecast said a dry cold front is coming through so too breezy and disturbed for an IOP.
449 Profiler working well, apart from a couple of dropped files around 18 & 21Z in both winds and rass. Occasional sound drop-outs. Average amp power 123W. Backup disk installed on DM to copy 449 data (using a Voyager docking station). DVD writing also tested - plan to use this to give lidar data to the ASU folks.
Sodar working very well since Eckhard made a couple of parameter changes yesterday. He reduced the gates to 15m (from 20m) which involved the first gate. He also switched the DBS operation to different beams - was A0 & A1 (SW & NW), now A1 & A2 (NW & NE) which improved the apparent quality of the winds considerably, although the range is a little less thanks to the shorter pulse. Today both winds and rass height ranges are quite variable varying from about 75 to 180m.
The lidar is getting returns out to about 2km range and a km or more altitude. We shut it off during the afternoon to avoid over heating.
WXT was moved to the lidar this morning from around 17Z. Last night it ran between 449 trailer and the surface met tower. A quick comparison shows that the temperature and RH agreed very well (within 1C and ~3%); the pressure and winds were a little higher on the WXT (approx 1hPa and 1m/s) but this could have been due to placement with flow around the trailer (although with the surface met wind head at 40' and the WXT at 6', I would have expected the WXT to be lower).
Generator diesel still showing well above full on dipstick, oil temp 180 F, pressure 60 psi.
Gary is returning home tomorrow, leaving Bill & Tim for the first ops period - thanks Gary!
Daily Report October 2
Sunny and warm with southwesterly breeze. No IOP tonight - too breezy.
Last night's lidar only IOP went well, however some tweaking of lidar orientations are needed.
Comms link to sodar when down yesterday evening, Gary did some adjustments today, including realigning the link antenna which seemed to help. Sodar data is now being transferred back to Boulder and plotted. Eckhard adjusted the sodar range gates, and now there is 15 meter sampling (prev 20 meter gates).
Ceilometer MLH plot software needed restarting.
Generator fuel down to 1/3 so refueled this afternoon.
449 Profiler working well, also still intermittent problem with RASS sound. Looked at some RIM plots, looks promising although phase offsets need adjusting.
WXT test at 449 site, positioned near 449 met tower from 2330UT, agreeing well so far (data is being saved in iss/data stream).
Daily Report for Tuesday Oct 1.
Today was clear and warm with a light mainly southern or southwestly breeze although it was northerly briefly this morning.
Eckhard made some repairs to the sodar speakers and amplifiers today and now it is fully operational (previously 2 or 3 speakers and amps weren't functioning, although the sodar was still working well). We were also working on the lidar on tests and tracking down an orientation problem.
Tonight we have a mini IOP to test lidar operations and coordination - unofficially we are calling this IOP-0.
The NCAR lidar (windcube 200 s leased from Leosphere) is operating on the north rim of the crater and is doing a series of coordinated test scans with the University of Utah lidar operating on the floor of the crater. An example of the scans is the NCAR lidar doing RHI scans at 190 azimuth between +10 and -10 deg elevation, while the Utah lidar is doing the same azimuth but from about 45 deg thru zenith to -45 deg, which will allow detailed dual-Doppler in the vertical plane. Both lidars are now working well, but getting their beam directions exactly calibrated is tricky.
The 449 is running well, picking up quite a few birds this evening. RASS sound was turned off for the film crew until about 4 UT. The sound intermittently cuts out at other times, Charlie is working on the problem. When it is running correctly, it works well - typically seeing up to over 1km, however the lower gates need some adjustment.
Earlier in the evening, the 449 site was visited by a film crew to record part of a documentary for middle schoolers.
From Lou:
For Mextcrax II the dedicated ISS operators phone number will be 303-579-1534. This is the phone number which will be given to the generator and fuel companies etc.