Friday, July 28. A terrible day in terms of discovering system problems. On this day, the 6th official day of data collection, I finally "woke up" to the "SHOTS" variable in the REAL housekeeping data. This is the number of laser pulses since the last scan. Given that we are completing a scan every 17 s, it should be about 170. But the housekeeping files showed it hovering around 85 or 86. Then, I connected this to the smaller than expected BSCAN file size that I had noticed days ago (but didn't react!) and had the sickening realization that we may not be collecting the full amount of data. By about 22:30 UTC Bruce found a bug in REAL's LabVIEW data acquisition code and repaired it.
There are a few lessons to learn here. One is that naming a variable "SHOTS" without providing a clear description of what it means can be hazardous. 8+ years ago when we ran the REAL last, I probably was "in touch" with that variable and knew what to expect. But time passed and I forgot. This week I had observed it scrolling by for almost 6 days and didn't bother to ask myself "What should the number be?" I assumed it was correct. It took me a quiet day just sitting and looking at data to realize it was too small. The second lesson is that you can't assume software is working properly and you must open and read your data files to make sure you are saving what is required.
The only photo I took this day was at 17:47 PDT:
Bruce returned to Bishop this afternoon.
New problem: While at dinner I noticed the REAL azimuth values were not decreasing. After dinner I worked with Robert Stillwell at NCAR in Boulder to try to find the cause. (Background: on January of 2015 we had the same problem with the elevation motor for which Dr. Eric Ayars determined was a bad quadrature angle pulse encoder. It required replacement of the elevation motor.)
Robert and I stopped the system, reinitialized the BSU, and started it again. After starting we noticed that the SHOTS variable was no longer up at 170 and was very erratic (around 4 UTC on 29th).
We also noticed that many other variables in the housekeeping data were much noisier including Nd:YAG energy, 1543 nm pulse energy, beam profile centroid position, radius, and fluence. Even the platform tiltmeter angles were looked noisier. It was as if the system had become possessed. Without any idea of what was wrong, and given the late hour, we shut the system down.
1 Comment
Unknown User (shane)
Friday, July 28. A terrible day in terms of discovering system problems. On this day, the 6th official day of data collection, I finally "woke up" to the "SHOTS" variable in the REAL housekeeping data. This is the number of laser pulses since the last scan. Given that we are completing a scan every 17 s, it should be about 170. But the housekeeping files showed it hovering around 85 or 86. Then, I connected this to the smaller than expected BSCAN file size that I had noticed days ago (but didn't react!) and had the sickening realization that we may not be collecting the full amount of data. By about 22:30 UTC Bruce found a bug in REAL's LabVIEW data acquisition code and repaired it.
There are a few lessons to learn here. One is that naming a variable "SHOTS" without providing a clear description of what it means can be hazardous. 8+ years ago when we ran the REAL last, I probably was "in touch" with that variable and knew what to expect. But time passed and I forgot. This week I had observed it scrolling by for almost 6 days and didn't bother to ask myself "What should the number be?" I assumed it was correct. It took me a quiet day just sitting and looking at data to realize it was too small. The second lesson is that you can't assume software is working properly and you must open and read your data files to make sure you are saving what is required.
The only photo I took this day was at 17:47 PDT:
Bruce returned to Bishop this afternoon.
New problem: While at dinner I noticed the REAL azimuth values were not decreasing. After dinner I worked with Robert Stillwell at NCAR in Boulder to try to find the cause. (Background: on January of 2015 we had the same problem with the elevation motor for which Dr. Eric Ayars determined was a bad quadrature angle pulse encoder. It required replacement of the elevation motor.)
Robert and I stopped the system, reinitialized the BSU, and started it again. After starting we noticed that the SHOTS variable was no longer up at 170 and was very erratic (around 4 UTC on 29th).
We also noticed that many other variables in the housekeeping data were much noisier including Nd:YAG energy, 1543 nm pulse energy, beam profile centroid position, radius, and fluence. Even the platform tiltmeter angles were looked noisier. It was as if the system had become possessed. Without any idea of what was wrong, and given the late hour, we shut the system down.