DSMs
Data acquired by the DSMs are archived locally on their flash drives, into files with names like Ah1_20121018_000000.dat, where Ah1 is the hostname of the DSM.
On Viper systems with Bluetooth radios, the local archive is on the Compact Flash drive, /media/cf.
On Titans, or the Viper DSMs at C and M, the local archive is on the USB flash drive, /media/usbdisk.
On each system the $DATAMNT environment variable is set to the archive media mount point, either /media/cf or /media/usbdisk.
If you log into a DSM, a cd raw_data will set your working directory to the $DATAMNT/projects/SCP/raw_data.
lsu will list the current contents of the raw_data directory. After 0Z the previous UTC day's file are rsync'd to the base laptop, and deleted from the DSMs. So, in general there should only be today's files on the local storage, along with any of yesterday's which are still being rsync'd.
The DSMs also send their data in real-time using UDP (user datagram protocol), over their network link to port 30009 on the "gully" system at 192.168.0.12.
Data Flow on "gully" Base System
The real-time UDP feed is received by the nidas_udp_relay process on gully, which then makes it available as a TCP feed on port 30009. Processes can connect to TCP port 30009 on gully to receive this data. One such process is dsm_server, which archives the data to /media/isfs0/projects/SCP/raw_data in files with names like isfs_20121018_200000.dat.bz2. Because of the nature of the UDP protocol, this real-time feed is not a 100% recovery of the data, but generally has a low-rate of dropped packets.
dsm_server then makes the data feed available on port 30000, which is the default port used by programs like data_stats, data_dump, mote_dump etc.
statsproc also reads the real-time feed on port 30009 from nidas_udp_relay and writes 5 minute statistics to NetCDF files on $ISFF/projects/SCP/ISFF/netcdf. These real-time statistics are therefore computed from the less-than-complete real-time feed.
Archive File Rsyncs
Every 4 hours the rsync_stns.sh script runs on the gully laptop to rsync any of the previous day's NIDAS archive files from the flash drives on the DSMs. Usually only at 00:00 UTC does the script have anything to do. Four rsyncs are run simultaneously. An rsync runs for each group of DSMs that are served by the three NAP base radios. Another rsync copies files from the WIFI DSMs: C20, M21 and Mu22. While those rsyncs are in progress, a station may not respond to every ping, and so a ping failure does not indicate a broken connection.
After a file has been successfully rsync'd from a DSM, it is deleted there.
The check_rsync.sh script checks the status of the rsync file copies, by listing the hidden files on /media/isfs0/projects/SCP/raw_data and the .rsync-partial directory
[aster@gully scripts]$ check_rsync.sh Files being currently copied to /media/isfs0/projects/SCP/raw_data: -rw-------. 1 aster eol 28311552 Oct 16 19:41 .A16_20121016_000000.dat.u7tB8S -rw-------. 1 aster eol 1835008 Oct 16 19:41 .Ah2_20121016_120000.dat.oxLl0V -rw-------. 1 aster eol 2359296 Oct 16 19:41 .Ap10_20121016_120000.dat.xlpsC0 -rw-------. 1 aster eol 29622272 Oct 16 19:41 .Mu22_20121016_120000.dat.HnBmVR Partial copies on .rsync-partial that failed and will be resumed: -r--r--r--. 1 aster eol 10891630 Oct 16 19:12 A16_20121016_000000.dat -r--r--r--. 1 aster eol 4108273 Oct 16 18:15 Ap14_20121016_000000.dat
A archive file with a leading '.' is a file being currently copied. Any files in the .rsync-partial directory indicate that an rsync failed, and will be resumed later. This happens on one or two stations every night, and is not a problem.
After a station's files are rsync'd to gully, statsproc is run on those files to re-write the 5 minute statistics in the NetCDF files. These statistics are therefore created from the full archive, and should not have the dropouts associated with the UDP feed.
Disks on gully
The LaCie external USB drive is mounted on the gully system as /media/isfs0. It is a 2 TB drive, which also currently contains some old data from previous projects, which I'd like to keep. Currently it has 500 GB available.
To provide a backup archive and a way to shuttle data back to Boulder, a pocketec USB drive is also connected to gully. Every 4 hours from aster's crontab on gully, an rsync process is run to synchronize the archives on /media/isfs0 with /media/usbdisk:
10 */4 * * * [ -d /media/usbdisk/projects ] || mount /media/usbdisk; [ -d /media/isfs0/projects/SCP -a -d /media/usbdisk/projects ] && rsync -a --exclude=merge --exclude=.rsync-partial --exclude="raw_data/.*.dat.*" --exclude=raw_data_save /media/isfs0/projects/SCP /media/usbdisk/projects
Swapping Pocketec
Approximately once a week, when someone is coming back to Boulder, the pocketec drive should be unmounted, removed from gully and brought back, and a empty one reconnected in its place.
From a shell window on gully, from the aster account (you don't have to be root), un-mount the drive:
umount /media/usbdisk
If it reports an error about being busy, you should try to figure out why. It might be an open shell window that is using it as a working directory. Look through the shell windows and do "cd" or "exit", and retry the umount.
Every 4 hours an rsync runs to copy the raw_data files from /media/isfs0 to the pocketec. That might be causing the busy error. but it It should finish in a minute or two.
Assuming the umount is successful,unplug the Pocketc, and bring it back to Boulder.
An empty pocketecs should be in the plastic box on the desk. Connect it to the USB port, and flip the switch on the drive to USB. You should see the LED flash green after a few seconds. Then, from the aster account:
mount /media/usbdisk
If it doesn't flash green, and reports an error when trying to mount, then set the switch to off/ext and then back to usb and try again.
Since the rsync of data to the pocketec only happens every 4 hours, you won't typically see activity on the LED.
NetCDF Files to Boulder, Web Plots and Table
Once an hour, the 5 minute NetCDF files are rsync'd back to Boulder.
Cockpit at EOL
(should work on this to display the whole data flow, statsproc, archive, rsync).
Here's the data flow from the DSMs to a cockpit process at EOL:
One link could be removed in the above scheme.
The dsm_server process on gully could actually send data as UDP directly to eol-rt-data, instead of via dsm_server on flux, which would remove one link. Other processes, namely statsproc connect to dsm_server on gully. It was easier to restart dsm_server on flux to set this up.
Or the dsm_server process on flux could get its data directly from the nidas_udp_relay on gully rather than dsm_server, which seems to be the simplest solution.
cockpit out at SCP attaches to the dsm_server on flux, which has a slightly different configuration for the real-time display, such that it doesn't process all GPS variables, and doesn't track the CSAT3 counts. Having dsm_server on flux attach to dsm_server on gully provides quick notification if dsm_server on gully went down.