Data Analysis Services Group - May 2013
News and Accomplishments
Scott Pearse has accepted an SEII position with the VAPOR team and will start work on Monday, June 10. We're are excited to begin working with Scott!
Mohammad Abouali, a SCIparCS summer intern from San Diego State, joined the VAPOR team on May 20.
VAPOR Project
Project information is available at: http://www.vapor.ucar.edu
XD Vis Award:
Planning resumed for VAPOR 3.0, a refactoring of the present code base that aims to address numerous current limitations. Development of 3.0 will proceed in parallel with the production 2.x code.
KISTI Award:
The KISTI award was finalized in May, with both NCAR and KISTI executing another year long agreement. The scope of this year's work will focus on performance improvements for windows platforms, and facilitating foreign data import into VAPOR.
Minsu Joh, the KISTI PI on the award, visited us for a week in May. While here we discussed year-three work. Minsu was recently promoted to the directory of Supercomputing Services at KISTI. Minsu visited numerous CISL staff. Thanks to Tom Engel, Dave Hart, Al Kellie, and Eric Nienhouse for spending time with Minsu.
Miles began work on a Windows port of the EasyThreads class object. Once complete various VAPOR utilities will execute thread-level parallelism on Windows platforms as is currently supported under Linux and Mac OSX.
Development:
Version 2.2.2 of VAPOR was released on May 29. This is primarily a bug-fix release for 2.2.0 that addresses numerous issues related to MOM, ROMS, and POP ocean data translation.
- We performed extensive testing of the new ocean data conversion routines and uncovered a number of bugs that were fixed
- The code was "purified" to check for memory errors.
- Alan ported the udunits2 package from Unidata to windows. Unidata only supports Linux and Mac, and we desired to have its units conversion capability for our 2.2.2 ROMS and MOM file reading.
- Due to deficiencies in the 32-bit NetCDF support on Windows, we found it necessary to develop a 64-bit version of VAPOR in the 2.2.2 release. This will enable our KISTI collaborators to read large MOM, POP, and ROMS models.
- The ROMS data converters, roms2vdf and romsvdfcreate, were re-architected and now make use of the new CF compliant data reader provided by the NetCDFCFCollection class object. Thus, the ROMS, MOM, and POP data converters all share a common code base. The new utilities were regression tested against numerous sample ROMS data sets provided to us by KISTI and other, local collaborators.
- Reference documentation was written for the new ocean data translators.
- Miles implemented a new utility, vdfcp, for copying selected subsets of a VDC data set from one location to another. Due to the potential substantial size of a VDC (10s or even 100's of terabytes), the ability to extract subsets (variables, time steps, or compression levels) is an important one.
- Miles performed regression testing and platform ports for the 2.2.2 release candidates. Thanks to Joey for providing us with Ubuntu and RH6 porting platforms.
Alan has been making improvements to our Drupal documentation on the website that will eventually enable VAPOR users to access context-sensitive documentation directly from the vaporgui application.
Administrative:
The SEII position on the VAPOR team that has been open since January was finally filled by Scott Pearse. Scott will be joining us on June 10. Scott recently completed his M.S. in C.S. at CU, while working with a meteorological team at NWRA in Boulder as a research associate.
Education and Outreach:
Mohammad Abouali, a Siparcs intern from San Diego State, joined us on May 20. Mohammad will be working with us as well as with Rick Brownrigg. His project is to enable visualization of WRF data in Google Earth. We have had discussions with Cindy Bruyere and Gabi Pfister to understand the requirements. Mohammad has already identified a plan for this implementation and is in the process of writing NCL code for the conversion.
John presented a poster on VAPOR at an R workshop at NCAR hosted by Doug Nychka.
Software Research Projects
Climate data compression:
With help from Ben Jamroz, John started working on compression utilities for CAM-SE data. The CAM-SE code uses HOMME's cube-sphere grid decomposition. The compression utilities will operate on each of the six cube faces separately. Once the data are compressed they will be compared with data compressed with other techniques used by John Dennis.
Systems Projects
Data Services
- Finished testing and configuration to activate four data-access nodes hosted in Linux containers on the gladedm servers. No users have
yet accessed these nodes. - Installed new X.509 host certificates for the GridFTP server hosts.
- Installed several Certificate Authority X.509 certificates for other sites for GridFTP and XSEDE automated monintoring use.
Security & Administration
- Updated artoms to backup files using tar that exceed htar's name length restrictions.
- Configured vapor1 into a RHEL6 test system for the VAPOR team.
- Removed old equipment from the racks in the ML computer room for excess.
- Started the reconfiguration of remaining systems in the ML computer room.
- Relocated the polynya backup server to another rack as part of DASG computer room cleanup efforts.
- Resumed work on the File Management Utility after a hiatus of 16 months with a review of the state where design efforts had previously reached. Will target the flexible parallelized backup functionality for the initial deployment to replace the clunky EMC Networker software.
- Had to reduce the retention period for /glade/u/home backups to 2 weeks due to resource constraints on the data deduplication server. Additional storage is to be installed in June to alleviate this constraint.
- Tested scripts to automate issuing DCS3700 CLI commands. Verified online update of controller firmware behaves correctly.
System Monitoring
- DASG continues to refine it's Nagios monitoring tools. These tools are used by both DASG and CASG for system monitoring and problem resolution.
- Continued monitoring GPFS to understand normal and abnormal operating conditions and how that is reflected in the monitoring tools output.
System Support
GLADE Client Support
- Helped VETS move their remaining systems to their cluster at NWSC.
- Provided CSG a script to show current license usage on yellowstone
GLADE Storage Cluster
- Restored old /glade/home user directories upon request for users who failed to migrate their files to the new GLADE directories before the old GLADE storage was shutdown.
Test Clusters
- Opened a case with DDN to fix the 9900 controllers not being able to boot.
- Started build of new filesystem test cluster. This cluster will support Lustre, GPFS, and OpenStack installation for evaluation and software builds.