Child pages
  • GlobusArchiver.py - Development Notes
Skip to end of metadata
Go to start of metadata

Overview

Definitions / Acronyms / Initialisms

  • CS - Campaign Store

Resources

Design

User Stories

Simple Copy

  • User wants to copy directories of data every day to the CS
    • from /ldm3_d2/grib/GFS002/%Y%m%d to /RAPDMG/grib/GFS002

Simple Tar

  • User wants to copy a directory of data into a tar file 
  • from /ldm1_d2/NLDN/DATEYYYYMMDD/ to /RAPDMG/LDM/ARCHIVE/DATEYYYY/DATEMMDD

  • put in tar file DATEYYYYMMDD.nldn.tar
  • include cdDirTar to put relative paths into the tar file

Globs in Source

  • User wants to copy several directories of data into a tar file
  • <source>/ldm1_d2/nids/raw/nids/*/BREF1/DATEYYYYMMDD</source>
    <destination>/RAPDMG/LDM/ARCHIVE/DATEYYYY/DATEMMDD</destination>
    <cdDirTar>/ldm1_d2/nids/raw/</cdDirTar>
    <expectedNumFiles>25000</expectedNumFiles>
    <expectedFileSize>245000000</expectedFileSize>
    <tarFilename>DATEYYYYMMDD_all.nids.tar</tarFilename>

  • Do we want to support this?  Maybe make user use python to build multiple entries into the archive items dictionary?

Multiple entries into the same tar file

  • User gives several archive items with the same destination tar file 
  • Items are added to the tar file before sending to the CS.


Requirements

Minimum Requirements

  • Support user stories - Simple Copy, Simple Tar, Multiple entries into the same tar file
  • Email user status of gat run

Stretch Goals

  • Support user stories - Globs in Source
  • Allow user to define expected file sizes, # of files, etc. and include results in email.
  • skip underscore files
  • set mode on uploads to CR
  • accept meta-data and make available for data discovery to other users.
  • zip files
  • encrypt files

Iterations

Proof of Functionality - Connect and transfer via globus

  • Uses globus-cli to connect to the CS and transfer a file.

Add ConfigMaster Support

  • Use ConfigMaster config file to define a set of archive items
  • Iterate over them to send to the CS

Support the "Simple Copy" user story


  • No labels