Marchive: Extending MarFS to a Long Term Archive

In response to the ever increasing bandwidth, capacity, and resiliency requirements of HPC data storage, Los Alamos National Laboratory developed MarFS, an open source filesystem providing a near-POSIX interface atop abstracted data and metadata storage implementations. For several years, the MarFS library has provided a high bandwidth, high resiliency storage tier for production data, known as Campaign Storage. The success of MarFS in this context, as well as the flexible nature of its underlying data and metadata storage, has spurred interest in extending the codebase to support the data archive needs of the Laboratory. Known as Marchive, this archive system will provide long term stability by storing parity protected data objects across magnetic tape media and expose a batch interface for efficient data ingest and retrieval. This presentation will review the concept of MarFS, describe the extension of that concept to form a Marchive system, and relate some of the more interesting solutions to have emerged from this effort.

Learning Objectives

Achieving familiarity with the core design concepts of MarFS,Understanding the goals and benefits of Marchive development,Grasping some of the nuanced aspects of the Marchive system structure