MarFS: A Scalable Near-POSIX File System over Cloud Objects for HPC Cool Storage - SDC 2016

Many computing sites need long-term retention of mostly cold data often “data lakes”. The main function of this storage tier is capacity but non trivial bandwidth/access requirements exist. For many years, tape was the best economic solution. Data sets have grown larger more quickly than tape bandwidth improvements and access demands have increased in the HPC environment. Disk can be more economically for this storage tier. The Cloud Community has moved towards erasure based object stores to gain scalability and durability using commodity hardware. The Object Interface works for new applications but legacy applications utilize POSIX for their interface. MarFS is a Near-POSIX File System using cloud storage for data and many POSIX file systems for metadata. Extreme HPC environments require that MarFS scale a POSIX namespace metadata to trillions of files and billions of files in a single directory while storing the data in efficient massively parallel ways in industry standard erasure protected cloud style object stores.

Learning Objectives

Storage tiering in future HPC and large scale computing environments
Economic drivers for implementing data lakes/tiered storage
HPC specific requirements for data lakes - multi way scaling
Overview of existing data lake solution space
How the MarFS solution works and for what types of situations