Gary Grider

In this tutorial, we will introduce the audience to the lunatic fringe of extreme high-performance computing and its storage systems. The most difficult challenge in HPC storage is caused by millions (soon to be billions) of simultaneously writing threads. Although cloud providers handle workloads of comparable, or larger, aggregate scale, the HPC challenge is unique because the concurrent writers are modifying shared data. We will begin with a brief history of HPC computing covering the previous few decades, bringing us into the petaflop era which started in 2009.

Many computing sites need long-term retention of mostly-cold data, often referred to as “data lakes”. The main function of this storage-tier is capacity, but non-trivial bandwidth/access requirements may also exist. For many years, tape was the most economical solution. However, data sets have grown larger more quickly than tape bandwidth has improved, such that disk is now becoming more economically feasible for this storage-tier.

Many computing sites need long-term retention of mostly cold data often “data lakes”. The main function of this storage tier is capacity but non trivial bandwidth/access requirements exist. For many years, tape was the best economic solution. Data sets have grown larger more quickly than tape bandwidth improvements and access demands have increased in the HPC environment. Disk can be more economically for this storage tier. The Cloud Community has moved towards erasure based object stores to gain scalability and durability using commodity hardware.

Many computing sites need long-term retention of mostly cold data often “data lakes”. The main function of this storage tier is capacity but non trivial bandwidth/access requirements exist. For many years, tape was the best economic solution. Data sets have grown larger more quickly than tape bandwidth improvements and access demands have increased. Disk can be more economically for this storage tier. The Cloud Community has moved towards erasure based object stores to gain scalability and durability using commodity hardware.

This talk will provide an overview of an extreme high end simulation computing site focusing on the storage challenges of yesterday, today, and tomorrow.

N/A

Scientific data is mostly stored in linear bytes in files but it almost always has hidden structure that resembles records with keys and  values, often times in multiple dimensions.  Further, the bandwidths required to service HPC simulation workloads will soon approach tens of terabytes/sec with single data files surpassing a petabyte and single sets of data from a campaign approaching 100 petabytes.  Multiple tasks from distributed analytical/indexing functions to data management tasks like compression, erasure, encoding, dedup, are all potentially more efficiently and econ

Los Alamos is working to revolutionize how scientific data management is done, moving from large Petabyte sized files generated periodically by extreme scale simulations to a record and column based approach. Along the journey, the NVME Computational Storage efforts became a strategic way to help accomplish this revolution. Los Alamos has been working on a series of proof of concepts with a set of data storage industry partners and these partnerships have proven to be the key to success.

High-performance computing data centers supporting large-scale simulation applications can routinely generate a large amount of data. To minimize time-to-result, it is crucial that this data be promptly absorbed, processed, and potentially even multidimensionally indexed so that it can be efficiently retrieved when the scientists need it for insights.

Gary Grider will kick off the talk with a brief background of the challenges currently faced in mass storage systems, and why it is so difficult for modern S3 based workloads to utilize them. Ben McClelland, who architected the system and did the bulk of the development, will introduce the Versity Gateway, an open-source high-performance object-to-file translation tool. He will explain that the Versity Gateway was written from scratch in Go, highlighting the benefits of this choice.