Scaling an Index to the Exabytes | SNIA

Abstract

At the exabyte scale, even the metadata becomes so large that single systems are unable to manage or even store it. The only way to cope with trillions of objects and their associated trillions of pieces of metadata is to rethink the fundamentals of how metadata is stored, accessed, and manipulated in a massive system with potentially huge numbers of users. We detail our solution, and show how it enables true horizontal scalability, and with the surprising property that as the system grows, metadata contention actually decreases.

Learning Objectives

How the distributed object storage cluster operates at scale.
Why a rich object API is more useful than a traditional file API.
How cluster-coherent POSIX file access scales