A Low Latency and Scalable Key Value Store from Modern off the Shelf Components

The current state of off the shelf software defined storage systems and high performance hardware is enabling simplified high performance systems that used to be complex and difficult to scale. By combining simple mainstream technologies like object storage systems with NVMeoF storage hardware it is possibly to implement a very low cost, very low latency, and very scalable key value store. The benefits include ease of deployment and management, use of open source software, up to date OS integration, and commodity component costs. The awareness of common use cases and past development efforts have resulted in the development of simpler and more attainable solutions for performance and scale using these components. I have built a demonstration based on MinIO and NVMeoF/TCP that has comparable latency performance to in-memory key value stores with the cost and scale associated with disaggregated flash storage systems.

Learning Objectives

Ability to deploy simple software and hardware systems for complex use cases at scale,Performance comparison of existing special purpose key value systems to a key value system made up of off the shelf components,Use of open source storage systems