Improving Azure File Service: Adding New Wings to a Plane in Mid-flight

For the past three years Microsoft Azure has provided a completely managed SMB3 file server in the cloud. Leveraging the Continuous Availability features of SMB3, the customer experience is an always available and reliable file share. As we push to add the most demanded new features, the complexity and caution required to achieve this in a transparent and safe way presents fundamentally new kinds of challenges due to the scale of Azure’s public cloud.

As background, the talk will begin with the architecture of Azure Files, which is based on Azure tables and blobs under the hood, not a conventional file system – let alone NTFS. Specific attention will be paid to those aspects that provide the seamless availability and reliability in spite of the constant din of hardware underneath it suffering failures and needing replacement.

An overview of recently added feature will be used as a segue to delve into the engineering challenges of making significant changes and additions to data schemas and the code that manipulates it, while not disturbing access to those many petabytes of data, or breaking the semantics that applications depend on.

Learning Objectives:
1. Learn how an SMB sever can be built on top of something other than a conventional file system
2. Gain an appreciation of the complexities involved in durably committing what is usually considered volatile handle state, which must be both highly available and high performance
3. See how imperfect hardware can be used to provide (near) perfect availability and perfect durability
4. Understand how methodical plodding patience, mixed with some engineering sleight of hand, can achieve impressive architectural changes with zero downtime