The SNIA Cloud Storage Technologies (CST) community hosted a live webinar, “Building Community to Tackle Cloud Object Storage Incompatibilities,” where we highlighted how a multi-vendor group of industry innovators came together to address incompatibility issues between various object storage implementations that challenge many organizations.  The webinar, along with presentation slides, are available on-demand here.   

Webinar panelists included engineers from Dell, Google, Hammerspace, IBM, Vast, and Versity Software, all of whom shared their Plugfest experiences, and the audience asked several intriguing questions.  This is a compilation of responses from our guest speakers; thanks to all for their insights and contributions. 

Q. Will you have more SNIA Cloud Object Storage (COS) Plugfests in 2025? 

A. Yes, to participate in the next SNIA COS Plugfest, April 28-30, 2025, (co-located with One Day Regional SDC in Denver CO on April 30th), please pre-register at SNIA Cloud Object Storage Plugfest.  Our second SNIA COS Plugfest for 2025 will be hosted during SDC’25, Sept 15-17, 2025, in Santa Clara, CA.  If you have questions, please contact us at: askcloudplugfest@snia.org.

Note: A Plugfest is a collaborative developer event where industry experts come together, test their cloud object storage solutions, find problems, and fix them. SNIA takes care of the space and tools needed, and everyone agrees to keep issues resolved secret under NDA.

Q. Please provide examples of how end-user customers may be impacted due to unsupported API calls and unexpected behavior? 

A. There are many different ways interoperability issues are introduced that can impact user experience, such as introducing new features or functionality. Third-party object storage server implementations are always a bit behind on the latest changes, so from an application developers’ perspective, it's not always obvious what features are new. Unfortunately, with new changes, even basic functionality can break, like CRUDs (Create, Read, Update, and Delete) or other essentials such as request signing. 

An example was a feature which introduced new methods for validating data integrity with support for additional checksum algorithms. By changing behavior of SDK clients, previous stable workflows halted, with clients reporting 500 errors.  This was a more friendly example, because the breakage was fairly obvious. 

We've also seen this kind of thing happen when a third-party implementation ignores unrecognized HTTP request headers, which has been the default behavior for unrecognized HTTP headers since HTTP 1.0 (1996). One more non-obvious example of how introduction of new features (along with version skew) may impact customers is the introduction of headers that implement conditional writes, which are designed to prevent modification or deletion of objects if they do or do not exist. As an example, if a client were to enable a newly introduced conditional write header, which was not previously implemented by a third-party implementation, users could potentially experience unexpected behavior. 

This points to the need for more collaboration, where developers on both sides of the wire (end-to-end) ensure successful introduction of features, as well as explore how flexible their applications are to configure around feature changes or explore how to bake some of that flexibility into their application.  This is why it is critical to engage all developer stakeholders in open collaborative developer discussions. 

Q. Is there anything that end users should be looking out for?

A. Yes. This is why API versioning is so important when you encounter things like new headers and new checksums. Ideally, you should be able to look at the version of any widely used protocol and identify whether or not it’s expected to have that feature, or not. This has historically been a fundamental expectation of user-friendly protocols – they should mandate certain features and certain versions of the API to ensure basic functionality remains consistent and stable. It’s one method that helps address unexpected behavior while building consensus around interoperability.   

However, the current state of development allows different vendors to choose various API options in their respective implementations (these may not always match or overlap sufficiently to attain proper compatibility support). This allows, or even creates, situations where each implementation may work well within its own limited ecosystem, yet unexpected results may occur when configuring a variety of implementations within a new configuration or end-to-end solution. 

End users expect that everything should just work. They may not even be aware of the backing storage implementation or vendor. This is the primary motivation why organizations are eager to participate in the SNIA Cloud Object Storage Plugfest, not only to proactively broaden the set of their compatibility testing (to find bugs before customers find them) but also to gain agreement on industry best practices. 

Note: up until the formation of this SNIA Cloud Object Storage Plugfest community, there had been no vendor-neutral community available to help define an accepted set of protocol options. Until now, it had been up to separate developer teams to choose independently, and that’s exactly the issue we are resolving. We are organizing a method to help developer teams get on the same page at the same time.   

Q. My organization mandates seamless data portability between cloud instances. What do I need to do to make sure I am not locked into one solution?   

A. The goal of the SNIA Cloud Object Storage Plugfest is to bring the multi-vendor community together to help organizations ensure data portability. Our team recommends customers and end users encourage their cloud object storage providers to take part in this SNIA Cloud Object Storage Plugfest effort, to make sure products and services have gone through rigorous interoperability efforts and the hard work of testing against each other. 

Q. Amazon and Microsoft were notably missing in this webinar, do they plan to join this effort in the future? 

A. Microsoft actively participated in the inaugural SNIA Cloud Object Storage Plugfest in September at SDC’24. In fact, Microsoft brought a team including protocol experts who contributed to the Plugfest bug assessment and the Birds of a Feather (BOF) session, where we gathered insight to formulate next steps for the industry.  At the beginning of the Plugfest, Microsoft and Plugfest contributors quickly compared notes to identify trouble spots. Most everyone in the room knew right away what to look out for. There was a quick consensus on where to test, and the team invested time where needed. Microsoft was an integral part of this. 

AWS has been invited. We are continuing to welcome and encourage them to join the SNIA Cloud Object Storage Plugfest discussion.

Q. Ultimately, doesn’t AWS dictate S3?  If a consortium of vendors represented here added extensions to the API, or new self-describing API, wouldn’t this break things assuming AWS is the genesis of S3 API going forward? 

A. Yes, AWS S3 is a proprietary API for Cloud Object Storage, and any non-AWS implementation of S3 must conform with AWS S3 API and SDKs in order to interoperate. 

A similar situation existed in the late ‘90s, with Microsoft and the SMB (CIFS) protocol. At that time, Microsoft owned the client, server, SMB protocol, and dictated everything that happened.  Yet, an evolution occurred when Microsoft worked with SMB developers, including both proprietary and open source implementations. In 2008, Microsoft sponsored the SNIA CIFS/SMB Plugfest organized / hosted by SNIA as a trusted third-party storage industry association.  This event is still ongoing (now called SNIA SMB Interoperability Lab or “IOL”) and has been proven as a model for timely information exchange among Microsoft and the global ecosystem of SMB developers.  Note, Microsoft continues to own their IP, drives the pace and direction of its SMB protocol roadmap, including changes and release timeframes. 

We anticipate a similar working arrangement may be possible between SNIA and AWS (and others) as we expand community participation focused on multi-vendor, heterogeneous interoperability.   

Thanks to all the SNIA Cloud Object Storage Plugfest team for your time, effort and insights, 

Michael Hoard, SNIA, Chair for Cloud Storage Technologies (CST) community