The SNIA Cloud Storage Technologies (CST) community hosted a live webinar, “Building Community to Tackle Cloud Object Storage Incompatibilities,” where we highlighted how a multi-vendor group of industry innovators came together to address incompatibility issues between various object storage implementations that challenge many organizations.  The webinar, along with presentation slides, are available on-demand here.   

Webinar panelists included engineers from Dell, Google, Hammerspace, IBM, Vast, and Versity Software, all of whom shared their Plugfest experiences, and the audience asked several intriguing questions.  This is a compilation of responses from our guest speakers; thanks to all for their insights and contributions. 

Q. Will you have more SNIA Cloud Object Storage (COS) Plugfests in 2025? 

A. Yes, to participate in the next SNIA COS Plugfest, April 28-30, 2025, (co-located with One Day Regional SDC in Denver CO on April 30th), please pre-register at SNIA Cloud Object Storage Plugfest.  Our second SNIA COS Plugfest for 2025 will be hosted during SDC’25, Sept 15-17, 2025, in Santa Clara, CA.  If you have questions, please contact us at: askcloudplugfest@snia.org.

Note: A Plugfest is a collaborative developer event where industry experts come together, test their cloud object storage solutions, find problems, and fix them. SNIA takes care of the space and tools needed, and everyone agrees to keep issues resolved secret under NDA.

Q. Please provide examples of how end-user customers may be impacted due to unsupported API calls and unexpected behavior? 

A. There are many different ways interoperability issues are introduced that can impact user experience, such as introducing new features or functionality. Third-party object storage server implementations are always a bit behind on the latest changes, so from an application developers’ perspective, it's not always obvious what features are new. Unfortunately, with new changes, even basic functionality can break, like CRUDs (Create, Read, Update, and Delete) or other essentials such as request signing. 

An example was a feature which introduced new methods for validating data integrity with support for additional checksum algorithms. By changing behavior of SDK clients, previous stable workflows halted, with clients reporting 500 errors.  This was a more friendly example, because the breakage was fairly obvious. 

We've also seen this kind of thing happen when a third-party implementation ignores unrecognized HTTP request headers, which has been the default behavior for unrecognized HTTP headers since HTTP 1.0 (1996). One more non-obvious example of how introduction of new features (along with version skew) may impact customers is the introduction of headers that implement conditional writes, which are designed to prevent modification or deletion of objects if they do or do not exist. As an example, if a client were to enable a newly introduced conditional write header, which was not previously implemented by a third-party implementation, users could potentially experience unexpected behavior. 

This points to the need for more collaboration, where developers on both sides of the wire (end-to-end) ensure successful introduction of features, as well as explore how flexible their applications are to configure around feature changes or explore how to bake some of that flexibility into their application.  This is why it is critical to engage all developer stakeholders in open collaborative developer discussions. 

Q. Is there anything that end users should be looking out for?

A. Yes. This is why API versioning is so important when you encounter things like new headers and new checksums. Ideally, you should be able to look at the version of any widely used protocol and identify whether or not it’s expected to have that feature, or not. This has historically been a fundamental expectation of user-friendly protocols – they should mandate certain features and certain versions of the API to ensure basic functionality remains consistent and stable. It’s one method that helps address unexpected behavior while building consensus around interoperability.   

However, the current state of development allows different vendors to choose various API options in their respective implementations (these may not always match or overlap sufficiently to attain proper compatibility support). This allows, or even creates, situations where each implementation may work well within its own limited ecosystem, yet unexpected results may occur when configuring a variety of implementations within a new configuration or end-to-end solution. 

End users expect that everything should just work. They may not even be aware of the backing storage implementation or vendor. This is the primary motivation why organizations are eager to participate in the SNIA Cloud Object Storage Plugfest, not only to proactively broaden the set of their compatibility testing (to find bugs before customers find them) but also to gain agreement on industry best practices. 

Note: up until the formation of this SNIA Cloud Object Storage Plugfest community, there had been no vendor-neutral community available to help define an accepted set of protocol options. Until now, it had been up to separate developer teams to choose independently, and that’s exactly the issue we are resolving. We are organizing a method to help developer teams get on the same page at the same time.   

Q. My organization mandates seamless data portability between cloud instances. What do I need to do to make sure I am not locked into one solution?   

A. The goal of the SNIA Cloud Object Storage Plugfest is to bring the multi-vendor community together to help organizations ensure data portability. Our team recommends customers and end users encourage their cloud object storage providers to take part in this SNIA Cloud Object Storage Plugfest effort, to make sure products and services have gone through rigorous interoperability efforts and the hard work of testing against each other. 

Q. Amazon and Microsoft were notably missing in this webinar, do they plan to join this effort in the future? 

A. Microsoft actively participated in the inaugural SNIA Cloud Object Storage Plugfest in September at SDC’24. In fact, Microsoft brought a team including protocol experts who contributed to the Plugfest bug assessment and the Birds of a Feather (BOF) session, where we gathered insight to formulate next steps for the industry.  At the beginning of the Plugfest, Microsoft and Plugfest contributors quickly compared notes to identify trouble spots. Most everyone in the room knew right away what to look out for. There was a quick consensus on where to test, and the team invested time where needed. Microsoft was an integral part of this. 

AWS has been invited. We are continuing to welcome and encourage them to join the SNIA Cloud Object Storage Plugfest discussion.

Q. Ultimately, doesn’t AWS dictate S3?  If a consortium of vendors represented here added extensions to the API, or new self-describing API, wouldn’t this break things assuming AWS is the genesis of S3 API going forward? 

A. Yes, AWS S3 is a proprietary API for Cloud Object Storage, and any non-AWS implementation of S3 must conform with AWS S3 API and SDKs in order to interoperate. 

A similar situation existed in the late ‘90s, with Microsoft and the SMB (CIFS) protocol. At that time, Microsoft owned the client, server, SMB protocol, and dictated everything that happened.  Yet, an evolution occurred when Microsoft worked with SMB developers, including both proprietary and open source implementations. In 2008, Microsoft sponsored the SNIA CIFS/SMB Plugfest organized / hosted by SNIA as a trusted third-party storage industry association.  This event is still ongoing (now called SNIA SMB Interoperability Lab or “IOL”) and has been proven as a model for timely information exchange among Microsoft and the global ecosystem of SMB developers.  Note, Microsoft continues to own their IP, drives the pace and direction of its SMB protocol roadmap, including changes and release timeframes. 

We anticipate a similar working arrangement may be possible between SNIA and AWS (and others) as we expand community participation focused on multi-vendor, heterogeneous interoperability.   

Thanks to all the SNIA Cloud Object Storage Plugfest team for your time, effort and insights, 

Michael Hoard, SNIA, Chair for Cloud Storage Technologies (CST) community

The SNIA Cloud Storage Initiative hosted a live Webcast “Cloud Object Storage 101.” Like any “101” type course, there were a lot of good questions. Here they all are – with our answers. If you have additional questions, please let us know by commenting on this blog.

Q. How do you envision the new role of tape (LTO) in this unstructured data growth?

A. Exactly the same way that tape has always played a part; it’s the storage medium that requires no power to store cold data and is cheap per bit. Although it has a limited shelf life, and although we believe that flash will eventually replace it, it still has a secure & growing foreseeable future.

Q. What are your thoughts on whether object storage can exist outside the bounds of supporting file systems? Block devices directly storing objects using the key as reference and removing the intervening file system? A hierarchy of objects instead of files?

A. All of these things. Objects can be objects identified by an ID in a flat non-hierarchical structure; or we can impose a hierarchy by key- to objectID translation; or indeed, an object may contain complete file systems or be treated like a block device. There are really no restrictions on how we can build meta data that describes all these things over the bytes of storage that makes up an object.

Q. Can you run write insensitive low latency apps on object storage, ex: virtual machines?

A. Yes. Object storage can be made up of the same stuff as other high performance storage systems; for instance, flash connect via high bandwidth and low latency networks. Or they could even be object stores built over PCIe and NVDIMM.

Q. Is erasure coding (EC) expensive in terms of networking and resources utilization (especially in case of rebuild)?

A. No, that’s one of the advantages of EC. Rebuilds take place by reading data from many disks and writing it to many disks; in traditional RAID rebuilds, the focus is normally on the one disk that’s being rebuilt.

Q. Is there any overhead for small files or object use cases? Do you have a recommended size?

A. Each system will have its own advantages and disadvantages for objects of specific sizes. In general, object stores are designed to store billions of objects, so the number of objects is usually not an issue.

Q. Can you comment on Internet bandwidth limitations on geographically dispersed erasure coded data?

A. Smart caching can make a big difference, but at the end of the day, a geographically EC dispersed object store won’t be faster than a local store. You can’t beat the speed of light.

Q. The suppliers all claim easy exit strategies from their systems. If we were to use one of the on-premise solutions such as ECS or Cleversafe, and then down the road decide to move off-premise, is the migration/egress typically as easy as claimed?

A. In general, any proprietary interface might lock you in. The SNIA’s CDMI is vendor neutral, and supported by a number of vendors. Amazon’s S3 is a popular and common interface. Ultimately, vendors want your data on their systems – and that means making it easy to get the data from a competing vendor’s system; lock-in is not what vendors want. Talk to your vendor and ask for other users’ experiences to get confirmation of their claims.

Q. Based on factual information, where are you seeing the most common use cases for Object Storage?

A. There are many, and each vendor of cloud storage has particular markets. Backup is a common case, as are systems in the healthcare space that treat data such as scans and X-rays as objects.

Q. NAS filers only scale up not out. They are hard to manage at scale. Why use them anymore?

A. There are many NAS systems that scale out as well as up. NFSv4 support high degrees of scale out and there are file systems like Gluster that provide very large-scale solutions indeed, into the multi-petabyte range.

Q. Are there any specific uses cases to avoid when considering object storage?

A. Yes. Many legacy applications will not generate any savings or gains if moved to object storage.

Q. Would you agree with industry statements that 80% of all data written today will NEVER be accessed again; and that we just don’t know WHICH 20% will be read again?

A. Yes to the first part, and no to the second. Knowing which 80% is cold is the trick. The industry is developing smart ways of analyzing data to help with the issue of ensuring cached data is hot data, and that cold data is placed correctly first time around.

Q. Is there also the possibility to bring “compliance” in the object storage? (thinking about banking, medical and other sensible data that needs to be tracked, retention, etc…)

A. Yes. Many object storage vendors provide software to do this.

There has been a lot of buzz around cloud object storage recently. But before you get deep into all that cloud object storage can do, it’s good to take a step back and make sure you understand the basics. That’s what the SNIA Cloud Storage Initiative is planning to do on July 14^th at our live Webcast “Cloud Object Storage 101.”

Many organizations, like large service providers, have already begun to leverage software-defined object storage to support new application development and DevOps projects. Meanwhile, legacy enterprise companies are in the early stages of exploring the benefits of object storage for their particular business and are searching for how they can use cloud object storage to modernize their IT strategies, store and protect data, while dramatically reducing the costs associated with legacy storage sprawl.

This Webcast will highlight the market trends towards the adoption of object storage, the definition and benefits of object storage, and the use cases that are best suited to leverage an underlying object storage infrastructure.

Join us on July 14^th to learn:

• How to accelerate the transition from legacy storage to a cloud object architecture
• Understand the benefits of object storage
• Primary use cases
• How an object storage can enable your private, public or hybrid cloud strategy without compromising security, privacy or data governance

I hope you’ll register today to join my colleague, Nancy Bennis, Director of Alliances at Cleversafe (an IBM company), and me for this tutorial on cloud object storage.