The most recent installment of our SNIA ESF webcast series “Everything You Wanted To Know About Storage But Were Too Proud To Ask” took on a broad topic – storage management. Our experts, Richelle Ahlvers, Mark Rogov and Alex McDonald did a great job explaining the basics and have now answered the questions that attendees asked here in this blog. If you missed the live webcast, check it out on-demand and download a copy of the slides if you’d like to take notes.

Q: What is the difference between storage and a database? Could a database be considered storage?

A: The short answer is no. The long answer relies on the fact that a database doesn’t just store data: it modifies the data to fit into its schema (table, index, etc.) A storage solution doesn’t mutate the data in any shape—the data is always preserved as is.

Q: Doesn’t provisioning a storage array mean setting it up?

A: Within the storage community, provisioning is akin to serving a cake at a party. To provision storage to a server means cutting a slice of usable capacity and allocating it to a very specific server. The record of the particular pairing is carefully recorded.

Q: Does deduplication fall into Configuration? Even when it is done only on cold data?

A: Great question! Deduplication is one of the services that a storage array may offer, therefore enabling it is configuring such service. To further clarify your question, the point of deduplication is irrelevant: it may happen with cold data (the data that is stored on the array but applications haven’t accessed it in a long time); it may happen to hot or in-flight data (frequently accessed data or data inside cache).

Q. Do Hyperscale vendors (like AWS) use any of the storage management?

A. Hyperscale vendors, like all consumers of storage, use storage management to configure their storage. They use a combination of vendor device tools and custom development scripts/tools, but are not heavy consumers of industry standard storage interfaces today. Swordfish’s RESTful interface will provide an easy-to-consume API for hyperscale vendors to integrate into their management ecosystem as vendors start delivering Swordfish-based solutions.

Q. It was mentioned that there was a ‘steep learning curve’ for previous SNIA storage management model. Any idea how much easier this is to learn?

A. One of the major advantages for Swordfish is that the RESTful API’s are standardized and can take advantage of readily available tools and infrastructure. With the JSON-based payload, you can use standard plug-ins for browsers, as well as Python scripting languages to immediately interact with the Swordfish API’s. This is a distinct difference from the SMI-S API’s, which although they are also XML-based APIs, required custom or third-party tools to interact with the SMI-S providers.

Q. You talked about how Swordfish is being designed as more user and client centric.  How are you doing this?   

A. We are starting with very specific use cases and scenarios  (rather than looking at “what is all the functionality we could expose”) as we build both the structure of the API and the amount of information returned.   We’ve also documented a lot of the basic use cases, and who might like to do them, in a user’s guide, and published that alongside the Swordfish specification.  You can find links to this at the SNIA Swordfish page: snia.org/swordfish

Q. You weren’t specific on storage management tools, and I was expecting more detail. I’m wondering why you did this at such a high level, as this really hasn’t helped me.

A. We were primarily referring to ITIL –(The Information Technology Infrastructure Library). It’s a framework designed to standardize the selection, planning, delivery and support of IT services to a business. Learn more here.

Q. While most of the products today support SMI-S, it’s not something that DevOps or Storage Admins use directly.  How, or is, Swordfish going to be different?

A. There are two primary ways we see the Swordfish API being much more accessible directly to the individual admins.  First, as a RESTful interface, it is very easy to access and traverse with the tools that they use daily – from web browsers directly, to REST plugins, to simple (or complex) python scripts.  The learning curve to start interacting with Swordfish is extremely small.  You can get a sense by going to an online “mockup” site here:  http://swordfishmockups.com – there are some simple instructions to either browse the system directly or some standard clients to make it easier.  That will give you an idea of how easy it will be to start interacting with Swordfish (plus security for a real system, of course).

Remember the “Everything You Wanted To Know About Storage But Were Too Proud To Ask” is a series. We’ve covered 8 storage topics to date and have a library of on-demand webcasts you can watch at your convenience. Happy viewing!

The most recent installment of our SNIA ESF webcast series "Everything You Wanted To Know About Storage But Were Too Proud To Ask" took on a broad topic – storage management. Our experts, Richelle Ahlvers, Mark Rogov and Alex McDonald did a great job explaining the basics and have now answered the questions that attendees asked here in this blog. If you missed the live webcast, check it out on-demand and download a copy of the slides if you'd like to take notes. Q: What is the difference between storage and a database? Could a database be considered storage? A: The short answer is no. The long answer relies on the fact that a database doesn't just store data: it modifies the data to fit into its schema (table, index, etc.) A storage solution doesn't mutate the data in any shape—the data is always preserved as is. Q: Doesn't provisioning a storage array mean setting it up? A: Within the storage community, provisioning is akin to serving a cake  at a party. To provision storage to a server means cutting a slice of usable capacity and allocating it to a very specific server. The record of the particular pairing is carefully recorded. Q: Does deduplication fall into Configuration? Even when it is done only on cold data? A: Great question! Deduplication is one of the services that a storage array may offer, therefore enabling it is configuring such service. To further clarify your question, the point of deduplication is irrelevant: it may happen with cold data (the data that is stored on the array but applications haven't accessed it in a long time); it may happen to hot or in-flight data (frequently accessed data or data inside cache). Q. Do Hyperscale vendors (like AWS) use any of the storage management? A. Hyperscale vendors, like all consumers of storage, use storage management to configure their storage. They use a combination of vendor device tools and custom development scripts/tools, but are not heavy consumers of industry standard storage interfaces today. Swordfish's RESTful interface will provide an easy-to-consume API for hyperscale vendors to integrate into their management ecosystem as vendors start delivering Swordfish-based solutions. Q. It was mentioned that there was a 'steep learning curve' for previous SNIA storage management  model. Any idea how much easier this is to learn? A. One of the major advantages for Swordfish is that the RESTful API's are standardized and can take advantage of readily available tools and infrastructure. With the JSON-based payload, you can use standard plug-ins for browsers, as well as Python scripting languages to immediately interact with the Swordfish API's. This is a distinct difference from the SMI-S API's, which although they are also XML-based APIs, required custom or third-party tools to interact with the SMI-S providers. Q. You talked about how Swordfish is being designed as more user and client centric.   How are you doing this?     A. We are starting with very specific use cases and scenarios  (rather than looking at "what is all the functionality we could expose") as we build both the structure of the API and the amount of information returned.     We've also documented a lot of the basic use cases, and who might like to do them, in a user's guide, and published that alongside the Swordfish specification.   You can find links to this at the SNIA Swordfish page:  snia.org/swordfish Q. You weren't specific on storage management tools, and I was expecting more detail. I'm wondering why you did this at such a high level, as this really hasn't helped me. A. We were primarily referring to ITIL –(The Information Technology Infrastructure Library). It's a framework designed to standardize the selection, planning, delivery and support of IT services to a business.  Learn more here. Q. While most of the products today support SMI-S, it's not something that DevOps or Storage Admins use directly.   How, or is, Swordfish going to be different? A. There are two primary ways we see the Swordfish API being much more accessible directly to the individual admins.   First, as a RESTful interface, it is very easy to access and traverse with the tools that they use daily – from web browsers directly, to REST plugins, to simple (or complex) python scripts.   The learning curve to start interacting with Swordfish is extremely small.   You can get a sense by going to an online "mockup" site here:   http://swordfishmockups.com  – there are some simple instructions to either browse the system directly or some standard clients to make it easier.   That will give you an idea of how easy it will be to start interacting with Swordfish (plus security for a real system, of course). Remember the "Everything You Wanted To Know About Storage But Were Too Proud To Ask" is a series. We've covered 8 storage topics to date and have a library of on-demand webcasts you can watch at your convenience. Happy viewing!

The most recent installment of our SNIA ESF webcast series “Everything You Wanted To Know About Storage But Were Too Proud To Ask” took on a broad topic – storage management. Our experts, Richelle Ahlvers, Mark Rogov and Alex McDonald did a great job explaining the basics and have now answered the questions that attendees asked here in this blog. If you missed the live webcast, check it out on-demand and download a copy of the slides if you’d like to take notes. Q: What is the difference between storage and a database? Could a database be considered storage? A: The short answer is no. The long answer relies on the fact that a database doesn’t just store data: it modifies the data to fit into its schema (table, index, etc.) A storage solution doesn’t mutate the data in any shape—the data is always preserved as is. Q: Doesn’t provisioning a storage array mean setting it up? A: Within the storage community, provisioning is akin to serving a cake at a party. To provision storage to a server means cutting a slice of usable capacity and allocating it to a very specific server. The record of the particular pairing is carefully recorded. Q: Does deduplication fall into Configuration? Even when it is done only on cold data? A: Great question! Deduplication is one of the services that a storage array may offer, therefore enabling it is configuring such service. To further clarify your question, the point of deduplication is irrelevant: it may happen with cold data (the data that is stored on the array but applications haven’t accessed it in a long time); it may happen to hot or in-flight data (frequently accessed data or data inside cache). Q. Do Hyperscale vendors (like AWS) use any of the storage management? A. Hyperscale vendors, like all consumers of storage, use storage management to configure their storage. They use a combination of vendor device tools and custom development scripts/tools, but are not heavy consumers of industry standard storage interfaces today. Swordfish’s RESTful interface will provide an easy-to-consume API for hyperscale vendors to integrate into their management ecosystem as vendors start delivering Swordfish-based solutions. Q. It was mentioned that there was a ‘steep learning curve’ for previous SNIA storage management model. Any idea how much easier this is to learn? A. One of the major advantages for Swordfish is that the RESTful API’s are standardized and can take advantage of readily available tools and infrastructure. With the JSON-based payload, you can use standard plug-ins for browsers, as well as Python scripting languages to immediately interact with the Swordfish API’s. This is a distinct difference from the SMI-S API’s, which although they are also XML-based APIs, required custom or third-party tools to interact with the SMI-S providers. Q. You talked about how Swordfish is being designed as more user and client centric.  How are you doing this?    A. We are starting with very specific use cases and scenarios  (rather than looking at “what is all the functionality we could expose”) as we build both the structure of the API and the amount of information returned.   We’ve also documented a lot of the basic use cases, and who might like to do them, in a user’s guide, and published that alongside the Swordfish specification.  You can find links to this at the SNIA Swordfish page: snia.org/swordfish Q. You weren’t specific on storage management tools, and I was expecting more detail. I’m wondering why you did this at such a high level, as this really hasn’t helped me. A. We were primarily referring to ITIL –(The Information Technology Infrastructure Library). It’s a framework designed to standardize the selection, planning, delivery and support of IT services to a business. Learn more here. Q. While most of the products today support SMI-S, it’s not something that DevOps or Storage Admins use directly.  How, or is, Swordfish going to be different? A. There are two primary ways we see the Swordfish API being much more accessible directly to the individual admins.  First, as a RESTful interface, it is very easy to access and traverse with the tools that they use daily – from web browsers directly, to REST plugins, to simple (or complex) python scripts.  The learning curve to start interacting with Swordfish is extremely small.  You can get a sense by going to an online “mockup” site here:  http://swordfishmockups.com – there are some simple instructions to either browse the system directly or some standard clients to make it easier.  That will give you an idea of how easy it will be to start interacting with Swordfish (plus security for a real system, of course). Remember the “Everything You Wanted To Know About Storage But Were Too Proud To Ask” is a series. We’ve covered 8 storage topics to date and have a library of on-demand webcasts you can watch at your convenience. Happy viewing!

We're all accustomed to transferring money from one bank account to another; a credit to the payer becomes a debit to the payee. But that model uses a specific set of sophisticated techniques to accomplish what appears to be a simple transaction. Today, we're also well acquainted with ordering goods online, reserving an airline seat over the Internet, or simply updating a photograph on Facebook. Can these applications use the same banking models, or are new techniques required? It's a question we'll tackle at our next Ethernet Storage Forum webcast on October 31st "Transactional Models & Their Storage Requirements." One of the more important concepts in storage is the notion of  transactions,  which are used in databases, financials, and other mission critical workloads. However, in the age of cloud and distributed systems, we need to update our thinking about what constitutes a transaction. We need to understand how new theories and techniques allow us to undertake transactional work in the face of unreliable and physically dispersed systems. It's a topic full of interesting concepts (and lots of acronyms!). In this webcast, we'll provide a brief tour of traditional transactional systems and their use of storage, we'll explain new application techniques and transaction models, and we'll discuss what storage systems need to look like to support these new advances. And yes, we'll decode all the acronyms and nomenclature too. You will learn:

• A brief history of transactional systems from banking to Facebook
• How the Internet and distributed systems have changed and how we view transactions
• An explanation of the terminology, from ACID to CAP and beyond
• How applications, networks & particularly storage have changed to meet these demands

You may have noticed this webcast is on Halloween, October 31^st. We promise it will be a treat not a trick! I encourage you to register today.

We’re all accustomed to transferring money from one bank account to another; a credit to the payer becomes a debit to the payee. But that model uses a specific set of sophisticated techniques to accomplish what appears to be a simple transaction. Today, we’re also well acquainted with ordering goods online, reserving an airline seat over the Internet, or simply updating a photograph on Facebook. Can these applications use the same banking models, or are new techniques required? It’s a question we’ll tackle at our next Ethernet Storage Forum webcast on October 31st “Transactional Models & Their Storage Requirements.” One of the more important concepts in storage is the notion of transactions, which are used in databases, financials, and other mission critical workloads. However, in the age of cloud and distributed systems, we need to update our thinking about what constitutes a transaction. We need to understand how new theories and techniques allow us to undertake transactional work in the face of unreliable and physically dispersed systems. It’s a topic full of interesting concepts (and lots of acronyms!). In this webcast, we’ll provide a brief tour of traditional transactional systems and their use of storage, we’ll explain new application techniques and transaction models, and we’ll discuss what storage systems need to look like to support these new advances. And yes, we’ll decode all the acronyms and nomenclature too. You will learn:

• A brief history of transactional systems from banking to Facebook
• How the Internet and distributed systems have changed and how we view transactions
• An explanation of the terminology, from ACID to CAP and beyond
• How applications, networks & particularly storage have changed to meet these demands

You may have noticed this webcast is on Halloween, October 31^st. We promise it will be a treat not a trick! I encourage you to register today.

Storage can be something of a “black box,” a monolithic entity that is at once mysterious and scary. That’s why we created “The Everything You Wanted To Know About Storage But Were Too Proud to Ask” webcast series. So far, we’ve explored various and sundry aspects of storage, focusing on “the naming of the parts.” Our goal has been to break down some of the components of storage and explain how they fit into the greater whole. On September 28^th, we’ll be hosting “Everything You Wanted To Know About Storage But Were Too Proud To Ask – Part Cyan – Storage Management.” This time, we’re going to open up Pandora’s Box and peer inside the world of storage management, uncovering some of the key technologies that are used to manage devices, storage traffic, and storage architectures. In particular, we’ll be discussing:

• SNMP – The granddaddy of management protocols
• SMI-S – The bread-and-butter of vendor-neutral storage management
• SNIA Swordfish – The new storage management solution gaining widespread momentum
• Software-Defined Storage – The catch-all term for storage that includes architectures and management

There’s so much to say on each of these subject. In fact, we could do a full webcast on any one of them, but for a quick overview of many of the technologies that affect storage in one place, we think you will find your time has been well spent. As always, we’ve assembled a great panel of experts to discuss these topics. So I hope you will be join us on September 28^th, 2017, for our continuation of the “Too Proud To Ask” series with Cyan, the Storage Management Pod. Register here. And if you’ve missed any of the other “Too Proud To Ask” webcasts, they are all available on-demand. Happy viewing!

In Part Vermillion of our SNIA Ethernet Storage Forum (ESF) “Everything You Wanted To Know About Storage But Were Too Proud To Ask” webcast series – we examined the terms and concepts are at the heart of where compute, networking and storage intersect. That’s why we called it “What if Programming and Networking Had a Storage Baby” If you missed the live webcast, you can watch it on-demand. The discussion from our panel of experts generated a lot of good questions. As promised, here are answers to them all.  Q. With regard to persistent memory, how does one decide if it’s better to use load/store or access via I/O? A. Legacy applications will not change and hence will access the persistent memory the way they were written. If your legacy application needs a lot of memory and you want to use the new persistent memory as just a big and cheap (volatile) memory, then the access will be byte addressable (load/store). If your legacy application uses block storage then it will use the persistent memory using block addressing. New applications can take advantage of using byte addressing and persistency. They can keep all their data structures in memory, and these data structures will also be persistent! This saves applications the need to serialize before storing and to de-serialize when retrieving from storage and enables many other ways to optimize the software. Q. Can you go over again a bit more slowly how byte accessible and LBA change with persistent memory? A. Persistent memory can be accessed in three different ways.

1. Using byte addressing, in which case it behaves like a big (volatile) memory
2. Using logical block addressing, in which case it behaves like a block storage device
3. Using SNIA NVM Programming Model that enable byte addressing along with persistency. In this case byte being written into the device can be made persistent with special APIs

You can configure and decide what model is better use for your application. Q. Is that like flash? A. Persistent memory is a technology that is persistent like flash, but has byte addressing. It can be implemented using underlying flash, battery backed DRAM, Phase Change Memory and more. Q. You were going to parse out flash vs. NVMe, I think. Also, how will the elements discussed during the session impact these evolving technologies? A. Flash is a non-volatile memory technology that supports block addressing. PCM is another non-volatile technology which is newer that supports byte addressing (which implies that it can also do block addressing by emulation). NVMe describes an interface to access non-volatile memory technology, by placing the non-volatile memory over the PCI bus. Storage Class Memory is yet another interface to access non-volatile memory, by placing the non-volatile memory over the memory bus. With this in mind: 1) It is common to see NVMe devices with backing flash devices. They will support block addressable. They have the option to expose a small byte addressable memory buffer as well on the PCI (typically a DRAM), which may or may not be volatile. 2) It is common to see Storage Class Memory with backing PCM device, or with DRAM (that can backup itself to flash on power failure). They will support byte addressable. Q. Regarding SMB & CIFS protocols, is SMB or CIFS the deprecated one? A. The name CIFS hasn’t been used in a while; it’s now all SMB. SMB version1 is deprecated; see this Microsoft article. Don’t use CIFS! Q. Are there any rules of thumb in regards to the efficiency of block vs. file vs. object stores from the storage capacity overhead and network “busyness”? A. Effectively, as you get closer to the lower-level block storage devices, your storage networking architecture needs to become more deterministic. That is, you begin to start caring more about the number of hosts connecting to a particular storage target (fan-in ratio) and the ratio of bandwidth the target has compared to the bandwidth that the hosts connecting to it have (oversubscription). Highly-transactional block storage protocols, such as Fibre Channel, FCoE and lossless iSCSI, for example, will need to have very low oversubscription ratios (sometimes as low as 4:1, depending on the type of application/workload). Most are somewhat more forgiving, and 16:1 and 20:1 are not uncommon. When you move into file-based systems, that oversubscription can be a lot higher (there is no general rule of thumb here, but the oversubscription can be in the low hundreds:1). Object-based systems are so scaled and distributed, that there really are no oversubscription limits at all, because those systems are not highly transactional. Q. Does an object always have to be replaced in entirety? How do systems handle updates to large objects? A. The rule is that you shouldn’t take a lock on an object. Classically, the whole object should be replaced. Updating is not straightforward. Traditional “get/release” locking is too expensive in terms of latency over geographic distances, too hard to manage in a distributed environment, is hard to scale, needs recovery in the case of failure, and introduces state to what is basically storage built for stateless operations. Plus, the object may be sharded across multiple physical systems. Some object systems do allow what they call “pessimistic locking” (take a lock for a fixed period of time, say 10 seconds) but it’s not a true lock that you obtain then release. It’s more like a window of opportunity and is often called, and acts like, a lease. There are also other techniques, like “optimistic concurrency” (using a unique identifier, try and then check if your identifier was successful) and “last writer wins” (as it says, the last write is the one that the storage system remembers). Many systems do this by snapshotting the object, allowing updates on the copy, and then atomically swapping them. Object systems differ in what they permit. In general, applications need to be aware that they may, very occasionally, not be successful when modifying objects, and to have strategies to deal with it, like retrying or even simply giving up. Again, you can check out the recorded version of the webcast at your convenience and you can download the webcasts slides as well if you’d like to follow along. Remember, this webcast was part of series. I encourage you to register today for our next one, which will be on August 1^st at 10:00 am PT – Part Turquoise “Where Does My Data Go?” And please visit the SNIA ESF website for our full library of ESF webcasts.  

The colorful “Everything You Wanted To Know About Storage But Were Too Proud To Ask,” popular webcast series marches on! In this 6^th installment, Part – Vermillion – What if Programming and Networking Had a Storage Baby, we look into some of the nitties and the gritties of storage details that are often assumed. When looking at data from the lens of an application, host, or operating system, it’s easy to forget that there are several layers of abstraction underneath each before the actual placement of data occurs. In this webcast we are going to scratch beyond the first layer to understand some of the basic taxonomies of these layers. In this webcast we will show you more about the following:

• Storage APIs and POSIX
• Block, File, and Object storage
• Byte Addressable and Logical Block Addressing
• Log Structures and Journaling Systems

It’s an ambitious project, but these terms and concepts are at the heart of where compute, networking and storage intersect. Having a good grasp of these concepts ties in with which type of storage networking to use, and how data is actually stored behind the scenes. Register today to join us on July 6th for this session. You can ask all the questions that, until now, you’ve been too proud to ask and we promise not to to show you any baby pictures!

New solid state storage technologies are forcing the industry to refine distinctions between networks and other types of system interconnects.   The question on everyone's mind is: when is it beneficial to use networks to access solid state storage, particularly persistent memory? It's not quite as simple as a "yes/no" answer. The answer to this question involves application, interconnect, memory technology and scalability factors that can be analyzed in the context of a latency budget. On April 19^th, Doug Voigt, Chair SNIA NVM Programming Model Technical Work Group, returns for a live SNIA Ethernet Storage Forum webcast, "Architectural Principles for Networked Solid State Storage Access - Part 2" where we will explore latency budgets for various types of solid state storage access.  These can be used to determine which combinations of interconnects, technologies and scales are compatible with Load/Store instruction access and which are better suited to IO completion techniques such as polling or blocking. In this webcast you'll learn:

• Why latency is important in accessing solid state storage
• How to determine the appropriate use of networking in the context of a latency budget
• Do's and don'ts for Load/Store access

This is a technical seminar built upon part 1 of this series. If you missed it, you can view it on demand at your convenience. It will give you a solid foundation on this topic, outlining key architectural principles that allow us to think about the application of  networked  solid state technologies more systematically. I hope you will register today for the April 19^th event. Doug and I will be on hand to answer questions on the spot. Update: If you missed the live event, it's now available  on-demand. You can also  download the webcast slides.

New solid state storage technologies are forcing the industry to refine distinctions between networks and other types of system interconnects.  The question on everyone’s mind is: when is it beneficial to use networks to access solid state storage, particularly persistent memory?

It’s not quite as simple as a “yes/no” answer. The answer to this question involves application, interconnect, memory technology and scalability factors that can be analyzed in the context of a latency budget.

On April 19^th, Doug Voigt, Chair SNIA NVM Programming Model Technical Work Group, returns for a live SNIA Ethernet Storage Forum webcast, “Architectural Principles for Networked Solid State Storage Access – Part 2” where we will explore latency budgets for various types of solid state storage access. These can be used to determine which combinations of interconnects, technologies and scales are compatible with Load/Store instruction access and which are better suited to IO completion techniques such as polling or blocking.

In this webcast you’ll learn:

• Why latency is important in accessing solid state storage
• How to determine the appropriate use of networking in the context of a latency budget
• Do’s and don’ts for Load/Store access

This is a technical seminar built upon part 1 of this series. If you missed it, you can view it on demand at your convenience. It will give you a solid foundation on this topic, outlining key architectural principles that allow us to think about the application of networked solid state technologies more systematically.

I hope you will register today for the April 19^th event. Doug and I will be on hand to answer questions on the spot.