Author:

Marty Foltyn

Company : Channel Chargers, LLC

Title : Vice President

 
 
author

Calling All Real-World Workloads

Marty Foltyn

Jun 13, 2019

title of post
Video streaming is an easy-to-understand workload from the I/O perspective, right?  It’s pretty obvious that it’s a workload heavy on long, streaming reads. The application can be modeled with a consistent read flow, and the software tests should be easy.  However, an analysis of the real-world workload shows something very different. At the disk level, the reads turn out to be a rapid flow of 4k and 8k block reads from a solid-state-disk.  Further, other processes on the system also add in a small amount of 4k and 8k writes in the midst of the reads. All of this impacts the application –and an SSD — which was likely heavily tested on the basis of long, streaming reads. Understanding the real-world characteristics of a workload can be a significant advantage in the development of new hardware, new systems, and new applications.   The SNIA Solid State Storage Initiative (SSSI) and SSSI member company Calypso Systems are providing an opportunity to build a repository of workloads for the industry to use for real-world testing, as outlined in a new SSSI white paper How to Be a Part of the Real-World Workload Revolution. By going to the TestMyWorkload site, anyone can participate by providing a trace capture of an I/O workload that can be used by others to develop better products. The capture itself traces the block transfers, but does not capture actual data.  Any workload replay would use representative blocks, so there are no concerns about data security or integrity from these captures. The repository can be used by any participant to test hardware and software, and can help system vendors and users optimize configurations for the best performance based on real-world data.  By participating in this effort, organizations and individuals can provide insight and gain from the knowledge of all the contributors. Follow these three steps to be a part of the revolution today! 1.  Read the white paper. 2.  Download the free capture tools at TestMyWorkload.com. 3. Mark your calendar and register HERE to learn more in the free SNIA webcast How to Be a Part of the Real-World Workload Revolution on July 9 at 11:00 am Pacific/2:00 pm Eastern.

Olivia Rhye

Product Manager, SNIA

Leave a Reply

Comments

Name

Email Adress

Website

Save my name, email, and website in this browser for the next time I comment.

New Conference Seeking PIRLs of Wisdom

Marty Foltyn

May 14, 2019

title of post
UCSD Computer Science and Engineering, the Non-Volatile Systems Laboratory, and the Storage Networking Industry Association (SNIA) are inviting submissions of proposals for presentation at the first annual Persistent Programming in Real Life (PIRL) conference.  PIRL brings together software development leaders interested in learning about programming methodologies for persistent memories and sharing their experiences with others. This is a meeting for developer project leads on the front lines of persistent programming; not sales, marketing, or non-technical management. PIRL is small, with attendance limited to under 100 people, including speakers.  It will discuss what real developers have done, and want to do, with persistent memory.  It will involve what worked, what didn’t, what was easy and hard, what was surprising, and what others can learn from the experience.  Presenters are encouraged, and even expected, to show and write code live in the presentation in a comfortable and dynamic peer environment. Possibilities for presentations include, but are not limited to: •  Experiences on a particular project •  Live code development showing new concepts •  Code challenges •  New tools for programming All attendees will be provided access to a development environment to respond to code challenges, or to show their own work in small forums.  This is intended to be a competition-free atmosphere for peers to network with each other to advance the use of persistent memory in the industry and academia.  By combining many of the industry leaders with the academic lights driving practical applications of new technology, peers at PIRL will encourage forward progress for adoption of persistent memory in the marketplace. Keynote speakers include key personnel from Dreamworks, VMWare, Oracle, Eideticom, and Intel. PIRL will be hosted by the Non-Volatile Systems Laboratory at the University of California, San Diego.  It will be held at Scripps Forum on July 22nd to 23rd, 2019, with optional events starting July 21st. Pre-registration will be $400. We’re excited to present this new conference, and we’re excited for you to participate.  Submit your presentation or code challenge idea today. Submissions are due by Monday, June 10th.  

Olivia Rhye

Product Manager, SNIA

Find a similar article by tags

Leave a Reply

Comments

Name

Email Adress

Website

Save my name, email, and website in this browser for the next time I comment.

New Conference Seeking PIRLs of Wisdom

Marty Foltyn

May 14, 2019

title of post
UCSD Computer Science and Engineering, the Non-Volatile Systems Laboratory, and the Storage Networking Industry Association (SNIA) are inviting submissions of proposals for presentation at the first annual Persistent Programming in Real Life (PIRL) conference.  PIRL brings together software development leaders interested in learning about programming methodologies for persistent memories and sharing their experiences with others. This is a meeting for developer project leads on the front lines of persistent programming; not sales, marketing, or non-technical management. PIRL is small, with attendance limited to under 100 people, including speakers.  It will discuss what real developers have done, and want to do, with persistent memory.  It will involve what worked, what didn’t, what was easy and hard, what was surprising, and what others can learn from the experience.  Presenters are encouraged, and even expected, to show and write code live in the presentation in a comfortable and dynamic peer environment. Possibilities for presentations include, but are not limited to: •  Experiences on a particular project •  Live code development showing new concepts •  Code challenges •  New tools for programming All attendees will be provided access to a development environment to respond to code challenges, or to show their own work in small forums.  This is intended to be a competition-free atmosphere for peers to network with each other to advance the use of persistent memory in the industry and academia.  By combining many of the industry leaders with the academic lights driving practical applications of new technology, peers at PIRL will encourage forward progress for adoption of persistent memory in the marketplace. Keynote speakers include key personnel from Dreamworks, VMWare, Oracle, Eideticom, and Intel. PIRL will be hosted by the Non-Volatile Systems Laboratory at the University of California, San Diego.  It will be held at Scripps Forum on July 22nd to 23rd, 2019, with optional events starting July 21st. Pre-registration will be $400. We’re excited to present this new conference, and we’re excited for you to participate.  Submit your presentation or code challenge idea today. Submissions are due by Monday, June 10th.  

Olivia Rhye

Product Manager, SNIA

Find a similar article by tags

Leave a Reply

Comments

Name

Email Adress

Website

Save my name, email, and website in this browser for the next time I comment.

Trends in Media and Entertainment Storage – Your Questions Answered from Our Webcast

Marty Foltyn

Apr 1, 2019

title of post
Thanks to all who attended or listened on-demand to our recent SNIA Solid State Storage Initiative (SSSI) webcast on Trends in Worldwide Media and Entertainment Storage. Motti Beck of Mellanox Technologies and Tom Coughlin, SSSI Education Chair and analyst with Coughlin Associates, got rave reviews for their analysis of this important market.  Feedback comments included “Good overview with enough details for me to learn something”; “Really appreciate the insight into the ME businesses”; and “Just in time for the upcoming NAB Show!”.  We appreciate your interest and enthusiasm! Important to every SNIA webcast are the Questions – and we got quite a few on this one.  Thanks in advance to Tom Coughlin, who provided the answers below.  Send any more questions to us at asksssi@snia.org with the subject- M&E Webcast Questions.  Happy reading, and we hope to see you at one of our upcoming webcasts or events. Q.  What is the best form to store(age) the format video, NAS or SAN? A.  Well, it depends.  A SAN can directly access the data blocks that make up the video file, these can be quickly transported to the workstation.  There they are reassembled into the video file.  A properly configured SAN can provide faster access to data, particularly if many users are accessing the same data in the storage system.  SANs can be appropriate for a larger production facility.  A NAS may provide somewhat slower access, but provides individual access to individual files.  NAS storage can be an appropriate shared storage for smaller production facilities where there are fewer users or the users don’t access the same files at the same time.   Q.  Are there an M&E-specific performance benchmarks or other qualification tools recommended for storage subsystem selection? A.  That is an interesting question.  I know about several general storage performance benchmarks, such as SPC (https://spcresults.org/benchmarks).  There are storage performance tests offered by some M&E industry suppliers, such as one from AJA System Test (https://www.aja.com/products/aja-system-test).  This is probably an area that could use some additional development.   Q.  Revenue share by type and use case – is this on the decline or rise?  What are the YoY trends? A.  If I understand this right, you are asking questions about revenue growth for different media and entertainment use cases or different parts of the workflow on an annual basis.  That information is in the 2018 Digital Storage for Media and Entertainment Report (https://tomcoughlin.com/tech-papers/)   Q.  A question for Tom Coughlin.  You  said 66% will use private or public cloud for archiving in 2018.  Do you have the breakdown between the the two? A.  It is a combination but given the concerns of the industry, I suspect this is mostly private cloud in 2018.   Q.  When Tom says “post production” storage, is that primary, secondary/nearline, or both? A.  If I understand the question right, this is all storage used in post-production, which can use a primary and secondary storage tier, particularly in a larger facility which is economizing on its storage costs.   Q.  With regard to HDD storage, does the interface trend continue to be SATA/SAS?  Does the back end workload look to benefit from SMR or dual-actuator technology? A.  For the time being HDDs will be SATA and SAS.  There are now some HDD storage systems with NVMe on the back end and it will be interesting to see how this develops.  I am sure that M&E users will benefit from SMR and dual-actuator HDDs.  SMR will be good for active archiving in particular and dual actuator will allow faster access to HDD data, a benefit for video projects.   Q.  Unless I missed it, you made no mention of software-defined storage as a viable method for storing the growing amount of data in M&E.  Was that taken into consideration when you did your survey? A.  Software defined storage can be an important element in media and entertainment storage and is finding increasing use in this and other applications.   Q.  Is the cloud archive, primary copy or secondary (insurance policy with limited to no access)? A.  It depends upon the organization, although I think for many studios and larger organizations, they may keep content on tape and even off-line tape as well.  Cloud archives do allow access to data, the usual issue is the cost of egressing that content.

Olivia Rhye

Product Manager, SNIA

Find a similar article by tags

Leave a Reply

Comments

Name

Email Adress

Website

Save my name, email, and website in this browser for the next time I comment.

Your Questions Answered – Applications Take Advantage of Persistent Memory Webcast

Marty Foltyn

Mar 27, 2019

title of post
We hope you had time to check out our recent webcast on Applications Take Advantage of Persistent Memory Raghu Kulkarni of Viking Technology, a member of the SNIA Solid State Storage Initiative, did a great job laying the foundation for an understanding of Persistent Memory today, just in time for the SNIA Persistent Memory Summit. You can catch up on videos of Summit talks, along with the slides presented, here. During the webcast, we had many interesting questions.  Now, as promised, Raghu provides the answers.  Happy reading, and we hope to see you at one of our upcoming webcasts or events. Q.  Does NVDIMM-N encryption lower the performance levels that you presented? A.  It typically depends on the implementation and differs from each vendor. Generally speaking, Save and Restore operations will increase by a small factor – less than 10%.  Products from some vendors, like Viking, will not see a performance degradation as it is offset by a faster transfer rate Q.  What are the read/write bandwidth capabilities of NVDIMM-N? How does that compare to Intel’s Persistent Memory? A.  For Byte-addressable mode, NVDIMM-N in theory has the same high performance as DRAM, around 100ns. With the latest Linux drivers in DAX mode, NVDIMM-N are still expected to be better than Intel’s Persistent Memory. Q.  On the use cases, what are the use cases when Persistent Memory is attached to an accelerator chip compared to a Processor attached setup? A.  Mainly to accelerate the performance by storing the metadata or even data in Persistent Memory, so that the request can be acknowledged immediately without having to wait for commits to SSD/HDD. It also saves the rebuild time, which is a common practice for volatile memory. Q.  How does BIOS/MRC work when a Persistent Memory is attached to an accelerator (ASIC/FPGA/GPU) over PCIe, when trying to extended/increase the memory for the processor? A.  System BIOS will not detect the Persistent Memory sitting on PCIe; it only discovers Persistent Memory installed in DIMM slots. FPGA/ASIC, etc. have to build their own bottom up code to detect and present the Persistent Memory on PCIe depending on the use case. Q.  Do we need application changes to take advantage of Persistent Memory-aware file storage? how does it compare against the DAX mode? A.  To take advantage of the low latency/high performance nature of Persistent Memory, it would be beneficial to modify the applications. However, one can still leverage the existing IO stack if modifying the application is not an option. Check out pmem.io for pre-built libraries that can be directly integrated into applications. Q.  Should the Persistent Memory usage be compared against the Storage or Memory. Which is a more relevant use case for Persistent Memory? A.  Typically, a media that is Byte-addressable is called Persistent Memory (PM); however, you can also access it in Block mode. Again, depending on the application needs, use case, and other system level factors it can be used in either modes.  However, you will find best performance when accessing in Byte-addressable/Load-Store mode.    

Olivia Rhye

Product Manager, SNIA

Find a similar article by tags

Leave a Reply

Comments

Name

Email Adress

Website

Save my name, email, and website in this browser for the next time I comment.

Innovating File System Architectures with NVMe

Marty Foltyn

Mar 20, 2019

title of post

It’s exciting to see the recent formation of the Solid State Drive Special Interest Group (SIG) here in the SNIA Solid State Storage Initiative.  After all, everyone appreciates the ability to totally geek out about the latest drive technology and software for file systems.  Right? Hey, where’s everyone going? We have vacation pictures with the dog we stored that we want to show…

Solid state storage has long found its place with those seeking greater performance in systems, especially where smaller or more random block/file transfers are prevalent.  Single-system opportunity with NVMe drives is broad, and pretty much unquestioned by those building systems for the modern IT environments. Cloud, likewise, has found use of the technology where single-node performance makes a broader deployment relevant.

There have been many efforts to build the case for solid state in networked storage.  Where storage and computation combine -- for instance in a large map/reduce application -- there’s been significant advantage, especially in the area of sustained data reads.  This has usually comes at a scalar cost, where additional systems are needed for capacity. Nonetheless, finding cases where non-volatile memory enhances infrastructure deployment for storage or analytics.  Yes, analytics is infrastructure these days, deal with it.

Seemingly independent of the hardware trends, the development of new file systems has provided significant innovation.  Notably, heavily parallel file systems have the ability to serve a variety of network users in specialized applications or appliances.  Much of the work has focused on development of the software or base technology rather than delivering a broader view of either performance or applicability.  Therefore, a paper such as this one on building a Lustre file system using NVMe drives is a welcome addition to the case for both solid state storage and revolutionary file systems that move from specific applications to more general availability.

The paper shows how to build a small (half-rack) cluster of storage to support the Lustre file system, and it also adds the Dell VFlex OS implemented as a software defined storage solution.  This has the potential to take an HPC-focused product like Lustre and drive a broader market availability for a high-performance solution. The combination of read/write performance, easy adoption to the broad enterprise, and relatively small footprint shows new promise for innovation.

The opportunity for widespread delivery of solid state storage using NVMe and software innovation in the storage space is ready to move the datacenter to new and more ambitious levels.  The SNIA 2019 Storage Developer Conference  is currently open for submissions from storage professionals willing to share knowledge and experience.  Innovative solutions such as this one are always welcome for consideration.

Olivia Rhye

Product Manager, SNIA

Find a similar article by tags

Leave a Reply

Comments

Name

Email Adress

Website

Save my name, email, and website in this browser for the next time I comment.

Innovating File System Architectures with NVMe

Marty Foltyn

Mar 20, 2019

title of post

It’s exciting to see the recent formation of the Solid State Drive Special Interest Group (SIG) here in the SNIA Solid State Storage Initiative.  After all, everyone appreciates the ability to totally geek out about the latest drive technology and software for file systems.  Right? Hey, where’s everyone going? We have vacation pictures with the dog we stored that we want to show…

Solid state storage has long found its place with those seeking greater performance in systems, especially where smaller or more random block/file transfers are prevalent.  Single-system opportunity with NVMe drives is broad, and pretty much unquestioned by those building systems for the modern IT environments. Cloud, likewise, has found use of the technology where single-node performance makes a broader deployment relevant.

There have been many efforts to build the case for solid state in networked storage.  Where storage and computation combine — for instance in a large map/reduce application — there’s been significant advantage, especially in the area of sustained data reads.  This has usually comes at a scalar cost, where additional systems are needed for capacity. Nonetheless, finding cases where non-volatile memory enhances infrastructure deployment for storage or analytics.  Yes, analytics is infrastructure these days, deal with it.

Seemingly independent of the hardware trends, the development of new file systems has provided significant innovation.  Notably, heavily parallel file systems have the ability to serve a variety of network users in specialized applications or appliances.  Much of the work has focused on development of the software or base technology rather than delivering a broader view of either performance or applicability.  Therefore, a paper such as this one on building a Lustre file system using NVMe drives is a welcome addition to the case for both solid state storage and revolutionary file systems that move from specific applications to more general availability.

The paper shows how to build a small (half-rack) cluster of storage to support the Lustre file system, and it also adds the Dell VFlex OS implemented as a software defined storage solution.  This has the potential to take an HPC-focused product like Lustre and drive a broader market availability for a high-performance solution. The combination of read/write performance, easy adoption to the broad enterprise, and relatively small footprint shows new promise for innovation.

The opportunity for widespread delivery of solid state storage using NVMe and software innovation in the storage space is ready to move the datacenter to new and more ambitious levels.  The SNIA 2019 Storage Developer Conference  is currently open for submissions from storage professionals willing to share knowledge and experience.  Innovative solutions such as this one are always welcome for consideration.

Olivia Rhye

Product Manager, SNIA

Find a similar article by tags

Leave a Reply

Comments

Name

Email Adress

Website

Save my name, email, and website in this browser for the next time I comment.

Marty Foltyn

Feb 22, 2019

title of post

Olivia Rhye

Product Manager, SNIA

Find a similar article by tags

Leave a Reply

Comments

Name

Email Adress

Website

Save my name, email, and website in this browser for the next time I comment.

Marty Foltyn

Feb 22, 2019

title of post

It’s now less than three weeks for the next SNIA Persistent Memory Hackathon and Workshop.  Our next workshop will be held in conjunction with the 10th Annual Non-Volatile Memory Workshop (http://nvmw.ucsd.edu/) at the University of California, San Diego on Sunday, March 10th from 2:00pm to 5:30pm.

The Hackathon at NVMW19 provides software developers with an understanding of the different tiers and modes of persistent memory, and gives an overview of the standard software libraries that are available to access persistent memory.  Attendees will have access to system configured with persistent memory, software libraries, and sample source code. A variety of mentors will be available to provide tutorials and guide participants in the development of code. Learn more here.

In the last workshop, the feedback from the attendees pointed to a desire to work longer on code after the tutorial ended.  We will ensure that all the Hackathon attendees will have access to their environment through the length of the conference. So any participant in the Sunday session will be able to continue work until the conference completion on Tuesday afternoon.  While there won’t be an opportunity for formal follow-up, we’re planning an informal meet-up the final day of the conference. Stay tuned for details.

For those not familiar with NVMW, the program is replete with the latest in non-volatile memory research, which enables attendees to understand the practical advances in software development for persistence.  The workshop facilitates the exchange of ideas and advances collaboration and knowledge of the use of persistent memory. Registration for the conference is affordable, and grants are available for university student attendees.

For those not able to get to San Diego in March, enjoy the weather that obviously won’t be anywhere near as nice where you live.  Oh, sorry. For those not able to get to San Diego in March, SNIA is working on the next opportunities for a formal hackathon. But we can’t do it alone.  If you have a group of programmers interested in learning persistent memory development, SNIA would consider coming to you with a Host a Hackathon. We can provide, or even train, mentors to get you started, and show you how to build your own cloud-based development environment.  You’ll get an introduction to coding, and you’ll be left with some great examples to build your own applications. Contact SNIA at PMhackathon@snia.org for more details and visit our PM Programming Hackathon webpage for the latest updates.

Olivia Rhye

Product Manager, SNIA

Find a similar article by tags

Leave a Reply

Comments

Name

Email Adress

Website

Save my name, email, and website in this browser for the next time I comment.

New Capability in Familiar Places

Marty Foltyn

Jan 14, 2019

title of post
When it comes to persistent memory, many application developers initially think of change as hard work that likely yields incremental result.  It’s perhaps a better idea to look at the capability that’s new, but that’s already easily accessible using the methods that are in place today.  It’s not that enabling persistent memory is effortless, it’s more that normal code improvement can take advantage of the new features in the standard course of development. The concept of multiple memory tiers is ingrained in nearly every programming model.  While the matrix of possibility can get fairly complex, it’s worth looking at three variables of the memory model.  The first is the access type, either via load/store or block operation. The second is the latency or distance from the processing units; in this case the focus would be on the DIMM.  The last would be memory persistence. Adding persistence to the DIMM tier of memory provides opportunity to programmers in a variety of ways.  Typically, this latency is used for most of the program flow, while data eventually is moved to a farther tier such as disk or network for persistence.  Allocating the majority of data to a low-latency tier like a DIMM has significant potential. An example of this in the marketplace would be SAP’s HANA in-memory database.  However, it’s less well-known that more traditional database products in the same category have built-in methodologies for moving data that is repeatedly accessed into the DIMM tier, later committing changes to storage via background processes.  It’s likely that adding persistence to DIMMs in volume would be both valuable and also architecturally possible in a short period of development time. One way that this process is simplified for developers is the fact that the SNIA NVM Programming Model for DIMM-based persistence incorporates both load/store and block access modes.   Developers already familiar with using SSD over rotating media -- that would be a fourth memory vector, deal with the ambiguity -- would be able to see some incremental performance and potentially some system design simplification.  Those already using memory for data storage could utilize better recovery options as well as explore changes that high-performance storage could bring. Join other developers on Wednesday, January 23rd at the SNIA Persistent Memory Programming Hackathon to explore options for how your software can take advantage of this new opportunity. Complimentary registration is available at this link.

Olivia Rhye

Product Manager, SNIA

Find a similar article by tags

Leave a Reply

Comments

Name

Email Adress

Website

Save my name, email, and website in this browser for the next time I comment.

Subscribe to Marty Foltyn