Experts Speak at Flash Memory Summit

Marty Foltyn

Jan 7, 2021

title of post



2020 brought new developments in persistent memory and computational storage. SNIA Compute, Memory, and Storage Initiative was pleased to sponsor two tracks at the recent Flash Memory Summit where industry leaders captured the advances.  Videos and presentations are now available.

In the Persistent Memory Track, Dave Eggleston of Intuitive Cognition Consulting and Chris Petersen of Facebook combine to deliver a state of the union address for the industry effort underway to deliver persistent memory. They examine industry advances of persistent memory media, the new devices and form factors for persistent memory attachment, remote and direct-attached PM with low latency interfaces like CXL, and describe the best fit applications and use cases for persistent memory.

Jia Shi of Oracle and Yao Yue of Twitter then dive into a rapid-fire presentation on two examples of how persistent memory is changing the landscape – in appliances, in infrastructure, and in applications - from the perspective of a social networking company and a cloud and enterprise software provider.  They highlight the motivation for using persistent memory and the delivered results

Finally, Ginger Gilsdorf of Intel and Tom Coughlin of Coughlin Associates look ahead to how Persistent Memory technology is evolving, including maximizing performance in next-generation applications, and provide their perspective on PM market growth projections.

The track concludes with speakers reuniting in a panel to discuss the reasons that have stopped persistent memory from gaining wider usage and identifying breakthroughs that are beginning to appear.

The Computational Storage Track opens with an update by Chuck Sobey of Channel Science who discusses the shifting of compute power to the storage; use cases including database, big data, AI/ML, and edge applications; and how the framework for computational storage is driven by SNIA and the NVM Express standards groups.

Stephen Bates of Eideticom follows with an outline of the state of the nation in computational storage standards. He then describes computational storage examples already in use that illustrate ways storage challenges are being met, and comments on promising directions to explore for the future.

Andy Walls of IBM then discusses using computational storage to handle big data, allowing data to reside close to processing power, thus allowing processing tasks to be in-line with data accesses. He covers computational storage examples already in use for application distribution and other promising directions to explore for the future.

Neil Werdmuller and Jason Molgaard of Arm discuss flexible computational storage solutions, and how data-driven applications that benefit from database searches, data manipulation, and machine learning can perform better and be more scalable if developers add computation directly to storage.

A lively panel with Arm, Eideticom, NGD Systems, and ScaleFlux rounds out the track, discussing keys to making computational storage work in your applications.  

Enjoy these presentations and contact us at askcmsi@snia.org with your questions and comments!



Olivia Rhye

Product Manager, SNIA

Leave a Reply

Comments

Name

Email Adress

Website

Save my name, email, and website in this browser for the next time I comment.

Marty Foltyn

Feb 5, 2020

title of post

The SNIA Persistent Memory and NVDIMM Special Interest Group announced a programming challenge for NVDIMM-based systems in Q4 of 2019.  Participants get free online access to persistent memory systems based at the SNIA Technology Center using NVDIMM-Ns provided by SIG members AgigA Tech, Intel, SMART Modular, and Supermicro.  The goal of the challenge is to spark interest by developers in this new technology so they can understand more clearly how persistent memory applications can be developed and applied in 2020 environments and beyond.

Response to the NVDIMM Programming Challenge has been very positive.  Entrants to date have backgrounds from no experience programming persistent memory to those who develop persistent memory applications as part of their day jobs.

At the January 2020 Persistent Memory Summit, the SIG announced the first NVDIMM Programming Challenge winner:   Steve Heller of Chrysalis Software Corporation.

Steve submitted a closed-source project, the Three Misses Persistent Hash Table (www.threemisses.com), a key-value store application that uses persistent memory to enable significantly faster start-up and shut-down.  Its use of the DRAM speed of the NVDIMM modules enables faster look-up performance.

Steve's project met the challenge criteria as reviewed by the judges, including the use of multiple aspects of NVDIMM/Persistent Memory capabilities and the use of persistence to enable new features and appeal across multiple aspects of a system beyond persistence.  The Three Misses Persistent Hash Table also advanced the cause of Persistent Memory and applied to all types of NVDIMM/Persistent Memory systems.

Jim Fister, who directs the SNIA Hackathon Program, provided a lively summary of Steve’s winning entry during his talk Introduction to PM Hackathons at the Persistent Memory Summit.  Look for the details about 9 minutes, 30 seconds into the video.  You can watch all of the day’s videos on the SNIA Video Channel PM Summit playlist.

Steve also provided a live demonstration of his work during the day at the Persistent Memory Summit.

SNIA congratulates Steve and reminds you that the NVDIMM Programming Challenge is still LIVE!  Additional participants and submissions are welcome through March 31, 2020, and will be featured at upcoming SNIA events.  Send an email to PMhackathon@snia.org and get your credentials.  Read more about challenge details, and watch this space for future winners, as well as more challenge opportunities!

Olivia Rhye

Product Manager, SNIA

Find a similar article by tags

Leave a Reply

Comments

Name

Email Adress

Website

Save my name, email, and website in this browser for the next time I comment.

Judging Has Begun – Submit Your Entry for the NVDIMM Programming Challenge!

Marty Foltyn

Nov 19, 2019

title of post
We’re 11 months in to the Persistent Memory Hackathon program, and over 150 software developers have taken the tutorial and tried their hand at programming to persistent memory systems.   AgigA Tech, Intel SMART Modular, and Supermicro, members of the SNIA Persistent Memory and NVDIMM SIG, have now placed persistent memory systems with NVDIMM-Ns into the SNIA Technology Center as the backbone of the first SNIA NVDIMM Programming Challenge. Interested in participating?  Send an email to PMhackathon@snia.org to get your credentials.  And do so quickly, as the first round of review for the SNIA NVDIMM Programming Challenge is now open.  Any entrants who have progressed to a point where they would like a review are welcome to contact SNIA at PMhackathon@snia.org to request a time slot.  SNIA will be opening review times in December and January as well.  Submissions that meet a significant amount of the judging criteria described below, as determined by the panel, will be eligible for a demonstration slot to show the 400+ attendees at the January 23, 2020 Persistent Memory Summit  in Santa Clara CA. Your program or results should be able to be visually demonstrated using remote access to a PM-enabled server. Submissions will be judged by a panel of SNIA experts.  Reviews will be scheduled at the convenience of the submitter and judges, and done via conference call. NVDIMM Programming Challenge Judging Criteria include: Use of multiple aspects of NVDIMM/PM capabilities, for example:
  1. Use of larger DRAM/NVDIMM memory sizes
  2. Use of the DRAM speed of NVDIMM PMEM for performance
  3. Speed-up of application shut down or restart using PM where appropriate
  4. Recovery from crash/failure
  5. Storage of data across application or system restarts
Demonstrates other innovative aspects for a program or tool, for example:
  1. Uses persistence to enable new features
  2. Appeals across multiple aspects of a system, beyond persistence
Advances the cause of PM in some obvious way:
  1. Encourages the update of systems to broadly support PM
  2. Makes PM an incremental need in IT deployments
Program or results apply to all types of NVDIMM/PM systems, though exact results may vary across memory types. Questions? Contact Jim Fister, SNIA Hackathon Program Director, at pmhackathon@snia.org, and happy coding!

Olivia Rhye

Product Manager, SNIA

Leave a Reply

Comments

Name

Email Adress

Website

Save my name, email, and website in this browser for the next time I comment.

Your Questions Answered - Now You Can Be a Part of the Real World Workload Revolution!

Marty Foltyn

Jul 17, 2019

title of post

The SNIA Solid State Storage Initiative would like to thank everyone who attended our webcast: How To Be Part of the Real World Workload Revolution.  If you haven’t seen it yet, you can view the on demand version here.  You can find the slides here.

Eden Kim and Jim Fister led a discussion on the testmyworkload (TMW) tool and data repository, discussing how a collection of real-world workload data captures can revolutionize design and configuration of hardware, software and systems for the industry.   A new SNIA white paper available in both English and Chinese authored by Eden Kim, with an introduction by Tom Coughlin of Coughlin Associates and Jim Handy of Objective Analysis, discusses how we can all benefit by sharing traces of our digital workloads through the SNIA SSSI Real-World Workload Capture program.

In an environment where workloads are becoming more complex -- and the choices of hardware configuration for solid-state storage are growing -- the opportunity to better understand the characteristics of data transfers to and from the storage systems is critical.  By sharing real-world workloads on the Test My Workload repository, the industry can benefit overall in design and development at every level from SSD development to system configuration in the datacenter.

There were several questions asked in and after the webcast.  Here are some of the answers.  Any additional questions can be addressed to asksssi@snia.org.

Q: Shouldn't real world workloads have concurrent applications?  Also, wouldn’t any SQL workloads also log or journal sequential writes?

A: Yes.  Each capture shows all of the IO Streams that are being applied to each logical storage recognized by the OS.  These IO Streams are comprised of IOs generated by System activities as well as a variety of drivers, applications and OS activities.  The IOProfiler toolset allows you to not only see all of the IO Stream activity that occurs during a capture, but also allows you to parse, or filter, the capture to see just the IO Streams (and other metrics) that are of interest.

Q: Is there any collaboration with the SNIA IOTTA Technical Work Group on workload or trace uploading?

A:  While IOTTA TWG and SSS TWG work closely together, an IO Capture is fundamentally different from an IO Trace and hence is not able to be presented on the IOTTA trace repository.  An IO Trace collects all of the data streams that occur during the IO Trace capture period and results in a very large file.  An IO Capture, on the other hand, captures statistics on the observed IO Streams and saves these statistics to a table.  Hence, no actual personal or user data is captured in an IO Capture, only the statistics on the IO Streams. Because IO Captures are a series of record tables for individual time steps, the format is not compatible with a repository for the streaming data captured in an IO Trace.

For example, an IO Trace could do a capture where 50,000 RND 4K Write and 50,000 RND 4K Read IOPS are recorded, resulting in 100,000 4K transfers, or 40M bytes of data.  OTOH, an IO Capture that collects statistics would log the fact that 50,000 RND 4K Writes and 50,000 RND 4K Reads occurred… a simple two item entry in a table.  Of course, the IOPS, Response Times, Queue Depths and LBA Ranges could also be tracked resulting in a table of 100,000 entries times the above 4 metrics, but 400,000 table entries is much smaller than 40 MB of data.

Both of these activities are useful, and the SNIA supports both.

Q: Can the traces capture a cluster workload or just single server?

A: IO Captures capture the IO Streams that are observed going from User space to all logical storage recognized by the OS.  Accordingly, for clusters, there will be an individual capture for each logical unit.  Note that all logical device captures can be aggregated into a single capture for analysis with the advanced analytics offered by the commercial IOProfiler tools.

Q: Have you seen situation where the IO size on the wire does not matched what application request?  Example Application request 256K but driver chopped the IO into multiple 16K before sent to the storage. How would we verify this type of issue?

A: Yes, this is a common situation. Applications may generate a large block SEQ IO Stream for video on demand.  However, that large block SEQ IO Stream is often fragmented into concurrent RND block sizes.  For example, in Linux OS, a 1MB file is often fragmented into random concurrent 128K block sizes for transmission to and from storage, but then coalesced back into a single 1024K BS in user space..

Q: Will you be sharing the costs for your tools or systems?

A: The tool demonstrated in the webcast is available free at testmyworkload.com (TMW).  This is done to build the repository of workloads at the TMW site.  Calypso Systems does have a set of Pro tools built around the TMW application.  Contact Calypso for specific details.

Q: Can the capture be replayed on different drives?

A: Yes.  In fact, this is one of the reasons that the tool was created.  The tool and repository of workloads are intended to be used as a way to compare drive and system performance, as well as tune software for real-world conditions.

Q: How are you tracking compressibility & duplication if the user does not turn on compression or dedupe?

A: The user must turn on compression or duplication at the beginning of the capture to see these metrics.

Q: An end user can readily use this to see what their real world workload looks like.  But, how could an SSD vendor mimic the real world workload or get a more "realworld-like" workload for use in common benchmarking tools like FIO & Sysbench?

A: The benchmarking tools mentioned are synthetic workloads, and write a predictable stream to and from the drive.  IO Captures ideally are run as a replay test that recreates the sequence of changing IO Stream combinations and Queue Depths observed during the capture.  While the Calypso toolset can do this automatically, free benchmark tools like FIO and sysbench may not be able to change QDs and IO Stream combinations from step to step in a test script.  However, the IO Capture will also provide a cumulative workload that list the dominant IO Streams and their percentage of occurrence.  This list of dominant IO Streams can be used with fio or sysbench to create a synthetic composite IO stream workload.

Q: Is it possible to use the tool to track CPU State such as IOWAIT or AWAIT based on the various streams?

A: Yes, IO Captures contain statistics on CPU usage such as CPU System Usage %, CPU IO Wait, CPU User usage, etc.

Q: Can we get more explanation of demand intensity and comparison to queue depth?

A: Demand Intensity (DI) is used to refer to the outstanding IOs at a given level of the software/hardware stack.  It may be referred to simply as the outstanding Queue Depth (QD) or as the number of outstanding Thread Count (TC) and QD.  The relevance of TC depends on where in the stack you are measuring the DI.  User QD varies from level to level and depends on what each layer of abstraction is doing.  Usually, focus is paid to the IO Scheduler and the total outstanding IOs at the block IO level.  Regardless of nomenclature, it is important to understand the DI as your workload traverses the IO Stack and to be able to minimize bottlenecks due to high DI.

Q: In these RWSW application traces do these include non-media command percentages such as identify and read log page (SMART), sleep states, etc.?  Depending on the storage interface and firmware this can adversely affect performance/QoS.

A: IO Capture metrics are the IO Streams at the logical storage level and thus do not include protocol level commands.  Non performance IO commands such as TRIMs can be recorded, and SMART logs can be tracked if access to the physical storage is provided.

Q: Isn't latency a key performance metric for these workloads so collecting only 2 minute burst might not show latency anomalies?

A: IO Captures average the statistics over a selected time window.  Each individual IO Stream and its metrics are recorded and tabulated on a table but the time window average is what is displayed on the IO Stream map.  Of course, the min and max Response times over the 2 minute window are displayed, but the individual IO latencies are not displayed.  In order to track IO Bursts, the time window resolution should be set to a narrow time range, such as 100 mS or less, in order to distinguish IO Bursts and Host Idle times.

Olivia Rhye

Product Manager, SNIA

Find a similar article by tags

Leave a Reply

Comments

Name

Email Adress

Website

Save my name, email, and website in this browser for the next time I comment.

Register for the PIRL Conference Today

Marty Foltyn

Jun 25, 2019

title of post

Registration is now open for the upcoming Persistent Programming in Real Life (PIRL) Conference – July 22-23, 2019 on the campus of the University of California San Diego (UCSD).

The 2019 PIRL event features a collaboration between UCSD Computer Science and Engineering, the Non-Volatile Systems Laboratory, and the SNIA to bring industry leaders in programming and developing persistent memory applications together for a two-day discussion on their experiences.

PIRL is a small conference, with attendance limited to under 100 people, including speakers.  It will discuss what real developers have done, and want to do, with persistent memory. Most of the presentations will include demonstrations of live code showing new concepts.  The conference is designed to be a meet-up for developers seeking to gain and share knowledge in the growing area of Persistent Memory development.

PIRL features a program of 18 presentations and 5 keynotes from industry-leading developers who have built real systems using persistent memory.  They will share what they have done (and want to do) with persistent memory, what worked, what didn’t, what was hard, what was easy, what was surprising, and what they learned.

This year’s keynote presentations will be:

  • * Pratap Subrahmanyam (Vmware): Programming Persistent Memory In A Virtualized Environment Using Golang
  • * Zuoyu Tao (Oracle): Exadata With Persistent Memory – An Epic Journey
  • * Dan Williams (Intel Corporation): The 3rd Rail Of Linux Filesystems: A Survival Story
  • * Stephen Bates (Eideticom): Successfully Deploying Persistent Memory and Acceleration Via Compute Express Link
  • * Scott Miller (Dreamworks): Persistent Memory In Feature Animation Production

Other speakers include engineers from NetApp, Lawrence Livermore National Laboratory, Oracle, Sandia National Labs, Intel, SAP, Red Hat, and universities from around the world.  Full details are available at the PIRL website.

PIRL will be held on the University of California San Diego campus at Scripps Forum, a state-of-the-art conference facility just a few meters from the beach.  Discounted early registration ends July 10, so register today to ensure your seat.

Olivia Rhye

Product Manager, SNIA

Find a similar article by tags

Leave a Reply

Comments

Name

Email Adress

Website

Save my name, email, and website in this browser for the next time I comment.

Marty Foltyn

Feb 22, 2019

title of post

Olivia Rhye

Product Manager, SNIA

Find a similar article by tags

Leave a Reply

Comments

Name

Email Adress

Website

Save my name, email, and website in this browser for the next time I comment.

Persistently Fun Once Again – SNIA’s 7th Persistent Memory Summit is a Wrap!

kristin.hauser

Jan 28, 2019

title of post
Leave it to Rob Peglar, SNIA Board Member and the MC of SNIA’s 7th annual Persistent Memory Summit to capture the Summit day as persistently fun with a metric boatload of great presentations and speakers! And indeed it was a great day, with fourteen sessions presented by 23 speakers covering the breadth of where PM is in 2019 – real world, application-focused, and supported by multiple operating systems. Find a great recap on the Forbes blog by Tom Coughlin of Coughlin Associates. Attendees enjoyed live demos of Persistent Memory technologies from AgigA Tech, Intel, SMART Modular, the SNIA Solid State Storage Initiative, and Xilinx.  Learn more about what they presented here. And for the first time as a part of the Persistent Memory Summit, SNIA hosted a Persistent Memory Programming Hackathon sponsored by Google Cloud, where SNIA PM experts mentored software developers to do live coding to understand the various tiers and modes of PM and what existing methods are available to access them.  Upcoming SNIA SSSI on Solid State Storage blogs will give details and insights into "PM Hacking".  Also sign up for the SNIAMatters monthly newsletter to learn more, and stay tuned for upcoming Hackathons – next one is March 10-11 in San Diego. Missed out on the live sessions?  Not to worry, each session was videotaped and can be found on the SNIA Youtube Channel.  Download the slides for each session on the PM Summit agenda at www.snia.org/pm-summit.  Thanks to our presenters from Advanced Computation and Storage, Arm, Avalanche Technology, Calypso Systems, Coughlin Associates, Dell, Everspin Technologies, In-Cog Solutions, Intel, Mellanox Technologies, MemVerge, Microsoft, Objective Analysis, Sony Semiconductor Solutions Corporation, Tencent Cloud, Western Digital, and Xilinx.   And thanks also to our great audience and their questions – your enthusiasm and support will keep us persistently having even more fun!

Olivia Rhye

Product Manager, SNIA

Find a similar article by tags

Leave a Reply

Comments

Name

Email Adress

Website

Save my name, email, and website in this browser for the next time I comment.

New Capability in Familiar Places

Marty Foltyn

Jan 14, 2019

title of post
When it comes to persistent memory, many application developers initially think of change as hard work that likely yields incremental result.  It’s perhaps a better idea to look at the capability that’s new, but that’s already easily accessible using the methods that are in place today.  It’s not that enabling persistent memory is effortless, it’s more that normal code improvement can take advantage of the new features in the standard course of development. The concept of multiple memory tiers is ingrained in nearly every programming model.  While the matrix of possibility can get fairly complex, it’s worth looking at three variables of the memory model.  The first is the access type, either via load/store or block operation. The second is the latency or distance from the processing units; in this case the focus would be on the DIMM.  The last would be memory persistence. Adding persistence to the DIMM tier of memory provides opportunity to programmers in a variety of ways.  Typically, this latency is used for most of the program flow, while data eventually is moved to a farther tier such as disk or network for persistence.  Allocating the majority of data to a low-latency tier like a DIMM has significant potential. An example of this in the marketplace would be SAP’s HANA in-memory database.  However, it’s less well-known that more traditional database products in the same category have built-in methodologies for moving data that is repeatedly accessed into the DIMM tier, later committing changes to storage via background processes.  It’s likely that adding persistence to DIMMs in volume would be both valuable and also architecturally possible in a short period of development time. One way that this process is simplified for developers is the fact that the SNIA NVM Programming Model for DIMM-based persistence incorporates both load/store and block access modes.   Developers already familiar with using SSD over rotating media -- that would be a fourth memory vector, deal with the ambiguity -- would be able to see some incremental performance and potentially some system design simplification.  Those already using memory for data storage could utilize better recovery options as well as explore changes that high-performance storage could bring. Join other developers on Wednesday, January 23rd at the SNIA Persistent Memory Programming Hackathon to explore options for how your software can take advantage of this new opportunity. Complimentary registration is available at this link.

Olivia Rhye

Product Manager, SNIA

Find a similar article by tags

Leave a Reply

Comments

Name

Email Adress

Website

Save my name, email, and website in this browser for the next time I comment.

Opportunity for Persistent Memory is Now

Marty Foltyn

Jan 7, 2019

title of post
It’s very rare that there is a significant change in computer architecture, especially one that is nearly immediately pervasive across the breadth of a market segment.  It’s even more rare when a fundamental change such as this is supported in a way that software developers can quickly adapt to existing software architecture. Most significant transitions require a ground-up rethink to achieve performance or reliability gains, and the cost-benefit analysis generally pushes a transition to the new thing be measured in multiple revisions as opposed to one, big jump. In the last decade the growth of persistent memory has bucked this trend.  The introduction of the solid-state disk made an immediate impact on existing software, especially in the server market.  Any program that relied on multiple, small-data, read/write cycles to disk recognized significant performance increases. In cases such as multi-tiered databases, the software found a, “new tier,” of storage nearly automatically and started partitioning data to it.  In an industry where innovation takes years, improvement took a matter of months to proliferate across new deployments. While the SSD is now a standard consideration there is unexplored opportunity in solid-state storage.  The NVDIMM form factor has been in existence for quite some time, providing data persistence significantly closer to processing units in the modern server and workstation.  Many developers, however, are not aware that programming models already exist to easily incorporate some simple performance and reliability, both for byte and block access in programs.  Moreover, new innovations of persistent memory are on the horizon that will increase the density and performance of DIMM form factors. Perhaps it’s time that more software architecture should be working on adapting this exciting technology.  The barriers to innovation are very low, and opportunity is significant. Over the year 2019, SNIA will be sponsoring the delivery of several workshops dedicated to opening up persistent memory programming to the developer community.  The first of these will be a Persistent Memory Programming Hackathon at the Hyatt Regency Santa Clara CA on January 23, 2019, the day before the SNIA Persistent Memory Summit.   Developers will have the opportunity to work with experienced software architects to understand how to quickly adapt code to use new persistent memory modes in a hackathon format.  Learn more and register at this link. Don’t miss the opportunity to move on a strategic software inflection point ahead of the competition.  Consider attending the 2019 SNIA Persistent Memory Summit and exploring the opportunity with persistent memory.

Olivia Rhye

Product Manager, SNIA

Find a similar article by tags

Leave a Reply

Comments

Name

Email Adress

Website

Save my name, email, and website in this browser for the next time I comment.

Exceptional Agenda – and a Hackathon – Highlight the 2019 SNIA Persistent Memory Summit

Marty Foltyn

Jan 5, 2019

title of post
SNIA 7th annual Persistent Memory Summit – January 24, 2019 at the Hyatt Santa Clara CA – delivers a far-reaching agenda exploring exciting new topics with experienced speakers:
  • Paul Grun of OpenFabrics Alliance and Cray on the Characteristics of Persistent Memory
  • Stephen Bates of Eideticom, Neal Christiansen of Microsoft, and Eric Kaczmarek of Intel on Enabling Persistent Memory through OS and Interpreted Languages
  • Adam Roberts of Western Digital on the Mission Critical Fundamental Architecture for Numerous In-memory Databases
  • Idan Burstein of Mellanox Technologies on Making Remote Memory Persistent
  • Eden Kim of Calypso Systems on Persistent Memory Performance Benchmarking and Comparison
And much more!  Full agenda and speaker bios at http://www.snia.org/pm-summit. Registration is complimentary and includes the opportunity to tour demonstrations of persistent memory applications available today from SNIA Persistent Memory and NVDIMM SIG, SMART Modular, AgigA Tech, and Viking Technology over lunch, at breaks, and during the evening Networking Reception.  Additional sponsorship opportunities are available to SNIA and non-SNIA member companies – learn more. New Companion Event to the Summit – Persistent Memory Programming Hackathon Wednesday January 23, 2019 9:00 am – 2:00 pm Join us for the inaugural PM Programming Hackathon on the day before the Summit –a half-day program designed to get software developers an understanding of the various tiers and modes of Persistent Memory and what existing methods are available to access them.  Learn more and register at https://www.snia.org/pm-summit/hackathon

Olivia Rhye

Product Manager, SNIA

Find a similar article by tags

Leave a Reply

Comments

Name

Email Adress

Website

Save my name, email, and website in this browser for the next time I comment.

Subscribe to persistent memory