A couple of months ago, we posted a video on social media where we unboxed Intel's Optane DC Persistent Memory and discussed how this new technology has the potential to revolutionise storage infrastructure in the data centre.
Since then, we’ve been busy running performance benchmarks to see just how fast these Intel Optane Persistent Memory devices really are. In this blog article we present our findings.
What is Persistent Memory?
Intel’s persistent memory technology, also known as DCPMM, is a new type of storage technology that provides non-volatile storage at near memory speeds. In other words, data that is stored on DCPMM can be accessed at ultra-high speeds (comparable with data stored in memory), with the benefit that the data persists across reboots.
The following table shows how Persistent Memory bridges the gap between byte-addressable memory and block-addressable storage:
For the rest of this article we will use PMEM as an abbreviation for Persistent Memory.
PMEM in a VMware VM?
VMware announced support for PMEM devices with the latest version of their software. This means that virtual machines running on a VMware node with PMEM installed will be able to leverage PMEM technology.
There are two ways of consuming PMEM in a VMware VM:
- vPMEMDisk In vPMEMDisk mode, PMEM is presented as a regular hard drive attached to the VM. From the guest OS perspective, it is unaware that the hard drive resides on PMEM. The advantage of this is that it works with legacy applications, and you will see significant performance improvements compared to traditional storage. However, as the OS accesses the PMEM through the vSCSI bus, it is unable to leverage the full capabilities of PMEM.
Figure 1: vPMEMDisk mode - /dev/sdc is backed by PMEM, but to the guest OS it just looks like a regular hard drive.
- vPMEM In vPMEM mode, PMEM is presented as a NVDIMM device attached to the VM. In this mode, the guest OS is able to directly access PMEM through the memory bus. In order to leverage the full capabilities of PMEM, your application will also need to be PMEM-aware. This is where your application issues memory calls (load/store/commit) directly to the PMEM device instead of issuing storage calls (fsync/write/read).
Figure 2: vPMEM mode - /dev/pmem0 is a new type of device that can be seen by the guest OS.
The following table ranks the different PMEM modes in terms of performance, from most optimal to least optimal:
- Running a PMEM-aware application (e.g. SQL Server 2019 or FIO) in vPMEM mode
- Running a legacy application in vPMEM mode
- Running a legacy application in vPMEMDisk mode
What are the implications of PMEM?
Advantages:
- Quite simply, performance. With PMEM you get close to the speeds of DRAM, but with the benefit that the data persists through power cycles.
Disadvantages:
- Single point of failure. With PMEM, all you data is stored on a single device. If that device fails, or the data becomes corrupt, then you’ll have to restore from backup.
- VMware HA is not supported. In the event of physical hardware failure, your VM will not be automatically restarted on another host in the cluster.
Enough theory, give us the results of the performance benchmarks!
FIO is an open-source I/O benchmarking tool which allows us to test the raw I/O performance of underlying storage devices.
In our first test, we measured the number of IOPS we could achieve with a 4K block size. With vPMEM, we were able to hit 2 million IOPS, obtained during the random read test case. In comparison, with the same test we achieved 155,000 IOPS with local SSD, and 110,000 IOPS on a SAN backed disk.
In our next test, we measured the throughput we could achieve with a 512K block size. In the random read test case, we achieved 11GBps with the vPMEM device, compared to 1.2GBps with SAN, and 1,1GBps with local SSD.
In our final test, we measured the latency in microseconds with a 512B block size. Accessing the vPMEM device was sub-microsecond, while the local SSD access times were approximately 200 microseconds, and SAN approximately 400 microseconds.
In addition to raw I/O performance tests, we also wanted to measure how PMEM will impact application performance, where there are many other factors in play.
SQL Server 2019 on Linux supports what’s known as “enlightenment”, which essentially means that SQL is PMEM-aware. Database files can be placed directly on PMEM devices, thereby reducing latency considerably. If you’re used to seeing WRITELOG waits in your SQL database, then PMEM will significantly improve your performance.
To load test SQL Server, we used an application called ’HammerDB’, which is an industry standard database benchmarking tool. The performance metric measured by HammerDB is the number of transactions per minute (TPM). Our test showed that moving the database and transaction logs to vPMEM improved the TPM by 1.6x compared to local SSD.
Conclusion
In summary:
- vPMEM gives up to 10x improved raw I/O performance compared to SSD.
- I/O latency with vPMEM is under 1 microsecond, compared to over 400 microseconds with SAN based disks.
- 60% application performance improvement with vPMEM compared to SSD.
If you would like to see us offer PMEM in GleSYS Cloud, then please let us know by sending an email to support@glesys.se to help us gauge interest.
