I’ve been generally disappointed with the performance of my MS S2D storage array due to the way MS lays writes down onto parity pools. All the SSD caching and NVMe write cache just wasn’t doing enough and it was time to go a different route. I’ve written about my Datrium install a few times now, and for those who are unfamiliar with the solution the short version is that Datrium takes host populated flash and builds a dedicated Read / Write pool on the host. It does an extremely good job of this and if you would like to know more check out their website here https://www.datrium.com/open-convergence-architecture I like that I am able to populate my host with “cheap[” SSD for performance, and then dump the data to cheaper spinning disk for cold storage.
So how can I do this at home without spending a bunch of money? My first though was Dell’s Cachecade which I have on the PERC H700 I used for this test. The problem with Cachecade however is that it has a hard limitation of 512GB per host, reads above 64kb are not cached, and there is no support for Write cache. I then thought about VSAN, and using the vFlash caching in that product, however given my lab and limited resources, VSAN would be a no-go as well.
Finally I came to VFRC. Like cachecade it is only a read side cache, however unlike Cachecade it will cache all reads, it can generate up to a 32TB pool with a maximum of 400GB of cache per VMDK. The downsides of VFRC are that you have to enable it per VMDK, and you have to set the cache block size to the average of your workload’s IO size otherwise you will degrade the performance of the cache. That said it is extremely easy to get setup and running, and it is included in your ESXi Enterprise Plus licensing, so you don’t get hit with additional cost for VSAN licensing (though that said, VSAN cache tiering would provide a significantly better performance experience).
Let’s look at the test VM. The server is a Dell R510 w/ 2x Xeon E5640 CPUs, 64GB of RAM, and the PERC H700 1GB BBWC card. I have 3 WD RE4 2TB drives + 1 Samsung 850 Pro 256GB SSD. The server is running ESXi 6.5 U1, and the VM is a 4 vCPU 6GB vRAM PVSCSI controller VM.
First up let’s see what this machine does with a VM running directly on the OCZ RD400 NVMe AIC. This is a Thin provisioned .vmdk living on a VMFS v6 volume.
This test shows that a 8k 70/30 Random Read/Write test against a 5GB file with 30 minutes of sustained IO generated 165MBps Throughput, 21,193 IOPS, and averaged 0.3ms of read latency. This is fantastic and exactly what I would expect of an NVMe disk.
So now let’s run exactly the same test against my VFRC volume. This is a RAID 5 of 3x SATA6 2TB Western Digital RE4 drives + 50GB of VFRC at an 8k block size.
For a SATA RAID5 that’s not bad. 3.3MBps, 429 IOPS, and an average read latency of 18ms. This isn’t amazing performance by any means, but again for a 3 disk SATA array this is pretty fantastic.
Let’s compare that to a disk with no vFlash Cache.
1.87MBps, 239 IOPS, and 33ms average latencyu. The VFRC disk is literally 2x the performance of the traditional disk running on the same array.
So as you can see VFRC is a great option for those who have traditional storage, either local or SAN, where a flash tier would normally not be available.
Finally, let’s look at the same workload on my Datrium cluster.
As you can see, this is not going to be a replacement for having dedicated flash resources, and Datrium caches much more efficiently than VMware does, however if you need a quick “Cheap” fix, VFRC is a great answer.
Like this:
Like Loading...
Get-SysadminBlog 
I’ve been generally disappointed with the performance of my MS S2D storage array due to the way MS lays writes down onto parity pools. All the SSD caching and NVMe write cache just wasn’t doing enough and it was time to go a different route. I’ve written about my Datrium install a few times now, and for those who are unfamiliar with the solution the short version is that Datrium takes host populated flash and builds a dedicated Read / Write pool on the host. It does an extremely good job of this and if you would like to know more check out their website here https://www.datrium.com/open-convergence-architecture I like that I am able to populate my host with “cheap[” SSD for performance, and then dump the data to cheaper spinning disk for cold storage.
So how can I do this at home without spending a bunch of money? My first though was Dell’s Cachecade which I have on the PERC H700 I used for this test. The problem with Cachecade however is that it has a hard limitation of 512GB per host, reads above 64kb are not cached, and there is no support for Write cache. I then thought about VSAN, and using the vFlash caching in that product, however given my lab and limited resources, VSAN would be a no-go as well.
Finally I came to VFRC. Like cachecade it is only a read side cache, however unlike Cachecade it will cache all reads, it can generate up to a 32TB pool with a maximum of 400GB of cache per VMDK. The downsides of VFRC are that you have to enable it per VMDK, and you have to set the cache block size to the average of your workload’s IO size otherwise you will degrade the performance of the cache. That said it is extremely easy to get setup and running, and it is included in your ESXi Enterprise Plus licensing, so you don’t get hit with additional cost for VSAN licensing (though that said, VSAN cache tiering would provide a significantly better performance experience).
Let’s look at the test VM. The server is a Dell R510 w/ 2x Xeon E5640 CPUs, 64GB of RAM, and the PERC H700 1GB BBWC card. I have 3 WD RE4 2TB drives + 1 Samsung 850 Pro 256GB SSD. The server is running ESXi 6.5 U1, and the VM is a 4 vCPU 6GB vRAM PVSCSI controller VM.
First up let’s see what this machine does with a VM running directly on the OCZ RD400 NVMe AIC. This is a Thin provisioned .vmdk living on a VMFS v6 volume.
This test shows that a 8k 70/30 Random Read/Write test against a 5GB file with 30 minutes of sustained IO generated 165MBps Throughput, 21,193 IOPS, and averaged 0.3ms of read latency. This is fantastic and exactly what I would expect of an NVMe disk.
So now let’s run exactly the same test against my VFRC volume. This is a RAID 5 of 3x SATA6 2TB Western Digital RE4 drives + 50GB of VFRC at an 8k block size.
For a SATA RAID5 that’s not bad. 3.3MBps, 429 IOPS, and an average read latency of 18ms. This isn’t amazing performance by any means, but again for a 3 disk SATA array this is pretty fantastic.
Let’s compare that to a disk with no vFlash Cache.
1.87MBps, 239 IOPS, and 33ms average latencyu. The VFRC disk is literally 2x the performance of the traditional disk running on the same array.
So as you can see VFRC is a great option for those who have traditional storage, either local or SAN, where a flash tier would normally not be available.
Finally, let’s look at the same workload on my Datrium cluster.
As you can see, this is not going to be a replacement for having dedicated flash resources, and Datrium caches much more efficiently than VMware does, however if you need a quick “Cheap” fix, VFRC is a great answer.
Share this:
Like this: