VERY Bad Performance on NVMe vs RAID on vSphere 6.7 - what gives?
I picked up a few Samsung 970 Pro NVMe (NAND 3D) (256GB) flash drives along with these Vantec M2 NVMe SSD PCI-e x4 Adapters (model # UGT-M2PC100). put them in a Dell R820. I was expecting to use the NVMe as flash caches (vFlash, vSAN, VirtuCache on my iSCSI, etc), as swap storage, etc. but the speed is miserable.
I ran tests with the VMWare I/O Analyzer appliance, one appliance running with testing on the NVMe and the other on a 8x 600GB 10k SAS RAID-6 w/ Dell PERC H710p 512MB RAID card. I ran a variety of tests, from IOPS / Max Throughput / Max Write Throughput / Max Write IOPS etc, but this one test of 512K 50% read 100% random really showed the speed problem. I figured this kind of test is more like regular use with lots of random accesses, which the NVMe should accel at.
Note from images: ioanalyzer is the NVMe, ioanalyzer4 is running on the RAID.
See test output below of the 512k_50%Read_100%Random.
RAID Performance (much better):
What is going on here? Why is this NVMe so much slower? Again, this isn't the only test I ran - I have ran many and while in some the sustained IOPS are a bit higher on the NVMe, its nothing spectacular and the random read and writes are horrid. These are Samsung 970 PRO NVMe M.2 V-NAND 3Ds
Any ideas? I am getting good feedback and responses on the VMware sub-reddit re: this exact thread. I was hoping I could get some chime-ins from VMware staff or certificated individuals, individuals who focus on a labor of love to their home-lab with experimental setup similar to this, etc.