NVMe disk/vmdk: Slower than iSCSI! Where is the claimed performance gain?

[SokoFromNZ]

Hi guys,

I'm currently in my trail period of v14 before I pay for the upgrade from v12.5.

Most important feature is the performance gain of such a disk as by the VMware Workstation 14 Pro Release Notes

My first tests although are quite disappointing!

Preface

My host is Windows 7 x64 on a Dell Precision 7510 with 8 Core Xeon, 64GB RAM, two NVMe m.2 drives (Samsung 960 Pro & SM951).

I've installed the latest Samsung NVMe driver on the host and everything else is up to date as well.

All guests are up to date and all NVMe disks are shown correctly in the device manager (under the VMWare NVMe Controller).

All my test runs are done with a complete idle host and guest and are repeated a couple of times to ensure consistency.

My first test was done wit the VM on the "old" SM951 NVMe.

The VM/guest is a newly installed Windows Server 2016 with full VMWare tools installation.

I created 4 disks/vmdk with 60GB each:

The performance inside the guest isn't the same as on the host - not even close!

But whats more troubling is, that the SCSI-vmdk outperforms the new NVMe!!

I did another test with the VM/guest on the brand new 960Pro.

The VM is a Windows 7 with just one vmdk as a NVMe:

Here the "loss" is even worse!

It even seems there there is a "cap" in the maximum performance on a NVMe-vmdk, doesn't matter how fast the physical NVMe is.

Does anyone have suggestions for reasons to this test results?

Can/Should I approach VMWare directly with this issue?

Thanks heaps!

Soko

[voidp]

Very interesting!  I never would have guessed that there would be such a performance difference among the different drive types.

[SokoFromNZ]

It sure is!

I've also investigated how to contact (get support) from VMWare directly... which doesn't seem to be possible. Even when I buy the upgrade to v14 I only get support for the installation process. I have to buy at least 10 licenses to be able to buy more support

[SokoFromNZ · Accepted solution]

Heya,

I've opened a support ticket with VMWare after I purchased v14 in the beginning of this year and we finally came to a conclusion on this issue after a couple of testing, logging, etc.

The good news:

They acknowledged the problem, can reproduce it and already started to work on it.

The bad news:

Its going to be a major change in the source code of Workstation and might not be fixed in the coming months... maybe not even in v14.

Finally I'd like to point out the great and competent support I've got on this issue. Sandeep was a great representative to work with.

They will update the documentation according to this issue and might put up a knowledge-base article for it.

When it is fixed its definetly mentioned in the release notes.

so long

Soko

[TimothyHuckabay]

As of version 15.0.2 of Workstation, this is still NOT fixed.  VMware's claim of "performance improvements for virtual NVMe storage" are just hot air.  (It's funny--that is, unprofessional and disgusting--how VMware always touts "performance improvement" and then offers zero metrics, zero independent testing or reviews, zero evidence whatsoever that such "improvements" are real and exist.  This is marketing hype, and we should all expect more from a company like VMware.  Why do we put up with this?)  I have extensively tested the latest versions of Workstation Pro 15 and Workstation Pro 14, and can definitively state that the SCSI drive option continues to be faster than the NVME drive option, and both are much slower than they should be within VMs.  (Even under the SCSI option, read/write performance is DRAMATICALLY worse in the VM than on the host for NVME drives.)  I tested using a Samsung 970 Evo 2TB NVME drive in all cases for this.  There is not even a recognizable marginal improvement in NVME performance between Workstation 14 and 15.  So, VMware has yet to actually address this problem.  Notably, Workstation 15 continues to recommend SCSI drives over all others for performance, so at least that much is yet true.  NOTE: Within a VM, NVME drives particularly suffer for multithreaded reads and writes, as may be seen in the AS SSD Benchmark tool's 4K-64Thrd (i.e., 64-threads) read and write tests.

There are very few reasons to upgrade to 15 Pro from 14 Pro IMO.  Even 4K support is questionable: one can achieve workable 1080P across 1080P and 4K screens with Windows 10 (1809) for example.  (Just set the host's resolution to 1080P before launching Workstation 14 Pro, AND ensure that you have set the "high DPI settings" appropriately: using the Properties for the Workstation Pro shortcut, go to the Compatibility tab, then "Change high DPI settings," and select "Program DPI" with "I signed in to Windows" for the corresponding dropdown option.)  This said, the 4K support is nice if you do decide to fork out the bucks for 15 Pro.  If you are using 15 Pro, you may also want to enable (check) "Automatically adjust user interface size in the virtual machine" (under VM --> Settings --> Hardware --> Display).

Now, regarding overall VM performance, including drive performance, here are your best options to date: use a SCSI disk (NOT NVME), select (enable) the "Disable memory page trimming" option under VM --> Settings --> Options --> Advanced, and select "Fit all virtual machine memory into reserved host RAM" under Edit --> Preferences --> Memory.  (For this last option, you will want to ensure you have plenty of physical host RAM that you can spare (i.e., dedicate exclusively) to the VM (should the VM need it).  Lastly, if you are using 15 Pro, you may want to set Graphics memory (under VM --> Settings --> Hardware --> Display) to 3GB.

Finally, if you are working on a system that has hybrid graphics (e.g., a laptop with an nVidia video card besides in-built Intel display graphics), you may want to use the relevant "control panel" application to instruct your system to use the discrete graphics card for vmware.exe and vmware-vmx.exe.

I hope this helps.  In the meantime, everyone should pepper VMware over the poor NVME performance of Workstation Pro.

[roblevine]

Thank you for this post - I wanted to echo exactly the same sentiments.

This problem has been around for over a year now and has totally destroyed the performance of certain VMs for me on two separate machines for any tasks involving reasonable disk I/O (including compiling software, one of my primary use cases).

I also raised an issue against it here: Very poor SSD performance inside Windows 10 guest running on VMWare Workstation on a Windows 10 host .

Initially it looked there was interest from someone at VMWare, but they disappeared after asking a single question. I too have reverted to the SCSI drive option. Every time there is a Workstation update which mentions drive performance improvements, I switch back to NVMe re-run my benchmarks and come away disappointed. SCSI performs better than NVMe, but still not especially well.

VMWare - PLEASE fix this problem, or at least give us an update on progress or if you are intending to fix it. I have now stopped using VMWare for my development environments (a real pity because virtualised development environments can be a real bonus). That's two licences I won't be renewing if the issue isn't fixed.

Thanks again for updating us - I'll open another issue for this.

[daphnissov]

/cc Mikero​ (for visibility)

[roblevine]

Hi daphnissov,

Can you be a bit more specific about where you feel the problem is in the posts? Was there something in particular that you think was lacking in my original post, that I should add/update in order to make it a better request for help?

Very poor SSD performance inside Windows 10 guest running on VMWare Workstation on a Windows 10 host

I tried to be detailed in my problem statement, including evidence and also outlining experiments and investigations I'd undertaken. I also immediately followed up when asked a further question. To be honest, I was expecting to be asked for further information (such as logs, or to try changing other settings) but the thread went quiet.

I would say the same is true for this thread - people have been pretty clear on the problem, supplied some evidence and requested help.

If you have some constructive suggestions, they would be appreciated.

Thanks,

Rob

[daphnissov]

I'm mentioning the PM here for visibility only. The article is in my signature and is not part of the previous post (therefore wasn't commentary on lack of clarity on your part).

[roblevine]

Hi daphnissov,

My mistake - I totally misread that and thought you were posting the guidelines to let me know something was wrong with the original post.

(Your post was so short my brain didn't quite spot the separation of post to sig).

Thank you for helping to raise the visibility, it is very much appreciated.

Rob

[TimothyHuckabay]

Still no improvement through Workstation 15.1.  This remains the situation:

As of version 15.0.2 of Workstation, this is still NOT fixed.  VMware's claim of "performance improvements for virtual NVMe storage" are just hot air.  (It's funny--that is, unprofessional and disgusting--how VMware always touts "performance improvement" and then offers zero metrics, zero independent testing or reviews, zero evidence whatsoever that such "improvements" are real and exist.  This is marketing hype, and we should all expect more from a company like VMware.  Why do we put up with this?)  I have extensively tested the latest versions of Workstation Pro 15 and Workstation Pro 14, and can definitively state that the SCSI drive option continues to be faster than the NVME drive option, and both are much slower than they should be within VMs.  (Even under the SCSI option, read/write performance is DRAMATICALLY worse in the VM than on the host for NVME drives.)  I tested using a Samsung 970 Evo 2TB NVME drive in all cases for this.  There is not even a recognizable marginal improvement in NVME performance between Workstation 14 and 15.  So, VMware has yet to actually address this problem.  Notably, Workstation 15 continues to recommend SCSI drives over all others for performance, so at least that much is yet true.  NOTE: Within a VM, NVME drives particularly suffer for multithreaded reads and writes, as may be seen in the AS SSD Benchmark tool's 4K-64Thrd (i.e., 64-threads) read and write tests.

There are very few reasons to upgrade to 15 Pro from 14 Pro IMO.  Even 4K support is questionable: one can achieve workable 1080P across 1080P and 4K screens with Windows 10 (1809) for example.  (Just set the host's resolution to 1080P before launching Workstation 14 Pro, AND ensure that you have set the "high DPI settings" appropriately: using the Properties for the Workstation Pro shortcut, go to the Compatibility tab, then "Change high DPI settings," and select "Program DPI" with "I signed in to Windows" for the corresponding dropdown option.)  This said, the 4K support is nice if you do decide to fork out the bucks for 15 Pro.  If you are using 15 Pro, you may also want to enable (check) "Automatically adjust user interface size in the virtual machine" (under VM --> Settings --> Hardware --> Display).

Now, regarding overall VM performance, including drive performance, here are your best options to date: use a SCSI disk (NOT NVME), select (enable) the "Disable memory page trimming" option under VM --> Settings --> Options --> Advanced, and select "Fit all virtual machine memory into reserved host RAM" under Edit --> Preferences --> Memory.  (For this last option, you will want to ensure you have plenty of physical host RAM that you can spare (i.e., dedicate exclusively) to the VM (should the VM need it).  Lastly, if you are using 15 Pro, you may want to set Graphics memory (under VM --> Settings --> Hardware --> Display) to 3GB.

Finally, if you are working on a system that has hybrid graphics (e.g., a laptop with an nVidia video card besides in-built Intel display graphics), you may want to use the relevant "control panel" application to instruct your system to use the discrete graphics card for vmware.exe and vmware-vmx.exe.

I hope this helps.  In the meantime, everyone should pepper VMware over the poor NVME performance of Workstation Pro."

Clearly, Workstation has some sort of performance LIMITATION / RESTRICTION in-built where is disk drivers are concerned: I have found that no matter what nvme drive I use, the performance level does NOT change, from the most expensive and performant drives (e.g., 970 Pro 1TB) to the cheapest ones.  It looks like we are going to have to bail on VmWare for better drive performance, and go to virtualization products that offer real nvme drivers with corresponding performance improvements in virtual environments.  VirtualBox anyone?  (I am tired of being scammed by VmWare with its phony claims of "improved performance!") 

[indothai]

v15.1 and the problem still exists.

[roblevine]

Ditto in VMWare Workstation 15.5.

I've given up, and stopped using VMWare Workstation and moved my development workflow to my main OS rather than in a VM.

Such a pity - I've used VMs for all my dev for 15+ years, but there is such a significant performance issue for anything even vaguely IO intensive, and it's been here for several years, on multiple host machines (all running Windows 10) for multiple guests (all Windows 10).

Apart from the apparent interest from VMWare at the beginning of this thread, there's been basically nothing. I'm pretty sure they're not working on this or monitoring these threads as we'd hear the odd update.

[hammerm]

As of  VMware Workstation version 15.5.6 build-16341506 (on CentOS 7), this problem still exists. Switching to SCSI hoping it will help disk performance. But of course converting to SCSI which uses vs NVMe's EFI boot method does not appear to be a trivial task. This is a production system with 7 Windows guests, one of them a Win2019 server AD server in a single server environment with 2.4 TB of share user data. So outages have to arranged and planned with lengthy backup which complicate and prolong outages.

If anyone can point to a tool or procedure to do revert to SCSI from NVMe VMDK, it would be greatly appreciated.

[wila]

Hi,

The trick is to first install the drivers...
The best/easiest way to do so is to temporarily add a new disk of the disk type you want to use so that Windows configures the required drivers. Then shut down, remove the temp disk again and change the disk type you wanted to change.
Switching between EFI and BIOS is another matter (not sure why you would even want that, once you're on EFI there is not reason to go back to BIOS)

Note that it is _always_ a good idea to have good backups before you do any of this, especially with production style VMs.

--
Wil

[FlirtyToad]

The WKST Pro 16.1.2 NVMe driver is still slower than the SCSI driver. I tested this on a WIN10-21H1 Guest and the RND4K-Q1T1 NVMe is about 40% slower than the SCSI driver (using CrystalDIskMark 8.0.4).

Also the RND4K-Q1T1 I/O is about 5 times slower in the Guest compared to bare-metal host I/O:

See my other post 

-Toad

[Br_Silva]

Goodnight!

I'm having this same problem on Vmware Workstation version 17.0.2.

Were you able to find out the reason for the problem?