ESXi 6.0 & Win 10 1809: Slow disk performance on RAID datastore

[Meathook]

Hello Everyone,

I'm currently facing a strange issue, where a Windows 10 1809 VM exhibits slow disk performance on our ESXi 6.0 hypervisor. For example, copying a file within the same folder will show write rates of about 10MB/s max.

The VM is currently running on a datastore consiting of a RAID-5 located within the server itself. I have 2 other VMs running on this datastore, both Windows Server 2012, and both show write rates of about 70-80MB/s, which seems much more acceptable to me. I also migrated the Windows 10 VM to a backup-datastore located on a QNAP NAS, here write speeds were about 50MB/s.

Looking at the performance graphs of the ESXi, it looks like the Win10 VM is somehow throttled to 10MB/s on the datastore, while the other VMs can happily go up to 30MB/s, so I'm kinda at a loss here.

Red marks the beginning/end of the file copy on the Win 10 VM

Green marks the beginning/end of the file copy on one of the Windows Server 2012 R2 machines

Task Manager looks like this during the file copy:

Performance Graph of the Win 10 VM (Datastore):

And CPU:

This are the current settings of the VM:

Here is what I tried so far:

- Disabling Anti Virus software

- Disabling Network Adapter

- Adding more vCPU and RAM

- Trying less vCPU and RAM

- Enable "Optimize Drives" Service

- Setting Custom Pagefile Size

- Disabling Accelerated Graphics

- Adding a new VMDK and copying there

- Using VMware OS Optimization Tool

- Cloning the VM

Has anyone encountered a similar issue yet and can help me solve this? I'm currently not sure how to proceed any further. If you need more information about the system, please let me know.

[Meathook]

No ideas? Any silver lining on the horizon would be appreciated. I'm honestly out of ideas, and at this point I'm pondering just walking into the sea.

[Meathook]

I'm still struggling with this issue. If anyone has the slightest hint of an idea, it would be really appreciated. Some more informations about the system:

ESXi Version:

[root@esxi:~] esxcli system version get

   Product: VMware ESXi

   Version: 6.0.0

   Build: Releasebuild-3825889

   Update: 2

   Patch: 37

Hardware Info:

[root@esxi:~] esxcfg-info | less -I

+Host :

   \==+Hardware Info :

      |----BIOS UUID................................................0xe 0x60 0x3e 0x9 0xe8 0x11 0xb6 0x41 0x8d 0xa5 0x66 0xde 0x56 0xe9 0xf8 0x87

      |----Product Name.............................................PRIMERGY TX2540 M1

      |----Vendor Name..............................................FUJITSU

      |----Serial Number............................................YLQK025729

      |----Hardware Uptime..........................................16362760340202

      |----Ipmi Supported...........................................true

      \==+PCI Info :

         \==+All Pci Devices :

            \==+PCI Device :

|----Segment.........................................0x0000

|----Bus.............................................0x00

|----Slot............................................0x00

|----Function........................................0x00

               |----Runtime Owner...................................vmkernel

               |----Has Configured Owner............................false

               |----Configured Owner................................

               |----Vendor Id.......................................0x8086

               |----Device Id.......................................0x0e00

               |----Sub-Vendor Id...................................0x1734

               |----Sub-Device Id...................................0x11f8

               |----Vendor Name.....................................Intel Corporation

               |----Device Name.....................................Xeon E7 v2/Xeon E5 v2/Core i7 DMI2

               |----Device Class....................................1536

               |----Device Class Name...............................Host bridge

               |----PIC Line........................................0

               |----Old IRQ.........................................255

|----Vector..........................................0

               |----PCI Pin.........................................0

               |----Spawned Bus.....................................0

|----Flags...........................................521

               \==+BAR Info :

                  \==+BAR0 :

|----Type......................................0

|----Address...................................0

|----Size......................................0

|----Flags.....................................0

                  \==+BAR1 :

|----Type......................................0

|----Address...................................0

|----Size......................................0

|----Flags.....................................0

                  \==+BAR2 :

|----Type......................................0

|----Address...................................0

|----Size......................................0

|----Flags.....................................0

                  \==+BAR3 :

|----Type......................................0

|----Address...................................0

|----Size......................................0

|----Flags.....................................0

                  \==+BAR4 :

|----Type......................................0

|----Address...................................0

|----Size......................................0

|----Flags.....................................0

                  \==+BAR5 :

|----Type......................................0

|----Address...................................0

|----Size......................................0

|----Flags.....................................0

               |----Module Id.......................................4294967295

|----Chassis.........................................0

               |----Physical Slot...................................4294967295

               |----VmKernel Device Name............................PCIe RP[0000:00:00.0]

               |----Slot Description................................

               |----Passthru Capable................................false

               |----Parent Device...................................

               |----Dependent Device................................

               |----Reset Method....................................5

               |----FPT Shareable...................................true

            \==+PCI Device :

|----Segment.........................................0x0000

|----Bus.............................................0x00

|----Slot............................................0x01

|----Function........................................0x00

               |----Runtime Owner...................................vmkernel

               |----Has Configured Owner............................false

               |----Configured Owner................................

               |----Vendor Id.......................................0x8086

               |----Device Id.......................................0x0e02

               |----Sub-Vendor Id...................................0x0000

               |----Sub-Device Id...................................0x0000

               |----Vendor Name.....................................Intel Corporation

               |----Device Name.....................................Xeon E7 v2/Xeon E5 v2/Core i7 PCI Express Root Port 1a

               |----Device Class....................................1540

               |----Device Class Name...............................PCI bridge

               |----PIC Line........................................11

               |----Old IRQ.........................................255

|----Vector..........................................0

               |----PCI Pin.........................................0

               |----Spawned Bus.....................................1

|----Flags...........................................515

               \==+BAR Info :

                  \==+BAR0 :

|----Type......................................0

|----Address...................................0

|----Size......................................0

|----Flags.....................................0

                  \==+BAR1 :

|----Type......................................0

|----Address...................................0

|----Size......................................0

|----Flags.....................................0

               |----Module Id.......................................0

|----Chassis.........................................0

               |----Physical Slot...................................4294967295

               |----VmKernel Device Name............................PCIe RP[0000:00:01.0]

               |----Slot Description................................

               |----Passthru Capable................................false

               |----Parent Device...................................

               |----Dependent Device................................

               |----Reset Method....................................5

               |----FPT Shareable...................................true

            \==+PCI Device :

|----Segment.........................................0x0000

|----Bus.............................................0x00

|----Slot............................................0x01

|----Function........................................0x01

               |----Runtime Owner...................................vmkernel

               |----Has Configured Owner............................false

               |----Configured Owner................................

               |----Vendor Id.......................................0x8086

               |----Device Id.......................................0x0e03

               |----Sub-Vendor Id...................................0x0000

               |----Sub-Device Id...................................0x0000

               |----Vendor Name.....................................Intel Corporation

               |----Device Name.....................................Xeon E7 v2/Xeon E5 v2/Core i7 PCI Express Root Port 1b

               |----Device Class....................................1540

               |----Device Class Name...............................PCI bridge

               |----PIC Line........................................11

               |----Old IRQ.........................................255

|----Vector..........................................0

               |----PCI Pin.........................................0

               |----Spawned Bus.....................................2

|----Flags...........................................515

               \==+BAR Info :

                  \==+BAR0 :

|----Type......................................0

|----Address...................................0

|----Size......................................0

|----Flags.....................................0

                  \==+BAR1 :

|----Type......................................0

|----Address...................................0

|----Size......................................0

|----Flags.....................................0

               |----Module Id.......................................0

|----Chassis.........................................0

               |----Physical Slot...................................4294967295

               |----VmKernel Device Name............................PCIe RP[0000:00:01.1]

               |----Slot Description................................

               |----Passthru Capable................................false

               |----Parent Device...................................

               |----Dependent Device................................

               |----Reset Method....................................5

               |----FPT Shareable...................................true

            \==+PCI Device :

|----Segment.........................................0x0000

|----Bus.............................................0x00

|----Slot............................................0x03

|----Function........................................0x00

               |----Runtime Owner...................................vmkernel

               |----Has Configured Owner............................false

               |----Configured Owner................................

               |----Vendor Id.......................................0x8086

               |----Device Id.......................................0x0e08

               |----Sub-Vendor Id...................................0x0000

               |----Sub-Device Id...................................0x0000

               |----Vendor Name.....................................Intel Corporation

               |----Device Name.....................................Xeon E7 v2/Xeon E5 v2/Core i7 PCI Express Root Port 3a

               |----Device Class....................................1540

               |----Device Class Name...............................PCI bridge

               |----PIC Line........................................11

               |----Old IRQ.........................................255

|----Vector..........................................0

               |----PCI Pin.........................................0

               |----Spawned Bus.....................................3

|----Flags...........................................515

               \==+BAR Info :

                  \==+BAR0 :

|----Type......................................0

|----Address...................................0

|----Size......................................0

|----Flags.....................................0

                  \==+BAR1 :

|----Type......................................0

|----Address...................................0

|----Size......................................0

|----Flags.....................................0

               |----Module Id.......................................0

|----Chassis.........................................0

               |----Physical Slot...................................4294967295

               |----VmKernel Device Name............................PCIe RP[0000:00:03.0]

               |----Slot Description................................

               |----Passthru Capable................................false

               |----Parent Device...................................

               |----Dependent Device................................

               |----Reset Method....................................5

               |----FPT Shareable...................................true

            \==+PCI Device :

|----Segment.........................................0x0000

|----Bus.............................................0x00

|----Slot............................................0x05

|----Function........................................0x00

               |----Runtime Owner...................................vmkernel

               |----Has Configured Owner............................false

               |----Configured Owner................................

               |----Vendor Id.......................................0x8086

               |----Device Id.......................................0x0e28

               |----Sub-Vendor Id...................................0x1734

               |----Sub-Device Id...................................0x11f8

               |----Vendor Name.....................................Intel Corporation

               |----Device Name.....................................Xeon E7 v2/Xeon E5 v2/Core i7 VTd/Memory Map/Misc

               |----Device Class....................................2176

               |----Device Class Name...............................System peripheral

               |----PIC Line........................................255

               |----Old IRQ.........................................255

|----Vector..........................................0

               |----PCI Pin.........................................255

               |----Spawned Bus.....................................0

|----Flags...........................................520

               \==+BAR Info :

                  \==+BAR0 :

|----Type......................................0

|----Address...................................0

|----Size......................................0

|----Flags.....................................0

                  \==+BAR1 :

|----Type......................................0

|----Address...................................0

|----Size......................................0

|----Flags.....................................0

                  \==+BAR2 :

|----Type......................................0

|----Address...................................0

|----Size......................................0

|----Flags.....................................0

                  \==+BAR3 :

|----Type......................................0

|----Address...................................0

|----Size......................................0

|----Flags.....................................0

                  \==+BAR4 :

|----Type......................................0

|----Address...................................0

|----Size......................................0

|----Flags.....................................0

                  \==+BAR5 :

|----Type......................................0

|----Address...................................0

|----Size......................................0

|----Flags.....................................0

               |----Module Id.......................................4294967295

|----Chassis.........................................0

               |----Physical Slot...................................4294967295

               |----VmKernel Device Name............................

               |----Slot Description................................

               |----Passthru Capable................................false

               |----Parent Device...................................

               |----Dependent Device................................

               |----Reset Method....................................5

               |----FPT Shareable...................................true

            \==+PCI Device :

|----Segment.........................................0x0000

|----Bus.............................................0x00

|----Slot............................................0x05

[root@esxi:~] esxcfg-scsidevs -l | egrep -i 'display name|vendor'

   Display Name: Local HL-DT-ST CD-ROM (mpx.vmhba0:C0:T0:L0)

   Vendor: HL-DT-ST  Model: DVDRAM GHC0N      Revis: WF01

   Display Name: Local USB Direct-Access (mpx.vmhba32:C0:T0:L0)

   Vendor: InnoDisk  Model: USB EDC H 2SE     Revis: 0225

   Display Name: Local LSI Disk (naa.600300570286867020f90a8027796c4c)

   Vendor: LSI       Model: MR SAS 6G 1GB     Revis: 3.22

   Display Name: QNAP iSCSI Disk (naa.6e843b60639252add174d4aeed81e0d8)

   Vendor: QNAP      Model: iSCSI Storage     Revis: 4.0

   Display Name: QNAP iSCSI Disk (naa.6e843b66489ff06d52e4d4b4ddba4dd0)

   Vendor: QNAP      Model: iSCSI Storage     Revis: 4.0

According to the compatibility guides, the system should be supported:

Am I still missing anything here?

[daphnissov]

Windows 10 1809 build is brand new and almost certainly not supported on that build of ESXi. It's also known to cause problems with other current versions of other VMware products. So, like with most bleeding-edge releases, my recommendation would be to not use it at the moment and, if you had to, to do it on the latest available version of ESXi and not a build from years ago.

[Meathook]

Hello daphnissov,

thank you for your reply, it's actually a good point you brought up, that I should keep in mind working in this environment. I have migrated an older Win10 VM onto the host, running Build 1609, the issue still persists.

I also tested the original 1809 Win10 Vm on another ESXi host, running ESXi 6.0 U1a, Build 3073146, I didn't notice any performance issue here, though that test is questionable at best, as it is not the same hardware.

It could be a driver issue for all I know, but since this is a productive machine located a few 100km away, I would prefer to do a reinstall of the ESXi as a last resort action.

Nevertheless, good point that I will keep in mind during my investigations on this issue, daphnissov, and thank you again for actually responding to this topic!

[ChrisFD2]

What's the disk provisioning? Thick/thin etc?

Have you tried with a paravirtual adapter as you're using LSI? Not that it should matter, but just some things you can try if you can't update the host version. I would be wanting to run it on at least 6.5 if not 6.7 (or 6.7 update 1).

[Meathook]

Hello ChrisFD2,

I just tested running the VM using the Paravirtual adapter, sadly this din't improve the performance in any way.

The disk of the VM is configured as Thick-Provision, Lazy-Zeroed.

Still thank you very much for the suggestion, I didn't test the paravirtual adapter before and it was worth a try.

[ChrisFD2]

Okay, well we know the local RAID5 is performing (well, I wouldn't say performing, but then again RAID5 has poor write performance), due to the other VMs getting higher throughput. If VMware tools is running and up to date I would suggest updating the host and trying again, or looking at the OS itself, since we can eliminate the host and RAID array.