Windows Server 2012 R2 Bad Performance

[DXK5314]

Hello,

I've a vSphere cluster running with 7x IBM HS22V blades / 2x Intel Xeon X3650 / 256 GB memory in every host.

Running at vSphere 5.5 23202651, with around 150 VM's.

Since a few months we are implementing Windows Server 2012 R2, and since a few weeks we've a new Citrix environment running at 2012 R2 to.

Currently the performance of the Windows Server 2012 machines is really really bad.

The respond slow and it doesn't feel right, the older VM's (W2K8 R2) are running great.

I've searched the internet, and I found that a lot of people had/have problems running 2012 on vSphere.

But I didn't found any solutions or things I could try to solve our problems.

Hopefully you guys have some tips for me, because I'm stuck...

Thanks in advance!

Best regards,

DXK

[vSphere_Admin]

to downgrade the virtual hardware just download and edit the vmx file from the VM datastore folder and change the value from 13 to 10 or 11, save the file to your desktop and upload back into the VM datastore folder, see links below.

How to Downgrade VMware ESXi 5.5 Virtual Machine Hardware Version to 9 from 10

Editing configuration files in VMware ESXi and ESX (1017022) | VMware KB

Note: make a backup copy of the original .vmx file just in case of any problems!

Steve

[peterriederer]

Thank you so much Steve, thats much faster and more efficient. Works like a treat.

After testing with my 2016 Servers, going back to HW11 is not enough, i had to put them back to HW10 in order to get the performance back.

Peter

[mvduijn]

Just be aware changing the vmx file to downgrade a VM is not supported by VMware!

The only supported methods to downgrade a VM are:

• Revert to a snapshot created before upgrading the virtual machine hardware.
• Use VMware vCenter Converter Standalone and select the required virtual hardware version in the Specify Destination wizard.
• Create a new virtual machine with required hardware version and attach the existing disk from the virtual machine.

Mike

[mvduijn]

Vmware contacted us yesterday to inform us about the progress of the performance issue.

Patch 1 wich will resolve this issue is delayed to 27July.

It seems the issue is related to a bug with the UNMAP command.

This process reclaims deleted blocks on thin provisioned disks and luns.

Vmware also suggested a possible workaround for Windows VM's to disable the unmap command inside the windows OS.

You can change a registry key or use VMware PowerCLI.

You can make the changes in the registry by any of these methods.

• Using a registery editor, change this key to 1:
  
  HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\FileSystem\DisableDeleteNotification
  
• Use this PowerCLI command:
  
  fsutil behavior set DisableDeleteNotify 1

The perfomance seems a little bit better.

Mike

[peterriederer]

Good to know, Thanks Mike.

Nevertheless a delay of another 4 weeks for the patch is really really bad!!!

Going to test that immediatelly. Thats also why Windows Server 2008R2 and lower are not affected.

Peter

[peterriederer]

Works much better now, Machines are fast again.

BTW: Found another thread with the same issues:ESXi 6.5 Slow vms, High "average response time"

[E07256]

This issue seems to be directly addressed in VMware KB 2129176 https://kb.vmware.com/kb/2129176​

"Symptoms

• You notice performance degradation for some client/servers workloads.
• Packets have up to a 0.5 second delay over the expected arrival in the application.
  
  This issue is observed under these conditions:
  
  
  • The guest operating system is Windows Server 2012,  Windows 8 or later.
  • The virtual machine is on hardware version 11/ ESXi 6.0 compatibility.
  • The virtual NIC is vmxnet3 and the driver version is 1.6.6.0 and later.
  • The Receive Side Coalescing (RSC) feature is enabled globally and on the vmxnet3 adapter.
  • This issue is more prevalent when:
  • Running Microsoft SQL/TDS based workloads
  • Using Jumbo Frames
  • The client and server are on two different hosts"

[...]

Resolution

This issue is resolved in ESXi 6.0 Update 2, available at VMware Downloads.

Note: Applying ESXi 6.0 Update 2 may cause a Purple Diagnostic screen.  For more information, see ESXi 6.0 Update 2 host fails with a purple diagnostic screen containing the error: Vmxnet3VMKDevRxWi.... This issue is resolved in ESXi 6.0 Patch Release ESXi600-201605001.  For more information, see VMware ESXi 6.0, Patch Release ESXi600-201605001 (2136186).

If you are unable to upgrade, you can work around any performance degradation by disabling RSC on the affected guest operating system.

To disable RSC on the virtual machine, perform one of these options:

• Use the PowerShell command Disable-NetAdapterRsc.
  
  To disable for all interfaces:
  
  Disable-NetAdapterRsc *
  
  To disable for a specific interface:
  
  Disable-NetAdapterRsc -Name Ethernetx
  
  
• Disable RSC globally using the netsh int tcp set global rsc=disabled command.
  
  Note: This command requires an elevated command prompt.

To verify that RSC is disabled on the virtual machine:

• Use the PoweShell command Get-NetAdapterRsc. The IPv4OperationalState and IPv6OperationalState should both be set to FALSE.
• Use the netsh int tcp show global command.  The Receive Segment Coalescing State should be set to disabled.

Note: This change is immediate and does not require rebooting the guest operating system.

Alternatively, you can disable LRO/RSC for all virtual machines on an ESXi host using:

esxcli system settings advanced set -o /Net/Vmxnet3SwLRO -i 0

esxcli system settings advanced set -o /Net/Vmxnet3HwLRO -i 0

Note: This will disable LRO for all virtual machines on the ESXi host. Virtual machines must be powered off and then powered on or migrated using vMotion to another hosts for the changes to take effect.

"

[PGinhoux]

Hi,

I'm surprised by the detail of the fix provided. It talks about a fix in ESXi 6.0 Update 2, whereas this case is related to the 6.5 level.

I encounter the same issue on my ESXi hosts running the update-from-esxi6.5-6.5_update01 level.

On a first ESXi host newly installed I have created a first VM created from an OVF then when I want to log in, I experience a big delay about 20 minutes) to get in.

Can somebody explain me, if possible, more about the fix provided (in ESXi 6.0 Update 2) ?

And what can I look at on my side?

Thanks in advance.