Windows Server 2012 R2 Bad Performance

[DXK5314]

Hello,

I've a vSphere cluster running with 7x IBM HS22V blades / 2x Intel Xeon X3650 / 256 GB memory in every host.

Running at vSphere 5.5 23202651, with around 150 VM's.

Since a few months we are implementing Windows Server 2012 R2, and since a few weeks we've a new Citrix environment running at 2012 R2 to.

Currently the performance of the Windows Server 2012 machines is really really bad.

The respond slow and it doesn't feel right, the older VM's (W2K8 R2) are running great.

I've searched the internet, and I found that a lot of people had/have problems running 2012 on vSphere.

But I didn't found any solutions or things I could try to solve our problems.

Hopefully you guys have some tips for me, because I'm stuck...

Thanks in advance!

Best regards,

DXK

[rblaas]

I am very interested in what VMWare will say about this issue.

I just installed a fresh Windows Server 2016 on our 6.5 vSphere platform using HW level 13 and it is not performing at all. I can't even update.

VMware tools are installed

Running on 2 CPU

4GB Ram

VMXNET3 Network

LSI Logic controller

(if important running on BIOS)

[peterriederer]

i am wondering, why this issue didnt show up earlier from more customers. vsphere 6.5 is already out some time now.

[adamsavagex]

I upgraded my ESXi hosts from 6.0 versus doing a clean install.. Is that the same for the rest of everybody else or were there some clean installs of ESXi hat are seeing the same issue?

[mvduijn]

All our ESXi hosts were upgraded from 6.0

[adamsavagex]

Hey mvduijn I need some confirmation.. I've been doing multiple tests today to try and resolve the issue while being on Hardware 13.. Can you install ALL of the features offered up in the VMware Tools install - specifically the VMCI drivers on a Server 2012 R2 system? i went to custom and installed from there.. I don't want to jump the gun but i did this on my 2012 R2 HW13 server and the queue length appears to be gone but i need confirmation from someone else before I can go back to my VMware ticket and point the finger at VMCI.

Thank you,

Adam

[peterriederer]

Hey Adam,

which VMware Tools Version did you use? i will try to test it on one of our machines. Currently we are are on 10.1.0 shipped with vSphere 6.5 but also tested 10.1.5.

You installed all VMCI-Options or just File and/or Network introspection driver ?

2 ESXi Hosts, where upgraded from 5-> 5.5->6->6.5

1 ESXi Host, was clean installed with 6 then upgraded to 6.5

Now 10.1.7 is out since a week which solved some issues with VMNEXT NIC Drivers:

Receive Side Scaling is not functional for vmxnet3 on Windows 8 and Windows 2012 Server or later

This issue is caused by an update for the vmxnet3 driver that addressed RSS features added in NDIS version 6.30 rendering the functionality unusable. It is observed in VMXNET3 driver versions from 1.6.6.0 to 1.7.3.0.

Regards

Peter

[mvduijn]

Hello adamsavagex,

I will install the vmware tools with VMCI driver tonight to one of our servers to see if there is any difference.

If the disk queue length is gone i will let you know.

peterriederer​

Currently we are are on 10.1.0 shipped with vSphere 6.5 but also tested 10.1.5.

I missed the release of VMware tools 10.1.7 and will give this a try to.

Most of our ESXi servers where upgraded from 5.5->6->6.5

What server brand and storage do you use?

We have HP Proliant BL460c Gen8 servers with HP 3par storage

Maybe i give it a shot to install a server from scratch with the latest HP Custom image.

Thank you,

Mike

[peterriederer]

Hello Mike,

i tried 10.1.5 and 10.1.7 with and without VMCI -> NSX File and Network Option in Setup for some 2016 Servers (Cannot Test with 2012R2 Fileservers at the moment) and it seems / "feels" a bit better with 10.1.7 but the issue still happens from time to time, didnt figured out exactly when. Getting load while Starting a full scan with Anti-Virus e.g. is no problem at the moment.

The Update from 10.1.0 to 10.1.5 or 10.1.7 took a very long time, Disk Queue up to 5 or 10 and more than 1000ms latency, 2 machines freezed completely and i had to hardreset them.

We have Fujitsu Primergy RX300 S8 Hosts, connected to a DataCore SANSymphony-V Storage System via iSCSI 10 GB.

BTW: We also have some 2008R2 Servers running with Tools 10.1.0 and HW13 there are not such Problems. Only 2012R and 2016 Servers.

Do you have still some 2008R2 Servers too, without any issues?

Regards Peter

[adamsavagex]

Hey Guys,

I've had this happen on 2012 R2 VMs only on Hardware 11 and 13 (we dont run any 2016 yet in our Datacenter)..

Our VM Tools versions vary between some of the 2012R2 VMs

They got from 9.10.5, 10.1.5, and 10.1.7 and they all have the same problem.. I was hoping the VMCI drive would have fixed the issue but alas like you said too it "looks" to perform better but in the end it's still crappy performance..

I have Cisco UCS B440s. 230s, 200s, HP DL980s, HP BL490C, Dell FC630, and Dell R610s on EMC VNX/CX/VMAX/Xtreme-IO Storage. The problem happens on all platforms and storage types. Although on the all SSD storage the performance is fine because it's being masked by the extreme performance of the SSDs.

This is the last response I received from VMware Support last night.

"We are still investigating this issue. We will keep you posted about this case.

I look forward to your response. Feel free to contact me if you have any questions or clarification

Thank you for choosing VMware!"

Hopefully they will come back with an answer soon. I'm running out of ideas here. I noticed that between a v9 and v11 2012 R2 VM there are multiple Generic Bus Devices under Other Devices in device manager on the v11 system versus just not being there on the v9 VM.

I'll keep you guys updated with anything new from VMware.

[controlsforall]

I found the problem to be crashPlan software, it speeds up a lot but the memory leak still exists with  vsphere 6.5 with latest patch.

[peterriederer]

Hello Adam,

just wanted to know if you already have any new information, or did some new tests which might be helpful?

Regards

Peter

[adamsavagex]

Hey Peter - I just got an email back from VMware today wanting to setup a webex. So nothing yet.. I've done some more testing specifically on my VSAN Cluster which is a Dell FX2 chassis running FC630s and i can't seem to re-produce the problem there - but I can on my Cisco UCS blades.. That may be because my VSAN is all-flash and wouldnt see the queue length issue because its so fast.

I'll keep you in the loop when and if I get a resolution out of VMware.

[adamsavagex]

Good afternoon everyone,

I just got off the phone with VMware and they have identified the problem and it will be fixed in 6.5 Update1. Update1 will be released sometime in July. I'm not sure of the exact day for the release.

Here is the summary of my call with VMware.

As per the current issue, we see that the disk Queueing is recorded once the hardware version for the VM is higher than the 10. The issue is only seen when the host is installed with vSphere 6.5 and the hardware version is higher than 10.

We already have the same reported with the engineering team and we are currently working on the same. The current workaround is to retain the hardware version to 10 or lower where the OS version is set to Win 2012 or higher. This issue is temporarily pointing to be caused due to the virtual disk polling enhancement. We do not have a definite solution yet and are currently working on the same.

As we have agreed, your support request is ready to be closed. If you come across any additional queries within 21 days, please contact me and I will be happy to reopen the ticket and continue working with you.

[peterriederer]

Thanks Adam

these are really good news, except for waiting so long till the update will arrive.

[mvduijn]

Hello Adam,

Good to hear there you have some progress on the issue.

Since last friday the problem is getting worse every day.

A simple copy job of 5GB takes about 9 minutes to complete.

Disk queue length on VM's are skyhigh.

Our storage provider and HP Enterprise have finished their investigating and confirming this is not a storage Hardware issue.

Can you share your call number of VMware so we can  communicate this with our VMware support engineer to confirm we have the same issue?

Thank you in advance.

Mike

[adamsavagex]

My support request number was 17467861205

[AdamViusaSerrat]

The same issue is happening to us. Vsphere 6.5, virtual hardware verssion 13 with windows 2016 and windows 2012 is not working well. In one w2012 R2 we have a file server and intermitent disconnections occurred... a big disaster as we had to downgrade the virtual hardware to all 2012 an 2016 vms.

After downgrade to virtual hardware 10 (w2012) and virtual hardware 11 (w2016) the issue disappears and all comes to life as usual...

[peterriederer]

and its hard doing a downgrade of the virtual Hardware.

at the moment i just create a new VM with HW10 (2012R2) or HW11 (2016) and connect the VMDKs in there, or is there an other more easier way?

[adamsavagex]

You could throw the VM through VMware Converter and select the Hardware Version from there.. Then delete the old VM once you get the new one online.

All ways are kind of a PITA.

[AdamViusaSerrat]

But this takes a lot of time if the vmdks are big and it's not an option if your users can't work and all enterprise is waiting your "repair" of the situation. Create a new machine and attach the vmdks is the faster solution I think... but I'm going to open a new ticket also for this bug.