Re: ESXi 5.5 U1 freezes Win7 and 2008R2 VMs

ViennaAustria · ‎07-31-2014

We recently upgraded many 5.5 hosts to 5.5 U1. Since then, several VMs with Windows 7 64 Bit or Windows 2008R2 64 Bit guest OS freeze - sometimes twice a week, usually once a day, in extreme cases every hour! Sometimes we see a bluescreen (everytime some other), mostly the OS is just frozen (you see the desktop but it won't respond to mouse/keyboard input). We also had a frozen taskbar (no response to windows key), but the explorer itself (icons on the desktop) did respond. But after a few klicks with the mouse it froze, too. In any cases you have to reset the VM.

These Issues also occoured on a freshly installed 5.5.U1 machine with a freshly installed Win7 VM.

As we couldn't find any problem in the guest OS, we tried to downgrade several hosts to ESXi 5.5 with no updates (build 1331820) by reinstallation last weekend. All problems are gone since 5 days now. On hosts where we didn't yet perform the downgrade/reinstallation, the VMs still freeze.

So 5.5.U1 definitely has a problem. [at lease on Supermicro X9SCM-F mainboards, which we mostly use]

As far as I remember, when we updated 5.5 hosts to build 1892794 we experienced no problems of that kind. But I can't say for sure. We'll try that after my vacation.

Anybody else having problems with 5.5.U1?

DavoudTeimouri · ‎07-31-2014

Hi,

Do you have NFS on your environment? vSphere 5.5u1 has NFS issue and you need to apply a patch.

-------------------------------------------------------------------------------------
Davoud Teimouri - https://www.teimouri.net - Twitter: @davoud_teimouri Facebook: https://www.facebook.com/teimouri.net/

ViennaAustria · ‎07-31-2014

Yes, many have NFS datastores.

Where to get the patch? The Update Manager of the vCenter hosts reports "compliant" for critical and non-critical patches.

DavoudTeimouri · ‎07-31-2014

Hi,

Please read this blog:http://www.yellow-bricks.com/2014/06/11/vsphere-5-5-u1-patch-released-nfs-apd-problem/

And this article:http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=207639...

You can download the patch from VMware Patch Portal.

-------------------------------------------------------------------------------------
Davoud Teimouri - https://www.teimouri.net - Twitter: @davoud_teimouri Facebook: https://www.facebook.com/teimouri.net/

ViennaAustria · ‎07-31-2014

Fond a description of the issue:

http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=207639...

I'll try that.

But I don't believe, that this issue is responsible for our problems, because only Win7 64 Bit and 2008R2 guests are affectet. All other guests on the very same host/datastore continue to run. And it is about specific VMs that habve problems. On one host there are about 10 Win7 VMs. Only one freezes. And it freezes frequently (5-10 times a day). The others show no problem. Same ESX host. Same NFS datastore.

DavoudTeimouri · ‎07-31-2014

Ok, you can try it and as I know, just NFS is affected on vSphere 5.5 U1.

Also try apply latest patches for resolve some security issues.

Anyway, you should plan to apply critical patches immediately.

-------------------------------------------------------------------------------------
Davoud Teimouri - https://www.teimouri.net - Twitter: @davoud_teimouri Facebook: https://www.facebook.com/teimouri.net/

ViennaAustria · ‎07-31-2014

The NFS APD issue is not responsible for the freezing problems. I have no messages of that kind in the logfile:

~ # grep -i path /var/log/vmkernel.log

2014-07-29T23:02:15.216Z cpu3:243709 opID=3ba5904c)NFS: 157: Command: (mount) Server: (192.168.1.210) IP: (192.168.1.210) Path: (/backup) Label: (backup) Options: (None)

2014-07-30T23:02:14.875Z cpu1:246603 opID=28e53910)NFS: 157: Command: (mount) Server: (192.168.1.210) IP: (192.168.1.210) Path: (/backup) Label: (backup) Options: (None)

I also see [find] no other error messages invmkernel.log at the time of the last freeze, that I can assign to the problem. Only the event 'Alarm 'Virtual machine CPU usage' changed from Green to Red' in vCenter which informs us about another frozen VM. Strange.

Regarding patches: we keep all systems up to date. Whenever Update Manager informs us about new patches, we install them a.s.a.p. But obviuosly there are not ony "critical" and "non-critical" patches, but also "express" patches. Is there a doanload source available for the Update Manager? How else can I be notified?

DavoudTeimouri · ‎07-31-2014

About virtual machine CPU usage, do you have any running background process that can affect CPU and Disk usage on VM such as AV?

Of curse, you mentioned that the issue is not happening on vSphere 5.5 without update 1.

About patch notification, please check this article: VMware KB: Receiving notifications for patches, security updates, and End Of Life announcements for ...

You should active it on your VMware profile.

-------------------------------------------------------------------------------------
Davoud Teimouri - https://www.teimouri.net - Twitter: @davoud_teimouri Facebook: https://www.facebook.com/teimouri.net/

ViennaAustria · ‎08-06-2014

A downgrade/new installation to/of ESXi5.5 without U1 including uninstall and install the VMWare tools did not solve the problem.

In an other posting (http://itnervecenter.com/content/windows-bugchecks-vmware-esxi-xeon-e5-2670-cpus) about BSOD 0xA on WIN7 I found a possible workaround for the problem. Inside the settings of the VM under the tab Options you can select the "CPU/MMU Virtualization Type". We changed the option to Software. Some VMs (WIN7) did not freeze since days, other (WIN7 and WIN2008R2) still do. But not all of the WIN7 / Win2008R2 VMs have this problem.

admin · ‎08-06-2014

Please post a vmware.log file for the affected VM.

ViennaAustria · ‎08-11-2014

Since my last posting no VM freeze til yesterday. Close to 2014-08-10T07:50 the VM freeze. The vmware.log of the affected VM: http://www.avalon.at/vmware-freeze.log .

bravobrawi · ‎08-11-2014

Hello,

since a few days some of our WIN7, WIN2008R2 and yesterday also a WIN2012 Std VM freeze. All VMs are 64bit. The ESX Server have Supermicro Boards with Intel Xeon E3-1230V2 CPUs. On the affected VMs I activated the logging and when the VMs freeze, i will post the vmware.log as soon as the freeze.

admin · ‎08-11-2014

bravobrawi wrote:

Hello,

since a few days some of our WIN7, WIN2008R2 and yesterday also a WIN2012 Std VM freeze. All VMs are 64bit. The ESX Server have Supermicro Boards with Intel Xeon E3-1230V2 CPUs. On the affected VMs I activated the logging and when the VMs freeze, i will post the vmware.log as soon as the freeze.

These processors may be impacted by Intel erratum BW113. Please make sure that you have the latest microcode for these processors. See http://www.intel.com/content/dam/www/public/us/en/documents/specification-updates/xeon-e3-1200v2-spe... for more information.

bravobrawi · ‎08-19-2014

In the latest BIOS of the Supermicro X9SCM-F boards is no option for EPT (BW113). I put the Logfile from a VM, with crash to BSOD after freeze under http://www.selfnet.at/vmware/ .

admin · ‎08-19-2014

These log files are not from a Xeon E3-1230V2 CPU; they're from a Xeon 5150.

bravobrawi · ‎08-19-2014

sorry, this was the wrong server ( mac pro ).

EDIT: But there is the same problem. There must be a problem with ESX 5.5 U1 and Windows 7/2008R2/2012.

bravobrawi · ‎08-19-2014

At about 2014-08-14T13:33:23 the VM freeze. The Logfiles can be found at http://www.selfnet.at/vmware2/.

mathanmailid · ‎08-19-2014

There is a known issue with vSheild driver with Win 7 which gets installed along with VMware Tools. Try reinstalling VMware Tools without vSheild drivers on the Win 7 VMs and check if that helps.

admin · ‎08-19-2014

You have microcode patch level 0x19. You probably need at least microcode patch level 0x1b to fix erratum BW113. You should contact your system vendor for a BIOS update.

bravobrawi · ‎09-08-2014

Hello,

since the uninstallation of the vSheild driver no VM freeeze.

Thx