ViennaAustria
Contributor
Contributor

ESXi 5.5 U1 freezes Win7 and 2008R2 VMs

We recently upgraded many 5.5 hosts to 5.5 U1. Since then, several VMs with Windows 7 64 Bit or Windows 2008R2 64 Bit guest OS freeze - sometimes twice a week, usually once a day, in extreme cases every hour! Sometimes we see a bluescreen (everytime some other), mostly the OS is just frozen (you see the desktop but it won't respond to mouse/keyboard input). We also had a frozen taskbar (no response to windows key), but the explorer itself (icons on the desktop) did respond. But after a few klicks with the mouse it froze, too. In any cases you have to reset the VM.

These Issues also occoured on a freshly installed 5.5.U1 machine with a freshly installed Win7 VM.

As we couldn't find any problem in the guest OS, we tried to downgrade several hosts to ESXi 5.5 with no updates (build 1331820) by reinstallation last weekend. All problems are gone since 5 days now. On hosts where we didn't yet perform the downgrade/reinstallation, the VMs still freeze.

So 5.5.U1 definitely has a problem.  [at lease on Supermicro X9SCM-F mainboards, which we mostly use]

As far as I remember, when we updated 5.5 hosts to build 1892794 we experienced no problems of that kind. But I can't say for sure. We'll try that after my vacation.

Anybody else having problems with 5.5.U1?

0 Kudos
19 Replies
DavoudTeimouri
Virtuoso
Virtuoso

Hi,

Do you have NFS on your environment? vSphere 5.5u1 has NFS issue and you need to apply a patch.

** If you found this note/reply useful, please consider awarding points for "Correct" or "Helpful" ** Davoud Teimouri - https://www.teimouri.net - Twitter: @davoud_teimouri Facebook: https://www.facebook.com/teimouri.net/
0 Kudos
ViennaAustria
Contributor
Contributor

Yes, many have NFS datastores.

Where to get the patch? The Update Manager of the vCenter hosts reports "compliant" for critical and non-critical patches.

0 Kudos
DavoudTeimouri
Virtuoso
Virtuoso

Hi,

Please read this blog:http://www.yellow-bricks.com/2014/06/11/vsphere-5-5-u1-patch-released-nfs-apd-problem/

And this article:http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=207639...

You can download the patch from VMware Patch Portal.

** If you found this note/reply useful, please consider awarding points for "Correct" or "Helpful" ** Davoud Teimouri - https://www.teimouri.net - Twitter: @davoud_teimouri Facebook: https://www.facebook.com/teimouri.net/
0 Kudos
ViennaAustria
Contributor
Contributor

Fond a description of the issue:

http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=207639...

I'll try that.

But I don't believe, that this issue is responsible for our problems, because only Win7 64 Bit and 2008R2 guests are affectet. All other guests on the very same host/datastore continue to run. And it is about specific VMs that habve problems. On one host there are about 10 Win7 VMs. Only one freezes. And it freezes frequently (5-10 times a day). The others show no problem. Same ESX host. Same NFS datastore.

0 Kudos
DavoudTeimouri
Virtuoso
Virtuoso

Ok, you can try it and as I know, just NFS is affected on vSphere 5.5 U1.

Also try apply latest patches for resolve some security issues.

Anyway, you should plan to apply critical patches immediately.

** If you found this note/reply useful, please consider awarding points for "Correct" or "Helpful" ** Davoud Teimouri - https://www.teimouri.net - Twitter: @davoud_teimouri Facebook: https://www.facebook.com/teimouri.net/
0 Kudos
ViennaAustria
Contributor
Contributor

The NFS APD issue is not responsible for the freezing problems. I have no messages of that kind in the logfile:

~ # grep -i path /var/log/vmkernel.log

2014-07-29T23:02:15.216Z cpu3:243709 opID=3ba5904c)NFS: 157: Command: (mount) Server: (192.168.1.210) IP: (192.168.1.210) Path: (/backup) Label: (backup) Options: (None)

2014-07-30T23:02:14.875Z cpu1:246603 opID=28e53910)NFS: 157: Command: (mount) Server: (192.168.1.210) IP: (192.168.1.210) Path: (/backup) Label: (backup) Options: (None)

I also see [find] no other error messages invmkernel.log at the time of the last freeze, that I can assign to the problem. Only the event 'Alarm 'Virtual machine CPU usage' changed from Green to Red' in vCenter which informs us about another frozen VM. Strange.

Regarding patches: we keep all systems up to date. Whenever Update Manager informs us about new patches, we install them a.s.a.p. But obviuosly there are not ony "critical" and "non-critical" patches, but also "express" patches. Is there a doanload source available for the Update Manager? How else can I be notified?

0 Kudos
DavoudTeimouri
Virtuoso
Virtuoso

About virtual machine CPU usage, do you have any running background process that can affect CPU and Disk usage on VM such as AV?

Of curse, you mentioned that the issue is not happening on vSphere 5.5 without update 1.

About patch notification, please check this article: VMware KB: Receiving notifications for patches, security updates, and End Of Life announcements for ...

You should active it on your VMware profile.

** If you found this note/reply useful, please consider awarding points for "Correct" or "Helpful" ** Davoud Teimouri - https://www.teimouri.net - Twitter: @davoud_teimouri Facebook: https://www.facebook.com/teimouri.net/
0 Kudos
ViennaAustria
Contributor
Contributor

A downgrade/new installation to/of ESXi5.5 without U1 including uninstall and install the VMWare tools did not solve the problem.

In an other posting (http://itnervecenter.com/content/windows-bugchecks-vmware-esxi-xeon-e5-2670-cpus) about BSOD 0xA on WIN7 I found a possible workaround for the problem. Inside the settings of the VM under the tab Options you can select the "CPU/MMU Virtualization Type". We changed the option to Software. Some VMs (WIN7) did not freeze since days, other (WIN7 and WIN2008R2) still do. But not all of the WIN7 / Win2008R2 VMs have this problem.

0 Kudos
admin
Immortal
Immortal

Please post a vmware.log file for the affected VM.

0 Kudos
ViennaAustria
Contributor
Contributor

Since my last posting no VM freeze til yesterday. Close to 2014-08-10T07:50 the VM freeze. The vmware.log of the affected VM: http://www.avalon.at/vmware-freeze.log .

0 Kudos
bravobrawi
Contributor
Contributor

Hello,

since a few days some of our WIN7, WIN2008R2 and yesterday also a WIN2012 Std VM freeze. All VMs are 64bit. The ESX Server have Supermicro Boards with Intel Xeon E3-1230V2 CPUs. On the affected VMs I activated the logging and when the VMs freeze, i will post the vmware.log as soon as the freeze.

0 Kudos
admin
Immortal
Immortal

bravobrawi wrote:

Hello,

since a few days some of our WIN7, WIN2008R2 and yesterday also a WIN2012 Std VM freeze. All VMs are 64bit. The ESX Server have Supermicro Boards with Intel Xeon E3-1230V2 CPUs. On the affected VMs I activated the logging and when the VMs freeze, i will post the vmware.log as soon as the freeze.

These processors may be impacted by Intel erratum BW113.  Please make sure that you have the latest microcode for these processors.  See http://www.intel.com/content/dam/www/public/us/en/documents/specification-updates/xeon-e3-1200v2-spe... for more information.

0 Kudos
bravobrawi
Contributor
Contributor

In the latest BIOS of the Supermicro X9SCM-F boards is no option for EPT (BW113). I put the Logfile from a VM, with crash to BSOD after freeze under http://www.selfnet.at/vmware/ .

0 Kudos
admin
Immortal
Immortal

These log files are not from a Xeon E3-1230V2 CPU; they're from a Xeon 5150.

0 Kudos
bravobrawi
Contributor
Contributor

sorry, this was the wrong server ( mac pro ).

EDIT: But there is the same problem. There must be a problem with ESX 5.5 U1 and Windows 7/2008R2/2012.

0 Kudos
bravobrawi
Contributor
Contributor

At about 2014-08-14T13:33:23 the VM freeze. The Logfiles can be found at http://www.selfnet.at/vmware2/.

0 Kudos
mathanmailid
Enthusiast
Enthusiast

There is a known issue with vSheild driver with Win 7 which gets installed along with VMware Tools. Try reinstalling VMware Tools without vSheild drivers on the Win 7 VMs and check if that helps.

0 Kudos
admin
Immortal
Immortal

You have microcode patch level 0x19.  You probably need at least microcode patch level 0x1b to fix erratum BW113.  You should contact your system vendor for a BIOS update.

0 Kudos
bravobrawi
Contributor
Contributor

Hello,

since the uninstallation of the vSheild driver no VM freeeze.

Thx

0 Kudos