Windows 2008 R2 + ESX4 + vCenter Client Console = FREEZE

[joergriether]

Hello,

i reproduced this several times now and also found a coressponding thread in the microsoft newsgroups, ( http://social.technet.microsoft.com/Forums/en-US/windowsserver2008r2general/thread/bc0f3c12-eda7-440... ) so here you go:

Install a fresh W2008 R2 to an ESX4 machine, do this all with vCenter Client Console, not with reomte RDP. Add the role

Remote-Desktop-Services to the newly created W2008 R2 machine, reboot, after reboot when the machine tries to finish

the changes (after logon) it totally freezes.

Now the interesting part on that:

If you log on via RDP after the reboot (oh yeah, you urgently need to turn rdp on before reboot!!!), then

the installer of the role WILL FINISH!!!

Now how strange is that?

Any ideas?

best,

Joerg

[jefmes]

Thank you for putting the time into this Tim. I've been wanting to myself but haven't had time otherwise...

[jaspain]

Very interesting, Tim. This is behaving like a User Account Control issue. The built-in local Administrator account (or the built-in domain Administrator account) auotmatically runs all processes with elevated privileges unless you specify otherwise with local security policy (or group policy). Other accounts that are members of the local Adminstrators group do not, and generate User Account Control prompts when elevated privileges are required. The video lock-up may result from the fact that the UAC system is trying to prompt for privilege elevation, but for some reason is not able to display the dialog. I would not be surprised to hear that if User Account Control is turned off, the video lock-up problem goes away when running as another account with local administrative rights. It remains a mystery to me why privilege elevation would be an issue for the video driver.

I still have a case open with VMware TS, but have not been able to speak to them yet today. Thanks. Jeff.

[TimKleen]

While waiting I have done some more testing. I made sure UAC was set to elevate w/o prompting. Logged into the system as Administrator (account 0) and browsed files to my hearts content. Logged off and then logged back on with my domain account (has Domain Admin, Ent. Admin, Domain User, is a test domain not our production) and started to browse the files......Smack! Locked up tighter than chunky peanut butter thru a baby (I have 3 boys and their mother (my ex-wife now) only bought chunky peanut butter so I know this to be a fact). This is with the SVGA II driver. It does not look to be a UAC issue per-say but there appears to be some kind of permissions issue with this driver.

Well back to resetting this system and working on my many other tasks.

Tim

This is from the vmware-#.log file that the second tech was interested in until he was told not to bother working on this as it is a known issue (before I got the corrected information). It does not mean much to me but to a VMware developer it might. If they read the posts of course.

Nov 30 08:46:19.149: vcpu-0| Guest: vmx_fb: tap: init

Nov 30 08:46:19.168: vcpu-0| Guest: vmx_fb: tap: init

Nov 30 08:46:19.194: mks| SVGA: enabling escape command

Nov 30 08:46:19.194: mks| HostOps hideCursor before defineCursor!

Nov 30 08:46:19.195: vcpu-0| Guest display topology changed: numDisplays 1

Nov 30 08:46:19.203: vcpu-0| Guest: vmx_fb: This is the primary surface: PPDEV c00c4020

Nov 30 08:46:19.204: vcpu-0| Guest: vmx_fb: Display driver is out of sync with virtual hardware. Disabling 3d.

Nov 30 08:46:19.205: vcpu-0| Guest: vmx_fb: Current hardware revision: 0.0.

Nov 30 08:46:19.206: vcpu-0| Guest: vmx_fb: Driver compiled against: 2.0.

Nov 30 08:46:19.206: vcpu-0| Guest: vmx_fb: DrvGetDirectDrawInfo: Overlay flags set

Nov 30 08:46:19.207: vcpu-0| Guest: vmx_fb: This is the primary surface: PPDEV c00c4020

Nov 30 08:46:19.207: vcpu-0| Guest: vmx_fb: Display driver is out of sync with virtual hardware. Disabling 3d.

Nov 30 08:46:19.208: vcpu-0| Guest: vmx_fb: Current hardware revision: 0.0.

Nov 30 08:46:19.208: vcpu-0| Guest: vmx_fb: Driver compiled against: 2.0.

Nov 30 08:46:19.209: vcpu-0| Guest: vmx_fb: DrvGetDirectDrawInfo: Overlay flags set

Nov 30 08:46:19.523: mks| HostOps showCursor before defineCursor!

Nov 30 08:46:33.927: vcpu-0| DecoderIsInvalidInstruction:691 0xfffffa80057c115d #UD e00d6306 sz=1 ct=14

Nov 30 08:46:33.928: vcpu-0| DecoderIsInvalidInstruction:691 0xfffffa80057c115d #UD e00d6306 sz=1 ct=15

Nov 30 08:46:33.928: vcpu-0| DecoderIsInvalidInstruction:691 0xfffffa80057c117d #UD a00d6306 sz=1 ct=16

Nov 30 08:46:33.929: vcpu-0| DecoderIsInvalidInstruction:691 0xfffffa80057c117d #UD a00d6306 sz=1 ct=17

Nov 30 08:46:33.929: vcpu-0| DecoderIsInvalidInstruction:691 0xfffffa80057c117d #UD a00d6306 sz=1 ct=18

Nov 30 08:46:33.930: vcpu-0| DecoderIsInvalidInstruction:691 0xfffffa80057c117d #UD a00d6306 sz=1 ct=19

Nov 30 08:46:33.930: vcpu-0| DecoderIsInvalidInstruction:691 0xfffffa80057c117d #UD a00d6306 sz=1 ct=20

Nov 30 08:46:33.930: vcpu-0| DecoderIsInvalidInstruction:691 0xfffffa80057c117d #UD a00d6306 sz=1 ct=21

Nov 30 08:46:33.931: vcpu-0| DecoderIsInvalidInstruction:691 0xfffffa80057c1184 #UD d630649 sz=2 ct=22

Nov 30 08:46:33.931: vcpu-0| DecoderIsInvalidInstruction:691 0xfffffa80057c11ab #UD 1a4e1982 sz=1 ct=23

Nov 30 08:46:33.932: vcpu-0| DecoderIsInvalidInstruction:691 0xfffffa80057c11b5 #UD 48cd6306 sz=1 ct=24

Nov 30 08:46:33.932: vcpu-0| DecoderIsInvalidInstruction:691 0xfffffa80057c11d4 #UD 5c524ed9 sz=3 ct=25

Nov 30 08:46:33.933: vcpu-0| DecoderIsInvalidInstruction:691 0xfffffa80057c11dd #UD 638dd906 sz=1 ct=26

[jaspain]

Just spoke to VMware TS on this issue. They are saying that they have withdrawn ESX 4.0.0 Update 1 due to various technical problems with it. The worst one, which I have unfortunately experienced myself, is that installing Update 1 on an ESX server with the HP Management Agents installed causes a PSOD and renders the server unrecoverable. My technician advised against installing either the SVGA II or the WDDM driver supplied with Update 1, and in fact advised against upgrading the VMware Tools at all, until a new version of Update 1 is released. He thought this might occur within the time frame of a few weeks.

My situation is that I have installed Update 1 on three of four ESX servers via the VMware Update Manager. I have upgraded the VMware Tools on the VMs on one of these, but not the other two. The fourth ESX server, rebuilt after being rendered unrecoverable, is running the origninal ESX 4.0.0. I'll probably wait to do anything else until a new version of Update 1 is released.

[john_gallucci]

I would like to cover a few things as I am unable to reproduce the console freeze, and have a working system. First off, could someone explain the steps to reproducing the error? I have tried windows explorer for several minutes (browsing files, folders, etc.) and also gone to the internet with IE8. No Freeze! So maybe I'm doing something wrong.

Here is my configuration:

-VMware ESX 4.0 Update 1 (updated from 4.0 using VUM)

-VMware Virtual Center 4.0 Update 1 (installed on Windows Server 2008 R2 Enterprise on a Dell 2950 III)

-Virtual Machine: Windows Server 2008 R2 Enterprise Edition (non-activated)

-Template originally had VMware Tools installed, but never the bad SVGA driver (template existed before Update 1)

-Upgraded VMware tools and manually updated the graphics driver using the Have Disk... option as I described in an earlier post

-Custom configuration is applied using the Specifications Manager (adds to domain and uses DHCP network)

-The virtual graphics card has 8MB of RAM (default is 4MB)

I have used both the SVGA II and WDDM video drivers successfully now without any console freeze. I also installed the drivers using both the local admin and domain account with the same result.

Some way or another we will get to the bottom of this. If someone with the freeze could try to create a VM exactly like I have AND still get the freeze, we might be able to narrow it down further. I also strongly recommend the method of using the Have Disk... option as using any other method to install the drivers can cause issues. Again see my instructions in the previous post for installing the graphics driver in this manner.

[Markisha1979]

I'm very intereted about this problem.

I installed a w2008R2 Std on 4.0u1 and i noticed the standard vga driver.

By now my vm works fine, with no freeze, so i keep the standard driver waiting the fix for this problem....

If I can help you in any way let me know.

Thanx.

[jaspain]

John: The video freeze is kind of random in my experience. Try opening a PowerShell prompt and running some commands. That has done if for me on some occasions.

I think we're at the point where VMware is aware of the issue and the developers are working on it. Now we have to wait for a supported fix. For now they are saying in http://kb.vmware.com/kb/1011709 not to use the VMware video drivers. We have a reasonable workaround in that the problem doesn't occur when accessing the VM using RDP as opposed to the VMware remote console. If I have to open a remote console session, say to change the network configuration, then I can put up with the poorer video performance that occurs without the VMware video drivers. My own practice with respect to VMware is to stick with officially supported hardware and software on production systems. Otherwise it's difficult to call upon them for help if things go wrong.

Jeff.

[-_DK_-]

I've got the same problem.

Luckily I have all demo hardware and VMs, so I can blow them up as necessary (or in this case as R2 thinks it's necessary).

The lockup for me has occured mainly when:

-Adding "features" or "roles"

-Changing the video driver

It has also happened randomly under general system usage

I had one of them lockup almost immediately after login. By the 5th reboot I was frantically hitting Windows+D to keep all the default windows minimized. I was then able to get in and make changes by doing that.

My testing shows that none of the drivers work. I have manually changed drivers. I have completely removed VMware Tools and reinstalled. When it reinstalled it did not use the .31 version, but used the .9. A manual update to .31 resulted in a system freeze once I selected OK to the driver.

I get lockups with both the built-in domain admin account, a local admin account on a system that is not a DC, and on domain admin accounts I create. It locks up under any user I login with.

I did just download Update 1 yesterday, but saw the notice it was being pulled last week. I figured by now they would have fixed the problems. Plus the warning for U1 no longer says don't install if you have 3rd party mgmt apps installed. Now it just specifically points out the HP one.

-Dan

[john_gallucci]

I finally got the VMware console to freeze. It took about 45 minutes after installing Java, Tomcat, and MySQL into a server. I was just about to install the Web application when it froze. Can't believe it took me this long to finally see the freeze, but it did occur.

So even after a millions tests/suggestions from everyone on this thread it looks like we are in the hands of the developers.

Back to the Standard VGA driver for now (...or Xen--J/K we love you VMware!)

[joergriether]

This is a shame. And as the well known song of talk talk makes fun about it, i suggest you take it very serious, vmware. w2008r2 is out for months and yet you are not even able to provide a working vga driver, nor are you able to backup this os vss aware with your "enterprise class product". i have been using esx since v2. next week, i´ll install my very first w2k8r2 hyper-v server. well done, vmware.

[jjahns]

Well. One bug since the rollout of vSphere (for us). I am kind of surprised though that this made it out of testing.

I think threatening to go to Hyper-V is really childish here. There is no point. Why would you go to that anyways if you do have vSphere?

As long as a prompt resolution is made for this, then I have no problems.

[joergriether]

now let´s get more concrete here and let´s talk about backup, this may be off-topic in this thread but it fits to this discussion in general. if i am unable to backup my win2k8r2 system vss aware neither with vcb nor with the new gui driven vmware-backup, what would be the point? yes, i can tolerate this buggy vga stuff a few more weeks, that would be no problem. but if i can´t backup vss aware, if would be stupid to install w2k8r2 systems in my existing virtual environment. i didn´t say i would replace esx, becuase i won´t! I am just saying i can´t afford to use my w2k8r2 guests with esx. i have to use em with hyper-v. so i have to install hyper-v systems because i have to use Exchange 2010 on w2k8r2 systems.

[jjahns]

See now that made more sense! Forgive my remarks then. It was just portrayed as a "I am unhappy, so bye" comment. We too are looking at putting in Exchange 2010, but have not done so yet. There is no hurry for us right now, because we aren't ready.

[jjahns]

Using the WDDM driver.

Have not had any freezing issues, but my issues might not be related. Running server 2008 core, and I am unable to now change the resolution of the console, as well as enabling hardware acceleration.

[jjahns]

So, in summary, this problem occurs:

• On Any VI Client That Has The Console For The VM Open (VM Freezes)

• Only Occurs When The VMware SVGA II Driver Is Installed

• Does Not Occur On A RDP Session

• RDP Sessions, If You Specify Them To Shutdown, Freeze At The System Event Notification Service

Is that all right so far? Now, the VMware Update 1 download links are unavailable, but available for ESXi. Can anyone say update 2 please?

[john_gallucci]

Yes, you are correct. It sounds like it occurs with the WDDM driver as well (not supported by VMware for 2008 R2), but I cannot confirm.

[jjahns]

How long, does everyone think, that they will get around to this? Personal opinion is that an update 1.5 will more than likely get released. At any rate, the bugs are starting to add up, and my previous statement about lack of problems... well... that could change soon. I learned a lesson long ago when dealing with 3.5 that it is best not to deploy an update until its been worked on. Good thing I heeded my advice and did not do this on a production box.

[jaspain]

VMware ESX 4.0 Update 1a was made available today as a patch via the VMware Update Manager. See http://kb.vmware.com/kb/1014842. I don't see any related download on the vSphere download page.

After remediating one of my ESX hosts with this patch, the build number remained 208167, the same as with the previous Update 1. Update 1a does not appear to include any update to the VMware Tools. VMs that had their VMware Tools updated with ESX 4.0 Update 1 still indicated VMware Tools status "OK" after the installation of Update 1a.

KB Article 1011709 "Disable SVGA drivers installed with VMware Tools on Windows 7 and Windows 2008 R2" (http://kb.vmware.com/kb/1011709) has still not been updated since November 11, and it still recommends against using the SVGA drivers included with the VMware Tools.

Looks like the SVGA freeze issue will be with us for a while longer.

[Markisha1979]

So there's no update for vcenter?

I proceed with the host update directly?

[jaspain]

The only way I have seen to update ESX hosts to 4.0 Update 1a is to use the VMware Update Manager. I don't see anything for Update 1a on http://downloads.vmware.com/ for either ESX hosts or vCenter Server. The available downloads are all the original Update 1 releases from 11/19/2009.