Poor performance Windows 2008 R2 under ESX 3.5

[hellraiser]

Hi,

I've recently been building a few 2008 R2 VMs for use on our ESX 3.5 servers, these are highly specced Dell R710's with an abundance of both RAM and processing power. For some reason, they run like crap - this is, in part, not helped by the lack of a VMware video driver, however the issue I have been seeing is that the VMs have been maxing out the available RAM, resulting in them paging out to disk. I have disabled the balloon driver and this immediately frees up the consumed RAM and performance returns to a sensible level, but even then we get occasional stutters where performance slows considerably for a few seconds before continuing.

None of the VMs are doing anything particularly intensive, and it appears many people are having the same issues - I have built VMs with 1, 2 and 4 vCPUs and all behave in the same way. Surely vmware are aware of this problem, and have a solution? I'm running ESX 3.5 update 5, patched to build 213532.

Help!

JD

[AWo]

How many cores does the host have? If you have some guests running with "too" much vCPU's thatwill affect guest with less vCPU's, as well.

Can you tell us how many guests run with how many vCPU's?

AWo

VCP 3 & 4

Author @ vmwire.net

\[:o]===\[o:]

=Would you like to have this posting as a ringtone on your cell phone?=

=Send "Posting" to 911 for only $999999,99!=

[hellraiser]

Many We are running a pair of Dell R710's that each have a pair of quad-core Nehalem Xeon X5570 chips, which with hyperthreading enabled effectively gives it a total of 32 logical cores, 48GB of RAM across the cluster and pair of Equallogic iSCSI boxes (PS4000 and PS4000XV). It's mighty quick, and any 2003 boxes run like lightning, but the 2008 R2 ones run like crap. We currently only have about 10 VMs across the two boxes as it's quite a new build, and resource-wise it's barely using anything, so can't think of any reason as to why the 2008 R2 boxes should run so poorly.

JD

CompTIA A+ Network+

vmware VCP 3

[tallyho]

In the BIOS of each of your 2008 VM's disable your Serial Ports, Printer Ports, and Floppy. This helpled me release IRQ's and my 2008 VM performed much better.

[hellraiser]

Already done, still no major difference. The balloon driver seems to be the biggest culprit, disable that from vmtools and performance improves enormously, but still nowhere near as quick as 2003. Likewise the lack of video driver is a major pain, trying to work on the local console with a juddery mouse is not very pleasant

JD

CompTIA A+ Network+

vmware VCP 3

[hellraiser]

<bump>

Anyone else had issues with this? I've google'd and it seems lots of people are having issues with poor performance of 2008 R2 under ESX 3.5, so where are you all???

JD

CompTIA A+ Network+

vmware VCP 3

[TxTechLaw]

HR:

I've run into an issue with mine freezing, on ESX 3.5 U4, and removed the VMTools, which so far has worked ok. Are the VMtools installed on yours?

Chad

[hellraiser]

I have the vmtools installed, the only thing I have disabled is the balloon driver for the memory - with this enabled, my VMs kept maxing out their RAM and not giving it back to ESX, causing slowdowns. Have tried it without vmtools altogether, and seems to make no difference, still runs quite poorly compared to the 2003 boxes (which absolutely fly along).

If you're experiencing issues with freezing with VMtools installed, I seem to recall there was an issue with the video driver on 2008 R1 causing lockups - the fix was not to install the driver and use the standard VGA one instead which works, but gives you a performance hit. I don't suppose vmware have any plans of updating vmtools with a suitable video driver for 2008 on ESX 3.5 do they?

JD

[TxTechLaw]

JD:

I don't know what it would be, but I wonder if there is a common denominator between Win2008R2 and RHEL5.4? There's been some traffic about 5.4 being inexplicabley slow:

http://communities.vmware.com/message/1501106#1501106

[hellraiser]

Hmmmm, will have to investigate further. Cheers for that link, gives me another avenue to explore

JD

CompTIA A+ Network+

vmware VCP 3

[Rumple]

quote - the only thing I have disabled is the balloon driver for the memory - with this enabled, my VMs kept maxing out their RAM and not giving it back to ESX, causing slowdowns.

The standard behaviour for Windows 2008 is to take every running process and load it into memory. If you are seeing your entire memory space used up, you might be under allocating RAM for the machines. Typically I go with 1GB on Windows 2008 but if you increase RAM, at some point you should see some free RAM avalable (meaning you are over allocated).

If you add 4GB of RAM to an idle Windows 2008 server and you have nothing free in task manager, there must be something else going on...

[mail2ibrahim21]

r u running your ESX 3.5 server with the update 4? or Update 5?

From U5 onwards Windows Server 2008 is officially compatible with ESX.

[VMmatty]

This isn't going to help with your memory issue, but if you're running the latest build of ESX then the VMware Tools package should include an updated video driver you can use. It is located in C:\Program Files\Common Files\VMware\Drivers\wddm_video. Just go into Device Manager and update the driver and point it to that location. I've seen a great improvement in performance when using the console of Server 2008 R2 (and Windows 7) VMs.

What else is running on those R2 guests? Do you see the high memory usage when deploying a clean R2 image from template without any software installed?

[hellraiser]

Alas no - I was allocating between 1 and 4GB and was seeing the same issue

JD

CompTIA A+ Network+

vmware VCP 3

[hellraiser]

When you say latest build, are you referring to vSphere or ESX 3.5? I'm running update 5 which has official 2008 support, plus it is patched to around build 213352 I seem to recall - can't say I've seen any video drivers available for Windows 2008 (at least not the R2 version). Will investigate when back in the office tomorrow, I have a new R710 to play with so will build a couple of guests on there and see what I can do.

Thanks for all the replies so far!

JD

CompTIA A+ Network+

vmware VCP 3

[pelletierr]

I can't find the driver in my 2k8R2 server hosted on ESX3.5u5 build 238493.

I've got a server on 2008 and an other on 2008R2 doing more or less the same job. I'm seeing a lot more hard faults/s in ressources monitor on the 2008R2 server. RAM usage goes up steadily for a while, until the hole "cache" portion is moved to "active". Than, I get a lot of hard errors, the "active" part is flushed while I get a ton of hard faults, and the cache starts to be rebuilt.

It may be caused by McAfee.. I'll have a look at that.

[hellraiser]

Me either, same build of ESX 3.5 u5 as yourself. I have built a new ESX box, patched up to date and have installed a bare copy of 2008 R2 with 4VCPUs which seems to be running fine (although the lack of a decent video driver is really irritating), will experiment with that and see if I can trigger the behaviour I am seeing on our SAN-attached hosts.

I am currently running no AV software on the SAN-based 2008 R2 hosts (due to the issues I have been experiencing) but they still seem to perform poorly so guessing it's not AV related

JD

CompTIA A+ Network+

vmware VCP 3

[VMmatty]

I honestly thought that the video driver was updated in the VMware Tools package on ESX 3.5. They've updated it on vSphere so I'm really surprised to hear it isn't updated on 3.5.

I zipped up the contents of the driver folder from my Server 2008 R2 VM on vSphere 4 and attached it. Maybe this will help with at least the lack of a decent video driver for Server 2008 R2. Make sure you test this before putting it on anything in production - I've never tested this on ESX 3.5 and have no idea how it will work.

Extract this somewhere, and then open Device Manager and right click on Standard VGA Adapter and click Update Driver. Browse to the extracted contents of this zip and see if it accepts that driver.

[hellraiser]

Shame.... Whilst it installs, it doesn't work correctly following a reboot and Windows automatically disables it - I guess it's because it's a WDDM driver which I don't think work under ESX 3.5

Come on vmware, give us a working driver for 3.5!

JD

CompTIA A+ Network+

vmware VCP 3

[Adam-ProInteg]

Beware - when I upgraded my Windows 2008 R2 VM with the VMware VGA II video driver the console connection locked up and the server would not boot after trying to restart. Wow, I guess that's why they don't install it by default. I booted in Safe-Mode and from device manager, uninstalled the VMware video driver (even asked it to delete the files) and rebooted. So far everything is okay but that's what I get for wanting it all.

Haven't tried the VMware mouse drive but I can say I'm not optimistic.

So the question is: When will we get the real driver so we can have good mouse and display performance?

Adam