Server 2008 extremely slow on ESXi 4 (208167)

[carrothospital]

A few months ago I started having serious performance issues on our Terminal server VM, and I have yet to track down the problem. I know that I'm inevitably going to leave out some sort of vital information, so please bare with me and slap my wrist when I forget.

Our Vmware ESXi system:

2 Quad-core processors(8 logical cores)

12GB Memory

RAID 1 array for OS disks and RAID 5 for data.

Vm's on the system:

SBS 2008 = 4 cores, 9GB RAM

Server 2008 STD (32-bit) = 4 cores, 2.5GB RAM

XP Pro = 1 core, 512MB RAM

This system used to run fine without any trouble, but now sometimes it is unusable. The CPU spikes and takes forever, and it's not just with one process either. It's used to serve a small website with a SQL database(used internally and not very intensive) and Terminal services so accountants can remote in and use Quickbooks.

I started wondering if it possibly started happening when I created the XP Pro VM and if it had to do with assigning it a processor when technically all 8 were already used. Could that cause issues? Also, RAM is only at 2.5(was at 3, until I gave some to XP). I realize that's low, even though 32-bit can only go to 4, but if I look at the memory usage in the system, it's only at 60% or so. I'm probably going to upgrade the RAM anyway, but couldn't nail that down as the problem.

Could this be something with system overhead and reservation? It seems it's only bad with this one VM that's not really set up any differently from the others. Most of the settings for the VM are default, other than allocating resources.

Even when doing something as simple as logging in, or opening the Reliability and Performance monitor, I watch the CPU usage in the vSphere client jump way up, but disk and memory usage stay relatively the same.

What should I look for and check to troubleshoot this performance issue?

[golddiggie]

Try reducing at least one of the servers to just two cores/vCPU's. What are the processors, such as Xeon 5520's? Are you using thick or thin provisioning on these VM's? Unless you're running software that requires the 9GB of RAM on the SBS, try reducing that too. Is the storage JUST for the VM's to use, or are other things allowed access to it? If they are local drives, are the SAS or SATA? What's the make/model of the host server?

VCP4

[mclark]

Just a couple of suggestions.

1) It's been said many times on the forums and elsewhere that you should always try just one vCPU first, then only add processors if proven necessary. Is there some reason you've assigned 4 VCPUs to the 2008 servers? You should probably try going to just 1 vCPU. I have several 2003 and 2008 servers, and I've never had to have more than 1 vCPU. Not saying you don't have reasons, but it could hurt you more than help you.

2) Any reason why you're assigning 9 GB to a 32-bit machine? Since it can only use 4 GB anyway, that's overkill. I would take it back down to 4 GB.

I personally would try 1 vCPU on the servers, and bump down that memory and see what happens.

[golddiggie]

I assumed that since he called out one of the servers as 32 bit that the other was 64 bit. Otherwise, as you mentioned, it's pointless (and none to smart) to assign more than 4GB of RAM to the VM. You'll almost never see the full 4GB either.

I've found that the 'sweet spot' for VM's vCPU count is two. I've tried with just one, but I end up getting CPU usage alerts periodically on those VM's. Increasing them to just two vCPU's makes those alerts evaporate. I also start off with either 1GB or 2GB of RAM per VM, unless the system is going to actually USE more. Even then, I've never gone above 4GB of RAM in a VM.

I would also split off any SQL server duties to it's own VM and not have other things running on that VM. It's far easier to trouble shoot when you don't need to worry about additional software running on a server being used for a SQL db. I have a SQL 2005 server that's using just two vCPU's, only one GB of RAM, a 20GB C drive, and a 60GB D drive (where SQL is installed and the db's reside) and I get no alerts for it's performance. I use it for a handful of db's without issue. If I was to have heavier traffic, I would watch for alerts and then start with adding a little memory at a time (1GB at a time, stopping at 4GB since it is a 32 bit OS).

I do have an Exchange 2010 server, installed on Server 2008 R2 64-bit, with 4GB of RAM allocated. IF I see any performance issues, once it goes live for email routing, I'll adjust it's spec's... That also only has two vCPU's assigned to it.

VCP4

[carrothospital]

Thanks for the replies, guys. Here are the answers to some of your questions..

To clear some confusion, SBS is 64-bit, hence the 9GB. I think that SBS 2008 is only available in 64, so I didn't specify that one. I think initially we had 6GB for the SBS box, but we had issues with it running out of memory and then spitting back smtp connections...bringing incoming mail to a halt. Since we gave it 3GB more, we haven't had that issue.

As far as why I assigned 4 vCPU's to each, I didn't know there was a reason not to. If each server is only using 1, doesn't that leave a lot of extras that are just sitting there? Or is my view of how resources are shared in vmware a bit off? I had read another post where someone suggested 1 vCPU, but I wasn't sure of the reasoning.

The server is a Dell PowerEdge 2950 with 2 Xeon E5430's. The VM's are thick and stored on local SAS storage. Nothing else is using this storage space but the VM's.

Does that help? I can try reducing the vCPU's down to 2 later tonight and see how it goes. This VM used to be fairly fast, so I'm not sure why it would get slow now, but thanks again for the help. I'm sure we'll get to the bottom of it.

[mclark]

I would definitely reduce the vCPUs to 2 at least, if not just 1, on both of your 2008 servers. I just know that if you look around there are threads about how sometimes more than 1 vCPU can cause slowdowns. I've never personally used more than 1, as I mentioned, but I would try 2 at the most.

I would also reduce the memory on that one 2008 SBS server from 9 GB to 8 GB. You have specified exactly as much RAM as your server has, but you have not left any for ESXi itself. The docs say ESXi needs 1 GB minimum. You may find that by reducing the number of CPUs your 2008 mail box performs better even with less RAM anyway.

[carrothospital]

Alright, i just switched both VM's down to 2 vCPU's and the RAM on SBS from 9GB to 8GB. I'm guessing vmware claims that 1GB back automatically, or is there something I need to do to make sure it uses it?

I'll see how it performs tomorrow under load and let you know.

[golddiggie]

Look in the Resource Allocation tab in the vSphere Client (for the host server)... You never want the Reserved Capacity to equal the Total Capacity.

VCP4

[Rumple]

I would suspect you are probably overdoing it even giving those servers 2 vCPU's and if you need 8GB of RAM assigned to SBS you must have upwards of 50 people running on it pretty hard (or you need to outsource your spam filtering because your network is getting hammered by the smtp connection.

Better to have a company like mxlogic accept mail and block all smtp requests to your server but for them...you will probably see a huge drop in resource requirements.

I had a client with 25 people getting hit with 250k worth of smtp requests/day (and they were not an open relay - PS you have checked that right?). Once I moved spam to mxlogic their server started idling again...

[J1mbo]

As said throughout, there is too much use of vSMP and too much RAM committed. The SMTP point is also a good one IMO.

I would echo the other comments - use 2 vCPU (max) for each of the three guests, and allocate a total of (say) 9GB across all guests. Ensure vmware tools are installed in all guests, and look at memory balloon and swap counters.

Please award points to any useful answer.

[carrothospital]

Whoops, didn't realize that I could only choose 2 helpful answers. They have all been helpful so far, and I'll find out soon enough which one is correct(if not more than one..)

In the Resource Allocation tab, since taking a gig away from SBS, shows the following:

Memory -

Total Capacity: 9817 MB

Reserved Capacity: 662 MB

Available capacity: 9155 MB

CPU -

Total Capacity: 19019 MHz

Reserved Capacity: 0 MHz

Available Capacity: 19019 MHz

Where did that other gig go that I freed up? Does VMware automatically claim that or do I need to specify it somewhere?

The swap and balloon counters are both at 0.

You do have a valid point about SBS and SMTP. We did have about 50 users using email pretty heavily. The company has since split and there are fewer people, so the load may not be as bad. Like I said, we had 6GB for SBS before, and I read a few other places that with SBS 2008 you should have at least 8GB because it's such a hog. We also have a few other services running on there besides just Exchange.

The system seems more responsive to me, but I'm waiting on some feedback from users. Hopefully I'll have that soon.

Thanks again for all the help.

[mclark]

From reading your postings, it seems like you are concerned about giving every CPU core and every MB of RAM to some VM. That's the wrong idea. You don't want to allocate every single resource to something. As I mentioned in my posting last night, ESXi needs some RAM and CPU for itself. If you can free up even more RAM than 1 GB, it is not a bad thing. For instance, I just looked at 2 of my ESXi servers, and each of the servers themselves (not the VMs) are using a little over 2.5 GB of RAM. My advise is to only allocate the resources the VMs really need and leave some free, I would not max out each VM with arbitrary huge numbers unless proven necessary.

[J1mbo]

You're looking in the wrong place. The capacities listed on the "resources" tab just shows how much for each resource ESX can guarantee to be available for VMs based on the current configuration.

.

Please award points to any useful answer.

[carrothospital]

Well, all of this seemed to help a lot. The only thing that was still slow was SQL, but I that's a separate issue.

Now I'm fighting with these other admins on how much to assign to each VM. (I work for an IT company that handles IT needs for small/medium businesses, and this particular client has recently decided they want to take care of most things on their own).

After my changes, the other admins added 8 GB more ram(20 total). The 2008 Standard server now has 4GB, SBS 2008 14.5GB, and XP Pro 512MB. They also bumped SBS back up to 4 procs.

When I questioned them about this they said that the whole idea of virtualization is to oversell resources, because most hosts only use 20%-40% of the resources allocated to them. Then the VM server has resources on demand. Also that not allocating resources does no good and upgrade to the point where applications benefit from the multiprocessing.

Is that right? From all I've heard here, you should only give the VM what it needs and save the rest, but their logic is completely opposite. I may be in the wrong here, and that's fine, but you guys seem like experts and I'd like to hear your opinion.

[golddiggie]

I would say that their reasoning/logic is flawed... Sounds like none of them have had any (real) training on VMware's ESX/ESXi products. Leaving the resources unallocated to VM's means that the entire cluster can better leverage the resources. Using DRS means that you take into account the memory on ALL the hosts, but with HA you also factor in what each VM needs to run so that you have properly configured hardware. I would give that SBS VM enough memory so that it functions properly, and not a lick more. The way they have things configured, IF you need to add any more VM's to that host, your performance will start to tank again.

Even in my lab, I don't have that large a chunk of RAM allocated to any one VM. I have one with 4GB (Server 2008 with Exchange 2010 installed), one with 3GB, one with 2GB and the balance have either 1GB or less (vMA has 512MB allocated to it). Total actual memory used is under 9GB (of the 13.5GB allocated). If they're seeing that only 20-40% of the memory is actually being used, then they can reduce the memory given to the SBS server by a large chunk. Dropping it down to 10GB would probably have zero impact (negative at least, if anything it could be a positive impact).

I would suggest reaching out to VMware resources in your area, or have your sales rep route the questions, to get the officially 'blessed' answer on this. Still, I suspect the guys at your location have it bass ackwards.

VMware VCP4

Consider awarding points for "helpful" and/or "correct" answers.

[mclark · Accepted solution]

I found a couple of other community postings that go in-depth on the vCPU issue. You should take a good look at both of the threads below. The overwhelming consensus is to not use multiple vCPUs unless required. These posts explain why, and they do it well. I would say your other admins that are coming in and changing things are possibly harming your system performance. Can you take their rights away to change stuff? :smileycool:

http://communities.vmware.com/thread/157849

http://communities.vmware.com/thread/131269

[J1mbo]

TBH you'll probably get away with a 4 vCPU guest if all the others are 1 vCPU since you have 8 cores available. The problem with vSMP is more with multiple guests since the overall concurrency potential is reduced.

But overcommitting RAM to that degree is just foolish - the inevitable consequence is that OS paging (balloon) and vSwapping will swamp the RAID device leading to massive latencies. Read here for some detail.

If the client feels that they need 20GB, spend the money and upgrade the host to 24GB I would say. RAM is very likely the limiting factor in this machine.

Edit - mclark, sorry this reply is directed towards the OP, I clicked on the wrong 'reply' button

Please award points to any useful answer.

[carrothospital]

Unfortunately they are moving towards trying to manage their servers themselves, so I can't take the admin privileges away!

At this point, I just have to leave it in their hands, since they believe their way of overselling resources is the way to go. So far, in this situation, it hasn't seemed to have any negative effects, since I pulled some vCPU's off of that one VM. The memory paging and balloon counters are at 0, and everything is running pretty normal. It may rear it's ugly head in the future, but for now I guess it's alright.

Thanks again everyone for all the help. I wish I could award points to everyone, since every answer was helpful.