bidDAVE
Contributor
Contributor

Multiple vCPU's (SMP); convincing the business

As most have come to understand, it is not a good practice to designate multiple vCPU's for a VM unless the application running can actually use the 2nd processor effectively (see discussion http://communities.vmware.com/thread/131269). I am fairly new to the VM environment at my new job and noticed ~20% of the VM's have either 2 or 4 (i assume very bad) vCPU's assigned. We steadily get requests for VM's with multiple CPU's and it appears they are coming from requirements that assume physical servers are being used (ex:Two (2) VM Servers; Each with 2 Dual-Core/8GB RAM 108GB Usable disk space). The VMware support team understands the implications of using vSMP when not needed, but how do we convince the business that their application will work on a single CPU when their requirements call for 2 dual-core; 4 processor cores. Any documentation available that we can hand out or any suggestions is greatly appreciated. Thanks.

Dave

0 Kudos
11 Replies
beagle_jbl
Enthusiast
Enthusiast

There should be a zillion best practices documents out there.

https://h30046.www3.hp.com/campaigns/2006/events/fose/images/HP%20Virtualization.pdf

In our organization we try not to tell anyone what they are getting for hardware. Surely the underlying ESX host meets their requirements. If they ask tell them, "Oh ya, the server has 8 cores and 16GB of RAM". Will they even know? Hahaha.

Essentially, we tell our folks that they get a single core and an appropriate amount of RAM and disk, and if there are not enough resources then we can scale it from there. The main reason for virtualization is efficient use of resources. We tell our people that when their server proves they need more resources that's when we'll give it more. Scaling it out is a piece of cake. Scaling down not so much... for example, in the case of Windows you have to manually reconfigure from multikernel to unikernel. While that works it is not supported by Microsoft. Their solution is to reinstall the OS from scratch. Surely they could understand that arguement.

It's not bad using vSMP - one just has to do it intelligently and sparingly. It's only really bad if you have the same number of cores on the VM as the ESX host. Also, if you assign too many multi-core VMs you could run into really high CPU Ready times and therefore poor performance. Simply put, the ESX host needs to have 4 IDLE cores to satisfy a 4-core VM. On a fairly busy 8-core ESX host, 4 available cores may be hard to come by. In that instance, that 4-core VM could potentially run slower than a single-core VM (depending on the CPU shares assigned and such).

VMWare recommends having at least double the amount of cores on the ESX host as compared to any multicore VMs. I think that works OK with the odd 2-core on a 4-core ESX. However, I've insisted on using 16-core ESX hosts for any 4-core VMs. In those two scenarios... with a little monitoring and tweaking... I have seen no issues with CPU Ready counters.

0 Kudos
RParker
Immortal
Immortal

Basically I have learned that watching Linux teaches you a lot. Applications seldom if ever peg the CPU identically at precisely the same time. That's pretty much impossible to do, considering you can't really split a process. You can multi thread it to run in semaphore and take advantage of CPU to run concurrent processes, but you don't see exactly the same performance.

So basically if you see an app that pegs both CPU at the same time, it's a good bet that the app is multithreaded but not necessarily SMP aware. So testing is the best course of action, but of course they won't take the time TO test , they want what the want.

If they ask for 2 CPU, give them one, and whatever memory they want, cut it by at least a third. Then give them their VM. Chances are they won't look for it. Then when the ask about where their missing resources is, you can push back and say.. "oops, I guess I missed that, well everyone needs to log off, then I can make the changes...". Chances are they are already using it, and they won't worry about it.

If they force the issue, then put a performance meter on it BEFORE you shut it off.. stall them. Ask them to run an app that they use or something that they would normally run daily, and log the performance. Then give them what they asked for and re-run the tests over time.

Then you now have pure bonafide test data to PROVE your case. So future requests will now have to PROVE they need the additional resource. Test data is pretty much the only way to prove it.

But just keep in mind there is a vast difference between multi threaded applicaitions and SMP (symetrical multi processor) apps. They are not the same. You can multi thread on one CPU no problem. Then my favorite quote is: "My VM runs at 100% all the time, when my XXX app is running".

Your response: "Good! That means it's working, that's outstanding news, thanks for the feedback!".

In reality you want to see 100% (not 24 hours a day) but when they are running it means they are using their resource, instead of wasting it. If they want to know why its good tell them, it's because on a physical box you won't reach the 100% threshold because there are too many other drivers, and background polling (for usb, serial, video, sound, etc..) on a native box that's not needed in a VM, it's a more pure environment.

Dave_Mishchenko
Immortal
Immortal

Hi Dave, just to add a few comments to what has been said.

1) It helps to educate application owners that the "required hardware" specified by application vendors tends to be overkill and that the application can get by with fewer resources. Over estimating hardware requirements is one of the reasons we did end up with x86 virtualization in the first place. It may help them to see performance data for how your existing 2 / 4 vCPU VMs are running (assuming the charts will show low CPU use) or how their applications are running on physical hardware.

2) If you employ a charge back model for VMs, it may help to set pricing to discourage vSMP. For example, if you charge a single vCPU VM at $1000 per VM, then charge a dual vCPU VM at $2500. That's an overly simplistic model, but you get the idea.

3) Deal with this at the management level. In part that will require showing them data on how vSMP may negatively impact application performance. If you can make your case there, then the policy will be to start all VMs with one vCPU (with exception if you require them) and then inscrease as needed.

bidDAVE
Contributor
Contributor

Thanks for all your answers. I look forward to producing some data evidence around the multi-processor VM's.

0 Kudos
Texiwill
Leadership
Leadership

Hello,

One thing to note is that when you use vSMP the only possible negative impact is when you overcommit pCores on your server. If you have an 8 way 4 core system or 32 cores you would need minimally 32 vCPUs in use to see any issues. BTW., such a beast does exist (HP DL785).....

So if you are not overcommitting CPU resources, you may never see a performance issue. Most of us are over committing and we do see issues.


Best regards,

Edward L. Haletky

VMware Communities User Moderator

====

Author of the book 'VMWare ESX Server in the Enterprise: Planning and Securing Virtualization Servers', Copyright 2008 Pearson Education.

CIO Virtualization Blog: http://www.cio.com/blog/index/topic/168354

As well as the Virtualization Wiki at http://www.astroarch.com/wiki/index.php/Virtualization

--
Edward L. Haletky
vExpert XII: 2009-2020,
VMTN Community Moderator
vSphere Upgrade Saga: https://www.astroarch.com/blogs
GitHub Repo: https://github.com/Texiwill
0 Kudos
dominic7
Virtuoso
Virtuoso

"Simply put, the ESX host needs to have 4 IDLE cores to satisfy a 4-core

VM."

This is actually a common misconception of ESX CPU scheulding, in fact even the VMware support erroneously tell people this. I talked to one of the employees of VMware who works on the particular functionality and he said he's been trying to correct this myth internally since 3.0.x arrived. While this was true in ESX 2.5.x, in ESX 3.x VMware introduced relaxed cosheduling, which doesn't require all cores to be available to be available to schedule a task, and while relaxed cosheduling isn't perfect, the truth is that the problem has been largely fixed.

The downside is that this is very poorly documented both internally and externally to VMware.

0 Kudos
bidDAVE
Contributor
Contributor

So if we give out 2 CPU's for a VM when the application is single-threaded, will there be any performance increase?

0 Kudos
Texiwill
Leadership
Leadership

Hello,

Absolutely no performance increase but there could be a performance decrease. Basically that extra vCPU is sitting idle and just using up resources. Even with co-scheduling, there may be a performance hit. In essence, unless the app is seriously multithreaded/multiprograme such as exchange, SQL, or Terminal Services, then there is no benefit to having more than one vCPU and often a penalty.


Best regards,

Edward L. Haletky

VMware Communities User Moderator

====

Author of the book 'VMWare ESX Server in the Enterprise: Planning and Securing Virtualization Servers', Copyright 2008 Pearson Education.

CIO Virtualization Blog: http://www.cio.com/blog/index/topic/168354

As well as the Virtualization Wiki at http://www.astroarch.com/wiki/index.php/Virtualization

--
Edward L. Haletky
vExpert XII: 2009-2020,
VMTN Community Moderator
vSphere Upgrade Saga: https://www.astroarch.com/blogs
GitHub Repo: https://github.com/Texiwill
0 Kudos
Jae_Ellers
Virtuoso
Virtuoso

There could be a performance increase because all the other services for the OS may run on the other vCPU if the application uses most or all of one vCPU.

-=-=-=-=-=-=-=-=-=-=-=-=-=-=-

-=-=-=-=-=-=-=-=-=-=-=-=-=-=-

-=-=-=-=-=-=-=-=-=-=-=-=-=-=- http://blog.mr-vm.com http://www.vmprofessional.com -=-=-=-=-=-=-=-=-=-=-=-=-=-=-
0 Kudos
Ken_Cline
Champion
Champion

There could be a performance increase because all the other services for the OS may run on the other vCPU if the application uses most or all of one vCPU.

This is truly one of those "it depends" situations. For a given VM, you may see a small performance boost by allocating a second vCPU for use by the OS to run ancillary service threads on. However, the down side is that if your host is reasonably well loaded, that second vCPU is going to impact the pCPU scheduling availability for ALL VMs on the host. So, you may achieve minimally better performance for one VM, but you will likely hurt the overall workload performance.

The impact was reduced with the relaxed co-scheduling introduced into the vmkernel with ESX 3 - back in the ESX 2.x days, the kernel used hard co-scheduling which had a much greater overall negative impact on the host. With the relaxed co-scheduling, there can be some skew in the perceived "real time" between between the various vCPUs on a vSMP VM.

So -- short answer: if you don't really need multiple vCPUs, don't allocate them!

Ken Cline

Technical Director, Virtualization

Wells Landers[/url]

VMware Communities User Moderator

Ken Cline VMware vExpert 2009 VMware Communities User Moderator Blogging at: http://KensVirtualReality.wordpress.com/
0 Kudos
beagle_jbl
Enthusiast
Enthusiast

Great post from Ken.

I think advice that says "don't worry about it" is really, really, really bad advice. Relaxed co-scheduling doesn't magically give you more CPU than your hosts can provide. It may lessen the impact when CPU's are busy but it doesn't eliminate it. If the hosts in your environment aren't heavily taxed then maybe it just doesn't matter... but if you are trying to use VMWare to maximize your hardware utilization (like most of us are) then you still have to be smart about provisioning multi-core VMs. The VMWare reps still tell you to be judicious with multi-core VM's because you should be.

0 Kudos