I'm getting frustrated trying to virtualize high performance apps. We can't ever seem to get the performance we need vs physical boxes. I think this all comes back to the way vCPUs are handled by Physical processors. If I understand correctly, a vCPU corresponds with the power of one processor core. So say I build up a database server VM with 2 vCPU on a nice newer Quad core processor system. Well since each vCPU is the power of only one core, my 2 vCPUs in my VM only represent 2 cores. So thats only 1/2 a physical quad core processor. If I instead purchased a basic 2CPU server - it would come with 2 quad core processors. So when I installed windows and SQL on it, I would have access to 2 full physical processors and 8 cores. No wonder the VM is slower!
So I guess, if I am understanding this correctly, to get the same CPU power as the Physical server standalone, I would need to create a 16 vCPU VM - which if it is even possible, would require over $100,000 in software licensing. Am I off base here? How do I get around this? I need to create some high performance servers and we trying to virtualize everything, but it seems like it is impossible to catch up.
Where do I sign to receive my royalties check?
woah there mr. impetuous, I didn't say you SHOULD be paid, I gave you kudos.. :smileygrin:
... and it's in the mail, along with my donation to PETA ... Speaking of which, I could use a good steak.
Just to put my 2 canadian cents in...
If you have wait times of 600+ ms you really need to review that environment and you've overloaded the box with way too many SMP systems and its gonig to perform worse and worse. Anything over 5ms means you are seriously fighting for CPU resources...
Best practice is to assign a VM a single CPU with as little memory as you can get away with and then scale up from there, only after proving you need more horsepower, threads and/or memory.
SQL boxes work pretty well with multiple CPU's if they have lots of databases on them, domain controllers, Exchange Servers, web servers, 95% of all application servers, file servers and other general servers do not need nor even require multiple vCPU's in most if not all instances.
If you have 8 cores, I would not have more then a single, MAYBE 2 - SMP VM's (and then only 2 vCPU's per VM). Outside of that, everything should be running as a single CPU...
Also note, there are some Applications that use Kernel level calls and those also perform poorly under virtualization no matter what you do.
I would recommend you get one of the virtualization monitoring solutions (vizioncore,veeam, etc) and then use it to review how many resources you've given boxes..and what the utilization is and see if you've wildly over allocated resources to the environment (which I am willing to bet money you have).
And for the love of god, please tell me you have not played around with CPU affinity in that environment...
I agree with Dustin - before you convert it to physical (which might stil have to do) is wat Ready Times are you experiencing? What utilization are you seeing on the processors of the ESX server hosting this beast of a machine - because if you are seeing low to mid utilizatoins of the physical CPUs on the ESX server with low ready times its is not cpu virtualization that is the problem - also what else is running on this host? Is this the only VM with multiple vCPUs?
CPU WAIT times are averaging 1300-3000 with max at about 6000
CPU Ready times are ~600 with max ~2000.
CPU usage is 85% with max at 97 and min at 74.
There are a few other 2 vCPU hosts on this 4 CPU host.
That definitely sounds like an overallocation problem. A host with only 4 CPUs and you have several 2 vCPU VMs? And if you're trying to add an 8vCPU VM on the mix, your host CPU thread contention will go skyrocketing.
I would suggest you evaluate your VM's performance by looking at the CPU usage MHz in VI client (+Word of caution: Don't use Windows Perfmon for this unless you're using VMware-provided counters!+) and see if it's really taking advantage of all the CPUs that's assigned (VM Total MHz = # vCPU * CPU MHz speed of host CPU). Since it sounds like you have overcommitted the CPU resources, chances are that the MHz usage will show it is not even using a single host CPU core's worth of MHz power. Dial-down some of VM's vCPU to 1 or move some of the VMs to another host. If you don't have another host available or if 1 vCPU is not enough to give an acceptable performance for the application, maybe it's time to consider moving some of the VMs to physical. Some overcommitment of the CPU resource is good as long as you can make sure the aggregated CPU usage of the VMs on that host will never exceed the host's maximum capacity. If the aggregate usage exceeds the host's capacity, then that is a bad way of overcommitting resources.
You can also try playing around with the CPU/Memory reservation settings for each VM, but I will caution you against it since with the way you have overcommitted your CPU resources, it might prevent you from powering on the VM since the host can't provide the CPU resource you reserved for that particular VM.
Also, as someone pointed out, CPU and memory usage don't tell the whole story. Network and Disk performance can affect the VM's ability to perform well, especially for database servers. Have you followed the SQL server's best practices published by VMware and Microsoft? Like placing database partition and log partition on separate disk spindles and placing log partition into a RAID 10 or RAID 1 array.
Where do I sign to receive my royalties check?
woah there mr. impetuous, I didn't say you SHOULD be paid, I gave you kudos.. :smileygrin:
... and it's in the mail, along with my donation to PETA ... Speaking of which, I could use a good steak.
Hey, in this economy, you've got to take it where you can get it.
I'll keep an eye out in my mailbox. ![]()
_________________
Dustin Pike
VMware Certified Professional (VI3 & vSphere)
I agree with Dustin - before you convert it to physical (which might stil have to do) is wat Ready Times are you experiencing? What utilization are you seeing on the processors of the ESX server hosting this beast of a machine - because if you are seeing low to mid utilizatoins of the physical CPUs on the ESX server with low ready times its is not cpu virtualization that is the problem - also what else is running on this host? Is this the only VM with multiple vCPUs?
CPU WAIT times are averaging 1300-3000 with max at about 6000
CPU Ready times are ~600 with max ~2000.
CPU usage is 85% with max at 97 and min at 74.
There are a few other 2 vCPU hosts on this 4 CPU host.
That definitely sounds like an overallocation problem. A host with only 4 CPUs and you have several 2 vCPU VMs? And if you're trying to add an 8vCPU VM on the mix, your host CPU thread contention will go skyrocketing.
I'm not trying to add a 8 vCPU system, right now it has only 2 vCPU and I was wondering if I should try for 4 or go physical.
Anyway, I didn't think having three 2 vcpu hosts on a 4 socket system would be that big of a deal. There are plenty of cores to go around - or so I thought. CPU wait has gone NUTS today though - hitting about 18,000ms average. I definitely need to tackle that first.
And whoever asked, no I do not have any affinities or priorities assigned to VMs.
One thing I would do then if you want some better visibility is to download and install the trial of the vfoglight application from visioncore.
One of the things I've used it for in the past is to identify when an application is using a large number of Kernel level calls...Some applications, especially those that were old mainframe type solutions rebuilt to be windows apps typically perform poorly anyhow and since virtualization is much slower with kernel level calls, it can cause problems.
the vfoglight will alo give you some nice information on if machines are under or overallocated memory,etc
Other products will also probably do the same, I've just never tried them.
fgw, you are dead on.
This is why throwing resources at a problem isnt the solution for virtualization. Apps can actually run with worse performance with more vCPUs because of this exact behavior you stated.
Well, yes and no.
In vSphere this is still an issue, but it is less of an issue. Relaxed co-scheduling has allowed the kernel to better optimize its use of physical processors. This means that multi-vCPU VMs will generally perform better in vSphere. It is still an excellent rule of thumb to only provision as many vCPU as necessary for your app. Start with one vCPU and scale up as needed.
More info on CPU scheduling in vSphere here:
http://www.vmware.com/files/pdf/perf-vsphere-cpu_scheduler.pdf
fgw, you are dead on.
This is why throwing resources at a problem isnt the solution for virtualization. Apps can actually run with worse performance with more vCPUs because of this exact behavior you stated.
Well, yes and no.
In vSphere this is still an issue, but it is less of an issue. Relaxed co-scheduling has allowed the kernel to better optimize its use of physical processors. This means that multi-vCPU VMs will generally perform better in vSphere. It is still an excellent rule of thumb to only provision as many vCPU as necessary for your app. Start with one vCPU and scale up as needed.
More info on CPU scheduling in vSphere here:
http://www.vmware.com/files/pdf/perf-vsphere-cpu_scheduler.pdf
Yes...vSphere does manage it better, but I still think the general rule you stated applies: "Start with one and scale up." You still hit issues with too many multi vCPU servers hogging cores or waiting to get access. It was a pleasant sighting to see VMware is making progress in this regard though.
_________________
Dustin Pike
VMware Certified Professional (VI3 & vSphere)
We have approximately 6 SQL VM's as well as DB2 and Oracle. I realize that every environment is different and workload is relevant but we had to tweak and learn as we grew our virtual environment. Just when it looked as if some of our SQL boxes weren't candidates we started looking at the Parallism setting on the SQL servers. Here is a good explanation that describes the setting.
http://searchwinit.techtarget.com/tip/0,289483,sid1_gci1033746_mem1,00.html
This changed the performance drastically. We actually had users remark that performance was better once running on the virtual machine vs. the physical box! Please note that this setting is defintely specific to different applications and does not fix every issue but we have had very good luck with it.
Good Luck
Parallelism in SQL Server basically limits the cores that can be used for queries. Its definately a good feature to use for virtualized SQL servers though, as it keeps the required number of vCPUs down and shortens the time the VM has to wait for available cores.
_________________
Dustin Pike
VMware Certified Professional (VI3 & vSphere)
I agree with Dustin - before you convert it to physical (which might stil have to do) is wat Ready Times are you experiencing? What utilization are you seeing on the processors of the ESX server hosting this beast of a machine - because if you are seeing low to mid utilizatoins of the physical CPUs on the ESX server with low ready times its is not cpu virtualization that is the problem - also what else is running on this host? Is this the only VM with multiple vCPUs?
CPU WAIT times are averaging 1300-3000 with max at about 6000
CPU Ready times are ~600 with max ~2000.
CPU usage is 85% with max at 97 and min at 74.
There are a few other 2 vCPU hosts on this 4 CPU host.
That definitely sounds like an overallocation problem. A host with only 4 CPUs and you have several 2 vCPU VMs? And if you're trying to add an 8vCPU VM on the mix, your host CPU thread contention will go skyrocketing.
I'm not trying to add a 8 vCPU system, right now it has only 2 vCPU and I was wondering if I should try for 4 or go physical.
Anyway, I didn't think having three 2 vcpu hosts on a 4 socket system would be that big of a deal. There are plenty of cores to go around - or so I thought. CPU wait has gone NUTS today though - hitting about 18,000ms average. I definitely need to tackle that first.
And whoever asked, no I do not have any affinities or priorities assigned to VMs.
Evaluating WAIT times can be confusing.
CPU WAIT times represent time the CPU was waiting on something. This goes two ways. If the CPU isn't doing anything you're going to have high wait times because..well, its waiting on something to actually do. If you have a lot of activity on the system otherwise, CPU usage or IO and you've got high WAIT times then you've got a problem. In other words, if I see 80% CPU used but CPU WAIT of 15000ms then I'd seriously check out your IO - disk latency, network latency. If you've got CPU Ready times going up at the same time, then you've got CPU contention going on.
Or any combo of the above. By itself, CPU WAIT is meaningless.
@Rparker - Way back early on in this thread, you said to turn off Hyperthreading? Just out of curiosity, why turn off Hyperthreading? I thought the issue around HT was corrected in Nahelem.
adam
