taylorb
Hot Shot
Hot Shot

High Performance frustration - vCPU vs muti-core physical Processors

I'm getting frustrated trying to virtualize high performance apps. We can't ever seem to get the performance we need vs physical boxes. I think this all comes back to the way vCPUs are handled by Physical processors. If I understand correctly, a vCPU corresponds with the power of one processor core. So say I build up a database server VM with 2 vCPU on a nice newer Quad core processor system. Well since each vCPU is the power of only one core, my 2 vCPUs in my VM only represent 2 cores. So thats only 1/2 a physical quad core processor. If I instead purchased a basic 2CPU server - it would come with 2 quad core processors. So when I installed windows and SQL on it, I would have access to 2 full physical processors and 8 cores. No wonder the VM is slower!

So I guess, if I am understanding this correctly, to get the same CPU power as the Physical server standalone, I would need to create a 16 vCPU VM - which if it is even possible, would require over $100,000 in software licensing. Am I off base here? How do I get around this? I need to create some high performance servers and we trying to virtualize everything, but it seems like it is impossible to catch up.

0 Kudos
31 Replies
weinstein5
Immortal
Immortal

Yes there may be some workloads that will require 16 cores - which if it does might eliminate as a candidate for virtualization but however in my experiencethose types of wqorkloads are few - in fact since the release of vSphere and the ability to build a VM with 8 vCPUs I have not come across a workload requirig more than 8 vCPUs and these have been some rather large clients - including a few large enterprises with more than 2000 physical machines machines -

If you find this or any other answer useful please consider awarding points by marking the answer correct or helpful

If you find this or any other answer useful please consider awarding points by marking the answer correct or helpful
0 Kudos
khughes
Virtuoso
Virtuoso

While virtualizing every server is what everyone is pushing for, there are servers that just don't run the same as a VM, most of the time databases are the ones that get left in the cold (as Rparker would preach to you about). Have you had someone run capacity planner on your systems? Do you know if that server is in fact a good canidate to be virtualized. Are you sure it is CPU constraints and not any other bottleneck, like storage speeds (if you're moving from local disk to iSCSI)?

If you really wanted to go that way and assign a straight ESX host to that VM you could go that route but it would be extremely costly (as you pointed out) and it would not play well if you had a hardware failure on that box and HA kicked in trying to move a 16 vCPU VM to a different host. If you aren't getting anywhere near the performance with 2 vCPU's and have tried 4 vCPU and still not close to the performance I would say that maybe you should leave that server as a physical box.

  • Kyle

-- Kyle "RParker wrote: I guess I was wrong, everything CAN be virtualized "
0 Kudos
taylorb
Hot Shot
Hot Shot

Even with 8 vCPU, I am barely getting the CPU power of 1 dual CPU physical system. With the new six cores - not even 2 physical CPU. And then I have to pay for Enterprise versions of all software - to get the same performance.

What is happening is we had a new Financial app in about 18 months ago that I have been giving more and more resources to and they still complain about performance. Looking at moving it to Physical instead because I can't give it any more vCPUs without upgrading the Windows version and SQL version.

0 Kudos
khughes
Virtuoso
Virtuoso

Sounds like it would be more cost effective and a better peroformance to convert it back to a physical server

  • Kyle

-- Kyle "RParker wrote: I guess I was wrong, everything CAN be virtualized "
0 Kudos
DPike
Enthusiast
Enthusiast

What does the performance chart look like for this server? Just because end users are complaing and asking for more resources doesnt mean that is the end all, be all fix. Maybe there is a bottleneck on networking, or if you've moved from a local disk storage on the physical server to an iSCSI setup in VM. There are many aspects to look at, not just vCPU and Memory.

What are the physical requirements from the app vendor to run the application?


_________________

Dustin Pike

VMware Certified Professional (VI3 & vSphere)

Virtual Blocks Blog

_________________ Dustin Pike VMware Certified Professional (VI3 & vSphere) Virtual Blocks Blog
0 Kudos
weinstein5
Immortal
Immortal

I agree with Dustin - before you convert it to physical (which might stil have to do) is wat Ready Times are you experiencing? What utilization are you seeing on the processors of the ESX server hosting this beast of a machine - because if you are seeing low to mid utilizatoins of the physical CPUs on the ESX server with low ready times its is not cpu virtualization that is the problem - also what else is running on this host? Is this the only VM with multiple vCPUs?

If you find this or any other answer useful please consider awarding points by marking the answer correct or helpful

If you find this or any other answer useful please consider awarding points by marking the answer correct or helpful
0 Kudos
Rohail2004
Enthusiast
Enthusiast

I have the same issue. I have 24 logical processors, running 4 VM's on that host, and each host has 4vCPU's, but the ready time on one VM goes up to 1300ms and then users start complaning. The VM has 16GB, Windows2003,64 bit EP, the host has 128 memory. the disk is attached on the SAN, no error or anything on HBA.. Networking is fine, but DBA's are saying when they run multithreaded app they're seeing delay in query.

0 Kudos
RParker
Immortal
Immortal

Looking at moving it to Physical instead because I can't give it any more vCPUs without upgrading the Windows version and SQL version.

Well A) did you run capacity planner on this box BEFORE you attempted to run it on Virtual? Not all applications are candidates for Virtualization. No where will you find that Virtualization REPLACES Physical hardware for performance. B) Virtual Environments are for one thing, and one thing only, consolidation. Virtual machines do not replace or meet physical machines for performance. VMs are easier to manage, and make OS more accessible, while saving you power and space.

Physical machines however need their own power and separate management (KVM, separate keyboard / monitor, etc...).

Did you get help with Virtualization before you moved to this environment or did you do this via trial and error, this seems like you are trying this or that before you fully understand how Virtualization works.

0 Kudos
RParker
Immortal
Immortal

(as Rparker would preach to you about)

I came a little late to the party.. but I see it was started before me..

0 Kudos
RParker
Immortal
Immortal

I have 24 logical processors, running 4 VM's on that host, and each host has 4vCPU's

Problem number 1, logical processors, turn OFF hyperthread in the BIOS.

Problem number 2, make ALL VM's only 1 CPU. If you add more vCPU that's more processors threading the app, but it doesn't translate into more cycles PER VM, it may actually cause MORE overhead, which is probably why your apps are not faster than before.

Problem number 3, MOST applications are NOT SMP capable, so simply throwing CPU at a program will not make it faster, since the apps were not designed to take advantage of multiple CPU's in the first place.

0 Kudos
Rohail2004
Enthusiast
Enthusiast

When I change the vCPU to 1 then the cpu runs high most of the time.

how about vCenter server? doesnt it require more than 1 vCPU?

0 Kudos
DPike
Enthusiast
Enthusiast

When I change the vCPU to 1 then the cpu runs high most of the time.

By CPU running high, what is high? Is this the same observation in the physical environment? What application is this? Ive seen my SQL servers use 80-100% CPU and a huge chunk of RAM. To the untrained eye, yes, this will cause a alert and subsequent troubleshooting. To someone knowledgable, we know SQL like to cache resources for its own silly reasons. Know your app and how it behaves before virtualizing. Then know whether it plays nice in a virtual space. VMs aren the end all, be all for data centers. Can some companies go 100% VMs, yes. Are all applications/servers VM canidates? Not at all. You can just go in wildly swinging a stick and virtualize everything.

how about vCenter server? doesnt it require more than 1 vCPU?

</div>vCenter Server was desinged to take advantage of multiple CPUs. As RParker pointed out, not all apps have this ability.

_________________

Dustin Pike

VMware Certified Professional (VI3 & vSphere)

Virtual Blocks Blog

_________________ Dustin Pike VMware Certified Professional (VI3 & vSphere) Virtual Blocks Blog
0 Kudos
taylorb
Hot Shot
Hot Shot

Looking at moving it to Physical instead because I can't give it any more vCPUs without upgrading the Windows version and SQL version.

Well A) did you run capacity planner on this box BEFORE you attempted to run it on Virtual? Not all applications are candidates for Virtualization. No where will you find that Virtualization REPLACES Physical hardware for performance. B) Virtual Environments are for one thing, and one thing only, consolidation. Virtual machines do not replace or meet physical machines for performance.

You should talk to the sales guys. They'll tell you EVERYTHING should be virtualized. Smiley Happy

To answer the question, no I did not run capacity planner. I don't own it, and by my understanding, capacity planner analyzes an existing physical workload. These apps were never run physical. Vendor said virtual should be OK, and I guess it is. The app runs - just not that fast.

>VMs are easier to manage, and make OS more accessible, while saving you power and space. Physical machines however need their own power and separate management (KVM, separate keyboard / monitor, etc...).

Got that, thanks. Physical machines don't need like 45 IO connections, either, though! Smiley Wink

Did you get help with Virtualization before you moved to this environment or did you do this via trial and error, this seems like you are trying this or that before you fully understand how Virtualization works.

I've been using Vmware since ESX 2.0, I know how virtualization works just fine. But what good is all the marketing nonsense about 8vCPU mean anything, when 8vCPU is really only 1.33-2 real CPUs? Like I said, I am just getting frustrated. Management pushes virtualization, users want performance. And I am finding "high-performance VM" is a bit of an oxymoron.

0 Kudos
taylorb
Hot Shot
Hot Shot

I agree with Dustin - before you convert it to physical (which might stil have to do) is wat Ready Times are you experiencing? What utilization are you seeing on the processors of the ESX server hosting this beast of a machine - because if you are seeing low to mid utilizatoins of the physical CPUs on the ESX server with low ready times its is not cpu virtualization that is the problem - also what else is running on this host? Is this the only VM with multiple vCPUs?

CPU WAIT times are averaging 1300-3000 with max at about 6000

CPU Ready times are ~600 with max ~2000.

CPU usage is 85% with max at 97 and min at 74.

There are a few other 2 vCPU hosts on this 4 CPU host.

0 Kudos
RParker
Immortal
Immortal

Know your app and how it behaves before virtualizing.

We should adopt this as the "communities" motto... That's fantastic!

0 Kudos
RParker
Immortal
Immortal


And I am finding "high-performance VM" is a bit of an oxymoron.

Well yes and no. I have the same argument when people ask for things like green ketchup, and 100 brands of cereal. Appease the public, just so they shut up! Who asked for this stuff, and better yet, who buys it?

You want 8 CPU in a VM, here you go! Knock yourself out. You want a million dollars, here you go, but it won't make you happy.. people need to learn some things for themselves.. like Virtualizing an app that LOOKS like a physical server doesn't mean it will PERFORM like a physical server, but if they just gave VM's 1 vCPU nobody would take it serious, so when people see the bells and whistles and same functions as a REAL server, well then they have something to compare it to.

And I always believe that just because something is there, you don't necessarily have to trip over it.. Smiley Wink Knowledge is power, and that's why these experts are here, to explain when you have a problem, they help sort it out. So you did the right thing, you posted your question, now you will get the answer to make it work right.

Sales people have the same problem around the world, they PUSH you to BUY, they don't care if it works or not. Sales people PUSH software, hardware isn't working, throw more hardware at it, that will fix it! Not a good solution.

Technically speaking VM Ware will get you 90% of everything you need, it's the 90 / 10 rule, 10% of your VM's are 90% of the problem, we just have to figure what to fix to make the VM's (and their users) happy....

Got that, thanks. Physical machines don't need like 45 IO connections, either, though!

Pound for Pound consider the BIG picture, host servers need a LOT less connections then the Physical counterpart.. when you consider a properly built ESX machine can host 40-100 similar Physical environments.

0 Kudos
DPike
Enthusiast
Enthusiast

We should adopt this as the "communities" motto... That's fantastic!

</div>

Where do I sign to receive my royalties check? Smiley Wink

_________________

Dustin Pike

VMware Certified Professional (VI3 & vSphere)

Virtual Blocks Blog

_________________ Dustin Pike VMware Certified Professional (VI3 & vSphere) Virtual Blocks Blog
0 Kudos
fgw
Contributor
Contributor

as i understand cpu scheduling, there is another point to consider: the number of vcpus configured for a vm has to be available before a vm gets cpu resources!

when a vm is using more than one vcpu this vm will wait for cpu resources until the number of configured vcpu's (physical cores) is available.

if you run two vm's configured with 4 vcpu each on a host equipped with two quadcore processors, this vm's will get their resources (physical cores) whenever they require it.

if you throw another vm configured with only one vcpu, onto this host the situation changes: as long as this single vcpu vm is using a physical core, only one of the two vms configured with 4 vcpu will be able to get physical cpu power, as the second vm configured with 4 vcpu is requesting 4 physical cores, but the host has just three physical cores available at this time! this may leed to extended times when vm's are waitiong for physical cpu.

in simple words, a vm with less vcpus might get physical cores more often than a vm with more vcpus.:

e.g. a vm with 4 vcpus might be waiting for 4 physical cores most of the time while a vm with a single vcpu is happily getting the requested single physical core.

its a simple calculation: if the single vcpu vm is getting the requested physical core four times as often as the vm configured with 4 vcpu's the reqiuesed 4 physical cores, i would expect both vm's will show about the same performance.

if the total number of vcpu's configured for all vm's on a host is greater than the number of physicals cores available on this host (and i assume this is usually the case) will result in a disadvantage for vm's configured with more vcpu's.

pls. correct me if i'm worng!

0 Kudos
DPike
Enthusiast
Enthusiast

fgw, you are dead on.

This is why throwing resources at a problem isnt the solution for virtualization. Apps can actually run with worse performance with more vCPUs because of this exact behavior you stated.


_________________

Dustin Pike

VMware Certified Professional (VI3 & vSphere)

Virtual Blocks Blog

_________________ Dustin Pike VMware Certified Professional (VI3 & vSphere) Virtual Blocks Blog
0 Kudos