VMware Cloud Community
Shocko
Enthusiast
Enthusiast

5% CSTOP but no overcommitment

Hi all

I have a single 8 Core Windows 2008 R2 Enterprise VM running on a Single 2 x Quad Core with HT (16 logical core) ESX 5.1 u1 server. Veeam one is telling me that over the last 1 hr the CO-STOP is on average 5.39% . If I look at the metrics raw in milliseconds in vCentre , over the 1 hr interval it says average 16391. Now, doing the math (http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=200218...

(CPU summation value / (<chart default update interval in seconds> * 1000)) * 100 = CPU ready %

therefore (5 min data interval for 1hr)

(16391 / (300 * 1000) ) * 100 = 5.46

Why am i seeing Co_STOP if the vm is the only thing with access to the hardware?

Reply
0 Kudos
7 Replies
DavoudTeimouri
Virtuoso
Virtuoso

Do you have any other virtual machine with 4 or more vCPUs? If your virtual machines is not overloaded and it's not under pressure, reduce your vCPU to 4 and check CPU ready.

Please share some information about your virtual machines with us.

-------------------------------------------------------------------------------------
Davoud Teimouri - https://www.teimouri.net - Twitter: @davoud_teimouri Facebook: https://www.facebook.com/teimouri.net/
Reply
0 Kudos
JPM300
Commander
Commander

If you are seeing co-stop but no cpu ready% it sounds like VMware scheduler is having issues scheduling CPU request. This isn't like ready where the request is ready to process and your waiting on a CPU because of a load issue, this is your scheduler having issues requesting your vCPU's to your pCPU's.  Try and reduce some of your VM's vCPU's then you should see this drop.  This will also probably help performance as with a higher co_stop the VM can have a "laggy" feel to it.

I hope this has helped.

Reply
0 Kudos
vfk
Expert
Expert

This also happens when you have too many vCPU assigned to a VM, but the VM is not actually making use of all the vCPU.  With relaxed co-scheduling in ESXi, the progress of each vCPU in a VM is tracked individually. The skew is measured as a difference in progress between the slowest vCPU and each of the other vCPUs.   Trying reducing the number of vCPU assigned to the VM.

--- If you found this or any other answer helpful, please consider the use of the Helpful or Correct buttons to award points. vfk Systems Manager / Technical Architect VCP5-DCV, VCAP5-DCA, vExpert, ITILv3, CCNA, MCP
Reply
0 Kudos
Shocko
Enthusiast
Enthusiast

Thanks for the comments guys but I'm not sure any of them address my questions. I'm quite familiar with co-scheduling but this is, and will be, the only vm running on this ESX host. We have a single host for this application to take advantage of vmware snapshots etc. for development. Since the the vm has 8 vCPU and the hosts have 8 pCPU and 16 cores, why would co-scheduling have an issue  scheduling the vCPU? The only thing I can think of is that the guest requests 8 threads to run on its vCPU at the exact same time. The application spawns 8 JVMs hence the 8 vCPUs.

Reply
0 Kudos
JPM300
Commander
Commander

Hey Shocko,


It could be the scheduler trying to maintain the numa settings.  Could you dump out the vmware.log file from this particular VM and post it here.

RubyIvy
Enthusiast
Enthusiast

Hi Shocko,

I saw that you mentioned that your single vm spawns about 8 instances of jvm . It is  not recommend to run multiple instances of jvm on the same vm.


Installing VMware ESX on a physical machine makes it capable of running a number of virtual machines, all of which are independent of each other from a performance monitoring perspective. In the virtualized world, it is more common to run one instance of Java in any one virtual machine


vSphere is pretty smart when it comes to managing physical memory and determining where best to place a virtual machine’s memory given how busy each NUMA node is in the physical server.You might also try and tweak the advance NUMA settings for the VM but it may become difficult to manage manually balancing NUMA resources as your environment grows.


There are several documents which talks about this as below:


If you find this or any other answer useful please consider awarding points by marking the answer correct or helpful.
Shocko
Enthusiast
Enthusiast

Thanks for the suggestions guys. The underlying ESXi 5.1U1 host is a HP Proliant BL460c G6 with 196 GB of RAM evenly balanced between processors.. Since there is only a single VM running on it the VM has full memory reserved so no swapping going on. During my load tests I have also checked the guest OS and no swapping going on there either. I know generally we would run 1-2 JVMs but for this project and due to licensing constraints etc., i need to run 2+. I notice even when idle, the WAIT time was high so I flipped the power mode of the Proliant into Static High Performance and that seemed to help there. Logs attached.

Reply
0 Kudos