Sounds like you are experiencing the exact same issue. And from my understanding on how processor's work you cant simply give a process 200mhz and another process 800mhz of a single 2000mhz processor. When the processes are scheduled on the processor, the processor works on that process until the process is complete or has been unscheduled. With this limitation alone I don't see how the scheduler in ESX/ESXi (or any virtualization software package) could ever get around having a limit of to many processes potentially being available and not requiring all the possible CPU power available. And with such a limit would also limit the number of VM's that can created per host. Just because processors can complete tasks quicker and quicker as the generations progress you still can over saturate the processor's with the number of processes requesting processor time. I as well would like a little better explination and understanding on how the scheduler works. We have had vmware come in and do a "health check" and there recommendations were to add additional processor core's into the environment.
One thing I would be interested in understanding better is when you run into this issue, are the VM's that are being scheduled with the %RDY% times in acceptable ranges recieving the correct amount of CPU time allowing them to obtain the required speed's they need or are they being limited to the amount of processor power they can achieve by the scheduler de-scheuling the VM due to other VM's requesting processor time? And when you do add additional resources bringing the average %RDY% time's down across the board, will the VM's in acceptable ranges prior to the resource additions see a performance increase? I would think that if you have more processes available than processor time can handle, that you would have the CPU's running at nearly 100% for the time that each processes is schedule on the processor in an effort to complete the tasks at hand as quickly as possible in order to move onto the next process needing available processor time. Otherwise you are affectively governing how quick a process can complete. Which I suspect the schedulre is actually doing as it needs to keep multiple thread's in sync when working with SMP vm's.
It just bugs me that vmware (and I am sure others) teach it the way they do and when confronted about it have no real answer's. I'm not certain tech support even fully understands the scheduler in detail (and wouldn't expect any real front line support to.)
