Performance impact of too many (in-active share-based) resource pools...?

[paetecsfb]

I got a call the other day that a VM was down in our vSphere cluster...

I went in, restarted the machine and ran a perpetual ping on the VM waiting for it to come back up. Well it did but the response times were ugly... We are talking from as low as <1 to greater than 800. So I got freaked out a bit and pinged another server. It was fine. Always at 1 or <1. So I felt a ton better. Then I decided to try one more VM for good luck, and it turned out not to be my day... another crazy ping result. So I went to another host in the cluster, same result.

VM's with no commonalities at all were showing different sets of ping results. They were on different vLANs running different OS's deployed from different templates at different times.

I though it might be the fact that our vSwitch is using an ip hash policy instead of virtual port id. I know this adds more overhead to the kernel but didn't really think it would add a significant amount in our large cluster. I also thought that maybe it was just affecting vm's with high CPU (which for the most part it was). I went into ESXTOP and saw that our %ready times were actually quite higher than normal, and the only thing that really changed was our resource pool layout.

Our resource pools are all share based, with no reservations, and are all set to be expandable up to the root of the cluster. We've never had resource issues as each host hovers at about %50-%60 cpu and memory. The share design is really only there to make sure that production machines, in the event that we lose some hosts or add a lot of vm's, always get the correct share of the pie. With that said, i've not once seen the shares kick in. Additionally, all the graphs showed that the vm's were receiving 100% of their entitled resources (which I now disagree with the way that thing presents data).

So a bit flustered, I did what anyone with Platinum support would do... I called support.

The support engineer took about a minute to analyze the situation, told me there were CPU scheduling issues (which I knew of, but couldn't really help due to funding). What I didn't know was really how much of a scheduling issue there was. I've always wanted to make sure I was within spec for the %ready times and I always thought I was (5-15%), till he told me that anything in the double digits was really nothing to be pleased with if it was in a constant double digit state. 1/3 of vm's were between 11,17, and 20 percent (there were a fair bit in the single digits), some as high as 80, and a few hitting over 100 (those were the ones we had limits on, so they can be ignored- old apps that don't tame well).

We ran the ping test again from VM's on the same network but also on the same host, so that the ping would not leave the vSwitch and stay internal. This is where it got interesting. We hit the same results, ping times that were all over the place. He then looked at my resource pools and asked if we could remove them for the sake to prove a point he was about to make. I told him before doing so that there were no reservations set, everything was expandable, and my hosts were no where near being maxed. He reiterated that these pools were likely contributing to the issue and that the vmkernel does it's job best when left alone to the job by itself. At this point I was a bit confused, I thought our share based resource pool structure was only having an effect when resource contention existed, and as previously mentioned, all charts, graphs, and otherwise told me there wasn't resource contention (minus scheduling issues).

Long story short, we removed the resource pools and presto... Almost all vm's were in the single digits for %ready times now. Those not in the single digits spiked up for a short while into the double digits and came right back down. Those that were still high (above 50%) were still high because of my limits, so I can't really help those (nor do i want to). My resource pool structure before wasn't that deep either (2 equal parents with six childs each).

Anyway... I no longer have my resource pools and am now thrown off a bit about what I saw. The VM's are great and the only ones with crazy ping results are the ones with the limits.

Do you guys have any thoughts on this or have you experienced or heard of this? I was not aware of this and would be curious as to how other people are setting up their resource pools. I also don't remember reading anything about the type of impact a share based resource pool would have on the kernel when the shares were not in effect due to resource contention.