First, in all the documents I have read, VMware recommends that you install VMware Tools in your VM's so they can "manage" better the resources and the resources are at the disposal of the VM who needs it, maybe you all ready know this but just in case.
I have a similar problem, not so critical VM's are taking resources that top App VM's do need them, so I have put a resource pool for the low priority VM's and the TOP VM put them at the root directory so far every thing it's working fine and my TOP VM's are operating very well
Using Vcenter 4.1 and trying to set up some resource management to ensure a few critical machines perform well. So I set up resource pools. I make basically high, medium and low tier pools and assign the high, normal, a low shares respectively. But then I only have a few machines in the high pool and a few in the low pool, with the bulk in normal.
First mistake Resource pools are NOT performance containers, they are Memory and Security containers, you can keep machines from running your hosts into the ground with RP, but RP make things MUCH easier to manage, I believe you are overthinking this.
Just so you know.. high, medium low makes ZERO difference with respect to priority. I will type that again.. there is NO priority on ESX.. Priority is there *IF* --- *IF* your hosts are low on resource.. ONLY if *AND* when yours hosts are low do priorities matter. Because if the hosts have enough CPU and RAM, there is no need to proritize your VM's.. the idea is to get enough hardware for your VM's, which is why do VM capacity planning in the begginning.. if you have issues with not enough CPU or RAM you didn't do your homework and didn't buy enough machines for your environment. You NEVER want to run that lean.. Priorities shouldn't ever become an issue as a result. If you want to ensure certain VM's have enough resource that's when you set reservations and set them to high priority In CASE a VM (or more) takes over and consumes the resources... But that means you didn't setup the resource pools correctly either, that really should not happen. VM configuration is key as well, yeah users complain they WANT a 4G RAM VM but do they REALLY need it, probably not.. Transparent Memory share, DRS are key tools and Windows VM's can share about 30% of the kernel on same host.. so you don't need to give VM's that much RAM. This should also be a part of your planning. Good performance monitoring will give you clues where your VM hogs are as well.
Planning is key with ESX, you MUST look at your WHOLE environment and plan accordingly. RP are not a managment nightmare, they are a neccessity, and they have the added benefit of security and VM isolation (for user rights) so when users need a group of VM's you can easily identify which RP they need access to.