Oh forgot to say the max vcpu any of the VMs on that host is 4
the bios is set to OS controlled and the hosts are set to high performance. This is the same across all 10 hosts in the cluster and if i move a vm of this one host the contention drops. when i move it back it jumps up again
So you see this behavior on one single node of that cluster.. Rest all dont show the contention?
Can you also check the CPU Latency counter in vCenter to see what it is reporting for this VM when it is running on that host.
Also, are these VMs running high Network Throughput or Storage I/O. When I say high, I mean very high :-)