We have slightly less than 600 vms.
We are on 4.0 waiting for 4.1 u1 to consider upgrading.
Our team argued for days on whether it would make sense to switch from e1000 to vmxnet3.
We are split down the middle in opinion on whether it make sense to switch.
E1000 is slower, but in practice the difference is negligible for most work loads.
E1000 supports kickstart install an important point for mass deployment of rhel.
On some workloads vmxnet2 is faster, on others vmxnet 3 is faster, it depends on the average packet size.
Awhile ago I did some stress testing, vmotioning machines back and forth in perpetuity.
Occasionally perhaps less than 1% or perhaps .01% of the time the vmxnet3 cards wouldn't see the network, this never happened with the flexible card.
My original argument was that vmxnet3 was the newest and fastest and should be the default.
My argument has changed to the opinion that the e1000 or flexible driver has been around the longest and should be the absolute most stable, as it has been timetested the longest.
My default is to just go with whatever vmware suggests as the default, unless I think the machine will be heavily utilized.
If it's a new server with a new OS I will also give it a vmxnet3.
If I need to change the MAC address from inside the vm I give it vmxnet2.
Some of our system's do scalability loadtesting with e1000 card's. (rhel5.5)
The limiting factor is not the cards. I think vsphere 4.0 with update 2, now supports tso offload for the e1000 cards.
I cannot remember at what exact point this support appeared.
Proper kernel settings have more of an affect on cpu as it reduces the timer interrupts, with any network card.
The newer network cards generally have lower background cpu load.
I just spent the last month migrating some machines from vmware server to esxi 4.0.
If you are considering windows machines, you have to remember that windows activation can bite you in the ass if you change:
cpu's.
memory.
Disk controller.
Hardware version.
In addition the foreign language versions of windows are pickier on deciding when to reactivate.
If you don't have the newer vmware tools installed and you upgrade from hw 4 to hw 7 it will always reactivate.
Just relying on synthetic benchmarks to determine the optimal performance is suboptimal.
For example synthetic benchmarks determined that virtualizing mysql would give us terrible performance, as we compared raw disk io between physical hardware and virtual hardware. In practice our virtualized mysql servers worked better than physical.
Mysql does row locking on databas drops which prevents multiple database creates and drops from occurring asynchronously, so having multiple little mysql servers instead of one big physical server, dramatically improved our multi database performance.
On vm upgrades or conversions leave the hardware alone unless forced to change.
Then if you find that a machine has performance issues or you think it will, you can focus on that small subset, and change the hardware.
Focus your manual effort on the items that have immediate benefit.
