We have now been running the Intel ESX hosts for 2 months. I thought I would send out what ESX is showing for the performance of VM’s that have made the move from AMD to Intel hosts.

The two metrics in the graph, CPU Usage is measured in percent and is the blue line. CPU Ready time is measured in the number of milliseconds that the VM spent waiting for a CPU to become available. CPU Ready is a summation of all of the wait time in the reporting interval which in this report is 1 day. So the red line represents the amount of time the VM spent waiting for a CPU through the entire day. Server Admins care about the percent of time that represents, so for ease of viewing, here are the numbers that I look for.

86400000 = milliseconds in a day. If a VM hits that number that would mean that it is getting no time on the CPU. Very bad
4320000 = 5% of the milliseconds in a day. This is what ESX admins target.
The right side of the graph shows the milliseconds scale.
The left show the percent CPU usage scale.

Ready time metrics were not retained prior to the vSphere update. That is why the red line is at zero for the old data.

I expected CPU Ready Time to drop. The unexpected benefit is what it did to the percent of CPU that the VM’s use. This is a very nice side effect that will make me more comfortable about putting more VM’s on 8 Intel cores than we do on the AMD cores.

Some of this can be attributed to Intel’s use of Hyper Threading and ESX4. ESX4 makes very good use of hyper threads. So, given enough RAM, 15 VM’s on an 8 core AMD host, may perform similarly to 30 VM’s on an 8 core, 16 thread Intel Host.

Our Intel hosts are currently running as many VM’s per host as the AMD hosts. So this should be very close to apples to apples.

Time will continue to tell the story, but I think this has been a very successful change.

Host details
Dell R805, 2 Quad Core AMD Opteron 2376, 32 GB RAM
Dell R710, 2 Quad Core Intel Nehalem L5520, 32 GB RAM
Both connect to the same EqualLogic SAS iSCSI SAN.
Both have 2 Quad port Intel Pro 1000 VT NICs with 8 cables to connect the SAN, LAN, and Management.

I have been impressed with many features of ESX4 and was again today. We currently, in ESX 3.5 U4, have VM's connect to their iSCSI volume with a software iSCSI initiator within the VM for performance reasons. This way we can give the VM multiple NICs in the SAN network and have it use multiple Gbps links to the volume. When using a VMDK on an iSCSI volume, we are llimited to a single Gbps link. That extra Gbps link or two really makes a difference.

With ESX4 supporting MPIO, we can now justify removing the iSCSI software from the VM and move all the data to a VMDK with a performance improvement, as the host has more connections to the SAN than the VM did.

Problem: In our current environment, when we snapshot the VM, system state is all that is snapped. Reverting only affects the system. If we move to VMDKs, now the snapshot affects data as well. This would be bad to roll back to a snapshot for an Exchange server.

Solution: Put the data VMDK into persistent mode so snapshots do not affect it. Great!

Another problem: Independent disks cannot be migrated with the VM online as snapshots cannot be taken.

Today I tested this on ESX4. It works. I have a VM with two disks, one is independent. Storage vmotion moved both disks.

Again, very impressed.

Charles Killmer