VMware Cloud Community
Schorschi
Expert
Expert

10Gb throughput just not adding up when doing VM to VM throughput testing across separate ESXi 4 hosts?

Ok, straight Linux RHEL 5.4 on some hardware that has 10Gb Chelsio NIC, we get near 10Gb throughput. With same hardware running ESXi 4 and 16 VMs, we get 3-4 Gb/s total throughput across physical NIC. This had us scratching our heads, thinking we did not have NetQueue (VMDq) enabled right. Of course the VMs themselves max out at 1Gb/s, but with 16 all running IO generation, we should be pushing the 10Gb NIC.

We double checked settings, retested, no joy. We tested INTEL and Mellanox cards, similar issues. Validated firmware and drivers for ESXi 4 where at recommended levels. No matter what we do, the total throughput is never above 3-4Gb/s total. With 16 VMs, running 1GB NICs in the VMs, we really expected better than total 3-4GB/s. We used IOzone, InetPerf, etc.

I have seen a few comments where others with various other NIC vendors have experienced similar issues. Anyone out there tacked this issue? Got anything better than 3-4Gb/s total throughput from the physical NIC?

We have talked to VMware, and each of the vendors, they all quote maximum throughput above 9GB/s achieved, but the vendors quote any OS other than ESXi 4. VMware says, it should work, and do better than 3-4Gb/s but so far no settings or configuration options have made the magic happen.

Tags (3)
Reply
0 Kudos
5 Replies
christianZ
Champion
Champion

Have you checked the cpus utilisation by running the tests?

IRQ sharing? There were problems with network throughput by irq sharing.

Reg

Christian

Reply
0 Kudos
Schorschi
Expert
Expert

CPU load from ESX host perspective is fine, we are using Nehalem of course, and there is more than enough capacity for 16 VMs generating simple network IO. The IRQ sharing is interesting, we have 1 - 10GB card that we use for all network IO traffic in this specific scenario. So am not sure we expected that to be an issue. We are using HP 490 blades and IBM IDataPlex dx360 servers, rather simple devices compared to typical rack mount systems.

Reply
0 Kudos
christianZ
Champion
Champion

Have you a chance to check the irq mappings in the server bios?

The problematic sharing was usb and nic.

Reply
0 Kudos
Schorschi
Expert
Expert

If IRQ sharing was an issue, we would see it impacting the RHEL instance. RHEL on the same hardware performed well, 9GB/s or better. Same hardware same setup, running ESXi 4, not even half that. Even though I question the IRQ sharing impact, checking it out just the same. Smiley Happy

Reply
0 Kudos
J1mbo
Virtuoso
Virtuoso

Out of interest, how many physical cores does the box have?

Please award points to any useful answer.

Reply
0 Kudos