Vi3 on ESX 3.5 U3 - Poor network speed

[Metalgalle]

Hi everybody!

I have a 3 way Vi3 infrastructure built on a HP C3000 Blade enclosure.

I installed ESX 3.5 U3 on 3 blades, any of those having 32 Gb RAM, dual Opteron processor,4 Gb nic and 2 FC HBA.

My enterprise storage is built on a MSA2000 FC with combination of SAS and SATA disks.

My problem is that I report a slow neworking performance in my Vinfrastructure, and so I have done some tests transferring a dummy file of 500MB between VMs (all Windows XP SP3 or Windows 2003 SP2 and all with VMtools installed) and I reported that:

- Transferring files between VM on the same ESX server has a medium upload speed of 110 Mbps and a medium download speed of 140 Mbps

- Transferring files between VM on different ESX servers has a medium upload speed of 110 Mbps and a medium download speed of 70 Mbps

I also tried to transfer files between different Vswitches that belongs to different nic teams and different VLAN connected by a full Gbit firewall, but I noticed that my firewall nics never gone above 70 Mbps peak speed.

So, did you have some ideas about that?

Thanx in advance!

---

Metalgalle

[djciaro]

Firstly I would say that 140Mbps could not be considered as poor.

Second: check your NIC settings and make sure that they are running at the correct speeds: from the console you can use the following command:esxcfg-nics -l

also check that you have not changed any settings in the Virtual Switch, for example: such as enabling traffic shaping

and lastly: make sure that your ESX is not connected to a firewall/switch/hub that runs at lower speeds

If you found this information useful, please consider awarding points for "Correct" or "Helpful". Thanks!

[Metalgalle]

Firstly I would say that 140Mbps could not be considered as poor.

Ok, it is also my though, but I think also that the gigabit medium speed is al little bit higher....

Second: check your NIC settings and make sure that they are running at

the correct speeds: from the console you can use the following

command:esxcfg-nics

OK, checked, and all 12 nics are running at 1000Mbps

also check that you have not changed any settings in the Virtual Switch, for example: such as enabling traffic shaping

I have made no changes on the Vswitch except for the VLAN tag.

and lastly: make sure that your ESX is not connected to a firewall/switch/hub that runs at lower speeds

No, I'm sure that in my serverfarm all the equipment is Gbit and that there are no Hubs.

---

Metalgalle

[Jackobli]

I agree with djciaro, seems quite fast... No, these are MBit not MBytes...

How did you measure the speed? What kind of guest OS is involved? Guests with more than one vCPU?

What is the usual speed/throughput of the arrays on your SAN?

[Metalgalle]

OK, let's go to explain :smileyblush:

I agree with djciaro, seems quite fast... !http://communities.vmware.com/images/emoticons/happy.gif!No, these are MBit not MBytes... - My measurements are in Mega Bit Per Second, and so 140 Mbps would be that my 1Gbps backbone is used at 14%

How did you measure the speed? - I measured in a simple mode: NetBios file transfer of a 500 MB dummy file from/to the hosts involved in the measurement and measure the time that was needed to complete the transfer and so Mb / Sec = Mbps

What kind of guest OS is involved? - Windows 2003 R2 x86 and x64 Enterprise ENG, without AV installed

Guests with more than one vCPU? - Every server VM has a minimum of 2 vCPU installed

What is the usual speed/throughput of the arrays on your SAN? - If you mean the FC backbone, it is a 4Gbps multipath connection from FC Switches and the HBAs/Storage controllers.

---

Metalgalle

[Jackobli]

This are dual socket, dual core opterons?

For test purposes, could you reduce to one vCPU and retest?

There is no Vista or W2K8 involved (SMB v2)?

Related to your SAN, it read about FC, but how much throughput are the spindles (Array / RAID-mode) delivering?

For metering raw ip, you could use iperf or jperf.

[Metalgalle]

This are dual socket, dual core opterons?

This is my HW configuration (3 identically nodes)

- ProLiant BL465c G5

- 8 CPU x 2,3 GHz

- Quad-Code Opteron 2356

- Hyperthreading: Inactive

- Number of Nics: 4

- RAM 32 GB

For test purposes, could you reduce to one vCPU and retest?

It is a little bit complicated, but I will do this test ASAP

There is no Vista or W2K8 involved (SMB v2)?

No, there are no Vista or Windows server 2008 VMs/Hosts

Related to your SAN, it read about FC, but how much throughput are the spindles (Array / RAID-mode) delivering?

Excuse me, but I don't understand what you mean about that....what test I must do?

For metering raw ip, you could use iperf or jperf.

OK, testing with iperf I got about the same results, but specifing the tcp window size at 64K and the MSS between 1300 and 1500, i got a server (W2K3 SP2) to server (W2K3 SP2) comunication speed up to about 600 Mbps and a client (WXP SP3) to server (W2K3 SP2) communication up to about 250 Mbps tested with and without AV.

---

Metalgalle

[Metalgalle]

Hi everyone!

I've done a lot of tests in order to get an answer about this problem.

Also I formatted my 3 ESXi nodes and installed ESX 3.5 U4 on each one.

Networking speed is always the same, but I noticed that:

- Using File Sharing, Filezilla, ecc.. the network transfer speed is exagerated slow.

- Using iperf or other raw socket ip testers the network speed is good.

In my tests I changed also the driver of the Vnic and used the VMXNET as suggested, but none changed.

Watching the network task monitor in windows, I notice that when I use File Sharing or Filezilla, the network usage reaches

peaks of 25% but for a little little time (about 1-3 sec) and then falls down to zero and rest for 1-3 sec.

Instead of this, using the Iperf tool, the network usage reaches peaks of 30-35% and keep stayng there for as log as the connection

is open, ex. I tried to force the Iperf client to test the network for 3 mins. and the network monitor shows me that

it use 30% (average) of the gbit bandwith for all of the 3 minutes.

I'm thinking that there are a buffer issue somewhere that, when using windows sockets, get full and stops networking for a while.

I also watched for the disk usage in the storage manager and ti appears to be very very low...

I'm running out my ideas....

---

Metalgalle

[Metalgalle]

Hi Everybody!

Donig my tests I get onto the service console in order to get some useful debug info.

I have a question: if I have a problem related to physical disk performance or congestion, how can I see it?

---

Metalgalle

[AsherN]

Related to your SAN, it read about FC, but how much throughput are the spindles (Array / RAID-mode) delivering?

Excuse me, but I don't understand what you mean about that....what test I must do?

It means that although the FC channel is capable of delivering 4MBS, it does not mean that the drives themselves can.