iscsi perfomance

[nettech1]

I have a Dell MD3000i with 5 SATA drives (RAID5) connected to a Dell 5424 switch.

ESXi 5.5 has two nics connected to the same switch for iscsi traffic.

The set up as follows:

MD3000i

RAID Controller 0,0 - 192.168.1.1 (Jumbo Frames set to 9000)

RAID Controller 0,1 - 192.168.2.1 (Jumbo Frames set to 9000)

RAID Controller 1,0 - 192.168.1.2 (Jumbo Frames set to 9000)

RAID Controller 1,1 - 192.168.2.2 (Jumbo Frames set to 9000)

Dell switch 5424 has jumbo frames enabled

ESXi has two vmkernal ports (1 for each nic) 192.168.1.10 & 192.168.2.10 and jumbo frames are set to 9000 on both.

when I read data (copy a 4GB file to a VM hosted on local storage of ESX) I get a steady 50MB/s transfer, but when I try to save something on SAN (copy the same 4GB file from desktop of a VM using local storage to SAN) the transfer starts at 270MB/s drops to a 0 somewhere in the middle and then flattens out at 50MB/s

Any ideas what could cause such a sporadic transfer rates?

Thanks

[JPM300]

That looks like a Windows 2012 Server Windows.  Is this only happening on a Windows 2012 Server.  If you try the same datastransfer to a Windows 2008R2 or 2003 R2 do you get the same speeds?

Also add an additional 2 drives to the current system you are testing drive 1 SCS LSI logic controller, Drive 2 Paravirtual Controller.  Test the copy to both drives and see if there is any difference.

Let me know,

[nettech1]

correct this is happening on 2012 R2 with LSI.

Will test results on other versions and post them here

should I look in to disabling delayed ack?

Link

[nettech1]

transfer rates look worse on a 2003 server

Writing at 15MB/s

Reading at 31MB/s

[JPM300]

Hmmm, okay, in the past I have seen some weird speeds come out of Server 2012.  If you check ESXTOP what is your DAVG this is the metric that shows latency or how long it takes to send a command to the SAN and back.

This is the setup for ESXi 4.0 but its still the same in 5.5:

http://en.community.dell.com/techcenter/virtualization/w/wiki/3043.aspx

I'm sure you have followed all of this already since your settings look similar.

Also your nics that your using for iSCSI in the ESXi hosts, what kind of nics are they?  Broadcom, intel, ?

[vfk]

I think this has something to do with flow control at the switch level, try enabling flow-control.  VMware KB: High disk latency observed on Dell MD3000i storage array

Dell recommends that you enable Flow Control on the switch ports that handle iSCSI traffic http://www.dell.com/downloads/global/solutions/public/white_papers/IP-SAN-BestPractice-WP.pdf

[nettech1]

Switch is configured based on this post. Flow Control is ON

http://en.community.dell.com/techcenter/storage/w/wiki/2721.configuring-a-powerconnect-5424-or-5448-...

Software Version  2.0.0.46    Boot Version  2.0.0.0    Hardware Version  00.00.02     

[nettech1]

copying 4GB from iscsi Storage to the same iscsi Storage

[schepp]

Have you also set the 9000 MTU on the vSwitch where the two iSCSI Nics are connected?

SSH into the host and test if you can reach your storage with 9000 MTU:

vmkping -d -s 8972 x.x.x.x

VMware KB: Testing VMkernel network connectivity with the vmkping command

[JPM300]

What kind of NIC's do you have in the host that is having the problem?  Do all the hosts get the same slow transfer rates?  What kind of disks are in the Dell SAN? SATA7.2k, SAS10k, SAS15K?

[nettech1]

vmkping -d -s 8972 192.168.1.1
PING 192.168.1.1 (192.168.1.1): 8972 data bytes
8980 bytes from 192.168.1.1: icmp_seq=0 ttl=64 time=1.947 ms
8980 bytes from 192.168.1.1: icmp_seq=1 ttl=64 time=1.626 ms
8980 bytes from 192.168.1.1: icmp_seq=2 ttl=64 time=1.766 ms
3 packets transmitted, 3 packets received, 0% packet loss
round-trip min/avg/max = 1.626/1.780/1.947 ms

[nettech1]

they are 5 SATA7.2k drives

[nettech1]

[nettech1]

there is only one host with 4 onboard NICs and 2 port BCM5720 NIC in PCIx slot

[JPM300]

Is your driver up to date for the Broadcom 5720?

I also found this article, however I don't think this is related to your problem

VMware KB: Broadcom 5719/5720 NICs using tg3 driver become unresponsive and stop traffic in vSphere

Driver:

https://my.vmware.com/group/vmware/details?downloadGroup=DT-ESXI55-BROADCOM-TG3-3134EV551&productId=...

So this host with the Broadcome 5720 nic, is this the only nic having the problem?

[nettech1]

it's using tg3 driver.
Version: Version 3.123c.v55.5, Build: 1623387, Interface: 9.2 Built on: Feb 21 2014
will try updating. onboard nic is the same make and model have not used it for iscsi
I also came across this KB - Link

[JPM300]

Hey Nettech1,

yeah I think updating the driver would be best, if that doesn't work try turning off TSO.  I used to always disable TSO by default in our deployments however some of the broadcom drivers this can only be done with the windows ones.  Hopefully the new driver works or at least gives you the ability to disable TSO to try it.

[JPM300]

I acutally found a newer driver off the VMware site on the driver sectioN:

VMware ESXi 5.5 Driver CD for Broadcom NetXtreme I Gigabit Ethernet including support for 5717/5718

tg3-3.135b.v55.1-1502707.zip

So it looks like you have a few options, but this appears to be the newest, at least form what I can find.

[vfk]

It would be interesting to know if the updating the network drivers makes a significant difference to performance issue.  Please keep us updated, one would usually assume the latest esxi would contains reasonably up-to-date drivers unless released since the image went out.

[nettech1]

this is what I saw in the KB I linked and was gonna look for version 3.135b. Since all nics in this ESX are the same make and model. when I upgrade iscsi driver it's gonna update the vm network drives as well or I can tell it to update specific nics ?