PabloJP
Contributor
Contributor

Horrible iSCSI performance - ESXi 5.5 + Storwize V7000

Hello folks,

Subject says it all. Here's the scoop:

Back end: IBM Storwize V7000, two canisters, each one with 2 x 10Gb connections (total of 4 x 10Gb connections)

Front end: ESXi 5.5 U1., configured with iSCSI,

2 x 10Gb Intel 82599 network cards;

2 vmdk portgroups for iSCSI, each bound to a different physical adapter as per best practices;

MTU is enabled at 9000 all the way to the storage.

I initially configured the host with FC to the same storage. Using a single VM running IOmeter, I was able to get 64,000 IOPS from SAS drives and 96,000 IOPS from SSD drives. Thus, I know my back end can provide 64,000 / 96,000 IOPS.

Upon replacing the HBA with the iSCSI software adapter, I get a mere 46,000 IOPS for BOTH SAS and SSD. I was expecting a drop, but not that much. Also, the fact that SAS and SSD run at the same speed gives me the pointer that it's not the storage, it is the medium.

Here's what we've tried:

- it is not the path: we suppressed the network switches (Nexus 5K) and plugged the storage directly into the ESXi server.

- We played disabling/enabling Delayed ACKs.

- We played disabling/enabling TSO/LRO

- We tuned the queue depth

- We tuned the ixgbe driver for the Intel 82599, disabling interrupt moderation

- We played with multipath: disabled it, re-enabled it, changes the IOPS rotation to 1.

- We changed the TCP/IP stack heap size to give it more room.

- Upon monitoring, we don't see the ESXi network interfaces as being busy, nor the iSCSI heads on the other end being busy. Low network, low CPU, and low Storage utilization on the SAN.

The only think I can come up with is the TCP/IP stack on ESXi. The stack is optimized for VMs data transfer, and I think vmdk traffic goes to a second place. Also, I haven't found any optimization for TCP/IP Window size and Segment size (as Windows has).

Is there anything that you folks can come up with? Any insight will be really appreciated.

Thanks all,

Pablo

Tags (3)
0 Kudos
5 Replies
rDale
Enthusiast
Enthusiast

Did you find a solution to this ?

0 Kudos
VMikeZ
Contributor
Contributor

Hello PabloJP,

have you tried to set MTU on Storwize eth ports?

Example:

cfgportip -mtu 9000 -iogrp 0 1

0 Kudos
lsonzogni
Contributor
Contributor

Hello Pablo,    

  I have the same situation.    

With some differences.

I have a Flex, x240 Nodes and Flex V7000 storage node.  All is working with ISCSI CNA 10Gbps.  The performance is horrible. And i can't get no much support by IBM people.

Can you get any advance???

Already configure my V7000 with VMikeZ suggestion but...no changes.

0 Kudos
joergriether
Hot Shot
Hot Shot

Hi,

i´d like to point you to a thread i answered to recently. You have tried so much - but you haven´t tried something which is actually very easy to try - you just have to touch your esxi ;-). So - to eleminate an MTU issue i suggest you disable jumbo frames at esxi level. Just for a short test.

Please also read my statement regarding jumbo frames in general: Re: VSAN "misconfiguration detected"

Best,

Joerg

0 Kudos
lsonzogni
Contributor
Contributor

Just updating this issue. Finally all is working.

All my config was OK. After update V7000 to 7.2.0.11 version, firmware update to emulex components to version 10.3.144. VMWare Esxi 6.0 and drivers be2iscsi to version 10.4.255, the performance is normal.

By the way, after update V7000 software the iSCSI port go back MTU to 1500...so again must be changed to 9

0 Kudos