MTU question

[BartekS]

Hi,

What are the correct MTU settings for EMC VNX 5300 (iSCSI) and HP BL490c G7 to benefit from hwiscsi(integrated NIC) and JumboFrames ?

On EMC VNX 5300 I can set MTU to (..., 8000, 9000) . EMC recommends not to set MTU to values different than available in drop down in Unisphere.

HP DL490 G7 comes with integrated NIC that offers ISCSI offloading. ESXi4.1 recognizes it as "Emulex NC553i". When Jumbo frames are enabled MTU is set to 8342, and it seems that Emulex recommends not to change it.

~ # esxcfg-hwiscsi -l vmhba0

DHCP Configuration Disabled

ARP Redirection Disabled

MTU Size = 8342

Question: What should I do, since I can't set MTU on both ends to match?

a) Assume that devices will negotiatiate optimal MTU size.

b) Use software iSCSI

c) Use hardware iscsi without jumboframes

Cheers,

Bartek

[Sreejesh_D]

Its a knows issue with NC553i. Refer the following KB from HP.

HP NC553i/NC553m 10Gb 2-port FlexFabric Converged Network Adapter - MTU Not Getting Up to 9k

Issue

When configuring the NC553i/NC553m card personality to be iSCSI from Emulex Oneconnect can be noticed the MTU for this card showing as 8342 instead of 9k when connected to a Virtual Connect module even when the VC modules has Jumbo Frames enabled by default as 9k.

Solution

When these cards are configured for HW iSCSI MTU would not go any bigger than 8342 as per card design, if 9k is needed by any specific reason then SW iSCSI must be configured instead of HW iSCSI.

http://bizsupport1.austin.hp.com/bizsupport/TechSupport/Document.jsp?lang=en&cc=us&taskId=110&prodSe...

[BartekS]

Hi Yezdi,

Yes, I have seen that article by HP. What about my question?

[Sreejesh_D]

I could see two options.

1. Use swiSCSI, so that you cans set the MTU to 9000 at both side.

2. Continue using hwiSCSI with MTU 8000.

I would recommend the second one coz the load on vmkernel will be less when we use HWiSCSI since TCP processing will be offloaded to the card.

[BartekS]

Thank you for your response. I posted this question because I would like to use hw iSCSI. Your advise is to set MTU 8000 on EMC and 8342 on esx? How different is it from MTU 9k and 8342?

Is it just a gut feeling or you have experience with setup like that?

[Robbk]

Never assume anything.

Don't use integrated ports. Bad practice. Put a NIC in there and you can control the frame size accordingly. Do some research for the appropriate card.

Failing that, set the frame to 9000 on the server and run an analyser against the data to see what is happening. the specification for the HP card is 8000-9000. If that isn't being set auto, then set it yourself. that should be fine.

MTU isn't usually, I might be wrong, set automatically at the filer/mover and at the server.

Hope this helps.

[Sreejesh_D]

its my gut feeling based on the inputs in first comment and HP KB.

on EMC VNX 5300 I can set MTU to (..., 8000, 9000) . EMC recommends not to set MTU to values different than available in drop down in Unisphere.

We cant use 9000 since its not supported at initiater end due to the HP bug. So the next option is 8000 at Unisphere and HWiSCSI initiator.

When these cards are configured for HW iSCSI MTU would not go any bigger than 8342 as per card design, if 9k is needed by any specific reason then SW iSCSI must be configured instead of HW iSCSI.

the KB says avalue  > 8342 is not supported, 8000 which is less than 8000 hence its supported. I havent tried it, i feel it should work fine, ie MTU 8000 at both side.

[BartekS]

Hi  Rob, THX for the tips.

I have sw iSCSI running on that hardware with MTU 9k without any problems. I just don't like the hit on CPU when I stress-test I/O.

MTU on the card is limited to 8342 when hw iscsi is enabled.

I use iSCSI on the block side of VNX.(I think it is not supported on file anymore)

Honestly I hoped that hardware I'm using is so common that someone did it before and can share the trick.

[BartekS]

Sreejesh@  I didn't find a way to change MTU on that harware iSCSI adapter in ESXi4.1U2. JumboFrames can be either enabled or disabled(and it chanegs MTU size). Do you know how to do it?

[Sreejesh_D]

Usucally the option will be available in HBA BIOS. Can you pls check there?

[BartekS]

yezdi@ I can't see that option in bios or OneConnect(Emulex) console.

[aqualityplacem]

We are using OCE11102 iscsi hardware initiators and had the same question. Just had this back from emulex

"There are no MTU settings because the adapter automatically adjusts the MTU based on the devices it communicates with"