How to enable jumbo frames for iSCSI traffic?

[Joshn]

Hi all,

I have my iSCSI traffic going out of a dedicated NIC in a separate vlan.

Now I have enabled the switch for jumbo frames, the qnas 869 pro for jumbo frames also and I have (as far as I can tell) enabled my ESXI host for jumbo frames.

However on the switch stats I cannot the jumbo frames counters increase.  others are but not jumbo.

After reading about this I though that jumbo frames can help increase performance? 

Therefore I'd appreciated it if you could help me!

Thanks,

Josh

[julienvarela]

Hi josh,

The best way is to use 2 network card for iSCSI traffic and use port binding  : http://kb.vmware.com/selfservice/microsites/search.do?cmd=displayKC&docType=kc&docTypeID=DT_KB_1_1&e...

You need to enable jumbo frame on your vmkernel interface and physical switch :VMware KB: iSCSI and Jumbo Frames configuration on ESX/ESXi

And configure round robin multipathing.

Regards,

Julien.

[Joshn]

thanks for the reply mate but I don't see how port binding will help me?

I just want to make sure my configuration is correct to enable jumbo frames.

Should the below not be enough?

Or do I need to create a vDS ?

[julienvarela]

Hi,

Port binding is a best practice for help you to increase performance.

As i can see, you have correctly configure the MTU size on your ESX. you need to configure jumbo frame on your physical switch, did you already do that?

regards,

Julien

[Joshn]

Right I'll go have a play around with it and try to get this port binding working!

And yes physical switch is set to 9200 (to give headroom) and the Qnas is set to 9000.

[Joshn]

okay it's bound now - just the 1 physical NIC as the qnas only has 1 physical NIC dedicated to iSCSI traffic.

[a_p_]

I see you have Promiscuous Mode set to Accept on the vSwitch and also enabled vMotion on the iSCSI port group!? Was this done by intention?

Anyway, you may want to take a look at http://rickardnobel.se/troubleshoot-jumbo-frames-with-vmkping/ for some hints on how to troubleshoot Jumbo frames.

André

[Joshn]

yes it was done intentionally.

I thought vmotion traffic should be there as this is the only connection to the SAN?

[Joshn]

Right I've had a look at that link you gave me mate.

It does work - it sends the frame unfragmented as far as the ESXI host is concerned.

HOWEVER...the switch does not see the counters increasing for jumbo frames (any frame over 1522 bytes in size)

I've checked and double checked my switch config and it looks right.  As does the SAN.

[a_p_]

vMotion traffic isn't really related to SAN traffic and should be separated. If - in your configuration - vMotion starts migrating VM workloads, you may run into performance (or even more severe) issues with SAN traffic, due to the bandwidth vMotion uses.

If all the Jumbo frame tests succeed, I'd start checking whether there's a setting (or firmware update) available on the switch!?

André

[Joshn]

Guys I really can't see any reason for my ESXI host to not be sending out jumbo frames...

Can someone help with this?

I think I have it all setup right - screen shots of my MTU settings on my iSCSI network adapter:

And my port binding:

And those are the only settings I can see relating to this...

Something Interesting I noticed was that when I send a ping out of my ESXI host to the SAN as per that article: http://rickardnobel.se/troubleshoot-jumbo-frames-with-vmkping/

when the frame size is 1000 it takes 0.1 ms to return.

But when it is 9000 in size it takes 0.9 ms to return.

This would suggest that something is having to fragment the packets.  Most likely my esxi host as the switch cannot see ANY jumbo frames coming in on that interface.

[a_p_]

Did you try vmkping -s 8972 -d IP? Does it succeed or does it return "message too long"?

André

[Joshn]

it succeeds..but the counters on the switch are not increasing.

plus it takes about 8 times as long to send/receive the packets which leads me to believe that something is getting fragmented.

I also did a packet capture and all the other packets are under 1500 in size apart from the pings.

Surely it shouldn't be like that?

[a_p_]

If vmkping -s 8972 -d IP succeeds, but e.g. vmkping -s 9000 -d IP does not, this would prove that Jumbo frames generally work from end to end. Unfortunately I don't see anything wrong in the host's iSCSI network setup.

Do you have another system which you could configure for Jumbo frames to see whether this works with the storage system as expected?

André

[Rafel]

Yes or not ?

[Joshn]

Rafel - I don't understand what you are trying to say mate?

[a_p_]

@Rafel

Welcome to the Community,

if you want to be notified of new replies in a discussion, you can simply click the "Receive email notifications" in the "Actions" box, and don't need to post a reply (which in this case is rather confusing).

André

[Joshn]

Still stuck on this gents..

re-cap:

So far as I can tell Jumbo frames are enabled and tested working end to end using the command line on the esxi host.

But when I do a packet capture I cannot see ANY packets over about 1500 in size.

It's as if the ESXI host just doesn't want to use jumbo frames even though the interfaces allow it.

[Joshn · Accepted solution]

Well some how I made the jumbo frames work on my ESXI host.

I enabled Jumbo frames for every single interface on the ESXI host and on my NAS which is providing the iSCSI (qnap 869 pro) I enabled jumbo frames on both interfaces on that.

Now it's working no problem!