Joshn
Enthusiast
Enthusiast

How to enable jumbo frames for iSCSI traffic?

Jump to solution

Hi all,

I have my iSCSI traffic going out of a dedicated NIC in a separate vlan.

Now I have enabled the switch for jumbo frames, the qnas 869 pro for jumbo frames also and I have (as far as I can tell) enabled my ESXI host for jumbo frames.

However on the switch stats I cannot the jumbo frames counters increase.  others are but not jumbo.

After reading about this I though that jumbo frames can help increase performance? 

Therefore I'd appreciated it if you could help me!

Thanks,

Josh

1 Solution

Accepted Solutions
Joshn
Enthusiast
Enthusiast

Well some how I made the jumbo frames work on my ESXI host.

I enabled Jumbo frames for every single interface on the ESXI host and on my NAS which is providing the iSCSI (qnap 869 pro) I enabled jumbo frames on both interfaces on that.

Now it's working no problem!

View solution in original post

0 Kudos
18 Replies
julienvarela
Commander
Commander

Hi josh,

The best way is to use 2 network card for iSCSI traffic and use port binding  : http://kb.vmware.com/selfservice/microsites/search.do?cmd=displayKC&docType=kc&docTypeID=DT_KB_1_1&e...

You need to enable jumbo frame on your vmkernel interface and physical switch :VMware KB: iSCSI and Jumbo Frames configuration on ESX/ESXi

And configure round robin multipathing.

Regards,

Julien.

Regards, J.Varela http://vthink.fr
0 Kudos
Joshn
Enthusiast
Enthusiast

thanks for the reply mate but I don't see how port binding will help me?

I just want to make sure my configuration is correct to enable jumbo frames.

Should the below not be enough?

Or do I need to create a vDS ?

http://i.imgur.com/VaEViL8.png

http://i.imgur.com/wNy45c6.png

0 Kudos
julienvarela
Commander
Commander

Hi,

Port binding is a best practice for help you to increase performance.

As i can see, you have correctly configure the MTU size on your ESX. you need to configure jumbo frame on your physical switch, did you already do that?

regards,

Julien

Regards, J.Varela http://vthink.fr
0 Kudos
Joshn
Enthusiast
Enthusiast

Right I'll go have a play around with it and try to get this port binding working!

And yes physical switch is set to 9200 (to give headroom) and the Qnas is set to 9000.

0 Kudos
Joshn
Enthusiast
Enthusiast

okay it's bound now - just the 1 physical NIC as the qnas only has 1 physical NIC dedicated to iSCSI traffic.

http://i.imgur.com/TPibL9A.png

0 Kudos
a_p_
Leadership
Leadership

I see you have Promiscuous Mode set to Accept on the vSwitch and also enabled vMotion on the iSCSI port group!? Was this done by intention?

Anyway, you may want to take a look at http://rickardnobel.se/troubleshoot-jumbo-frames-with-vmkping/ for some hints on how to troubleshoot Jumbo frames.

André

Joshn
Enthusiast
Enthusiast

yes it was done intentionally.

I thought vmotion traffic should be there as this is the only connection to the SAN?

0 Kudos
Joshn
Enthusiast
Enthusiast

Right I've had a look at that link you gave me mate.

It does work - it sends the frame unfragmented as far as the ESXI host is concerned.

HOWEVER...the switch does not see the counters increasing for jumbo frames (any frame over 1522 bytes in size)

I've checked and double checked my switch config and it looks right.  As does the SAN.

0 Kudos
a_p_
Leadership
Leadership

vMotion traffic isn't really related to SAN traffic and should be separated. If - in your configuration - vMotion starts migrating VM workloads, you may run into performance (or even more severe) issues with SAN traffic, due to the bandwidth vMotion uses.

If all the Jumbo frame tests succeed, I'd start checking whether there's a setting (or firmware update) available on the switch!?

André

Joshn
Enthusiast
Enthusiast

Guys I really can't see any reason for my ESXI host to not be sending out jumbo frames...

Can someone help with this?

I think I have it all setup right - screen shots of my MTU settings on my iSCSI network adapter:

http://i.imgur.com/3zLD78H.png

http://i.imgur.com/kBZFS5h.png

And my port binding:

http://i.imgur.com/LVVK8Qd.png

And those are the only settings I can see relating to this...

Something Interesting I noticed was that when I send a ping out of my ESXI host to the SAN as per that article: http://rickardnobel.se/troubleshoot-jumbo-frames-with-vmkping/

when the frame size is 1000 it takes 0.1 ms to return.

But when it is 9000 in size it takes 0.9 ms to return.

This would suggest that something is having to fragment the packets.  Most likely my esxi host as the switch cannot see ANY jumbo frames coming in on that interface.

0 Kudos
a_p_
Leadership
Leadership

Did you try vmkping -s 8972 -d IP? Does it succeed or does it return "message too long"?

André

0 Kudos
Joshn
Enthusiast
Enthusiast

it succeeds..but the counters on the switch are not increasing.

plus it takes about 8 times as long to send/receive the packets which leads me to believe that something is getting fragmented.

I also did a packet capture and all the other packets are under 1500 in size apart from the pings.

Surely it shouldn't be like that?

0 Kudos
a_p_
Leadership
Leadership

If vmkping -s 8972 -d IP succeeds, but e.g. vmkping -s 9000 -d IP does not, this would prove that Jumbo frames generally work from end to end. Unfortunately I don't see anything wrong in the host's iSCSI network setup.

Do you have another system which you could configure for Jumbo frames to see whether this works with the storage system as expected?

André

Rafel
Contributor
Contributor

Yes or not ?

0 Kudos
Joshn
Enthusiast
Enthusiast

Rafel - I don't understand what you are trying to say mate?

0 Kudos
a_p_
Leadership
Leadership

@

Welcome to the Community,

if you want to be notified of new replies in a discussion, you can simply click the "Receive email notifications" in the "Actions" box, and don't need to post a reply (which in this case is rather confusing).

André

Joshn
Enthusiast
Enthusiast

Still stuck on this gents..

re-cap:

So far as I can tell Jumbo frames are enabled and tested working end to end using the command line on the esxi host.

But when I do a packet capture I cannot see ANY packets over about 1500 in size.

It's as if the ESXI host just doesn't want to use jumbo frames even though the interfaces allow it.

0 Kudos
Joshn
Enthusiast
Enthusiast

Well some how I made the jumbo frames work on my ESXI host.

I enabled Jumbo frames for every single interface on the ESXI host and on my NAS which is providing the iSCSI (qnap 869 pro) I enabled jumbo frames on both interfaces on that.

Now it's working no problem!

View solution in original post

0 Kudos