MD3000i + S/W iSCSI + Jumbo Frames

[manfriday]

Hi folks,

I 'think' I have set up jumbo frames in vSphere for use with my Dell MD3000i.

I followed the directions as per:

http://blog.scottlowe.org/2008/04/22/esx-server-ip-storage-and-jumbo-frames/

I am having an issue that other folks mentioned in that blog, but I never saw an answer to them..

Namely that when I have jumbo frames enabled I cannot vmkping my iscsi controllers with 9000 byte frames.

I get a response if I ping with 8991, but anything above that is not returned..

initially things 'seemed' to be working.. The ESX machines saw and connected to the storage just fine...

But strange little anomalies have cropped up..

Events like "Lost access to volume 495blahblahblah... dues to connectivity issues. Recovery attempt in progress.."

followed by "Successfully restored access to volume 495blahblahblah.."

I am using intel nics capable of JF's.

Cisco 2960g's, with JF's turned on.

MD3000i with JF's set to 9000.

I had read that it may be necessary to turn the JF's on the storage up a little high than 9000 to account for possible overhead introduced by the switch..

however, this is not possible on the MD3000i... The max allowed is 9000.

Perhaps I should try setting the JF's everywhere else to 8991? Or maybe I should scrap the jumbo frames altogether?

Thanks for any insight.

Jason

[AndreTheGiant]

You can reduce the MTU size on the vSwitch properties or the vmkernel interface.

But you case is strange, cause must work with 9000.

Are you using VLAN tagging?

Andre

[manfriday]

There are two VLANS set up on the switch.

the default (1) is being used for iscsi traffic and vlan 2 is for vmotion traffic.

So you think setting the JF's on the vSwitch & cisco's to 8991 rather than 9000 might be a possible solution?

[AndreTheGiant]

So you think setting the JF's on the vSwitch & cisco's to 8991 rather than 9000 might be a possible solution?

Usually for MTU is used the lowest value.

Something is cut your packet, so you can try or to reduce the MTU on one end node (vmkernel interface could be a good choice).

Or try to increase MTU to 9010 (for example) con cisco switch and/or vSwitch to see if your reach the right size.

Andre

[AndreTheGiant]

There are some similar problem on this page (search at the bottom):

http://blog.scottlowe.org/2008/04/22/esx-server-ip-storage-and-jumbo-frames/?nomobile

Andre

[manfriday]

Hi Andre,

Thanks for the assistance.

I did see that other folks on that page had the same issue as me, but I didnt see any resolution.

Someone mentioned increasing the MTU's on the storage to9014, but it is not possible to increase the MTU's beyond 9000 on the MD3000i.

I can confirm that it seems that 8 bytes of overhead are being added somewhere along the line..

I can vmkping the iscs ports on other ESX hosts with 9000 bytes and I get a response back with 9008 bytes.

If I vmkping the md3000i with 8991, I get a response back with 8999 bytes.

[GeneNZ]

@manfriday

Did you have any luck with this? I'm currently researching enabling of Jumbo Frames on our virtualisation cluster. We have 2 x MD3000i's (latest firmware) and several PE2950's connecting to these server connecting across Dlink DGS-1224T switches. Supposedly all the hardware, including the switches, support Jumbo Frames. The only problem is that all these systems are currently in production, but we have enough spare capacity to bring down one ESX server at a time. I can try out Jumbo Frames on my setup and report back (once I'm happy I've done enough research and understand the situation completely for the migration). I do have a couple of questions if you would be so kind to answer them for me. They maybe simple, but they are important enough for me to ask, especially since the system is in production:

- When you enabled Jumbo frames on your MD3000i, did it cause any downtime (e.g. require a reboot or anything along those lines)? Or is it essentially an on/off switch?

- Did you recreate your vSwitches and port groups as per the guide? Or did you use an existing vSwitch for iSCSI traffic

- Out of interests sakes did you have to reboot your switches to enable Jumbo Frames on your Cisco switches? Problem with the Dlink switches is they have a web interface, and the Jumbo Frames look like an on/off switch, so I can't tell if they are enabled or not. Plus I'd rather not reboot a switch currently in use. Unfortunately they don't have an IOS like the cisco gear for me to query these things.

Would be great if you could help me with those questions.

Gene

[manfriday]

Hi GeneNZ,

Yes, I have been using my MD3000i with Jumbo Frames for a little while now, with success.

Enabling it did not really cause "downtime", but it did cause path failovers.

When you enable jumbo frames on the MD3000i it reboots the controller. I did it one at a time, so that the MD3000i would still be accessible.

Same goes for the Cisco switches. They did require a reboot to enable Jumbo frames.

I blew away my vSwitches and recreated them as per the guide, which I could get away with because I hade multiple iscsi vSwitches on different lans (i.e a iscsi vswitch on 192.168.1.0 and a vswitch on 192.168.2.0)

You have probably seen this, but if you gave not, it is worth looking at..

http://www.delltechcenter.com/page/VMwareESX4.0andPowerVault+MD3000i

[GeneNZ]

Thanks for that, helps me alot.

Have you managed to solve the above problem? If has it shown to aggravate anything down the track?

Gene

[manfriday]

Oh, the problems. yeah I forgot about those.

The "Lost access to volume" issues I was having did get resolved and they were not related to Jumbo Frames at all.

I had bad controllers. It took some fighting but I managed to get Dell to replace both controllers and the mid-plane and since then I have not had any issues with failovers or lost access messages.

The ability to ping with 9000 byte frames was not resolved, but I am not so sure anything was ever wrong..

I could be wrong, so if I am someone can feel free to correct me, but when you ping something 8 bytes of ICMP are added to the packet.

So when I was trying to ping with 9000 bytes it was sending the md3000i 9008..

The MD3000i would not respond to that..

But if I sent it 8991 it would reply just fine.

Since getting the controllers fixed, I have not had any issues with jumbo frames turned on.

I will also say that I have not noticed a huge performance boost either.

It gave me a little boost, but it's not like the clouds parted and rays of golden sunlight broke thru and shined down on my datacenter while an angelic choir sang or anything like that.

[J1mbo]

> it's not like the clouds parted and rays of golden sunlight broke thru and shined down on my datacenter while an angelic choir sang or anything like that.

Nice! Out of interest, are you using flow control on your switches?

[athlon_crazy]

Just wonder, vsphere support JF, storage must support JF & same goes to pswitch. But how bout pNIC, do pNIC must also support JF? do anyone have a list of supported pNIC for JF?

vcbMC-1.0.6 Beta

vcbMC-1.0.7 Lite

[manfriday]

@ J1mbo,

Yes, I believe we are using flow control on the cisco switches. I think I remember trying to turn that off, but being advised by the Dell guy to turn it back on.

@ Athlon_Crazy

Yes, the pNic does need to support Jumbo Frames.

[GeneNZ]

@manfriday,

Thank you very much, your responses have been beyond helpful. If you could be so kind as to answer one further question for me: For VMWare backups, did you enable Jumbo frames for the VCB Proxy? I assume its just a network card setting.

Thanks again.

Gene

[manfriday]

Hi GeneNZ,

I did indeed enable jumbo frames on the VCB proxy, and yes it is just a setting in the iscsi nic properties on your windows host.

I don't really have any before/after performance data to give you by way of comparison though, as I have only just started experimenting with VCB.

Previously, I had been using vizioncore products.

[athlon_crazy]

May I know technical reason why you changed from Vizioncore to VCB? Is it because of COS performance degraded on vSphere?

vcbMC-1.0.6 Beta

vcbMC-1.0.7 Lite

[manfriday]

Yes, that is exactly why.

[GeneNZ]

@manfriday,

Thanks again for your help, we've migrated some of our systems to Jumbo Frames for testing this weekend. But based purely on benchmarks, I get either the same performance or worse performance. It depends on which benchmarking tool I used. I tried IOmeter, but it took to long to run so I gave up. With other tools like HDTune Pro, my average transfer rate crashed down from 40-50mb/s to something like 20-5mb/s. Whereas other benchmarking tools, gave similar performance to without Jumbo frames (which I would be completely happy with). As you said CPU load dropped, mine dropped from about 5% to close to 0%.

I do have one question, have you set your path policy to Most Recently Used or Round Robin? I've got mine currently set to Round Robin, and based on the discussion I'm having with another person on that link you posted previously, its possible that Jumbo Frames and Round Robin causes problems?

Gene

[manfriday]

We are using Round Robin.

As far as I am aware there is no problem with RR & Jumbo frames, but who knows.

Maybe someone more knowledgable than I will weigh in on that.

[GeneNZ]

Hrmm, definitely strange. Out of interests sakes are you running your ESX machines on PowerEdge 2950's? If so, are you putting your iSCSI traffic through the onboard Broadcom NetXtreme II interfaces?

Thanks,

Gene