some iSCSI related questions, vmk, multipath, jumbo frames

[trickybits]

Recently I started playing around with iSCSI multipath in our lab environment, and also setting up jumbo frames.

Regarding jumbo frames I managed to set the MTU to 9000 on the vmk portgroup, on the vSwitch, and then on the physical switches and the SAN. My tests indicate that it works well.

I'm curious though, what exactly is the benefit of enabling Jumbo frames on the vmk portgroup? Some people mention that you should do it, and some don't mention it at all. What's your take on it?

Regarding iSCSI multipathing, I have done the following. Two vmk portgroups assigned to two physical nics (one active and one disabled, and then mirrored for the second vmk, so basically one physical nic for each vmk).

I have then assigned the two vmk's to the iSCSI Software Adapter. I then checked that I have all the different paths showing up, and finally I set the multipathing on each datastore to Round Robin. This seems to work.

In this process, what struck as surprising was the step of assigning vmk's to the iSCSI Software adapter through CLI. Is this something that needs to be done only in a multipath setup? Because on our production environment we're only using single-path, but I have never assigned a specific vmk portgroup to the iSCSI Software adapter in this fashion. If someone could shed some light on this, I'd appreciate it!

Thanks!

[rickardnobel]

Regarding jumbo frames I managed to set the MTU to 9000 on the vmk portgroup, on the vSwitch, and then on the physical switches and the SAN. My tests indicate that it works well.

I'm curious though, what exactly is the benefit of enabling Jumbo frames on the vmk portgroup? Some people mention that you should do it, and some don't mention it at all. What's your take on it?

I would guess that if you do not enable jumbo frames on the VMK interface/portgroup then it would not use it at all. If jumbo frames shall be actually used it has to be known to the logical tcp/ip driver on both sides to be able to negotiate a larger "MSS size" in the TCP handshake.

Remember also that if you use any ping utility to test jumbo frames you must use the "dont fragment" option to really see if a larger lenght could be used.

[AndreTheGiant]

I'm curious though, what exactly is the benefit of enabling Jumbo frames on the vmk portgroup?

Jumbo Frames is end-to-end...

All points must support and use it... so initiators, vSwitches, switches and targets.

Regarding iSCSI multipathing

Each vendor has his own specific configuration.

Have a look at storage documentations or suggested best practices.

Andre

[mrkelder]

Can you post you command to do this, I have no idea what you mean. (the assigning vmk's to the SW iSCSI adapter)

[trickybits]

Remember also that if you use any ping utility to test jumbo frames you must use the "dont fragment" option to really see if a larger lenght could be used.

Good point. I use the -d argument while pinging. The odd thing is that I get "Message too long" on my -s 9000 pings, but I can get -s 8972 through (with headers 8980 byte ping). It's still bigger than 1500 so the jumbo frames are in effect.

[trickybits]

Can you post you command to do this, I have no idea what you mean. (the assigning vmk's to the SW iSCSI adapter)

Not assigning as such. But specifying which physical nic(s) to uplink the vSwitch where the iSCSI vmkernel portgroups are located. I chose two nics and on each vmkernel portgroup I specified override failover rules so that each vmkernel port gets a dedicated nic. This is easily done using the vSphere client.

[rickardnobel]

>The odd thing is that I get "Message too long" on my -s 9000 pings, but I can get -s 8972 through

That is actually expected! The IP header is 20 bytes and ICMP header is 8 bytes, which leaves 8972 bytes for ICMP payload.

"Jumbo frames" is also not really standardized, so it just means something larger than default 1500 bytes MTU, often around 9000 bytes, but sometimes up to slighly above 9200.

[trickybits]

Thanks for clearing that up! I'm all set then

[mrkelder]

Ah, now I understand. youre just saying its easely done in the GUI instead of the CLI.

Well, probibly because you need to do a lot of settings in the CLI, andsome in the GUI, its a choice.

I also switch back and forward between CLI and GUI when setting up my iSCSI Virtual Switch.

To bad this still isnt working from the Host Profile, but it doesnt take long to setup manually...