vmkping -s 9000

[tdubb123]

I have enabled jumbo frames on my ISCSI vmkernel nic but when I try to ping the iscsi san with

vmkping -s 9000 iscsi_san_ip

it does not respond.

any idea?

does something need to be enabled on the switchport itself?

[Dave_Mishchenko]

You'll want to enable it on the vSwitch (esxcfg-vswitch -m) as well and you networking equipment and SAN also have to be enabled for jumbo frames.

[AndreTheGiant]

As written by Dave you must enable Jumbo Frame on each point.

Note that for physical switches, this operation may require a reboot.

Andre

[tdubb123]

i enabled jumbo frames on the switch. but vmkping -s 9000 to the equalogic still fails

my vmnics, vmknic and dvswitch are all set at 9000 as well as the switch.

any ideas?

on the cisco side this is what we have set.

System MTU size is 1500 bytes

System Jumbo MTU size is 9000 bytes

Routing MTU size is 1500 bytes

[AndreTheGiant]

Try with -s 8000 (for example) to see if it works...

And check again on switch side, where probably it's the problem.

Andre

[tdubb123]

i tried pinging

~ # vmkping -s 8000 x.x.x.x

PING x.x.x.x.(x.x.x.x): 8000 data bytes

--- x.x.x.x. ping statistics ---

3 packets transmitted, 0 packets received, 100% packet loss

~ # vmkping -s 1400 x.x.x.x

PING x.x.x.x (x.x.x.x): 1400 data bytes

1408 bytes from x.x.x.x: icmp_seq=0 ttl=255 time=0.463 ms

1408 bytes from x.x.x.x: icmp_seq=1 ttl=255 time=0.439 ms

1408 bytes from x.x.x.x: icmp_seq=2 ttl=255 time=0.460 ms

--- x.x.x.x ping statistics ---

3 packets transmitted, 3 packets received, 0% packet loss

round-trip min/avg/max = 0.439/0.454/0.463 ms

~ # vmkping -s 1500 x.x.x.x.

PING x.x.x.x. (x.x.x.x.): 1500 data bytes

--- x.x.x.x. ping statistics ---

3 packets transmitted, 0 packets received, 100% packet loss

~ #

looks like it is still only accepting 1500bytes.

its a global setting on the switch. when i show the gigabit interfaces the mtu is set to 9000 on the switch

[tdubb123]

esxcfg-vswitch -l

DVS Name     Num Ports   Used Ports  Configured Ports  MTU     Uplinks  

dvS-mgmt       256                           8                     256  9000    vmnic2,vmnic1,vmnic0

--------------

esxcfg-nics -l

Name    PCI           Driver      Link Speed     Duplex MAC Address       MTU    Description                  

vmnic0  0000:03:00.00 bnx2        Up   100Mbps   Full   00:22:19:02:f0:e0 9000   Broadcom Corporation Broadcom NetXtreme II BCM5708 1000Base-T

vmnic1  0000:07:00.00 bnx2        Up   1000Mbps  Full   00:22:19:02:f0:e2 9000   Broadcom Corporation Broadcom NetXtreme II BCM5708 1000Base-T

vmnic2  0000:0e:00.00 igb         Up   1000Mbps  Full   00:1b:21:22:97:70 9000   Intel Corporation 82575GB Gigabit Network Connection

------------------

esxcfg-vmknic -l

~ # esxcfg-vmknic -l

Interface  Port Group/DVPort   IP Family IP Address                              Netmask         Broadcast       MAC Address       MTU     TSO MSS   Enabled Type               

vmk0       264                 IPv4      x.x.x.x.                              255.255.0.0     x.x.255.255   00:50:56:72:9f:aa     1500    65535     true    STATIC             

vmk1       266                 IPv4      x.x.x.x                              255.255.0.0     x.x.255.255   00:50:56:75:ef:5c      9000    65535     true    STATIC             

vmk2       103                 IPv4      x.x.x.x                              255.255.0.0     x.x.255.255   00:50:56:70:ad:c3      9000    65535     true    STATIC             

vmk3       135                 IPv4      x.x.x.x                              255.255.0.0     x.x.255.255   00:50:56:79:72:42      9000    65535     true    STATIC   

[tdubb123]

I tried connecting a vm to the equalogic ISCSI1 portgroup but I it only connects at standard frame length

Level: INFO

Time:  6/4/11 12:13:58 PM

Member:  EQUALOGIC

Subsystem:  MgmtExec

Event ID:  7.2.13

iSCSI login to target 'x.x.x.x:3260, iqn.2001-05.com.equallogic:0-8a0906-24a0cff09-fad18c4478a4de1f-iscsi-vol1' from initiator 'x.x.x.x.:57027, iqn.1991-05.com.microsoft:win7.xxx.com' successful, using standard frame length.

but the physical switch is already set at System Jumbo MTU size to 9000 bytes

System MTU and routing MTU  is at 1500bytes though.

[tdubb123]

I tried connecting a vm to the equalogic ISCSI1 portgroup but I it only connects at standard frame length

Level: INFO

Time:  6/4/11 12:13:58 PM

Member:  EQUALOGIC

Subsystem:  MgmtExec

Event ID:  7.2.13

iSCSI login to target 'x.x.x.x:3260, iqn.2001-05.com.equallogic:0-8a0906-24a0cff09-fad18c4478a4de1f-iscsi-vol1' from initiator 'x.x.x.x.:57027, iqn.1991-05.com.microsoft:win7.xxx.com' successful, using standard frame length.

but the physical switch is already set at System Jumbo MTU size to 9000 bytes

System MTU and routing MTU  is at 1500bytes though.

[tdubb123]

finally figured this out. looks like vmkping uses the vmk0 interface to get out. vmk0 was my vmotion interface and I did not set that at 9000. I guess it was not possible to set an interface with vmkping.

after i enabled jumbo on vmk0. it works. vmk1 and vmk2 are my iscsi interfaces

[AndreTheGiant]

So you have vMotion interface in the same network of other vmkernel?

Andre

[tdubb123]

yes. same network

[Josh26]

tdubb123 wrote:

after i enabled jumbo on vmk0. it works. vmk1 and vmk2 are my iscsi interfaces

This would indicate that vmk1 and vmk2 are on the same IP address range as vmk0 - I would recommend against this.

[tdubb123]

i dont have a choice. Its a small network in a dc. why can i not have both iscsi and vmotion on the same network?

[mittim12]

It's just best practice to keep your storage network isolated from other types traffic. That way you don't have to worry about anything interfering with the storage.

Sent from my iPhone

[tdubb123]

why am I able to ping huge packets to the equalogic

root@esx3 ~]# vmkping -s 11000 10.x.x.x.x

PING 10.x.x.x (10.x.x.x.): 11000 data bytes

11008 bytes from 10.x.x.x.: icmp_seq=0 ttl=255 time=0.628 ms

11008 bytes from 10.x.x.x.: icmp_seq=1 ttl=255 time=0.560 ms

11008 bytes from 10.x.x.x: icmp_seq=2 ttl=255 time=0.558 ms

it says a max of up to 65507?

I thought jumbo frames is 9000

[mcowger]

It is, but IP supports fragmented packets, which is what's happening here,  Your 65K packet is getting fragmented into multiple packets and reassembled on the other side.

[mcowger]

This is whats going on:

http://en.wikipedia.org/wiki/IP_fragmentation

[tdubb123]

before jumbo frames were enabled on the switches/nics etc.. i as able to ping only up to 1500bytes and anything over that will just get no response. by enabling jumbo frames, it allows up to 9000bytes. why would any packets over 9000 get a response?  i read only packets under 9000 will get a response

[tdubb123]

Strange. now vmkping from host to the equalogic IP is failing.

nothing was changed. and esxcfg-vswitch, esxcfg-vmknic, esxcfg-nics shows that jumbo frames is enabled