vmkping -d -s 9000 sendto () (message too long)

[tdubb123]

I am getting an error when doing a jumbo packet ping to my iscsi storage

everything has been set to 9000mtu end to end

vmkping -s 9000 storageip

responds fine

but when i put the -d flag in I get

sendto () failed (message too long)

any idea?

I tried vmkping -I vmkernel_nic as well but same problem

[a_p_]

You need to subtract some header data bytes when you use the ping command. Use 8972 instead. See http://rickardnobel.se/troubleshoot-jumbo-frames-with-vmkping/ for details

André

[tdubb123]

still doesnt work

1 packets transmitted, 0 packets received, 100% packet loss

/vmfs/volumes/504d1fa5-1c455cec-0519-0024e85c1297/pa-vc # vmkping -I vmk5 -d -s 8972 10.60.1.106

PING 10.60.1.106 (10.60.1.106): 8972 data bytes

sendto() failed (Message too long)

sendto() failed (Message too long)

sendto() failed (Message too long)

[a_p_]

In this case you need to check which part in the chain isn't configured properly.

port groups -> vSwitch -> physical switch -> storage system

André

[zXi_Gamer]

If you are using 5.5, then you can use  pktcap-uw to identify where the drop is happening

[MKguy]

Packet tracing won't work, and the physical infrastructure can't be blamed (at least yet) because the packets are dropped by the host's own networking stack before they actually reach the wire.

I've seen that sometimes a host needs an actual reboot after enabling jumbo frames.If it still doesn't work after a reboot, try this:

1. Please post the output of the following commands on the local shell:

# esxcli network nic list

# esxcli network ip interface list

# esxcli network ip interface ipv4 get

# esxcli network ip route ipv4 list

# esxcli network vswitch dvs vmware list

# esxcli network vswitch standard list

# esxcli network nic list | awk {'print $1'} | grep vmnic | while read nic; do esxcli network nic get -n "$nic" ; done

# vmware -v

2. Try some smaller payload size values like 8000, 4000, 2000.

3. Can you vmkping the host on it's own localhost and vmkernel interface IP? E.g.:

# vmkping -I vmk5 -d -s 8972 [IP of vmk5]

# vmkping -d -s 8972 127.0.0.1

4. Try the tests with esxcli network diag ping utility as well, e.g.:

esxcli network diag ping --df --interface vmk5 --ipv4 --size 8972 --host 10.60.1.106

5. Also make sure your NIC drivers and firmware are up to date.

[tdubb123]

vmkping -d -s 8972 127.0.0.1 responds fine

but

vmkping -I vmk5 -d -s 8972 [IP of vmk5]fails

esxcli network diag ping --df --interface vmk5 --ipv4 --size 8972 --host 10.60.1.106

fails

[King_Robert]

by default the MTU size is 1500 bytes and there is Jumbo Frame enabled yet, that's why when run command Ping -d -s 9000 giving you error sendto() failed (Message too long).

[a_p_]

Please don't mind me asking this again, but you didn't respond to my previous reply. Can you confirm Jumbo frames (MTU 9000) is configured for the VMkernel port group as well as for the vSwitch?

André

[tdubb123]

hi

yes jumbo frames are enabled on the host vmk ports and the vswitch

DVS Name Num Ports Used Ports Configured Ports MTU Uplinks

dvS-UCS-iSCSI 256 7 256 9000 vmnic9,vmnic8

vmnic8 0000:10:00.00 enic Up 10000Mbps Full 00:25:b5:11:0a:cf 9000 Cisco Systems Inc Cisco VIC Ethernet NIC

vmnic9 0000:11:00.00 enic Up 10000Mbps Full 00:25:b5:11:0b:0f 9000 Cisco Systems Inc Cisco VIC Ethernet NIC

vmk4 101 IPv4 10.60.1.219 255.255.0.0 10.60.255.255 00:50:56:6c:aa:d6 9000 65535 true STATIC

vmk5 132 IPv4 10.60.1.220 255.255.0.0 10.60.255.255 00:50:56:6c:6c:34 9000 65535 true STATIC

[DRadford]

Hi guys,

I got this thread searching in google for problem in question/subject. Straight to the output:

~ # vmware -v

VMware ESXi 5.5.0 build-1623387

~ # esxcli network ip interface list

vmk0

   Name: vmk0

   MAC Address: 00:50:56:63:22:4c

   Enabled: true

   Portset: DvsPortset-0

   Portgroup: N/A

   Netstack Instance: defaultTcpipStack

   VDS Name: ASW1

   VDS UUID: c5 89 37 50 bc 43 c3 75-82 4e 52 f1 85 87 78 33

   VDS Port: 2229

   VDS Connection: 25747893

   MTU: 1500

   TSO MSS: 65535

   Port ID: 33554488

vmk1

   Name: vmk1

   MAC Address: 00:50:56:62:41:cc

   Enabled: true

   Portset: DvsPortset-0

   Portgroup: N/A

   Netstack Instance: defaultTcpipStack

   VDS Name: ASW1

   VDS UUID: c5 89 37 50 bc 43 c3 75-82 4e 52 f1 85 87 78 33

   VDS Port: 2230

   VDS Connection: 418560376

   MTU: 9000

   TSO MSS: 65535

   Port ID: 33554504

vmk2

   Name: vmk2

   MAC Address: 00:50:56:68:ad:97

   Enabled: true

   Portset: DvsPortset-0

   Portgroup: N/A

   Netstack Instance: defaultTcpipStack

   VDS Name: ASW1

   VDS UUID: c5 89 37 50 bc 43 c3 75-82 4e 52 f1 85 87 78 33

   VDS Port: 2233

   VDS Connection: 442667658

   MTU: 9000

   TSO MSS: 65535

   Port ID: 33554505

~ # esxcli network ip interface ipv4 get

Name  IPv4 Address   IPv4 Netmask   IPv4 Broadcast   Address Type  DHCP DNS

----  -------------  -------------  ---------------  ------------  --------

vmk0  192.168.100.4  255.255.255.0  192.168.100.255  STATIC           false

vmk1  192.168.100.6  255.255.255.0  192.168.100.255  STATIC           false

vmk2  192.168.100.7  255.255.255.0  192.168.100.255  STATIC           false

~ # esxcli network ip route ipv4 list

Network        Netmask        Gateway        Interface  Source

-------------  -------------  -------------  ---------  ------

default        0.0.0.0        192.168.100.1  vmk0       MANUAL

192.168.100.0  255.255.255.0  0.0.0.0        vmk0       MANUAL

~ # esxcli network vswitch dvs vmware list

ASW1

   Name: ASW1

   VDS ID: c5 89 37 50 bc 43 c3 75-82 4e 52 f1 85 87 78 33

   Class: cisco_nexus_1000v

   Num Ports: 1024

   Used Ports: 45

   Configured Ports: 1024

   MTU: 9000

   CDP Status: listen

   Beacon Timeout: -1

   Uplinks: vmnic2, vmnic3

   VMware Branded: false

   DVPort:

         Client:

         DVPortgroup ID: dvportgroup-268

         In Use: false

         Port ID: 96

[cut]

~ # esxcli network vswitch standard list

~ # esxcli network nic list | awk {'print $1'} | grep vmnic | while read nic; do esxcli network nic get -n "$nic" ; done

   Advertised Auto Negotiation: true

   Advertised Link Modes: 10baseT/Half, 10baseT/Full, 100baseT/Half, 100baseT/Full, 1000baseT/Full

   Auto Negotiation: true

   Cable Type: Twisted Pair

   Current Message Level: 7

   Driver Info:

         Bus Info: 0000:05:00.0

         Driver: igb

         Firmware Version: 1.59, 0x800008f8

         Version: 5.0.5.1

   Link Detected: false

   Link Status: Down

   Name: vmnic0

   PHYAddress: 1

   Pause Autonegotiate: true

   Pause RX: true

   Pause TX: true

   Supported Ports: TP

   Supports Auto Negotiation: true

   Supports Pause: true

   Supports Wakeon: true

   Transceiver: internal

   Wakeon: MagicPacket(tm)

   Advertised Auto Negotiation: true

   Advertised Link Modes: 10baseT/Half, 10baseT/Full, 100baseT/Half, 100baseT/Full, 1000baseT/Full

   Auto Negotiation: true

   Cable Type: Twisted Pair

   Current Message Level: 7

   Driver Info:

         Bus Info: 0000:05:00.1

         Driver: igb

         Firmware Version: 1.59, 0x800008f8

         Version: 5.0.5.1

   Link Detected: false

   Link Status: Down

   Name: vmnic1

   PHYAddress: 1

   Pause Autonegotiate: true

   Pause RX: true

   Pause TX: true

   Supported Ports: TP

   Supports Auto Negotiation: true

   Supports Pause: true

   Supports Wakeon: true

   Transceiver: internal

   Wakeon: Disabled

   Advertised Auto Negotiation: true

   Advertised Link Modes: 10baseT/Half, 10baseT/Full, 100baseT/Half, 100baseT/Full, 1000baseT/Full

   Auto Negotiation: true

   Cable Type: Twisted Pair

   Current Message Level: 7

   Driver Info:

         Bus Info: 0000:05:00.2

         Driver: igb

         Firmware Version: 1.59, 0x800008f8

         Version: 5.0.5.1

   Link Detected: true

   Link Status: Up

   Name: vmnic2

   PHYAddress: 1

   Pause Autonegotiate: true

   Pause RX: true

   Pause TX: true

   Supported Ports: TP

   Supports Auto Negotiation: true

   Supports Pause: true

   Supports Wakeon: true

   Transceiver: internal

   Wakeon: Disabled

   Advertised Auto Negotiation: true

   Advertised Link Modes: 10baseT/Half, 10baseT/Full, 100baseT/Half, 100baseT/Full, 1000baseT/Full

   Auto Negotiation: true

   Cable Type: Twisted Pair

   Current Message Level: 7

   Driver Info:

         Bus Info: 0000:05:00.3

         Driver: igb

         Firmware Version: 1.59, 0x800008f8

         Version: 5.0.5.1

   Link Detected: true

   Link Status: Up

   Name: vmnic3

   PHYAddress: 1

   Pause Autonegotiate: true

   Pause RX: true

   Pause TX: true

   Supported Ports: TP

   Supports Auto Negotiation: true

   Supports Pause: true

   Supports Wakeon: true

   Transceiver: internal

   ~ # esxcfg-vswitch -l

DVS Name         Num Ports   Used Ports  Configured Ports  MTU     Uplinks

ASW1             1024        45          1024              9000    vmnic2,vmnic3

~ # vmkping -4 -d 192.168.100.2 -I vmk1 -s 1600

PING 192.168.100.2 (192.168.100.2): 1600 data bytes

sendto() failed (Message too long)

sendto() failed (Message too long)

sendto() failed (Message too long)

--- 192.168.100.2 ping statistics ---

3 packets transmitted, 0 packets received, 100% packet loss

~ # vmkping -4 -d 192.168.100.2 -I vmk2 -s 1600

PING 192.168.100.2 (192.168.100.2): 1600 data bytes

sendto() failed (Message too long)

sendto() failed (Message too long)

sendto() failed (Message too long)

--- 192.168.100.2 ping statistics ---

3 packets transmitted, 0 packets received, 100% packet loss

~ #  vmkping 192.168.100.6 -s 7600 -d -I vmk1

PING 192.168.100.6 (192.168.100.6): 7600 data bytes

sendto() failed (Message too long)

sendto() failed (Message too long)

sendto() failed (Message too long)

--- 192.168.100.6 ping statistics ---

3 packets transmitted, 0 packets received, 100% packet loss

~ #

~ #  vmkping 192.168.100.6 -s 7600 -d -I vmk2

PING 192.168.100.6 (192.168.100.6): 7600 data bytes

sendto() failed (Message too long)

sendto() failed (Message too long)

sendto() failed (Message too long)

--- 192.168.100.6 ping statistics ---

3 packets transmitted, 0 packets received, 100% packet loss

~ #

It looks like the MTU is properly configured everywhere, but for some reason vmkernel is not able to use it 😐

My setup is as follow:

[iSCSI-TARGET]------[NEXUS3064]-------[ [NEXUS1000v] [SUPERMICRO] ]

here is some output from 1000V:

port-profile type ethernet system-port-channel-uplink

  vmware port-group

  switchport trunk allowed vlan 50,100

  switchport mode trunk

  mtu 9000

  channel-group auto mode active

  no shutdown

  system vlan 100

  state enabled

!

port-profile system-port-channel-uplink

type: Ethernet

description:

status: enabled

max-ports: 32

min-ports: 1

inherit:

config attributes:

  switchport trunk allowed vlan 50,100

  switchport mode trunk

  mtu 9000

  channel-group auto mode active

  no shutdown

evaluated config attributes:

  switchport trunk allowed vlan 50,100

  switchport mode trunk

  mtu 9000

  channel-group auto mode active

  no shutdown

assigned interfaces:

  port-channel2

  Ethernet3/3

  Ethernet3/4

port-group: system-port-channel-uplink

system vlans: 100

capability l3control: no

capability iscsi-multipath: no

capability vxlan: no

capability l3-vservice: no

port-profile role: none

port-binding: static

!

ASW1# show int ethernet 3/3

Ethernet3/3 is up

  Hardware: Ethernet, address: 0050.5653.1a8e (bia 0050.5653.1a8e)

  Port-Profile is system-port-channel-uplink

  MTU 9000 bytes

  Encapsulation ARPA

  Port mode is trunk

  full-duplex, 1000 Mb/s

  5 minute input rate 268104 bits/second, 195 packets/second

  5 minute output rate 310099816 bits/second, 1472 packets/second

  Rx

    209833845 Input Packets 211147151 Unicast Packets

    22289 Multicast Packets 5553 Broadcast Packets

    90787232006 Bytes

  Tx

    209377590 Output Packets 207623415 Unicast Packets

    58439 Multicast Packets 20585 Broadcast Packets 65636 Flood Packets

    530214333212 Bytes

    1 Input Packet Drops 0 Output Packet Drops

ASW1# show int ethernet 3/4

Ethernet3/4 is up

  Hardware: Ethernet, address: 0050.5653.1a8f (bia 0050.5653.1a8f)

  Port-Profile is system-port-channel-uplink

  MTU 9000 bytes

  Encapsulation ARPA

  Port mode is trunk

  full-duplex, 1000 Mb/s

  5 minute input rate 3574688 bits/second, 5062 packets/second

  5 minute output rate 307317560 bits/second, 1260 packets/second

  Rx

    206359322 Input Packets 207121928 Unicast Packets

    291654 Multicast Packets 1324 Broadcast Packets

    48693934052 Bytes

  Tx

    221563934 Output Packets 219762972 Unicast Packets

    50337 Multicast Packets 36333 Broadcast Packets 73025 Flood Packets

    457044944110 Bytes

    117 Input Packet Drops 0 Output Packet Drops

ASW1# show int po2

port-channel2 is up

  Hardware: Port-Channel, address: 0050.5653.1a8e (bia 0050.5653.1a8e)

  Description: === LINK TO N3064 ===

  MTU 9000 bytes, BW 2000000 Kbit, DLY 10 usec,

     reliability 255/255, txload 1/255, rxload 1/255

  Encapsulation ARPA

  Port mode is trunk

  full-duplex, 1000 Mb/s

  Beacon is turned off

  Input flow-control is off, output flow-control is off

  Switchport monitor is off

  Members in this channel: Eth3/3, Eth3/4

  Last clearing of "show interface" counters never

  300 seconds input rate 3842696 bits/sec, 5257 packets/sec

  300 seconds output rate 614845280 bits/sec, 2723 packets/sec

  Rx

    118063294 unicast packets  37583 multicast packets  1363 broadcast packets

    118102240 input packets  85807288965 bytes

    80 input packet drops

  Tx

    36317177 unicast packets  7133 multicast packets  17137 broadcast packets

    36341448 output packets  839871823663 bytes

    22599 flood packets

    0 output packet drops

  1 interface resets

I have configured VMGuest machine (Linux) with MTU9000 and managed to ping the physical Nexus3k switch with bigger packets:

root@debian2:~# ping 192.168.100.2 -s 8000 -M do

PING 192.168.100.2 (192.168.100.2) 8000(8028) bytes of data.

8008 bytes from 192.168.100.2: icmp_req=1 ttl=255 time=1.24 ms

8008 bytes from 192.168.100.2: icmp_req=2 ttl=255 time=1.09 ms

8008 bytes from 192.168.100.2: icmp_req=3 ttl=255 time=0.965 ms

So it looks like VMguest interfaces are OK, 1000v is OK, Nexus3064 is OK, but for some reason vmk1 & vmk2 interfaces cannot use mtu bigger than 1500 ..

I am really confused here Any help would be more than welcome!

Sincerely,

D.

[DRadford]

Hi chaps, I managed to find what was the problem... It was due to all interfaces being part of the same broadcast domain (vmk0,1 and 2). For some reason kernel decides to use vmk0 interface (perhaps owing to routing table?! but the remote host shouldn't be even routed as it is part of the same bcast domain..) which has mtu 1500... So I split iSCSI ifaces (vmk1 & 2) to another subnet and that fixed it.. Still strange though, why does ESXi kernel uses vmk0 when I have explicitly pointed vmk1 & 2.. ? Anyway! Thanks for your time and hope this post will help someone else 🙂