I am getting an error when doing a jumbo packet ping to my iscsi storage
everything has been set to 9000mtu end to end
vmkping -s 9000 storageip
responds fine
but when i put the -d flag in I get
sendto () failed (message too long)
any idea?
I tried vmkping -I vmkernel_nic as well but same problem
You need to subtract some header data bytes when you use the ping command. Use 8972 instead. See http://rickardnobel.se/troubleshoot-jumbo-frames-with-vmkping/ for details
André
still doesnt work
1 packets transmitted, 0 packets received, 100% packet loss
/vmfs/volumes/504d1fa5-1c455cec-0519-0024e85c1297/pa-vc # vmkping -I vmk5 -d -s 8972 10.60.1.106
PING 10.60.1.106 (10.60.1.106): 8972 data bytes
sendto() failed (Message too long)
sendto() failed (Message too long)
sendto() failed (Message too long)
In this case you need to check which part in the chain isn't configured properly.
port groups -> vSwitch -> physical switch -> storage system
André
If you are using 5.5, then you can use pktcap-uw to identify where the drop is happening
Packet tracing won't work, and the physical infrastructure can't be blamed (at least yet) because the packets are dropped by the host's own networking stack before they actually reach the wire.
I've seen that sometimes a host needs an actual reboot after enabling jumbo frames.If it still doesn't work after a reboot, try this:
1. Please post the output of the following commands on the local shell:
# esxcli network nic list
# esxcli network ip interface list
# esxcli network ip interface ipv4 get
# esxcli network ip route ipv4 list
# esxcli network vswitch dvs vmware list
# esxcli network vswitch standard list
# esxcli network nic list | awk {'print $1'} | grep vmnic | while read nic; do esxcli network nic get -n "$nic" ; done
# vmware -v
2. Try some smaller payload size values like 8000, 4000, 2000.
3. Can you vmkping the host on it's own localhost and vmkernel interface IP? E.g.:
# vmkping -I vmk5 -d -s 8972 [IP of vmk5]
# vmkping -d -s 8972 127.0.0.1
4. Try the tests with esxcli network diag ping utility as well, e.g.:
esxcli network diag ping --df --interface vmk5 --ipv4 --size 8972 --host 10.60.1.106
5. Also make sure your NIC drivers and firmware are up to date.
vmkping -d -s 8972 127.0.0.1 responds fine
but
vmkping -I vmk5 -d -s 8972 [IP of vmk5]fails
esxcli network diag ping --df --interface vmk5 --ipv4 --size 8972 --host 10.60.1.106
fails
by default the MTU size is 1500 bytes and there is Jumbo Frame enabled yet, that's why when run command Ping -d -s 9000 giving you error sendto() failed (Message too long).
Please don't mind me asking this again, but you didn't respond to my previous reply. Can you confirm Jumbo frames (MTU 9000) is configured for the VMkernel port group as well as for the vSwitch?
André
hi
yes jumbo frames are enabled on the host vmk ports and the vswitch
DVS Name Num Ports Used Ports Configured Ports MTU Uplinks
dvS-UCS-iSCSI 256 7 256 9000 vmnic9,vmnic8
vmnic8 0000:10:00.00 enic Up 10000Mbps Full 00:25:b5:11:0a:cf 9000 Cisco Systems Inc Cisco VIC Ethernet NIC
vmnic9 0000:11:00.00 enic Up 10000Mbps Full 00:25:b5:11:0b:0f 9000 Cisco Systems Inc Cisco VIC Ethernet NIC
vmk4 101 IPv4 10.60.1.219 255.255.0.0 10.60.255.255 00:50:56:6c:aa:d6 9000 65535 true STATIC
vmk5 132 IPv4 10.60.1.220 255.255.0.0 10.60.255.255 00:50:56:6c:6c:34 9000 65535 true STATIC
Hi guys,
I got this thread searching in google for problem in question/subject. Straight to the output:
~ # vmware -v
VMware ESXi 5.5.0 build-1623387
~ # esxcli network ip interface list
vmk0
Name: vmk0
MAC Address: 00:50:56:63:22:4c
Enabled: true
Portset: DvsPortset-0
Portgroup: N/A
Netstack Instance: defaultTcpipStack
VDS Name: ASW1
VDS UUID: c5 89 37 50 bc 43 c3 75-82 4e 52 f1 85 87 78 33
VDS Port: 2229
VDS Connection: 25747893
MTU: 1500
TSO MSS: 65535
Port ID: 33554488
vmk1
Name: vmk1
MAC Address: 00:50:56:62:41:cc
Enabled: true
Portset: DvsPortset-0
Portgroup: N/A
Netstack Instance: defaultTcpipStack
VDS Name: ASW1
VDS UUID: c5 89 37 50 bc 43 c3 75-82 4e 52 f1 85 87 78 33
VDS Port: 2230
VDS Connection: 418560376
MTU: 9000
TSO MSS: 65535
Port ID: 33554504
vmk2
Name: vmk2
MAC Address: 00:50:56:68:ad:97
Enabled: true
Portset: DvsPortset-0
Portgroup: N/A
Netstack Instance: defaultTcpipStack
VDS Name: ASW1
VDS UUID: c5 89 37 50 bc 43 c3 75-82 4e 52 f1 85 87 78 33
VDS Port: 2233
VDS Connection: 442667658
MTU: 9000
TSO MSS: 65535
Port ID: 33554505
~ # esxcli network ip interface ipv4 get
Name IPv4 Address IPv4 Netmask IPv4 Broadcast Address Type DHCP DNS
---- ------------- ------------- --------------- ------------ --------
vmk0 192.168.100.4 255.255.255.0 192.168.100.255 STATIC false
vmk1 192.168.100.6 255.255.255.0 192.168.100.255 STATIC false
vmk2 192.168.100.7 255.255.255.0 192.168.100.255 STATIC false
~ # esxcli network ip route ipv4 list
Network Netmask Gateway Interface Source
------------- ------------- ------------- --------- ------
default 0.0.0.0 192.168.100.1 vmk0 MANUAL
192.168.100.0 255.255.255.0 0.0.0.0 vmk0 MANUAL
~ # esxcli network vswitch dvs vmware list
ASW1
Name: ASW1
VDS ID: c5 89 37 50 bc 43 c3 75-82 4e 52 f1 85 87 78 33
Class: cisco_nexus_1000v
Num Ports: 1024
Used Ports: 45
Configured Ports: 1024
MTU: 9000
CDP Status: listen
Beacon Timeout: -1
Uplinks: vmnic2, vmnic3
VMware Branded: false
DVPort:
Client:
DVPortgroup ID: dvportgroup-268
In Use: false
Port ID: 96
[cut]
~ # esxcli network vswitch standard list
~ # esxcli network nic list | awk {'print $1'} | grep vmnic | while read nic; do esxcli network nic get -n "$nic" ; done
Advertised Auto Negotiation: true
Advertised Link Modes: 10baseT/Half, 10baseT/Full, 100baseT/Half, 100baseT/Full, 1000baseT/Full
Auto Negotiation: true
Cable Type: Twisted Pair
Current Message Level: 7
Driver Info:
Bus Info: 0000:05:00.0
Driver: igb
Firmware Version: 1.59, 0x800008f8
Version: 5.0.5.1
Link Detected: false
Link Status: Down
Name: vmnic0
PHYAddress: 1
Pause Autonegotiate: true
Pause RX: true
Pause TX: true
Supported Ports: TP
Supports Auto Negotiation: true
Supports Pause: true
Supports Wakeon: true
Transceiver: internal
Wakeon: MagicPacket(tm)
Advertised Auto Negotiation: true
Advertised Link Modes: 10baseT/Half, 10baseT/Full, 100baseT/Half, 100baseT/Full, 1000baseT/Full
Auto Negotiation: true
Cable Type: Twisted Pair
Current Message Level: 7
Driver Info:
Bus Info: 0000:05:00.1
Driver: igb
Firmware Version: 1.59, 0x800008f8
Version: 5.0.5.1
Link Detected: false
Link Status: Down
Name: vmnic1
PHYAddress: 1
Pause Autonegotiate: true
Pause RX: true
Pause TX: true
Supported Ports: TP
Supports Auto Negotiation: true
Supports Pause: true
Supports Wakeon: true
Transceiver: internal
Wakeon: Disabled
Advertised Auto Negotiation: true
Advertised Link Modes: 10baseT/Half, 10baseT/Full, 100baseT/Half, 100baseT/Full, 1000baseT/Full
Auto Negotiation: true
Cable Type: Twisted Pair
Current Message Level: 7
Driver Info:
Bus Info: 0000:05:00.2
Driver: igb
Firmware Version: 1.59, 0x800008f8
Version: 5.0.5.1
Link Detected: true
Link Status: Up
Name: vmnic2
PHYAddress: 1
Pause Autonegotiate: true
Pause RX: true
Pause TX: true
Supported Ports: TP
Supports Auto Negotiation: true
Supports Pause: true
Supports Wakeon: true
Transceiver: internal
Wakeon: Disabled
Advertised Auto Negotiation: true
Advertised Link Modes: 10baseT/Half, 10baseT/Full, 100baseT/Half, 100baseT/Full, 1000baseT/Full
Auto Negotiation: true
Cable Type: Twisted Pair
Current Message Level: 7
Driver Info:
Bus Info: 0000:05:00.3
Driver: igb
Firmware Version: 1.59, 0x800008f8
Version: 5.0.5.1
Link Detected: true
Link Status: Up
Name: vmnic3
PHYAddress: 1
Pause Autonegotiate: true
Pause RX: true
Pause TX: true
Supported Ports: TP
Supports Auto Negotiation: true
Supports Pause: true
Supports Wakeon: true
Transceiver: internal
~ # esxcfg-vswitch -l
DVS Name Num Ports Used Ports Configured Ports MTU Uplinks
ASW1 1024 45 1024 9000 vmnic2,vmnic3
~ # vmkping -4 -d 192.168.100.2 -I vmk1 -s 1600
PING 192.168.100.2 (192.168.100.2): 1600 data bytes
sendto() failed (Message too long)
sendto() failed (Message too long)
sendto() failed (Message too long)
--- 192.168.100.2 ping statistics ---
3 packets transmitted, 0 packets received, 100% packet loss
~ # vmkping -4 -d 192.168.100.2 -I vmk2 -s 1600
PING 192.168.100.2 (192.168.100.2): 1600 data bytes
sendto() failed (Message too long)
sendto() failed (Message too long)
sendto() failed (Message too long)
--- 192.168.100.2 ping statistics ---
3 packets transmitted, 0 packets received, 100% packet loss
~ # vmkping 192.168.100.6 -s 7600 -d -I vmk1
PING 192.168.100.6 (192.168.100.6): 7600 data bytes
sendto() failed (Message too long)
sendto() failed (Message too long)
sendto() failed (Message too long)
--- 192.168.100.6 ping statistics ---
3 packets transmitted, 0 packets received, 100% packet loss
~ #
~ # vmkping 192.168.100.6 -s 7600 -d -I vmk2
PING 192.168.100.6 (192.168.100.6): 7600 data bytes
sendto() failed (Message too long)
sendto() failed (Message too long)
sendto() failed (Message too long)
--- 192.168.100.6 ping statistics ---
3 packets transmitted, 0 packets received, 100% packet loss
~ #
It looks like the MTU is properly configured everywhere, but for some reason vmkernel is not able to use it 😐
My setup is as follow:
[iSCSI-TARGET]------[NEXUS3064]-------[ [NEXUS1000v] [SUPERMICRO] ]
here is some output from 1000V:
port-profile type ethernet system-port-channel-uplink
vmware port-group
switchport trunk allowed vlan 50,100
switchport mode trunk
mtu 9000
channel-group auto mode active
no shutdown
system vlan 100
state enabled
!
port-profile system-port-channel-uplink
type: Ethernet
description:
status: enabled
max-ports: 32
min-ports: 1
inherit:
config attributes:
switchport trunk allowed vlan 50,100
switchport mode trunk
mtu 9000
channel-group auto mode active
no shutdown
evaluated config attributes:
switchport trunk allowed vlan 50,100
switchport mode trunk
mtu 9000
channel-group auto mode active
no shutdown
assigned interfaces:
port-channel2
Ethernet3/3
Ethernet3/4
port-group: system-port-channel-uplink
system vlans: 100
capability l3control: no
capability iscsi-multipath: no
capability vxlan: no
capability l3-vservice: no
port-profile role: none
port-binding: static
!
ASW1# show int ethernet 3/3
Ethernet3/3 is up
Hardware: Ethernet, address: 0050.5653.1a8e (bia 0050.5653.1a8e)
Port-Profile is system-port-channel-uplink
MTU 9000 bytes
Encapsulation ARPA
Port mode is trunk
full-duplex, 1000 Mb/s
5 minute input rate 268104 bits/second, 195 packets/second
5 minute output rate 310099816 bits/second, 1472 packets/second
Rx
209833845 Input Packets 211147151 Unicast Packets
22289 Multicast Packets 5553 Broadcast Packets
90787232006 Bytes
Tx
209377590 Output Packets 207623415 Unicast Packets
58439 Multicast Packets 20585 Broadcast Packets 65636 Flood Packets
530214333212 Bytes
1 Input Packet Drops 0 Output Packet Drops
ASW1# show int ethernet 3/4
Ethernet3/4 is up
Hardware: Ethernet, address: 0050.5653.1a8f (bia 0050.5653.1a8f)
Port-Profile is system-port-channel-uplink
MTU 9000 bytes
Encapsulation ARPA
Port mode is trunk
full-duplex, 1000 Mb/s
5 minute input rate 3574688 bits/second, 5062 packets/second
5 minute output rate 307317560 bits/second, 1260 packets/second
Rx
206359322 Input Packets 207121928 Unicast Packets
291654 Multicast Packets 1324 Broadcast Packets
48693934052 Bytes
Tx
221563934 Output Packets 219762972 Unicast Packets
50337 Multicast Packets 36333 Broadcast Packets 73025 Flood Packets
457044944110 Bytes
117 Input Packet Drops 0 Output Packet Drops
ASW1# show int po2
port-channel2 is up
Hardware: Port-Channel, address: 0050.5653.1a8e (bia 0050.5653.1a8e)
Description: === LINK TO N3064 ===
MTU 9000 bytes, BW 2000000 Kbit, DLY 10 usec,
reliability 255/255, txload 1/255, rxload 1/255
Encapsulation ARPA
Port mode is trunk
full-duplex, 1000 Mb/s
Beacon is turned off
Input flow-control is off, output flow-control is off
Switchport monitor is off
Members in this channel: Eth3/3, Eth3/4
Last clearing of "show interface" counters never
300 seconds input rate 3842696 bits/sec, 5257 packets/sec
300 seconds output rate 614845280 bits/sec, 2723 packets/sec
Rx
118063294 unicast packets 37583 multicast packets 1363 broadcast packets
118102240 input packets 85807288965 bytes
80 input packet drops
Tx
36317177 unicast packets 7133 multicast packets 17137 broadcast packets
36341448 output packets 839871823663 bytes
22599 flood packets
0 output packet drops
1 interface resets
I have configured VMGuest machine (Linux) with MTU9000 and managed to ping the physical Nexus3k switch with bigger packets:
root@debian2:~# ping 192.168.100.2 -s 8000 -M do
PING 192.168.100.2 (192.168.100.2) 8000(8028) bytes of data.
8008 bytes from 192.168.100.2: icmp_req=1 ttl=255 time=1.24 ms
8008 bytes from 192.168.100.2: icmp_req=2 ttl=255 time=1.09 ms
8008 bytes from 192.168.100.2: icmp_req=3 ttl=255 time=0.965 ms
So it looks like VMguest interfaces are OK, 1000v is OK, Nexus3064 is OK, but for some reason vmk1 & vmk2 interfaces cannot use mtu bigger than 1500 ..
I am really confused here Any help would be more than welcome!
Sincerely,
D.
Hi chaps, I managed to find what was the problem... It was due to all interfaces being part of the same broadcast domain (vmk0,1 and 2). For some reason kernel decides to use vmk0 interface (perhaps owing to routing table?! but the remote host shouldn't be even routed as it is part of the same bcast domain..) which has mtu 1500... So I split iSCSI ifaces (vmk1 & 2) to another subnet and that fixed it.. Still strange though, why does ESXi kernel uses vmk0 when I have explicitly pointed vmk1 & 2.. ? Anyway! Thanks for your time and hope this post will help someone else 🙂