mtu 9000, iSCSI VLAN, ESXi - not work

[itsibgurman]

Dear Community. My configuration.

ESXi host 192.168.34.11 (vSwitch MTU 9000)

|| (2 links)

Dlink DXS-3400 x2 STACK;  VLAN ip interface 192.168.34.254/24; hw jumbo-frames - 9216; ip mtu 1500

  |||| (4 links; lacp)

Synology NAS 192.168.34.250; bond interface; mtu 9000

Problem: When i set vmk's mtu 9000, iscsi connection get lost.

max transmited frame bytes 1472.

What can i do to get 9000 frame size work?

i can set dlink "ip mtu vlan 9000" but I don’t know if this will help.

[IRIX201110141]

You need to enable "Jumbo" frames on all virtual and phyical devices.

1. On the virtual Switch
2. On the VMK
3. Rest from your environment

Btw. yes its supportet to create a single LAG on a vSS but  LACP it only supportet on vDS.

In the last 13 years of doing iSCSI i never have seen a single Storage that needs to setup with a LAG/LCAP. ESXi have inbuild network redundancy and the swISCSI offer multipathing as well. Even in the "free" vSphere Hypervisor.

Regards,
Joerg

[itsibgurman]

I seem to have been misunderstood. everywhere mtu 9000 costs, except for vmk, if i put mtu 9000 on vmk then the connection is lost.

[IRIX201110141]

Yes.
If you enable MTU 9000 on the VMK the ESXi starts to use the larger packets... if you have a misconfiguration somewhere on the COMPLETE chain for sure it cannot work and will fail.

If you use an unsupportet network setup... the screenshot doesnt show clearly if you use vSS or VDS expect problems.

Regards,
Joerg

[itsibgurman]

i use vSS.

iSCSI
Name: iSCSI
Class: cswitch
Num Ports: 7936
Used Ports: 7
Configured Ports: 1024
MTU: 9000
CDP Status: listen
Beacon Enabled: false
Beacon Interval: 1
Beacon Threshold: 3
Beacon Required By:
Uplinks: vmnic3, vmnic2
Portgroups: VMkernel_iSCSI_10G_P1, VMKerenel_iSCSI_10G_P0

vmk1
Name: vmk1
MAC Address: 00:50:56:66:8b:28
Enabled: true
Portset: iSCSI
Portgroup: VMKerenel_iSCSI_10G_P0
Netstack Instance: defaultTcpipStack
VDS Name: N/A
VDS UUID: N/A
VDS Port: N/A
VDS Connection: -1
Opaque Network ID: N/A
Opaque Network Type: N/A
External ID: N/A
MTU: 1500
TSO MSS: 65535
RXDispQueue Size: 1
Port ID: 50331652

vmk2
Name: vmk2
MAC Address: 00:50:56:61:9a:3f
Enabled: true
Portset: iSCSI
Portgroup: VMkernel_iSCSI_10G_P1
Netstack Instance: defaultTcpipStack
VDS Name: N/A
VDS UUID: N/A
VDS Port: N/A
VDS Connection: -1
Opaque Network ID: N/A
Opaque Network Type: N/A
External ID: N/A
MTU: 1500
TSO MSS: 65535
RXDispQueue Size: 1
Port ID: 50331655

Dlink ports to esxi:

Eth2/0/18 is enabled link status is up
Interface type: 10GBASE-R
Interface description:
MAC Address: 10-62-EB-DD-51-21
Auto-duplex, auto-speed, auto-mdix
Send flow-control: off, receive flow-control: off
Send flow-control oper: off, receive flow-control oper: off
Full-duplex, 10Gb/s
Maximum transmit unit: 9216 bytes
RX rate: 368 bits/sec, TX rate: 592 bits/sec
RX bytes: 70176982, TX bytes: 4824610449
RX rate: 0 packets/sec, TX rate: 1 packets/sec
RX packets: 672142, TX packets: 1419129
RX multicast: 116253, RX broadcast: 2657
RX CRC error: 0, RX undersize: 0
RX oversize: 0, RX fragment: 0
RX jabber: 0, RX dropped Pkts: 118
RX MTU exceeded: 0
TX CRC error: 0, TX excessive deferral: 0
TX single collision: 0, TX excessive collision: 0
TX late collision: 0, TX collision: 0

Eth1/0/18 is enabled link status is up
Interface type: 10GBASE-R
Interface description:
MAC Address: 10-62-EB-DD-10-99
Auto-duplex, auto-speed, auto-mdix
Send flow-control: off, receive flow-control: off
Send flow-control oper: off, receive flow-control oper: off
Full-duplex, 10Gb/s
Maximum transmit unit: 9216 bytes
RX rate: 188048 bits/sec, TX rate: 7744 bits/sec
RX bytes: 620446181491, TX bytes: 219744735249
RX rate: 18 packets/sec, TX rate: 10 packets/sec
RX packets: 432249736, TX packets: 199030579
RX multicast: 116384, RX broadcast: 3334
RX CRC error: 0, RX undersize: 0
RX oversize: 0, RX fragment: 0
RX jabber: 0, RX dropped Pkts: 222
RX MTU exceeded: 0
TX CRC error: 0, TX excessive deferral: 0
TX single collision: 0, TX excessive collision: 0
TX late collision: 0, TX collision: 0

dlink to nas:

Port-channel2 is enabled, link status is up
Interface type: port-channel
Interface description: Link aggregation group 2
MAC Address: 10-62-EB-DD-10-9D
Maximum transmit unit: 1536 bytes
RX rate: 10272304 bits/sec, TX rate: 586808 bits/sec
RX bytes: 4173320750561, TX bytes: 364406037276
RX rate: 939 packets/sec, TX rate: 832 packets/sec
RX packets: 3434147658, TX packets: 1707984512
RX multicast: 11945360, RX broadcast: 192754
RX CRC error: 7, RX undersize: 0
RX oversize: 0, RX fragment: 0
RX jabber: 1, RX dropped Pkts: 4607
RX MTU exceeded: 0
TX CRC error: 0, TX excessive deferral: 0
TX single collision: 0, TX excessive collision: 0
TX late collision: 0, TX collision: 0

Interface vlan34 is enabled, Link status is up
IP address is 192.168.34.254/24 (Manual)
ARP timeout is 240 minutes.
IP MTU is 1500 bytes
Helper Address is not set
Proxy ARP is disabled
IP Local Proxy ARP is disabled
gratuitous-send is disabled, interval is 0 seconds

NAS ifconfig
bond0 Link encap:Ethernet HWaddr 00:11:32:A6:63:11
inet addr:192.168.34.250 Bcast:192.168.34.255 Mask:255.255.255.0
UP BROADCAST RUNNING MASTER MULTICAST MTU:9000 Metric:1
RX packets:20327760172 errors:3 dropped:0 overruns:0 frame:3
TX packets:30241393852 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:26248357850761 (23.8 TiB) TX bytes:38891741027230 (35.3 TiB)

[a_p_ · Accepted solution]

I'm not familiar with the switches you're using, but it looks like the issue is related to the max MTU size on the ports (Port-channel2).

André

[IRIX201110141 · Accepted solution]

If you try the vmkping -d -s 8972 to reach a VMK on the other ESXi you can simple test if the edge port of you Dlink switch is configured properly or not. If that works than the culprit is the portchannel which of course listet a MTU 1500 at the moment.

I was not able to see the sceenshots from in the first post. Your portchannel from NAS to switch doesnt effect the ESXi in first place(thats what i thought after only reading the text).

Regards,
Joerg

[itsibgurman]

Yes it is. I set max-rcv-framesize 9216 on port-channel 1 interface.

Result:

Thank you all.