JinjiroSan
Contributor
Contributor

ESXi picks unlogical path when using separate subnets for storage vMotion networks

Hi all,

Hope you have fresh insights or can help me figure this out. I have moved from a single storage subnet to splitting the storage network into two subnets due to inconsistent performance issues, random dead paths and other issues.

I have three ESXi 6.7 nodes connected over two 10Gb switches (QNAP QSW-M1208-8C) to two storage arrays (QNAP TS-451DeU). The storage arrays have two 2.5Gb ports. I have iperf3 verified that each link is around 2.45 Gbits/s as expected.

I have created two separate subnets (yellow and orange). Each node has a dedicated vmnic, vmk port group to each subnet for vMotion using the vMotion TCP/IP stack. Management is on the default TCP/IP stack using a separate vmk and vmnic.

Storage Network.png

The problem

A Storage vMotion dus not use the most logical path from Storage A port 2 over the orange network to Storage B port 2. It is using the path from Storage A port 1 (yellow network) out over yellow gateway to the orange gateway and then to Storage B port 1.

You have to fill-in a gateway for each vmk adapter otherwise it does not perform a storage vmotion at all. The gateway goes over a 1 Gbps connection so it results in a throughput of 0.23 Gbits/s instead of a 2.5 Gbits/s link.

I've attached a couple of screenshots from node 2 (others are identical except for IP's ofc).

I really do not want to put in a 10Gb router as a single point of failure to make this work. Probably missing something super obvious but for the life of me can't figure out what 🙂

How can I make vCenter use the preferred route (A2 to B1, or B2 to A1) ?

 

-- EDIT ---

I also tried VMkernal Adapters on the normal TCPIP stack (separated from management) and setting the vMotion, Provisioning, Replication services. Same result.

0 Kudos
0 Replies