Hi all,
Hope you have fresh insights or can help me figure this out. I have moved from a single storage subnet to splitting the storage network into two subnets due to inconsistent performance issues, random dead paths and other issues.
I have three ESXi 6.7 nodes connected over two 10Gb switches (QNAP QSW-M1208-8C) to two storage arrays (QNAP TS-451DeU). The storage arrays have two 2.5Gb ports. I have iperf3 verified that each link is around 2.45 Gbits/s as expected.
I have created two separate subnets (yellow and orange). Each node has a dedicated vmnic, vmk port group to each subnet for vMotion using the vMotion TCP/IP stack. Management is on the default TCP/IP stack using a separate vmk and vmnic.
The problem
A Storage vMotion dus not use the most logical path from Storage A port 2 over the orange network to Storage B port 2. It is using the path from Storage A port 1 (yellow network) out over yellow gateway to the orange gateway and then to Storage B port 1.
You have to fill-in a gateway for each vmk adapter otherwise it does not perform a storage vmotion at all. The gateway goes over a 1 Gbps connection so it results in a throughput of 0.23 Gbits/s instead of a 2.5 Gbits/s link.
I've attached a couple of screenshots from node 2 (others are identical except for IP's ofc).
I really do not want to put in a 10Gb router as a single point of failure to make this work. Probably missing something super obvious but for the life of me can't figure out what 🙂
How can I make vCenter use the preferred route (A2 to B1, or B2 to A1) ?
-- EDIT ---
I also tried VMkernal Adapters on the normal TCPIP stack (separated from management) and setting the vMotion, Provisioning, Replication services. Same result.
