Should I use a custom TCP/IP stack with iSCSI?

[vvv850]

We have a cluster of 3 identical hosts with each connecting to the SAN through 2 pnics using 2 vmks. Each vmk resides on a separate broadcast domain thus they belong to separate vswitches. No bindings are used as per the vSphere guide.

So, as the title says, should we use a separate custom TCP/IP stack for the iSCSI vmks? or leave them in the default stack (though there isn't an option to mark the vmks as IP storage)

[akarydas2]

What exactly you mean by deploying a separate custom TCP/IP stack?

I guess you use already different network as layer 2 and 3 for the iSCSI communication.

What I did to confirm I am fine with my config was to deploy the I/O analyzer from vmware flings and benchmark my I/O by producing on demand high I/O utilization and when I noticed that I can reach my network capacity I convinced that my config was correct.

Best Regards

[MKguy · Accepted solution]

As long as your iSCSI vmkernel adapters and their respective iSCSI targets are in the same layer 3 subnet, i.e. you're not routing iSCSI traffic (vmkX<->TargetSPX and vmkY<->TargetSPY subnet separated iSCSI design does not apply, as traffic is still never routed between networks), OR you don't want to configure custom static routes for iSCSI targets in other subnets, then there is no point in using a separate TCP/IP stack.

The main point of using the new ESXi 6 feature with multiple TCP/IP stacks is having a completely separate routing table per stack. Most admins weren't able to handle very basic layer 3 static routing with dedicated routes per subnet, so they assigned default gateways on multiple vmkernel NICs and wondered why things broke or never worked.

It also enables more fine-grained control over which interfaces to send what data when you have multiple layer 3 paths or subnets to communicate with, but this is largely irrelevant and already an integral point of an iSCSI network.

[vvv850]

The iSCSI network is isolated and on the same subnet. No routing between hosts and SAN.

We were thinking of creating another TCP/IP stack so that in an event that vSphere reserves resources per stack (buffers/sockets) then iSCSI won't be affected by this.

[macalb]

"Note: At the time of writing the use of a custom TCP/IP Stack is not supported for iSCSI!"

Reference:
"Best Practices For Running VMware vSphere On iSCSI"

https://core.vmware.com/resource/best-practices-running-vmware-vsphere-iscsi#sec7262-sub5