vucit
Contributor
Contributor

Software iSCSI and Port Binding

Hello All,

I am looking for some additional information about software iSCSI and port binding based on the below articles.

VMware Knowledge Base

Review Port Binding Details

Here is my environment. I have two SANs, a Equallogic and a Nimble. They are both on the same subnet. I have a total of 6 physical adapters that I will bind to VMKernel Ports.

My HBA takes a long time to scan, and the question I am left with is, do my issues stem from having 2 VMKernel Ports on 1 virtual switch and the 4 VMKernel Ports on a second vswitch.

21 Replies
a_p_
Leadership
Leadership

This could be related to a couple of things, staring with connection issues, Jumbo frame configurations, ...

Also keep in mind that ESXi supports only up to 8 paths to an iSCSI LUN.

Do you see any entries in the host's log files (e.g. vmkernel.log)?

André

0 Kudos
kenbshinn
Enthusiast
Enthusiast

You might also want to check for Latency Events in your event logs. Also make sure that Jumbo Frames are properly configured on all switches and ports between the Host and the Storage.

0 Kudos
vucit
Contributor
Contributor

I am working with VMWare on the logs, I had a call and uploaded them yesterday. Jumbo frames is set correctly on the switches as well as the vswitch.

So if I have two SANs, should I only create two connections to each? Meaning that I have two connections to my equallogic and two to the Nimble? Right now there are three hosts. Each host has 2 connections to the Equallogic (6 total) and 2 hosts have 4 connections to the Nimble and the last host has 2 connections. so 10 total.

0 Kudos
kenbshinn
Enthusiast
Enthusiast

So I just re-read your original post and I don't believe I have ever seen More than one Kernel port setup on a vSwitch for iSCSI. I am sure that you can do it, I have just never seen it done. Most of the time I have seen 1 Kernel port per switch with multiple targets configured. So in the end you would have 2 Switches with 1 VM Kernel Port (you have have standby interfaces if I remember correctly)  each connecting to multiple targets. This is also dependent on whether or not there is a business requirement to keep all your storage on separate interfaces and if there is enough bandwidth on each port to be able to support that.

0 Kudos
ChrisFD2
VMware Employee
VMware Employee

What's your vSwitch, port group and adapter binding for the PGs?

How have you bound the software iSCSI adapters to vmk interfaces?

Please share your configuration.

Regards,
Chris
VCIX-DCV 2020 | VCP-DCV 2020 | VCP-NV 2020 | vExpert *** | CCNA R&S
0 Kudos
sjesse
Leadership
Leadership

For the nimble do you have the ncm plugin installed, if not that may be related. I only have the nimble, but we have two iscsi kernal adapters, but they are on the same switch that has two uplinks. There are two portgroups, which are overrriden to only use one of the uplinks, and then I bond the vmkernal adapter to the iscsi software adapter. I generally have a pretty quick rescan time.

0 Kudos
vucit
Contributor
Contributor

So in continuing to test, this is what I have done, I have one vswitch with 4 VMKernel ports on it, two for the equallogic and 2 for the Nimble, each one uses the physical adapter needed to get to the target.

The issue I see in the log is as follows :

Description Type Date Time Task Target User

Login to iSCSI target xxx.equallogic:xxx on vmhba64 @ vmk4 failed. The iSCSI initiator could not establish a network connection to the target. Error 1/15/2019 9:48:10 AM

vmk2, vmk3 and vmk4 are the kernels physically connected to the Nimble, so they are not able to get to the equallogic. Is there a way for me to tell the software iscsi adapter what kernels to use for what san? Seems to be that is the major part of my high scan times and timeouts.

And thank you to everyone for replying, I am doing my best to answer and understand all the information that you guys are giving me and how my infrastructure is setup.

Thank you!!!!

0 Kudos
ChrisFD2
VMware Employee
VMware Employee

The vSwitch should have all physical adapters active which are connected to the storage, and then a port group for each adapter with only 1 active NIC, others unused.

So PG1 has NIC1, PG2 has NIC2 and so on. You override this on the PGs themselves.

Then under software iSCSI binding you bind a new vmk to each PG, each with its own IP address.

Does that make sense?

Also you can use vmkping from ESXi command line to ensure full 9000 MTU all the way through to the storage for each path. Don't forget to subtract the ICMP header, so use 8972 (can't remember how vmkping behaves so it may actually work with 9000).

Regards,
Chris
VCIX-DCV 2020 | VCP-DCV 2020 | VCP-NV 2020 | vExpert *** | CCNA R&S
kenbshinn
Enthusiast
Enthusiast

I am pretty sure you can only have 1 Software iSCSI adapter per host.

So you are going to need to use Hardware iSCSI adapters to connect to your equallogic SAN or have a Single iSCSI Software Adapter and have both SANs connect to that.

ChrisFD2
VMware Employee
VMware Employee

^^ you can have multiple storage devices on one adapter. Just allocate vmk interfaces and PGs correctly.

Regards,
Chris
VCIX-DCV 2020 | VCP-DCV 2020 | VCP-NV 2020 | vExpert *** | CCNA R&S
0 Kudos
vucit
Contributor
Contributor

I think I might have found our issue (while not as big of a deal in 5.5, 6.5 seems not to like it)

We are using layer 2 for our connection to the sans from the hosts. WIth that said, we are using two different non stacked switches in this case. So the host connects to a switch with connections to the nimble and the host connects to a different switch with connections to the equallogic. No routing is in play here, all targets are on the same non-routed subnet.

I will be adding a switch and moving all connections to the same layer 2 switch where the host and both sans will connect.

I think that will solve my issues.

0 Kudos
a_p_
Leadership
Leadership

I will be adding a switch and moving all connections to the same layer 2 switch where the host and both sans will connect.

This should help, but may not perfect from a design perspective.

Note that the all vmk's added to the port binding are expected to be able to reach sll targets.

André

0 Kudos
kenbshinn
Enthusiast
Enthusiast

So are you using only Software iSCSI Adapters or is it mixed? If you are only using Software iSCSI then you still have the issue of only being allowed to have 1 per host. The math does not add up if you are binding it to a port.

0 Kudos
sjesse
Leadership
Leadership

you only have one iscsi adapter, but you can have multiple discovery addresses on that one adapter, and then multiple vkernel adapters that are attached to that adapter.

kenbshinn
Enthusiast
Enthusiast

Okay I got confused, I thought I read something above that said that iSCSI Port binding was being used. That is why I asked the question.

0 Kudos
sjesse
Leadership
Leadership

He is, its not the iscsi adapter thats the problem, its the multiple vswitches according to if they the vkernel adapters are bound to different vswitches

VMware Knowledge Base

When not to use port binding

Port binding should not be used when:

  • Array Target iSCSI ports are in a different broadcast domain and IP subnet.
  • VMkernel ports used for iSCSI connectivity exist in a different broadcast domain, IP subnet and/or vSwitch.
  • Routing is required to reach the iSCSI array.
  • Software iSCSI Port Binding is also contraindicated when LACP or other link aggregation is used on the ESXi host uplinks to the pSwitch, Limitations of LACP in VMware vSphere 5.5 (2051307)

0 Kudos
vucit
Contributor
Contributor

If they are not expected to be able to reach all targets, then the only thing I can think of is that there is an issue on the VMWare side of things. Because in this case, it makes sense that a vmk connected to switch one (nimble) will not find the IQN target (Equallogic) as it is not connected to that switch.

I guess that is where I am getting tripped up with idea that the vmk is talking to the switch, asking for targets on the equallogic based on the discovery ip, and getting nowhere cause it is not connected, which would make sense on the error we are seeing in the logs.

0 Kudos
vucit
Contributor
Contributor

"If they are not expected to be able to reach all targets"

I typed that wrong and originally read it wrong as well. If they are expected to reach all targets, then the only way to do this without routing would be to use a "san" switch in this case between the host and the storage need to be on the same network.

0 Kudos
ChrisFD2
VMware Employee
VMware Employee

If you set the vSwitch, PGs and vmk interfaces up as I suggested, and the physical NIC you are binding to each PG using overrides, to the correct interface on the storage, through the same physical switch it should work. There is no need to move to a SPOF by using one physical switch.

Sorry if that's bad english, it doesn't read great but the concept is the same if you have 2 paths or 8 paths.

As an example:

All iSCSI NICs are on vSwitch 1, all are active.

Storage box 1 NIC1 on 192.168.1.1, cabled to a physical switch, cabled to host physical NIC1 > vSwitch > Port Group 1 (with physical adaptor override so NIC1 is the ONLY adaptor) > port bound to vmk interface on 192.168.1.101

Storage box 1 NIC2 on 192.168.1.2, cabled to a physical switch, cabled to host physical NIC2 > vSwitch > Port Group 2 (with physical adaptor override so NIC2 is the ONLY adaptor) > port bound to vmk interface on 192.168.1.102

Storage box 2 NIC1 on 192.168.1.3, cabled to a physical switch, cabled to host physical NIC3 > vSwitch > Port Group 3 (with physical adaptor override so NIC3 is the ONLY adaptor) > port bound to vmk interface on 192.168.1.103

And so on.

One of the biggest things for beginners to get your head around with vSphere networking is the difference between a port group and a vmk interface.

Think of port groups as VLANs on a switch.

Think of vmk interfaces as NICs that provide services which the host requires, such as vMotion, vSAN, Managment, Software iSCSI etc.

Regards,
Chris
VCIX-DCV 2020 | VCP-DCV 2020 | VCP-NV 2020 | vExpert *** | CCNA R&S
0 Kudos