iSCSI setup but failover is slow.

[NWP_Tom]

Ok, I've acquired all of the kit and have plugged it all in but I need a little help with the setup of the nics and ip addresses, etc.

The host is a Dell 2950 with 6 nics. The san is a Dell MD3000i with dual storage controllers. Per Dell documentation the ports on each storage controller are on two different subnets (see below)

Storage Controller 0 Port 0 192.168.130.101/24

Storage Controller 0 Port 1 192.168.131.101/24

Storage Controller 1 Port 0 192.168.130.102/24

Storage Controler 1 Port 1 192.168.131.102/24

All 4 ports on the storage controller are connected to a gig switch. Two nics in the ESX host are connected to the same gig switch for access to the san.

Now my question is how do I go about setting up the ESX host so that I can access the SAN and have automatic load balance/fail over? I'm unsure of what IP's to assign the VMKernel port. Do I have 2 VMKernel ports with different subnets attached to different nics that go to the san? Or one VMKernel port with 2 nics? etc...

[christianZ]

Have you checked this doc: http://www.dell.com/downloads/global/solutions/pvault_esx_storage_deployment_guide_v1.pdf

[NWP_Tom]

Yes I have. I don't have a problem getting the host to see the SAN. My question is more on how I should setup the vswitches and pNics for good failover/load balancing.

[NWP_Tom]

Ok, I've been playing around and I think I have it setup right but I wanted to get some opinions... I used an Intel 2 port nic and used both ports to connect to the iSCSI network. They are both in active state and are set to see the 192.168.131.x network. Aside from not having dual switches (that comes in about a month when I can get another one) does this sound right? Attached is a pic from the networking configuration tab.

This config does work for connectivity. I can see the LUNS on the san and whatnot. I'm just wanting to know if this is a good setup for failover/load balance to the san.

Thanks for your help!!

[Mark_Bradley]

Hi Tom,

On the document that was mentioned previously there are some good diagrams of how to set up the networking... on Page 5...

So for the setup you have with the storage controllers having an IP in two different subnets I would say the ideal setup would be to have two physical switches for the iSCSI networks. One switch would be for the 192.168.130.x range and the other switch would be for the 192.168.131.x range. Then on ESX you would need to create two vSwitches for iSCSI, one for 192.168.130.x and one for 192.168.131.x. Then add one or more NICs to the vSwitches, for more resilience/redundancy add two or more NICs.

So as an example:

vSwitch2 - 2 NICs assigned - 192.168.130.x

vSwitch3 - 2 NICs assigned - 192.168.131.x

Physical Switch1 - 192.168.130.x

Physical Switch2 - 192.168.131.x

Storage Processor A - NIC0 - 192.168.130.x

Storage Processor A - NIC1 - 192.168.131.x

Storage Processor B - NIC0 - 192.168.130.x

Storage Processor B - NIC1 - 192.168.131.x

So with the items listed above, you can see that a single NIC failure in an ESX server would not take down either vSwitch or Paths to the storage. If a single physical switch were to fail you would still have connectivity to two NICs in ESX and also to the MD3000i's two storage controllers. If one of the storage controllers were to fail in the MD3000i you would still have connectivity to both physical switches and the ESX servers.

__________________________________________________________________________________________

Check out my blog at

[NWP_Tom]

Unfortunately I only have 1 switch and 2 nics to work with so those diagrams on page 5 are only partially helpful. I will eventually be able to get another switch but we'll still only have 2 nics to play with. The other 2 are set aside for vmotion in the future.

That said, what would be the best setup? Right now I have all 4 ports of the MD3000i going into my dell 5424 switch. Each Storage Controller in the MD3000i has 2 ports, and each of those ports are on different IP networks (192.168.130.x and 192.168.131.x)

Currently the 2 nics in the ESX host are on one vswitch in one port group. The iSCSI software adapter sees 2 targets. The Dynamic discovery tab was set with 1 ip address of 192.168.131.101. I can pull the cable on either of the 2 nics going from host to san switch and everything still works. I can pull either of the cables from san switch to MD3000i and everything still works.

Is this an ok config or could I do something better with what I have?

[Mark_Bradley]

Hi Tom,

Have you setup VLANs on the Dell 5424 switch? The reason I ask is that as you have setup the MD3000i Storage Controllers to have one NIC on 192.168.130.x and one on 192.168.131.x unless you have VLAN'ed they are going to both be broadcasting over the switch.

To be honest what I would do now if I were you is as follows:

1. Remove the SPA1 and SPB1 cables from the Dell 5424 switch, these will not be being used as they are on a IP different range.

2. Create a second vSwitch on ESX ready for when you have the second switch. Assign no NICs to it at this stage.

In that configuration you are protected from most failures, however the Dell 5424 switch is a single point of failure. If you can get the second switch for the other IP range/subnet you will then have removed that single point of failure, even having just one NIC per vSwitch on the ESX servers is not really a single point of failure as the failover paths are available. I would mention that if I were you I would have one iSCSI NIC on one physical NIC card and the other on a different card, that way if one physical NIC card fails iSCSI should still work.

__________________________________________________________________________________________

Check out my blog at http://www.ridethevirt.blogspot.com

[NWP_Tom]

I am not using Vlans.

I can't unplug any of the ports on the MD3000i as I have a non-virtualized server accessing a lun on the SAN using the ip's assigned to the other ports.

So right now I have only 2 pNics available for use on the iSCSI san network and I have them both on the same vSwitch with the vmkernel and second SC port (network 192.168.131.x).

What you are suggesting is to create a second vSwitch with a vmkernel and SC port on it with IP's belonging to the other IP network (192.168.130.x) but not assign any nics to it yet. Once I get the second switch I would re-arrange the network cables from the SAN so that each IP subnet is connected to it's own switch and then move the corresponding network cable from the host to "new" switch and move that pNic over to the other vSwitch correct?

[Mark_Bradley]

Hi Tom,

Thanks for the further information I have a better understanding now of what you have setup.

So please correct me if I have this wrong:

192.168.130.x - Has SPA0, SPB0 and ESX connectivity

192.168.131.x - Has SPA1, SPB1 and Physical Server connectivity

Quote:

"What you are suggesting is to create a second vSwitch with a vmkernel and SC port on it with IP's belonging to the other IP network (192.168.130.x) but not assign any nics to it yet. Once I get the second switch I would re-arrange the network cables from the SAN so that each IP subnet is connected to it's own switch and then move the corresponding network cable from the host to "new" switch and move that pNic over to the other vSwitch correct? "

In essence yes, however after further thought I would wait until the physical switch is installed. But the process would be the same as you state above.

__________________________________________________________________________________________

Check out my blog at

[NWP_Tom]

Hi Tom,

Thanks for the further information I have a better understanding now of what you have setup.

So please correct me if I have this wrong:

192.168.130.x - Has SPA0, SPB0 and ESX connectivity

192.168.131.x - Has SPA1, SPB1 and Physical Server connectivity

Quote:

"What you are suggesting is to create a second vSwitch with a vmkernel and SC port on it with IP's belonging to the other IP network (192.168.130.x) but not assign any nics to it yet. Once I get the second switch I would re-arrange the network cables from the SAN so that each IP subnet is connected to it's own switch and then move the corresponding network cable from the host to "new" switch and move that pNic over to the other vSwitch correct? "

In essence yes, however after further thought I would wait until the physical switch is installed. But the process would be the same as you state above.

__________________________________________________________________________________________

Check out my blog at

The IP's are assigned the other way around but yes, you have the picture.

Now since each storage controller has an IP from each subnet and each storage controller can get to all LUNs on the array is there any reason to change it from the way it is now other then adding an additional switch for switch redundancy? I guess I keep thinking in my mind that there is only one "Correct" way to hook things up but that's not always the case.

[malaysiavm]

You should try to avoid your ISCSI connection to be connected to the same physical switch and subnet on your production network. As the storage traffic on going may utilize most of the bandwidth available on the switch. Provide extra NIC for load balancing and failover on the VMkernel you utilize for ISCSI will improve redundancy and performance always.

Malaysia VMware Communities -

[NWP_Tom]

My iSCSI connection is actually on an isolated lan. It is for iSCSI data only. The only thing on that lan is the nics for iSCSI on my ESX server and the nics for iSCSI on a separate physical server. All "Production" traffic is on different nics and a completely separate lan.

[Mark_Bradley]

Hi Tom,

With regard to the switches, you are right in an ideal configuration you would have two physical switches for the iSCSI traffic. One would be connected to 192.168.130.x and the other would be 192.168.131.x. Then with two NICs in the ESX servers you would end up with two vSwitches configured for iSCSI each with a NIC each. This would protect you from a failure at a physical switch level and if a NIC failed.

If you move to having two switches, one per subnet, you could also provide dual redundant connections for all your servers not just ESX if you have available NIC ports in them.

__________________________________________________________________________________________

Check out my blog at http://www.ridethevirt.blogspot.com

[NWP_Tom]

I got my second switch yesterday and got it installed. I believe I have everything setup correctly however it seems that with 2 subnets the failover between the two subnets is really slow as it takes up to 1.5 minutes or so. I have the nics and SAN plugged in just like the image on page 5 of the Dell/ESX Storage Deployment guid (a screenshot is attached below.) I've also attached a screenshot of the Networking tab from the VI client.

Basically I'm looking to see if this is valid and maybe find out why failover is slow between the two subnets. My testing was basically to start a file copy on a VM from some server on the production network over to the desktop of the VM which is housed on the SAN. They copy works and doesn't error but it pauses for a long time after I pull one of the cables that go from the host to the san. It eventually picks the copy back up on the other NIC like it should and continues till finished.

[NWP_Tom]

So I've made some changes as indicated in my previous post. Failover works but it's slow. Any way to speed it up?

Thanks!

[KyawH]

You could try putting a link between the iSCSI switches.