DS4800 Cabling Best Practices

[tadsmith]

The SAN Configuration Guide

(http://pubs.vmware.com/vi301/san_cfg/wwhelp/wwhimpl/common/html/wwhelp.htm?context=san_cfg&file=esx_san_cfg_arrays.6.10.html)

notes the following:

• Connect each HBA on each ESX Server machine to a separate switch. For example, connect HBA1 to FC switch 1 and HBA2 to FC switch 2.

• On FC switch 1, connect SP1 to a lower switch port number than SP2, to ensure that SP1 is listed first. For example, connect SP1 to FC switch 1 port 1 and SP2 to FC switch 1 port 2.

• On FC switch 2, connect SP1 to a lower switch port number than SP2, to ensure that SP1 is listed first. For example, connect SP1 to port 1 on FC switch 2 and SP2 to port 2 on FC switch 2.

This example states that it assumes that all LUNs are owned by SP1. If I have LUNs owned by both SP1 and SP2, should I reverse this cabling on the second switch? For example, connect SP1 to a lower port on FC switch 1 and connect SP2 to a lower port on FC switch 2? Does it make a difference?

[mike_laspina]

Hi,

No that's not required. The architectural strategy is know as and ABBA connection format. The principle is to prevent thrashing from Storage Processor LU ownership takeovers. In other words if ESX host A connects to LU1 on it preferred path A to SP1 and a second ESX host connects on its preferred path B to LU1 and it this path fails the second host will transition to path A. Since the second hosts path A connects the the same SP1 it will not cause a Automatic Volume Transfer (AVT) to SP2 and thus prevent the thrashing event.

So the important part is make sure your ESX hosts startup on the correct preferred path.

The 1st port 2nd port SP configuration causes FC ESX hosts to enumerate targets in the same path order on startup without explicit settings.

vExpert 2009

[tadsmith]

So you're suggesting that I keep SP1 in a lower port on both switches? If I'm understanding that correctly, this would cause SP1 to be the first in the path order for all LUNs (even the LUNs owned by SP2). Wouldn't you want the LUNs owned by SP2 first in the path order for one of your HBAs? Otherwise, won't everything go through the first HBA by default? It seems as though I would want all LUNs on SP1 to connect through HBA1 and all LUNs on SP2 to connect through HBA2 for very minimal load balancing.

I'm not aware of anyway to make sure my ESX hosts start on the correct path since the DS4800 requires the MRU policy.

[mike_laspina]

I'm not suggesting you do not follow the SP connection best practice,

you did not indicate how you are currently connected. I am only suggesting not to be concerned about the host side because it will always enumerate in the same way provided that the storage side has SPs on ports that enumerate one on a lower address that the other.

vExpert 2009

[tadsmith]

I currently do not have a cross-connect SAN. This was the way IBM formerly advised the setup. They are now advising a cross-connect SAN so I am looking into adding an additional connection from each FC Switch to the opposite controller it already has a connection to (see image.)

I'm wondering if I need to change the ports on FC Switch 2 so that SP1 is in a lower port number. The best practice says to put SP1 in a lower port number, but then goes on to say that they are making the assumption that all LUNs are owned by SP1. Since I have LUNs balanced across SP1 and SP2, does this recommendation that SP1 always be in a lower port number still hold true?

Thanks for your help.

[mike_laspina]

Your welcome,

In my experience using 3 ports at the host FC ports will add little or no value. The reasoning is wrapped around the additional diagnostic complexity during issues and the fact that if your drive side connections are only running 2 paths which means you do not have the bandwidth to realize the additional FC connection capability and more likely the number of spindles to serve it. There is no change to the availability factor in the event of a cable/or switch issue.

While a cross fabric interconnect can improve a single fabrics scalability it does not improve availability. Separate fabrics are more important as you can take one fabric path offline or change it's configuration without any impact to the other fabric and this gives you the ability to perform non-disruptive maintenance tasks.

The simplest ABBA format is what I would normally focus on.

Regards,

Mike

vExpert 2009

[tadsmith]

The fabrics remain seperate. It is simply a matter of both storage processors having a connection to each fabric (much like each host is connected to both fabrics). IBM and VMWare are both requiring this type of connection now in their best practices. The real question I need answered is whether SP1 needs to be in a lower switch port than SP2 on both fabrics, or if it is ok to have SP1 in a lower port on Fabric 1 and SP2 in a lower port on Fabric 2.

[mike_laspina]

SP1 should be on the lower port number on each fabric for proper enumeration.

vExpert 2009

[tadsmith]

Thanks again. I should be able to make the path changes online correct? If I fail everything over to a path on Fabric 1, I should be able to reconfigure Fabric 2 and rescan the HBAs without any issues, right?

[mike_laspina · Accepted solution]

It will have some path transition issues and I would recommend you switch them at a low IO time window. Since you're in MRU policy mode it should not do any AVT's thus the interuption would be small. But with any SCSI connection change comes the unforseen elements.

Thus, yes you can do it online but not on prime time.

Regards,

Mike

vExpert 2009