vSphere deployment - network considerations..

[tiaccadi]

Hi all,

I'm going to build a vSphere infrastructure, with 2 ESX nodes and a NetApp FAS2020A as shared storage.

I've already done a project like this, but with only one switch and one storage controller. This time we want to eliminate every SPOF, therefore we're going to use a twin-controller storage and two switches.

I've attached a Visio diagram that I think will make me more easier to explain.

I've some doubt about the network configuration, convergence time, STP, MPIO, and something more.. I'll try to show you the environment I want to build, to expose later my questions:

1) Switches:

- 2 3Com 4200G, but could be something else.. they're going to be NOT stackable, however.

- Every port attached to ESXs or FAS controllers will be in port fast mode (I guess..)

- Link 5, 8 and 9 are 2-port link aggregation

2) NetApp:

- 2 NICs, bonded in 1 VIF in single-mode, with 3 IP aliases. The red cable for both controllers (as shown in the attachment) will be the favorite NIC (vif favor command).

3) ESX hosts:

- 2 NICs for iSCSI (red and green cable in the diagram) in Active/Unused mode (VLAN 101 will use one NIC, VLAN 102 the other one.. no standby NIC, as for best practices for iSCSI MPIO in vSphere). Fixed path, round robin should be unuseful, 'cause there's only one active link on the storage side (therefore max 1Gbps)

- 2 NICs for virtual machine network and service console access, load balancing based on port-ID

Let's imagine that I've one datastore (Datastore1) on controller 1 (10.0..221) and one (Datastore2) on controller 2 (10.0..222).. then, questions:

1) MPIO:

- Will it works if configured in this way? Fixed path, path on link 1-6 to reach Datastore1, link 2-5-6 to reach Datastore2.. I've done some test, it seems to work, but I've some doubt in case of failure..

- If link 1 goes down, ESX switches on the other path to reach Datastore1, and traffic flow will be link 2-5-6.. this is the only thing on which I'm almost sure

- If link 6 goes down, link 7 goes on (vif in single-mode).. Does than ESX switch to the second path to reach Datastore1 (link 2-5-6), or data will flow on link 1-5-7?

2) Layer-2 and STP:

- How Spanning Tree Protocol can interfere with behaviors relative to MPIO described above?

- Port fast on port attached to ESXs and FAS controllers should be enough to have this infrastructure work as desired?

- Will STP, or traffic in general, have problems with switches linked in this topology?

- I really need link 5? If yes, it's better to remove VLAN 103 (that can use link 8 and 9, up to that simil-core switch)?

- How the network react to a switch failure? MPIO switches paths, NetApp switches NIC and everything is working?

- How failback should be set on ESX vSwitch in this topology?

To summarize, I can't be sure that everything is going to work well, mostly for a strange fear in loop and in STP, both during normal activities and failover-failback situations..

Thank you so much and sorry for the length of the post!

[MKguy]

Pertaining your layer 2/STP concerns:

From what I can see, the only possible loop present in your configuration concerns VLAN 103. VLAN 101 and 102 dont even have a requirement for STP, since it's only 2 switches connected with an aggregated Link of 2 physical ports (5).

I'd say you don't have to tag VLAN 103 through link 5, which would eliminate the loop of VLAN 103 as well. Traffic across hosts would then always have to go through your central switch below, but I don't think it would be much of an issue (depending on your needs, of course). Also, STP would disable one link in that 'triangle' anyways to prevent a loop.

Your SC network is on the same VLAN as your VMs? You should put that on dedicated interfaces with a dedicated VLAN plus an additional private VLAN (or even crossover cable with your 2 hosts) for VMotion, if required.

I guess you are also aware of the SPOF that the semi-core switch could impose?

I highly recommend you this magnificent networking whitepaper:

http://www.cisco.com/application/pdf/en/us/guest/netsol/ns304/c649/ccmigration_09186a00807a15d0.pdf

It's a bit older and from Cisco, but it contains great general networking information that applies to every environment.

Sorry if I have missed something from your post after you so precisely described your considerations and environment and I'm not a networking +_expert+_.