Network design with 6 NICs and configuration of physical switches

[blueprint01]

Hello,

we  have two ESX Servers and both have 6 physical NICs, at this time there  is only one physical switch availiable (so no redundancy). Now I want to  configure my network and I'd like to discuss the following network  design:

vmnic0, vmnic1 -> vswitch0 -> Virtual Machine Network (Active/Active)

vmnic2, vmnic3 -> vswitch1 -> iSCSI Network (MPIO)

vmnic4 (Active), vmnic5 (Standby) -> vswitch2 -> vMotion

vmnic5 (Active), vmnic4 (Standby) -> vswitch2 -> Management

Now vMotion and Management would be on the same IP subnet but this is not best practise, that's why I want to use VLANs for it:

vmnic4 (Active), vmnic5 (Standby) -> vswitch2 (VLAN 10) -> vMotion

vmnic5 (Active), vmnic4 (Standby) -> vswitch2 (VLAN 20) -> Management

Is  this best practise? I have now separate networks for vMotion and  Management and a failover link is available. What do I have to configure  on my HP ProCurve 2510G-Switch? Let's say vmnic4 and vmnic5 are  connected to port 1 and port 2 of the HP Switch, do I have to add two  VLANs (10 and 20) on the Switch define define port 1 and port 2 as an untagged vlan port for these VLANs? The second ESX server will be connected to port 3 and 4 there is the same configuration?

Should I use VLANs for the other networks although it not seems to be necessary?

Greetings!

[weinstein5]

Welcome to the Community - using vlans is a perfectly acceptable method for seprating network traffic and remember each vlan will have to have its own IP Subnet scheme -

The physical switch you will need to configure the physical posrts for each vlan that will come across creating a trunk port.

I have also moved this to a more approriate forum -

[rickardnobel]

blueprint01 wrote:

we  have two ESX Servers and both have 6 physical NICs,

What version of ESX or ESXi do you have? How are the 6 NIC ports located (i.e. two built in and one quad-port card?)

vmnic0, vmnic1 -> vswitch0 -> Virtual Machine Network (Active/Active)

vmnic2, vmnic3 -> vswitch1 -> iSCSI Network (MPIO)

vmnic4 (Active), vmnic5 (Standby) -> vswitch2 -> vMotion

vmnic5 (Active), vmnic4 (Standby) -> vswitch2 -> Management

Typically you place the vmk used for managment on vSwitch0 and connected to vmnic0 + some other card, perhaps even on another physical network card if possible. It should work without problems to have the VM network on the first NICs, but they are most often placed on some other vSwitch.

I think in general your setup looks good. As noted from weinstein above you must really have separate IP subnets for the Management and vMotion separation. If using the same IP network, even with VLANs, will actually not work and all traffic will use the same vmknic.

What do I have to configure  on my HP ProCurve 2510G-Switch? Let's say vmnic4 and vmnic5 are  connected to port 1 and port 2 of the HP Switch, do I have to add two  VLANs (10 and 20) on the Switch define define port 1 and port 2 as an untagged vlan port for these VLANs?

When you define a VLAN on the vSwitch portgroup you must make sure that every port on the physical switch is a tagged member of that VLAN. This means for example that this must be done on both ports connected to the both active and standby adapter.

Note also that you should not configure the ports as trunk, as this on HP switches is something different than on Cisco devices, just to make things a bit harder..

[blueprint01]

Hi rickardnobel,

we have vSphere 5.1 with Essentials Plus license and two active hosts which share a Dell Equallogic via iSCSI. Both hosts have 10 NICs (two onboard NICs and two quad port NICs) but I need up to four NICs for other networks like DMZs:

vmnic0, vmnic1: onboard

vmnic2 - vmnic5: first quad port

vmnic6 - vmnic9: second quad port

I will change my network design as you described:

vmnic0 (active), vmnic2 (standby) -> vswitch0 -> Management (VLAN 10) -> 192.168.10.0/24

vmnic2 (active), vmnic0 (standby) -> vswtch0 -> vMotion (VLAN 20) -> 192.168.20.0/24

vmnic1, vmnic3 -> vswitch1 -> iSCSI (MPIO) -> 10.10.0.0/24

vmnic4, vmnic6 -> vswitch2 -> VM Network -> 192.168.40.0/24

This would mean that that vmotion and management is redundant and they don't use the same IP network. All networks are redundant and use two different hardware NICs. I have to configure the physical switch ports which are connected to vmnic0 and vmnic2 as tagged for VLAN 10 and VLAN 20 the same for the ports of the other host.

Is this correct and best practice?

May I use the same IP subnet and VLAN (default untagged VLAN) for management and VM network or is this a problem for performance or security? May I use all interfaces for VMs (VM network and DMZs) with one vswitch with VLAN to have redundancy and load balancing on all networks?

[rickardnobel]

blueprint01 wrote:

I will change my network design as you described:

vmnic0 (active), vmnic2 (standby) -> vswitch0 -> Management (VLAN 10) -> 192.168.10.0/24

vmnic2 (active), vmnic0 (standby) -> vswtch0 -> vMotion (VLAN 20) -> 192.168.20.0/24

vmnic1, vmnic3 -> vswitch1 -> iSCSI (MPIO) -> 10.10.0.0/24

vmnic4, vmnic6 -> vswitch2 -> VM Network -> 192.168.40.0/24

That seems good, now you have redundancy on network card level too. If you can (and if possible should) get another physical switch you could quite easy arrange your cables to have full physical switch redundancy as well without any changes on the ESXi network configuration.

All networks are redundant and use two different hardware NICs. I have to configure the physical switch ports which are connected to vmnic0 and vmnic2 as tagged for VLAN 10 and VLAN 20 the same for the ports of the other host.

Is this correct and best practice?

Yes, that is good. The vMotion network could use just any unused IP range and needs only addresses for the host, so no routing or default gateway is needed. If you need help with the switch configuration with tagging status let us know.

May I use the same IP subnet and VLAN (default untagged VLAN) for management and VM network or is this a problem for performance or security?

That will depend on your situation. If possible it is good to have a separate VLAN / IP subnet only for management, but depending on the size of your network (you mentioned having a single switch) then it might not be necessary. Be sure to set a long and complex password for the root account on the hosts.

[blueprint01]

Rickard Nobel schrieb:

That seems good, now you have redundancy on network card level too. If you can (and if possible should) get another physical switch you could quite easy arrange your cables to have full physical switch redundancy as well without any changes on the ESXi network configuration.

I was thinking about using a second switch for redundancy but what do I have to do to avoid loops? Every vSwitch would have two NICs, one connected to Switch A and one connected to Switch B. Switch A and Switch B have at least in the VM network an uplink to our core switch. What if I now connect Switch A with Switch B by using a trunk port? (tagged port for all VLANs) I think I have to enable STP to avoid loops? There is no STP enabled in our network, do I have to enable it for all switches?

Rickard Nobel schrieb:

That will depend on your situation. If possible it is good to have a separate VLAN / IP subnet only for management, but depending on the size of your network (you mentioned having a single switch) then it might not be necessary. Be sure to set a long and complex password for the root account on the hosts.

Perhaps I will use a separate IP subnet for management, may I connect my physical vcenter server to the management network and the "normal" network to be able to connect with vsphere client from our network? Do I need a default gateway for the management network or does it need something like DNS?

[rickardnobel]

blueprint01 wrote:

I was thinking about using a second switch for redundancy but what do I have to do to avoid loops? Every vSwitch would have two NICs, one connected to Switch A and one connected to Switch B. Switch A and Switch B have at least in the VM network an uplink to our core switch. What if I now connect Switch A with Switch B by using a trunk port? (tagged port for all VLANs) I think I have to enable STP to avoid loops? There is no STP enabled in our network, do I have to enable it for all switches?

STP is good to have, and if you are using HP Procurve devices it will default to the newer and much fast Rapid Spanning Tree. However, even as RSTP in my opinion is very good to keep the network overall in a robust state it is actually not needed for the ESXi network redundancy.

If you make sure you use the default NIC teaming policy on your vSwitch ("port id") you could actually connect a single vSwitch to two physical switches which connects to a core switch, and not getting any layer two loops. If wanted I can explain the exact workings of this, but the short answer is that as long as you keep the default NIC teaming settings you will not create any loops and you do not have to configure Spanning Tree for that reason (but is good for other purposes).

[blueprint01]

Rickard Nobel schrieb:

STP is good to have, and if you are using HP Procurve devices it will default to the newer and much fast Rapid Spanning Tree. However, even as RSTP in my opinion is very good to keep the network overall in a robust state it is actually not needed for the ESXi network redundancy.

If you make sure you use the default NIC teaming policy on your vSwitch ("port id") you could actually connect a single vSwitch to two physical switches which connects to a core switch, and not getting any layer two loops. If wanted I can explain the exact workings of this, but the short answer is that as long as you keep the default NIC teaming settings you will not create any loops and you do not have to configure Spanning Tree for that reason (but is good for other purposes).

Okay, that is clear. But what if I connect the two swtiches with each other? Then there are two ways for Switch A to reach Switch B, direct and via core switch. Isn't there a loop? I though it is necessary to connect the two swtiches with each other (perhaps twice with LACP) because of performance reason, isn't it?

[rickardnobel]

blueprint01 wrote:

Okay, that is clear. But what if I connect the two swtiches with each other? Then there are two ways for Switch A to reach Switch B, direct and via core switch. Isn't there a loop? I though it is necessary to connect the two swtiches with each other (perhaps twice with LACP) because of performance reason, isn't it?

If you connect the three physical switches to each other than that would indeed create a loop.

It does depend on your total physical network infrastructure how the physical switches should be best connected. There is not really a definitive need for two "close to ESXi"-switches to be directly connected, as long as there is connectity between them through the core switch.

With some advanced Spanning Tree configuration (called MSTP) you could create several spanning trees at the same time, keeping different links up and down depedning on the VLANs. However, that involves some planning and configuration before getting into that.

[blueprint01]

Rickard Nobel schrieb:

If you connect the three physical switches to each other than that would indeed create a loop.

It does depend on your total physical network infrastructure how the physical switches should be best connected. There is not really a definitive need for two "close to ESXi"-switches to be directly connected, as long as there is connectity between them through the core switch.

With some advanced Spanning Tree configuration (called MSTP) you could create several spanning trees at the same time, keeping different links up and down depedning on the VLANs. However, that involves some planning and configuration before getting into that.

Okay, I think I will add the second switch in a second step. Now I will focus on the deployment of my ESX Environment, if this is running I will decide to use STP or not.

Thank you for your help, I think my questions are now answered.

[rickardnobel]

blueprint01 wrote:

Thank you for your help, I think my questions are now answered.

You are welcome. Return if you have more questions on the setup of physical or virtual networking.