Network Best Practice for 6 Uplinks

[TheTone]

Can someone help me out and tell me a recommended best practice for segregation using 6 Uplinks. Ideally I would like to have 2 for Management, 2 for VMotion, 2 for Production and 2 for iSCSI (Software). However I only have 6 pNICS. What would people here recommend when using HA and DRS clusters?

I would also like to use load balancing on the iSCSI team but not sure how to set this up. The SAN has two NICS and are trucked using 802.1ad. How would I setup the load balancing on the iSCSI vSwitch to take advantage of the trunk to the SAN?

Any help or pointers to the right white papers or documentation would be really appreciated.

Thanks

Tony

[AndreTheGiant]

Have a look at:

http://www.networkworld.com/community/node/36359

For SAN check storage best practice cause on most storage you can't use "link aggregation".

Andre

[TheTone]

Ok from the link you have provided I should setup my network configuration like this:

pNIC0 -> vSwitch0 -> Portgroup0 (Management)

pNIC1 -> vSwitch0 -> Portgroup1 (VMotion)

pNIC2 -> vSwitch1 -> Portgroup2 (iSCSI Storage)

pNIC3 -> vSwitch1 -> Portgroup2 (iSCSI Storage)

pNIC4 -> vSwitch2 -> Portgroup3 (Production)

pNIC5 -> vSwitch2 -> Portgroup3 (Production)

My question now is what happens when either pNIC0 or pNIC1 fail?

Does that mean if pNIC1 fails VMotion will steal the Management networks pNIC therefore preventing access to the management network?

Thanks

Tony

[mokymura]

Hi

That could be one option but you won´t have redundancy in Management and VMotion....what does means:

1.- VMotion -> won´t go throught management

2.- Management -> if that nics fails depends of the configuration you have made of the HA: the esx can detect that it has being isolated and power off machines and power them on in other esx -> loose of service

How many machines are we speaking about? do you have any other trafic in the Management network segment: other servers into the network..or is separate network vlan....

I will go for pNIC0 and pNIC1 with both: SC and VMotion.

Regards

If you find this or any other answer useful please consider awarding points by marking the answer helpful or correct.

[TheTone]

I have just been reading LeftHand Networks documentation on how they recommend you setup the ESX networking when using 6 Uplinks and they recommend this:

pNIC0 -> vSwitch0 -> Portgroup0 (Management)

pNIC1 -> vSwitch0 -> Portgroup1 (Production)

pNIC2 -> vSwitch1 -> Portgroup2 (iSCSI Storage)

pNIC3 -> vSwitch1 -> Portgroup2 (iSCSI Storage)

pNIC4 -> vSwitch2 -> Portgroup3 (VMotion)

pNIC5 -> vSwitch2 -> Portgroup3 (VMotion)

Is this a better setup than mentioned above with the VMotion on the same switch as the Service console?

Thanks

Tony

[mokymura]

Hi

I just realise that i made a mistake in mi other answer....i thought management and vmotion was on diferent vswitch

Acutaly that's the best option

3 vswitches with 2 pnics each -> you need to configure the loadbalancing and the network failover detection

Sorry about the missunderstood

Regards

If you find this or any other answer useful please consider awarding points by marking the answer helpful or correct.

[AndreTheGiant]

Is this a better setup than mentioned above with the VMotion on the same switch as the Service console?

Cause VMotion can only work with a link (with default team policy) I suggest:

pNIC0 -> vSwitch0 -> Portgroup0 (Management)

pNIC1 -> vSwitch0 -> Portgroup1 (VMotion)

pNIC2 -> vSwitch1 -> Portgroup2 (iSCSI Storage)

pNIC3 -> vSwitch1 -> Portgroup2 (iSCSI Storage)

pNIC4 -> vSwitch2 -> Portgroup3 (Production)

pNIC5 -> vSwitch2 -> Portgroup3 (Production)

Andre

[TheTone]

So why would LeftHand Networks recommend this:

pNIC0 -> vSwitch0 -> Portgroup0 (Management)

pNIC1 -> vSwitch0 -> Portgroup1 (Production)

pNIC2 -> vSwitch1 -> Portgroup2 (iSCSI Storage)

pNIC3 -> vSwitch1 -> Portgroup2 (iSCSI Storage)

pNIC4 -> vSwitch2 -> Portgroup3 (VMotion)

pNIC5 -> vSwitch2 -> Portgroup3 (VMotion)

If I go with your recommendation of VMotion and Management on the same switch then is that not a security issue as VMotion does not encrypt its traffic?

Thanks

Tony

[mokymura]

they will suggest like this to have a dedicate vswitch with 2pnics for VMotion: no traffic is mixed

Depend's on your network team...what will you have easier

If you find this or any other answer useful please consider awarding points by marking the answer helpful or correct.

[AndreTheGiant]

is that not a security issue as VMotion does not encrypt its traffic?

You can use VLAN to isolate traffic.

Andre

[TheTone]

But if I am right in saying, if use a VLAN to seperate traffic I no longer have failover for the management or VMotion network but if I use Production and Management on same vSwitch I could use the same VLAN for both and still have failover?

Thanks

Tony

[TheTone]

I am the network team, and vmware team and the SAN team. Which is why things are really complicated. So many things to think about.

So would you recommend a dedicated vSwitch for VMotion. I think that is the way I am going to go. LeftHand must me doing things right so I am tempted to copy what they are recommending.

Thanks

Tony

[mokymura]

You are the Allin1 guy

I would go for that one...1 vswitch for vmotion dedicate: loadbalancing with GB nics

Regards

If you find this or any other answer useful please consider awarding points by marking the answer helpful or correct.

[AndreTheGiant]

But if I am right in saying, if use a VLAN to seperate traffic I no longer have failover for the management or VMotion network but if I use Production and Management on same vSwitch I could use the same VLAN for both and still have failover?

You can have failover, you have only to do tagging at portgroup level.

The 2 NIC are on "trunk" port of the "real" switch.

Andre

[TheTone]

I dont understand. If I have this setup:

vSwitch0 Portgroup1 (VLAN 100) contains pNIC1 connected to pSwitch1 (VLAN 100)

vSwitch0 Portgroup2 (VLAN 101) contains pNIC2 connected to pSwitch1 (VLAN 101)

If portgroup1 (Management) pNIC1 failed then it would try to use pNIC2 which is physicall connected to a different VLAN on the pSwitch. I would of though that it would not be able to use pNIC2? Can you explain to me how this would work as I am confused?

Thanks

Tony

[virtualportal]

Hi,

You will have to set the ports on the physical switch as trunks.

Then they can be configured to share the VLANs from the wider network. This could be using VLAN Trunking Protocol (VTP) or by allowing the VLANs through these trunks and having it VTP Transparent.

Then you will have two physical NICs on that vSwitch and you can Load Balance and have redundancy as the VLANs will be shared over both

Finally dont forget to add the VLAN ID's from the wider network to each Portgroup. In your case VLAN ID: 100 for Portgroup 0 and VLAN ID: 101 for Portgroup 1.

note: if you are using VLAN 1 in your switching network as your native VLAN, then do not configure the Port Group as VLAN ID 1, leave it blank or 0.

Hope this helps

Steve

[AndreTheGiant]

I dont understand. If I have this setup:

vSwitch0 Portgroup1 (VLAN 100) contains pNIC1 connected to pSwitch1 (VLAN 100)

vSwitch0 Portgroup2 (VLAN 101) contains pNIC2 connected to pSwitch1 (VLAN 101)

You have vSwitch0 with 2 uplink (pNIC1 and 2) connected to port in trunk mode (that allow both VLAN 100 and 101).

The two portgroup ARE connected to vSwitch, NOT directly to the tro pNIC.

Andre

[TheTone]

But surely that means the traffic is not seperated because it is going down the same trunk. I dont understand. What would be the difference with this method or just putting both portgroups in the same VLAN? I am new to VLAN's and managed switching so dont fully understand?

Thanks

Tony

[virtualportal]

Traffic is Seperated by using a different VLAN. Yes it does travel over the same uplink, but VLAN's are widely used as a way of Sperating Traffic.

As long as the Service Console and vMotion Port Groups have different VLAN IDs then the traffic cannot get from one VLAN to another.

This is used widely. For example if you have a Cisco Network with 10 Physical Switches using VLANS, you will always have trunks between these switches that are sending traffic for all different VLANS. However the Data is Tagged with a VLAN ID.

Using a network standard called 802.1Q or VLAN Tagging, this "allows multiple bridged networks to transparently share the same physical network link without leakage of information between networks".

(source: )

I believe that this is your best option if you require a redundancy and failover.

Steve

[TheTone]

Ok, I think I'm starting to get the hang of this? So Just to go over things if I have the following:

pNIC0 -> vSwitch0 -> Portgroup0 (Management) VLAN 100

pNIC1 -> vSwitch0 -> Portgroup1 (VMotion) VLAN 101

1. Now I have 2 physical switches that I will connect together using a 2 port trunk (ports 23 and 24).

2. I then setup VLAN 100 on both switches as ports 1, 23 and 24.

3. I then setup VLAN 101 on both switches as ports 1, 23 and 24.

4. I then connect pNIC0 to port 1 on pSwitch 1.

5. I then connect pNIC1 to port 1 on pSwitch 2.

6. I then connect port 23 on pSwitch1 to port 23 on pSwitch2.

7. I then connect port 24 on pSwitch1 to port 24 on pSwitch2.

8. I then trunk ports 23 and 24 together for Link Aggregation.

Is what I propose the correct way of setting up the switches and VLANS for using both switches?

What would be the best teaming for the vSwitch0?

Would it be better to setup prefered paths from each pNIC to go through seperate switches or to just setup the default load balancing using port ID?

Is there anything I am missing here?

Thanks again.

Tony