VMware Cloud Community
DSeaman
Enthusiast
Enthusiast
Jump to solution

Nexus 1000v and vSwitch best practices

I am working on the design of our Nexus 1000v vDS for use on HP BL490 G6 servers. The 8 pNICs are assigned as follows:

vmnic0,1: ESXi management, VSM-CTRL-PKT, VSM-MGT

vmnic2,3: vMotion

vmnic4,5: iSCSI, FT, Clustering heartbeats

vmnic6,7: Server and Client VM data traffic

Should I migrate all pNICs to the 1000v vDS, or should I leave vmnic 0,1 on a regular vSwitch and migrate all of the others to the vDS? If I did migrate all of the pNICs at a minimum I'd designate vmnic 0,1 as system so traffic could flow until the VSM could be reached. My inclination is to migrate all the pNICs, but I saw elsewhere on the forums comments that the VSM related networks and possibly the ESX(i) console are best left off the vDS.

Thoughts?

Derek Seaman
Reply
0 Kudos
1 Solution

Accepted Solutions
RBurns-WIS
Enthusiast
Enthusiast
Jump to solution

Here's a best practice/how-to guide specific for 1000v & HP VC you might find useful.

Cheers,

Robert

View solution in original post

Reply
0 Kudos
30 Replies
logiboy123
Expert
Expert
Jump to solution

If pNics 1-4 are on a different bus from pNics 5-8 then I would use the following configuration (presuming that each and every port group is separated using VLAN tagging):

vSwitch0

VM Network - vSphere Management = pNic1 active / pNic5 standby / pNic6 standby

Management = pNic1 active / pNic5 standby / pNic6 standby

VMotion = pNic5 active / pNic1 standby / pNic6 standby

Fault Tolerance = pNic6 active / pNic1 standby / pNic5 standby

dvSwitch1

iSCSI = pNic2 active / pNic7 - Typically used in a Round Robin configuration. See;

http://virtualgeek.typepad.com/virtual_geek/2009/09/a-multivendor-post-on-using-iscsi-with-vmware-vs...

dvSwitch2

VM Network Servers = pNic3 active / pNic4 active / pNic 8 active

VM Network Nexus PCK & CTRL = pNic3 active / pNic4 active / pNic 8 active

Consider using the IP Hash load balancing method if you have good enough switches as it will give you better load balancing and throughput from all NICs. See;

http://www.vmware.com/files/pdf/virtual_networking_concepts.pdf

http://communities.vmware.com/thread/136547

http://www.tcpdump.com/kb/virtualization/vmware-esx-server/esx-nic-teaming/load-balancing-methods.ht...

Alternate configuration;

Take 1 NIC from vSwitch0 and 1 NIC from dvSwitch2 and create a second standard switch vSwitch1. Put VMotion and FT on this switch.

Design Considerations;

vSwitch0

1) You only need the  Management network for your Nexus 1000V to be able to contact the ESX  hosts and vCenter and also needs to be routable, so it is easier to use a VM Network port group with  the same VLAN as the Management network that the hosts use. If vCenter  is a VM on the inside of the environment then put it here, not on a vDS.  This will create a circular dependency otherwise. Apparently the next  major release of vSphere will resolve this issue. Put the vCenter server and Nexus primary switch on the same VLAN as the ESX hosts.

2) Management is not on a vDS because this would create a  circular dependency that you absolutely do not want. One client of ours  put their management network on a vDS and the entire system died because  of an event. The only way to fix it was to reset all networking on each  host.

3) Put VMotion here for now, if you find it is going over 1GB capacity you can modify the configuration later.

4) Same as VMotion. FT is essentially constant VMotion action. You can have up to roughly three VM's doing FT on a single 1GB connection. Depends on their workload and IO of course.

5) Separation of networking is done at the VLAN tagging level as well as the NIC Teaming level on each port.

6) Put the Control and Packet networks as separate port groups with their own VLAN's on the dvSwitch2.

Regards,

Paul Kelly

If you found this or any other post helpful please consider making use of the points system.

Reply
0 Kudos
DSeaman
Enthusiast
Enthusiast
Jump to solution

Kelly,

Thanks for taking the time to provide a long response to my question. Let me clarify a couple of things about the environment. In my OP I already paired the pNICs across the physical buses, so that level of redundancy is covered. The 8 pNICs are distributed across two 10Gb LOMs, four per LOM. 

Second, I'm no Nexus 1000v expert by any means, but my understanding is that you only get one 1000v dVS per ESX host. You can use a standard vSwitch with the 1000v on the same host though. All of the networks I described are on different VLANs. So I'm confused by your multiple dVS switch suggestion since I'm not using the VMware vDS.

99% of the storage is on a Fibre Channel SAN, so the iSCSI VLAN is mostly just for testing purposes or future usage. Likewise for FT, this is more experimental at this point since we don't have VMs that need that level of protection. But I wanted it built into the design so we could test it. In the end state the vCenter will be a VM running on hosts with the 1000v, but vCenter is currently located in another physical datacenter not using the 1000v switches. That is a future migration project.

The 1000v VSM management IP is routable to the vCenter IP, and to all of the ESXi management IPs.

Derek Seaman
Reply
0 Kudos
logiboy123
Expert
Expert
Jump to solution

Couple of points;

1) a) vDS = Distributed Virtual Switch

    b) vSS = Standard Virtual Switch

    c) vSwitch0 = Standard Virtual Switch

    d) dvSwitch1 = Distributed Virtual Switch

2) You can have up to 32 vDS in vSphere 4.1 per datacentre, it doesn't matter which host they are sitting on.

3) VMware's recommended configuration for the Nexus 1000V is to have a primary and secondary switch. The secondary switch acts as a backup for the primary and only comes online if the primary fails. It is best to try and keep the primary and secondary switch on seperate hosts.

4) I like to seperate iSCSI on to a seperate switch, be it a vSS or a vDS as I like to set jumbo frames at the switch level. Doing this while having other port groups on the same switch means you have to get quite messy with your configuration. Having a seperate vDS for iSCSI won't cost you anything extra, in fact if your iSCSI was seperated physically to a different switch this would make it more secure.

5) Regarding your NICs I was running on the presumption that they were 8 x 1GB physical NICs. Since they aren't please ignore that portion of the design consideration.

6) Yes you can use vSS and vDS on the same datacentre without issues. It is best to keep management layers on vSS at this point. It is supported to run them from a vDS, but not recommended by VMware. I don't have a document to back that up, only a support call I logged not so long ago around supportability of a design like yours.

7) VEM = Virtual Ethernet Module. This is installed on each host and requires communications to each other host for Packet and Control, but requires a connection to vCenter and must be routable for Management layer. Hence putting the Management layer for the Nexus 1000V switch on the same subnet as the hosts, but putting Packet and Control on VM Network port groups on a vDS.

DSeaman
Enthusiast
Enthusiast
Jump to solution

Comments:

#3) Don't you mean VSM? Yes I was planning on two VSMs, for redundancy.

#4) iSCSI in my environment is just for testing, so I'm not too concerned about jumbo frames. Plus the NIC in the BL490c G6s only supports jumbo frames up to 4K. But if I did want to use jumbo frames, that looks like a pretty simple configuration option for the 1000v port profile (system mtu xxxx)?

#6) Back to my OP, it sounds like vmnic0,1 should only have ESXi-Management and VSM-Management? Move the VSM-CTRL-PKT to the vDS? Say, vmnic4,5?

#7) So best practices recommend putting the 1000v management console on the same VLAN as the ESXi console? For security reasons the network team always puts switch management ports on a restricted VLAN that only certain clients can access. The 1000v management VLAN is fully routable to VC and ESXi management console.

Derek Seaman
Reply
0 Kudos
logiboy123
Expert
Expert
Jump to solution

3) Yes. The VSM is a Nexus 1000V switch. It uses the same OS as a 5000 and 7000.

4) When I use iSCSI I typically use Round Robin configurations for load balancing as this typically gets the highest throughput and I try to have a seperate physical infrastructure. So my practises are typically based around this. In your case I think you can get away with putting iSCSI on the same vDS as your other networking. In production I wouldn't do this if I could avoid it. Remember, you can have up to 32 vDS per datacentre, so no issue there. But VMware recommends against using a vDS with only a single uplink; it just something to keep in mind when building a design.

6) Yes. I would use generic vDS VM Network port groups for Packet and Control networks on the VSM.

7) My suggestion wasn't best practise. It is what I have found to be a best tradeoff in security and usage in a shared networking infrastructure. Best practise would be to put all management layer networking on a seperate switching infrastructure and then route traffic through firewalls if required. However it seems a bit of a waste to assign 2 NIC's on a vSS just for your Nexus 1000V management network. Remember that you should not put any management functions on a vDS, so that doesn't leave you many choices for placing the VM Network port group on a switch. You can actually get a physical VSM from Cisco which is a physical server that you can then plug into seperate switching infrastructure and route as required. This costs a bit more then the VM setup however.

Reply
0 Kudos
DSeaman
Enthusiast
Enthusiast
Jump to solution

So I'm still a bit confused. One possbility is:

VMware vDS: vmnic0,1: ESXi management, VSM-CTRL-PKT, VSM-MGT

Nexus vDS: vmnic2,3: vMotion

Nexus vDS: vmnic4,5: iSCSI, FT, Clustering heartbeats

Nexus vDS: vmnic6,7: Server and Client VM data traffic

In your first response you seemed to indicate the PKT-CTRL should go on the (Nexus?) vDS but in your last response you said to avoid management traffic on the vDS. So for vmnic0,1 should I make that a standard vSwitch and not use any VMware vDS but just the Nexus vDS for the rest of the NICs? Maybe I misread your responses or a few wires crossed in my brain.

Derek Seaman
Reply
0 Kudos
logiboy123
Expert
Expert
Jump to solution

I don't consider Packet and Control networking as Management in the context of assigning them to a network. They don't require routing and only need to be able to see the other hosts in the cluster. Losing Packet and or Control on a VSM will not mean losing a core infrastructure component, your system will still work in a reduced function. Losing the Management network on a VSM will cause you issues though.

This is my recommendation in your formatting;

vSS: vmnic0,1

VMkernel - ESXi Management

VM Network - VSM Management and vCenter VM

vDS: vmnic2,3,4,5,6,7

VMkernel - iSCSI

VMkernel - VMotion

VMkernel - Fault Tolerance

VM Network - Server Data

VM Network - VSM Packet & Control

If you want you could seperate out iSCSI to a seperate vDS. By the way your inbuilt NIC's will probably support an MTU of 9000 if you set them to use the iSCSI software adapter. This involves creating a vSS with a VMkernel for each NIC assigned and binding each VMkernel to the software adapter. See the following guide for instructions on this;

http://virtualgeek.typepad.com/virtual_geek/2009/09/a-multivendor-post-on-using-iscsi-with-vmware-vs...

Reply
0 Kudos
jdewitt
Contributor
Contributor
Jump to solution

One thing that is confusing in this exchange...The Nexus 1Kv switch is a combination of the VSM (the Supervisor Module) and the VEM (the Ethernet Module - or virtualized line card). You can not have a "switch" without both, and as pointed out, you can only have 1 VEM per host and that VEM can only be managed by one VSM (or pair of clustered VSMs for redundancy). Therefore a single physical ESX(i) host can only have one distributed switch that is a Nexus 1Kv. You can have other non-Nexus distributed switches...but that kind-of begs the question of why you're implementing it for some of your traffic, but don't feel it's needed for the rest.

I understand the logic of creating a standard vswitch for the management port-group and for vCenter, and that's probably a good bet for a small deployment. For larger deployments, the typical recommendation is to use another small cluster of physical hosts to house your management components (your VSMs, vCenter, etc.) and to just dump all of your networking into the Cisco Nexus 1000v on the production clusters. This becomes one of the few viable options with a 10Gb infrastructure where you only have 2 physical nics to use.

Reply
0 Kudos
logiboy123
Expert
Expert
Jump to solution

When you install the Nexus 1000V solution including the VSM primary, VSM secondary and each VEM per host, it will take control of all your Distributed Virtual Switches, not just one of them.

Reply
0 Kudos
DSeaman
Enthusiast
Enthusiast
Jump to solution

jdewitt,

We have more hardware available to us than we will use anytime soon, so my thoughts are:

1. Create a 2-node "management" cluster that hosts two VSMs, vCenter/SQL VM , and one AD VM. I could use a standard vSwitch or possibly the VMware vDS. For two nodes I'm not sure the VMware vDS buys me much.

2. All other ESXi hosts would be in a different cluster and fully utilize the 1000v for all vNICs and pNICs.

This gets away from the confusion/dependency of the VSM running on hosts that have a VEM that it manages.

Derek Seaman
Reply
0 Kudos
RBurns-WIS
Enthusiast
Enthusiast
Jump to solution

Greetings All,

I'd like to include a few more best practice suggestions from my experience.

1. VSM VLANs/Interfaces.  There have been mixed views on the requirements for 3 separate VLANs for use with the VSM in L2 mode.  Control/Mgmt/Packet.  Some organizations don't want to dedicate a brand new VLAN specically for 1000v operation.  Cisco advises you can use the same VLAN for all three interfaces.  As a best practice I always suggest you at least separate your Control & Packet VLANs from Management.  Most customers already have a "Management" network, so one additional VLAN for your 1000v control plane is ideal.  I wouldn't want to chance any anomolies with my control traffic intefering wtih management my devices - and vice-versa.  Separating them by VLAN will at least provide some protection.  This also makes it easier if you want to sniff your controll traffic if/when you're troubleshooting LACP/IGMP type of communications.

2.  System VLANs.  The ONLY port profiles that should contain "system vlans" are those for Control, Packet, Management (including Service Console/VMKernel for Mgmt) and IP storage.  I see alot of people including VMotion and other port profiles which are definately not req'd.  There's a max of 16 port profiles containing system VLANs, if you exceed it you'll have issues such as VEM programming getting truncated. I've heard people say "Well if I make it a system VLAN then if my VSM is offline and my VEM reboots, I'll still be able to forward traffic".  Perhaps so, but a) how often do you plan on your redudant VSMs to be unavailable and b) you're circumventing any security the 1000v offers on standard port profiles (ACLs, QoS etc) which will not receive their feature level programming before coming online.

3.  Uplink Auto Port Channel Configuration.  Hands down the safest & least complicated type of port channels for your uplink profiles is "mac-pinning".  This is identical to the default hashing used by the vSS - route based on virtual port ID.  The beauty of this mode is that there's no special upstream switch config required other than making it a trunk and allowing the appropriate VLANs. Also you can add/remove uplinks on the fly, or move them to a vSwitch without "breaking" any upstream port channels you might have.  The only drawback is if you're using multiple interfaces to the same switch, the links are treated as separate entities and not an aggregate channel.  With 10G ethernet this is almost a none issue unless you have some very heavy network intensive VMs.  The simplicity and ability to add/remove links if ever needed make this my ideal choice.

4.  VSM Affinity.  This is a fairly straight forward practice.  When using HA/DRS in a cluster, create a DRS rule that will keep both your VSMs on "different" hosts at all times.  Protects against single point of failure from both VSMs running on the same host.

If I think of any more, I'll update this thread.  For the curernt version of 1000v (SV1.3b) these are all I can think of.

Regards,

Robert

logiboy123
Expert
Expert
Jump to solution

Having a 2 node cluster just for vCenter and the VSM's seems like overkill. Especially if this is a test environment. It would be very secure though if you could get seperate physical networking infrastructure.

If you plan on going to production and moving all your infrastructure in then maybe it would be worth it, depends on how many hosts you intend to bring into the environment.

Reply
0 Kudos
DSeaman
Enthusiast
Enthusiast
Jump to solution

Robert,

Thanks for the great feedback. After a lot of research, many of the suggestions you made were already on my list of design features.

1. I was planning on putting the control and packet interfaces on one non-routable VLAN. The VSM management interface is going on it's own VLAN, which will be routed.

2.  Totally concur with limiting service ports to those you mentioned.

3. Because I'm using HP virtual connect the only option I have is mac-pinning.

4. Very good point. Do you run your VSMs on the same hosts that have VEMs that it manages?

Derek Seaman
Reply
0 Kudos
RBurns-WIS
Enthusiast
Enthusiast
Jump to solution

In my lab yes.  I've been hosting my VSMs on DVS for 2 years without any issues.

BTW I work for Cisco TAC (the team that supports the 1000v).  I see the majority of our customers hosting the VSM on the DVS.  If you have a couple of available onboard 1G NICs you could run a standard vSwitch for your VSMs, but then you'll need those NICs and vSwitch available on every host in the cluster to support VMotioning the VSMs.

Robert

Reply
0 Kudos
RBurns-WIS
Enthusiast
Enthusiast
Jump to solution

Here's a best practice/how-to guide specific for 1000v & HP VC you might find useful.

Cheers,

Robert

Reply
0 Kudos
DSeaman
Enthusiast
Enthusiast
Jump to solution

Oh thank you very much Robert! I had a version from 2009, but this latest version is much cleaner. Thanks!!

Derek Seaman
Reply
0 Kudos
DSeaman
Enthusiast
Enthusiast
Jump to solution

Robert,

It looks like in the examples that the VSM management interface and the ESX console share the same VLAN? Is there any good reason to have the VSM interface on its own routed VLAN, or is sharing the recommended solution to help minimize the number of VLANs?

Derek Seaman
Reply
0 Kudos
RBurns-WIS
Enthusiast
Enthusiast
Jump to solution

If you're already using a "Management" VLAN for Service Console, vCenter access etc, I would use that.  I wouldn't bother creating a new VLAN just for the VSM's Management interface.   Treat the VLAN used for VSM management the same you would for any out-of-band access to any other switch/router in your infrastructure.

VLANs come at a cost, both for managing & tracking.  There's little traffic over the management interface of a VSM that would warrant its "own" VLAN.

In regards to the control interface, if the VSM is located in a different Layer 2 subnet than where the ESX hosts reside you can use the Layer3 mode and transport the Control traffic over IP.  For most deployments the VSM is deployed in the same L2 subnet as the VEM hosts which makes L2 mode an easy deployment option.

Regards,

Robert

Reply
0 Kudos
DSeaman
Enthusiast
Enthusiast
Jump to solution

Robert,

I noticed in the Cisco/HP VC slide deck that vMotion is on the same physical NICs as all of the management traffic. Since 1000v doesn't support NetIOC, and the HP VC doesn't support QoS, isn't there some concern of possible bandwidth starvation if vMotion is going full tilt?

Derek Seaman
Reply
0 Kudos