VMware Cloud Community
adi_dragomir
Contributor
Contributor

Cisco Nexus 1000V implementation

I am trying to implement a Cisco Nexus 1000V switch. I read all the documentation from the Cisco website and I even watch the videos. I worked with my VMware environment for a long time using just normal vSwitches without problems but I have to admit the implementation of the Cisco switch let me in the dark. There are a lot of gaps and no explanations why I should do whatever step. I expect to have a knowledge base document from the VMware where to be clearly specify the requirements for the VMware environments when you implement a Cisco Nexus 1000V switch. I have a working VMware DataCenter environment with normal Layer 2 switches and I would like to implement a Cisco Nexus 1000V switch and I do not know for example what kind of physical switches I need, how many extra physical NICs I need for every ESX host with VEM, how I should configure the networking physical and virtual environment for the ESX host with VSM (Cisco documentation about that is very poor and confusing). These informations should come from VMware because I will first setup the VMware environment. For configuring the Cisco Nexus 1000V switch I'll go after to the Cisco website.

Reply
0 Kudos
29 Replies
MauroBonder
VMware Employee
VMware Employee

see this link maybe helpful

http://www.cisco.com/en/US/docs/switches/datacenter/nexus1000/sw/4_0/layer2/configuration/guide/l2_1...

*If you found this information useful, please consider awarding points for "Correct" or "Helpful"*

*Please, don't forget the awarding points for "helpful" and/or "correct" answers. *Por favor, não esqueça de atribuir os pontos se a resposta foi útil ou resolveu o problema.* Thank you/Obrigado
Reply
0 Kudos
gary1012
Expert
Expert

IMHO, the reason for the lack of specifc design tips is due to the number of options available to you once you introduce a Cisco 1000v into environment. You'll need at least two NICs for the the System Uplink connection going up to your distribution/core data center switches. The system uplink will carry the Nexus traffic (control, management, packet) and the VMware service console. Each of these must be marked as System VLANs to allow opaque traffic flow in the event that there is a problem with the VSMs or the host VEM.

Once you've established the base setup for the 1000v, then additional NICs can be added to either the System Uplink or other new uplinks. I prefer to create new uplinks for virtual machine and vmkernel traffic so that I can further isolate each traffic type. The VM Uplink is typically a trunked connection with only the VLANs necessary for the VMs to communicate up through the core. Each VM Uplink should have at least two NICs to provide redundancy in the event of a path or NIC failure.

You can create another uplink for vMotion to provide dedicated bandwidth and to isolate the traffic due to security reasons. This uplink should have two NICs but I suppose you could get by with one if pushed.

As for physical switches; it will depend on your bandwitch requirements, NICs that will connect, and the number of ports required.

Community Supported, Community Rewarded - Please consider marking questions answered and awarding points to the correct post. It helps us all.

Community Supported, Community Rewarded - Please consider marking questions answered and awarding points to the correct post. It helps us all.
Reply
0 Kudos
adi_dragomir
Contributor
Contributor

I've read that document but it's not helpful, it does not show how should look like an VSM implementation. For example they say all the time you need 3 Port Groups for an VSM. Do I need a separate vSwitch for that? And how many physical NICs I should allocate to that vSwitch? The video show you can install the VSM mapping the existing VM Network port group to all Control, Packet and Management VLANs ....and no VLAN ID for those port groups !! One it is written in the documentation ,other thing is shown in the video...and you know what? I could install the VSM as shown in the video !!!!!....From now on anyway I am in the dark. Whay I should do? Edit the settings for the VSM virtual machine and allocate the network cards to those 3 port groups?Create uplink? And which physical NICS I should use from the ESX hosts which will be the VEMs? How I should configure my layer 2 switch where all the physicla NICs from the all ESX hosts are connected?

Thank you very much for your help.

Reply
0 Kudos
gary1012
Expert
Expert

Keep in mind that the 1000v is an enhanced DVS so everything can be running over a single 1000v's backplane. I would not create separate vSwitches (VSS or DVS) unless you want to keep your service console/vMotion traffic off the 1000v. So far it sounds like you have your VSM up and running and have it registered in vCenter. Now you must create your uplink profiles for system, VMs, and possibly vmkernel. You do this by creating the port-profile then invoking 'capability uplink' from the VSM CLI. Once the uplinks have been created, then you add the host to the 1000v via the VIC and then add the available NICs to the previously defined uplinks. Upon completion, then you can migrate the service console to the system uplink (if you choose to use the 1000v for this function), remove the NIC from the VSS, and then add the NIC to the system uplink to provide redundancy.

Recapping (presuming you have registered the vCenter plug-in and invoked svs connection vc:

  1. Add your system, VM and possibly vmkernel uplinks to the VSM via the VSM CLI. Do not change the NIC port-group assignments.

  2. Open the VIC, go to Networks, and add your VMware host to the 1000v by right-clicking on the VSM and choosing "Add Host"

  3. From the VIC, select the newly added host, and choose the Configuration tab

  4. Select Networking the the Distributed Virtual Switch button

  5. Select the Manage Physical Adapters and then add your NICs to the appropriate Uplinks

  6. If you elect to have the 1000v run the service console, then you will need to invoke the Manage Virtual Adapters and step through the instructions. As a tip, you may want to configure a secondary service console connection just in case the migration doesn't work.

Configure the remaining non-uplink port-profiles from the VSM's CLI and then these will be available to the VMs for network label assignment. Lastly, think of the 1000v having two sides; a forward-facing side with ports for each VM, and then the back-side with uplink ports to the physical distribution/core switch. The back-side is where each NIC is inserted into the most appropriate system, VM, or vmkernel uplink.

Community Supported, Community Rewarded - Please consider marking questions answered and awarding points to the correct post. It helps us all.

Community Supported, Community Rewarded - Please consider marking questions answered and awarding points to the correct post. It helps us all.
Reply
0 Kudos
adi_dragomir
Contributor
Contributor

Hi Gary,

Yes, my VSM virtual machine is up and running and it's connected to the VC. The problem is all its 3 vNICs are connected to the VM Network port group in a normal vSwitch (an existing one). It seems the first step is connect them to the Control port group, Packet port group and Management port group which I already created on a different vSwitch to which I attached a separate physical NIC. Is that correct? Should I put a different VLAN ID to those groups which should match the vlans used during port-profiles defined later for system and data uplinks? Is there any requirement for the physical switch ports where all the physical NICs of the ESX hosts are connected?

Thank you for your answer.

Reply
0 Kudos
gary1012
Expert
Expert

The 1.3 version of the 1000v allows you to run control, management and packet over the same VM NIC on the VSM. Also, they can be on same VLAN. Define a single access or data port with a single VLAN for these traffic types. Remove two NICs from the vSwitch. Configure the remaining NIC as a standby NIC on the VSS. Take the remaining two NICs and assign them to the VM Uplink on the 1000v. Configure VM port-profiles to traverse the VM uplink using either access or data ports. Once those are defined, then your VMs will see them in the Network Label drop down box.

Community Supported, Community Rewarded - Please consider marking questions answered and awarding points to the correct post. It helps us all.

Community Supported, Community Rewarded - Please consider marking questions answered and awarding points to the correct post. It helps us all.
Reply
0 Kudos
adi_dragomir
Contributor
Contributor

I am sorry Gary but it failed and I think it happen because of the VSM virtual machine configuration. There is no clear explanation how you should define the VSM settings. One thing is on the documentation, different way is in the video and it's becoming extremely confusing. Let's start with the beginning. I have an ESX host with one vSwitch and 4 NICs attached to it. How should I define the Control, Management and Packet port groups and the correspondent vSwitch and physical NIC? Should I create another vSwitch and move 2 NICs to it? ShouldI put a different VLAN ID than zero to those 3 port groups? I tried to put VLAN ID 1 and I lost connection with the VSM machine...

Sorry to bother you but the documentation is very poor and have many gaps.

Thank you for your help.

Reply
0 Kudos
gary1012
Expert
Expert

Leave the Service Console and two NICs configured on vSwitch0. Create a new VLAN that control, management and packet will traverse. Create a new port group on vSwitch0 called N1KV. Change your VSM to use the new N1KV port group for control, management, and packet. This is all that is required for VSM 1.3.

Once you have communications going with the VSM; which it sounds like you have if you have registered it with vCenter, then you can create your VM-Uplink, add your port-profiles, then add two NICs to the VM-Uplink. Once this is complete, you can add VMs to the newly create port-profiles.

Community Supported, Community Rewarded - Please consider marking questions answered and awarding points to the correct post. It helps us all.

Community Supported, Community Rewarded - Please consider marking questions answered and awarding points to the correct post. It helps us all.
Reply
0 Kudos
adi_dragomir
Contributor
Contributor

What means "create a new VLAN"? I should create a new port group named N1KV and give the VLAN ID 1 on vSwitch0?

Reply
0 Kudos
Rumple
Virtuoso
Virtuoso

Are you actually running LAyer 3 switches with VLAN capabilities or are all your physical switches Layer 2 unmanaged switches?

Reply
0 Kudos
adi_dragomir
Contributor
Contributor

Layer2 managed switches...

Reply
0 Kudos
gary1012
Expert
Expert

It can be native or another available VLAN ID. Just ensure that it is purpose built for the 1000v management.

Community Supported, Community Rewarded - Please consider marking questions answered and awarding points to the correct post. It helps us all.

Community Supported, Community Rewarded - Please consider marking questions answered and awarding points to the correct post. It helps us all.
Reply
0 Kudos
lwatta
Hot Shot
Hot Shot

Gary's recommendation is spot on. For a first time deployment keep the all the VSM connections on vSwitch0. Use one of the free NICs to add to the N1KV for your VM traffic. Please do not use VLAN 1 for the control traffic. Have your network admin create a new vlan on your upstream physical switch(say vlan 2) and use that.

You then need to make sure that VLAN 2 is trunked down on all the NICs that will attach your ESX hosts to the N1KV. All ESX hosts that will be part of the N1KV implementation will need to see control traffic on VLAN1.

A quick explanation of the control traffic. The Nexus 1000V is a virtual switch. The VSM is the virtual supervisor module (control plane) the Virtual ethernet modules (ESX hosts) are the virtual line cards. In a real switch VSM and VEM communicate via a backplane. There is no real backplane here so we use the control network as a virtual backplane so all the parts of the virtual switch can talk to each other.

Louis

adi_dragomir
Contributor
Contributor

To clarify I will resume your answers:

-I have a vSwitch0 with 4 NICs attached on the ESX host. Present VM traffic, management network and vMotion use this vSwitch0 with the default VLAN ID (which is zero). I will free 2 NICs for Cisco Nexus 1000V switch.

  • I will create a port group (let's say N1KV) on vSwitch0 with VLAN ID 2 and I will modify the VSM settings network cards to use that port group.

  • I will modify the physical upstrean switch layer 2 to allow VLAN ID 0 and VLAN ID 2 for every ports which will connect ESX hosts ( I will declare them as trunk ports). In fact which are the minimum requirements for the physical upstream switch? Layer 2? Layer 3?

  • I need to create 3 port profiles at the command line of the VSM machine. Here I am a little bit unclear about the commands, I need help.

  • I will attach ESX hosts to the Cisco Nexus 1000V switch including the ESX hosting the VSM machine.

These steps are Ok?

Thank you for your help, guys.

Reply
0 Kudos
gary1012
Expert
Expert

Looks right. Port-profiles are established by executing the following:

n1000v$ config t

n1000v(config)# port-profile data262

n1000v(config-port-prof)# switchport mode access

n1000v(config-port-prof)# switchport access vlan 262

n1000v(config-port-prof)# vmware port-group data262

n1000v(config-port-prof)# no shut

n1000v(config-port-prof)# state enabled

n1000v(config-port-prof)# copy run start

100%

n1000v(config-port-prof)#

In this example your VMs will now see data262 as an option under Network Label.

Community Supported, Community Rewarded - Please consider marking questions answered and awarding points to the correct post. It helps us all.

Community Supported, Community Rewarded - Please consider marking questions answered and awarding points to the correct post. It helps us all.
Reply
0 Kudos
adi_dragomir
Contributor
Contributor

It appears my physical switch ports are not able to carry VLAN ID 2 or any other VLAN than the native one. It's possible to put all the uplinks in the VLAN ID 0? Which will be the commands to create system-uplink and the vm-uplink in that case?

Thank you for your help.

Reply
0 Kudos
gary1012
Expert
Expert

Yes, it's possible to run everything over the same VLAN but you breaking a number of security best practices by doing so. Know that you are running the Nexus opaque management traffic, the service console, and the VMs over the same VLAN. If you cannot create new VLANs upstream, then create your system and VM uplinks and assigned each uplink the appropriate number of NICs. Creating uplinks is invoked by using the capability uplink command under port-profile.

Community Supported, Community Rewarded - Please consider marking questions answered and awarding points to the correct post. It helps us all.

Community Supported, Community Rewarded - Please consider marking questions answered and awarding points to the correct post. It helps us all.
Reply
0 Kudos
adi_dragomir
Contributor
Contributor

I did the following:

I created a port group N1KV on vSwitch0 with VLAN ID 0 and I modify the VSM settings network cards to use that port group.

I created 1 port profile with the following commands:

port-profile system-uplink

switchport mode trunk

no shutdown

capability uplink

vmware port-group

state enabled

I tried to attach an ESX host to the Cisco Nexus 1000V switch using the system-uplink and an unassigned NIC.

The result was a failure with the same exception: vmodl.fault.SystemError

Reply
0 Kudos
adi_dragomir
Contributor
Contributor

I think the problem is the vSwitch0 native VLAN value during VSM installation which should be different with zero. But I have just one VLAN on my vSwitch0 which is zero. So the upstream physical switch should support trunk ports.

Reply
0 Kudos