4 Replies Latest reply: Aug 22, 2010 4:50 PM by DOakman RSS

    Cisco Nexus 1000v VMotion

    eljow Novice

      Hi,

       

      I wonder if you could help me out though, I am struggling with Cisco Nexus 1000v.

       

      I am trying to figure out how to do vmotion on the Cisco Nexus 1000v. At the moment I have both cold/hot migrations taking place but I am not sure if its best practise the way I have configured it.

       

      I have created a vmotion port-profile that is used by the vmk interfaces, which is as follows,

       

      port-profile type vethernet VMotion

         vmware port-group

         switchport mode access

         switchport access vlan 50

         no shutdown

         system vlan 50

         state enabled

       

      Now if I add vlan 50 as a system vlan, the vmotion traffic will use the main system-uplink trunk, which the control, packet VLANS go over and it will work perfectly.

       

       

       

      Problem is I want to be able separate the vmotion traffic using a different vmnic. So I created another uplink, configuration is below,

       

       

       

      port-profile type ethernet VMOTION-LINK

      vmware port-group

      capability uplink

      switchport mode trunk

      switchport trunk allowed vlan 50

      no shutdown

      system vlan 50

      state enabled

       

      for that port-profile uplink I have added a different VMNIC and allowed the relevant configuration on the upstream switch. So now my vmk interface traffic for vmotion will now use the VMOTION-LINK, I have also remove the system vlan 50 from the system-uplink port-profile but it does not work and the migration bombs out at 10%. I have also tried creating the VMOTION-LINK as an access port aswell, which is as follows,

       

      port-profile type ethernet VMOTION-TEST

        vmware port-group

        capability uplink

        switchport mode access

        switchport access vlan 50

        no shutdown

        system vlan 50

        state enabled

       

      again changing the upstream switch to an access port from a trunk port but it does not work and bombs at 10%. I can only get it working on the system-uplink port-profile.

       

      My question is this, is what I am trying to achieve even possible with the Nexus 1000v and separating the Vmotion traffic onto a different VMNIC from the main system port-profile?

       

      Any help would be great.

        • 1. Re: Cisco Nexus 1000v VMotion
          mmehta76 Novice

           

          Hi,

           

           

          this is very much possible with Nexus 1000V to separate VMotion traffic on a VMNIC from the other NICs used for data VLANs. I am suspecting that you may have the VLAN 50 configured on both data UPlink and the VMotion UPlink. We do not allow system VLANs to be deleted from a live port-profile since they are carrying critical traffic.  In order to remove it from Data UPlink and easy way will be create another uplink port profile with all data VLANs except system VLAN 50, and then move the data PNICs to this new one in the VCenter.

           

           

          Let me know if it works. could you please provide information on the software version you are using, as well as configuration of all the uplink port-profiles.

           

           

           

           

           

          Munish.

           

           

          • 2. Re: Cisco Nexus 1000v VMotion
            eljow Novice

            Hi mate,

             

            I definatley don't have the VLAN tagged with any other profile. I learned the hard way trying to remove a system vlan for a live port-profile, it took me ages to figure out how to remove it. In any case i don't think that is the problem.

             

            As for port-profile configuration, you will find this information below,

             

            Main System-Uplink

            port-profile type ethernet system-uplink

              vmware port-group

              capability uplink

              switchport mode trunk

              switchport trunk allowed vlan 100 -102

              no shutdown

              system vlan 100 - 102

              state enabled

             

            Data Uplink Port Profile

            port-profile type ethernet data-uplink

              vmware port-group

              capability uplink

              switchport mode trunk

              switchport trunk allowed vlan 150 - 155

              no shutdown

              state enabled

             

            Above are the main uplinks, system uplink and data uplink that carries vm traffic. The VMotion i have tried three sets of configuration, firstly, if i configure the VMotion with the following below and assign an extra system vlan (VLAN 50) to the system-uplink and assign the below port-profle to the vmk interfaces, it will work perfectly but i don't want this i want it seperate from the system-uplink.

             

            Used by VMK Interfaces.

            port-profile type vethernet VMotion

              vmware port-group

              switchport mode access

              switchport access vlan 50

              no shutdown

              system vlan 50

              state enabled

             

            Secondly, if i remove the system vlan 50 from the system-uplink and configure a new uplink with a seperate VMNIC, as in the below,

             

            port-profile type ethernet VMOTION-LINK

              vmware port-group

              capability uplink

              switchport mode trunk

              switchport trunk allowed vlan 50

              no shutdown

              system vlan 50

              state enabled

             

            and assign the VMK interfaces the Vmotion VMK port-profile it won't work using the new VMOTION-LINK. Thirdly and just for a test i changed the VMOTION-LINK, so its a access port uplink instead of a trunk port as in the below,

             

            port-profile type ethernet VMOTION-TEST

              vmware port-group

              capability uplink

              switchport mode access

              switchport access vlan 50

              no shutdown

              system vlan 50

              state enabled

             

            it still does not work correctly.

             

            I have checked that the system vlan  50 is not part of any other uplink ports and is only configured on the VMOTION-TEST

            or VMOTION-LINK depending what i have configured at the time. The upstream switch matches all port-profiles, so i cannot not understand why it will only work using the system-uplink profile. Does the VMotion traffic have to pass down the same uplink as control and packet, i would doubt it but its the only way i can get it working?

             

            cheers

            • 3. Re: Cisco Nexus 1000v VMotion
              mmehta76 Novice

               

              Hi,

               

               

              thanks for sharing the profile configurations. Nothing obvious stands out. Can you please provide additional information as below:

               

               

              1) Which software version are you using?

               

               

              2) From ESX host:  "vemcmd show port", "vemcmd show trunk". "vemcmd show profile". Ideally "vem-support all" command captures all the needed info in a file.

               

               

              3) Configuration of corresponding ports on the upstream switch.

               

               

              4) Since you are vmotioning from one host to other, please provide the above on both hosts.

               

              Munish.

              • 4. Re: Cisco Nexus 1000v VMotion
                DOakman Lurker

                We ran into a similiar issue today with a similiar configuration. Vmotion was failing when we had vmotion on a dedicated Portgroup (isolated Layer 2 Vlan upstream) and Management on a seperate port group (routeable L3 subnet upstream).

                 

                We ESXi hosts could ping each other on both the Management and Vmotion subnets.

                Vcenter could access hosts through management IP, but not vmotion (due to L2 isolation).

                 

                We moved changed the vmkernel config to have vmotion and management on the management subnet and vmotion worked fine.

                 

                ESXi version 4.1

                Nexus 1000v version 4.0(4)SV1(3a)

                 

                We want to go back and move this back to dedicated vlan, but need to figure out why it is not working.

                 

                We are having 1 other problem after vmotion occurs in that Vms on a particular port group loose network access. We are only able to get the vm back up ad running after changing port groups and then changing back to original.

                 

                DaveO