Solved: Re: error adding host to Cisco N1000v

bedobash · ‎08-26-2009

Cannot complete a Distributed Virtual Switch operation for one or more host members.

DVS operations failed on host x, error during the configuration of the host: create dvswitch failed with the following error message: sysinfoexception:node (vsi_node_net_create); status(bad0003)=not found; message = instance(0); Input(3)DvsPortset-0 256 cisco_nexus_1000v got (vim.fault.PlatformConfigFault) exception

lwatta · ‎08-28-2009

I've got some info on the VUM issues. This might not pertain to your problem but apparently there is a problem when VUM is configured with a Proxy. VUM needs to talk to the outside world and to the VSM. When a proxy is configured VUM tries to reach the VSM through the proxy. This usually doesn't work. In the current version there is no way tell VUM to not use the proxy for local hosts. VUM needs to be able to talk to the VSM to install the VEM module. I know VMware is working on a fix for this. The work around is to manually install the VEM modules.

There are several patches that VMware has released that require a new VEM module. These new modules are available through the VUM repository or through www.cisco.com. In addition to the new patches you also have to make a change to the configuration of VUM. This change only needs to be made if you have patched your ESX boxes. Follow this KB article to make the change to VUM http://kb.vmware.com/selfservice/microsites/search.do?cmd=displayKC&docType=kc&externalId=1013134&sl...

If you are installing the VEM manually the following will help determine which VEM version should be loaded.

If you are running ESX or ESXi at build 164009 (essentially no patches) use VEM module cross_cisco-vem-v100-4.0.4.1.1.27-0.4.2-release.vib
If have loaded patch ESX400-200906001 or ESXi400-200906001 use VEM module cross_cisco-vem-v100-4.0.4.1.1.28-0.5.2-release.vib
If you have loaded patch ESX400-200907001 or ESXi400-200907001use VEM module cross_cisco-vem-v100-4.0.4.1.1.28-0.6.2-release.vib

Your network problem sounds like you are trying to migrate the nic that has your vswif and vmk interfaces. This is allowed but you have to make sure you have the following set.

Make sure your uplink port-profile has "system vlan w,xy,z" Where w = vswif vlan, x = vmk vlan, y = control vlan, z = packet vlan. If vswif and vmk use the vlan obviously just have one entry
Make sure your port-profile for the vswif and vmk also have a "system vlan x" command for the corresponding vlans

If you are just starting out its easier to add a nic that does not have the vswif or vmk interfaces configured on it. Use that nic for VM data traffic and then when you more comfortable with the N1KV migrate your vswif and vmk interfaces.

Let us know if you still have problems with the installation.

louis

View solution in original post

lwatta · ‎08-26-2009

The reason you are getting that error is that the VEM module is either not installed on the ESX host you are trying to add or VUM is not installed or working.

If the VEM module is not installed manually we try to invoke VUM to install the VEM module.

If you don't want to use VUM you can load the VEM module by hand with esxupdate for ESX or vihostupdate for ESXi.

You can pull the VEM vib file directly from the VSM via the following link

http://Nexus1000V-IP-Address/cisco/ then pull down the vib file in that directory.

louis

bedobash · ‎08-26-2009

I am using VUM, and it does have the VEM in the inventory.

How do I get it to install? All the documentation I see says VUM automatically installs it.

Also, I dont have VCLI installed, only PowerCLI. Can I use that to manually push? I'd still rather use VUM to push the VEM to the ESX and ESXi hosts.

-Ed

lwatta · ‎08-26-2009

Ed,

Is it a clean VUM install or did you download and apply patches to the ESX boxes? We've seen where some of the patches break the VEM install. We are trying to nail down the process for how to get around this issue. We should have something in day or so.

I'm not sure if PowerCLI can do the VEM install. If it can run vihostupdate for an ESXi box than it should work.

louis

bedobash · ‎08-26-2009

Ohh, I haven't manually installed anything outside of VUM. And that was clean. Build of the boxes is ESXi 4.0.0 181792.

ESX boxes are at 4.0.0 175625

lwatta · ‎08-26-2009

Ed,

Those builds are patched and there is an issue where VUM does not recognize that the VEM module is compatibile with those builds. If you use the FCS build of 164009 it works no problem. There is a lot of confusion around what is the right procedure to get VUM working again. I don't have a good answer right now other than install manually.

I'm going to work on it today and hopefully have a procedure for fixing VUM to work with the patched ESX and ESXi images.

bedobash · ‎08-26-2009

when running vihostupdate.pl -i -b ./cisco-vem-v100-4.0.4.1.1.27-0.4.2.zip --server testESXiserver

I get the following error:

There was an error resolving dependencies.

No VIB provides 'vmknexus1kvapi-0-4' (required by cross_cisco-vem-v100-esx_4.0.4.1.1.27-0.4.2)

Requested VIB cross_cisco-vem-v100-esx_4.0.4.1.1.2-0.4.2 conflicts with the host.

bedobash · ‎08-26-2009

I'm pushing the ESXi400-VEM-200907001.zip or ESX400-VEM-200907001.zip, which installs successfully through the vCLI. Going to try the whole process again after.

lwatta · ‎08-27-2009

Ed,

Did you have a proxy setup for VUMwhen you were initially using it?

louis

bedobash · ‎08-27-2009

No,

VUM is a default install. the entire setup is in a lab-type environment on non-routable IPs, and only VC/VUM has seperate network connection for updates.

When I add the hosts that have VEM installed manually via a VUM patch, the hosts disconnect from vCenter after a long while (ESX and ESXi) and then the process appears to time out.

Question: if I have any existing network configuration setup when I try to add the dV, will that interfere?

I'm going to go ahead and blow away all configs on an ESXi host, and try again. Just trying to come up with a repeatable procedure that reliably works, so I can document it, and move on.

lwatta · ‎08-28-2009

I've got some info on the VUM issues. This might not pertain to your problem but apparently there is a problem when VUM is configured with a Proxy. VUM needs to talk to the outside world and to the VSM. When a proxy is configured VUM tries to reach the VSM through the proxy. This usually doesn't work. In the current version there is no way tell VUM to not use the proxy for local hosts. VUM needs to be able to talk to the VSM to install the VEM module. I know VMware is working on a fix for this. The work around is to manually install the VEM modules.

There are several patches that VMware has released that require a new VEM module. These new modules are available through the VUM repository or through www.cisco.com. In addition to the new patches you also have to make a change to the configuration of VUM. This change only needs to be made if you have patched your ESX boxes. Follow this KB article to make the change to VUM http://kb.vmware.com/selfservice/microsites/search.do?cmd=displayKC&docType=kc&externalId=1013134&sl...

If you are installing the VEM manually the following will help determine which VEM version should be loaded.

If you are running ESX or ESXi at build 164009 (essentially no patches) use VEM module cross_cisco-vem-v100-4.0.4.1.1.27-0.4.2-release.vib
If have loaded patch ESX400-200906001 or ESXi400-200906001 use VEM module cross_cisco-vem-v100-4.0.4.1.1.28-0.5.2-release.vib
If you have loaded patch ESX400-200907001 or ESXi400-200907001use VEM module cross_cisco-vem-v100-4.0.4.1.1.28-0.6.2-release.vib

Your network problem sounds like you are trying to migrate the nic that has your vswif and vmk interfaces. This is allowed but you have to make sure you have the following set.

Make sure your uplink port-profile has "system vlan w,xy,z" Where w = vswif vlan, x = vmk vlan, y = control vlan, z = packet vlan. If vswif and vmk use the vlan obviously just have one entry
Make sure your port-profile for the vswif and vmk also have a "system vlan x" command for the corresponding vlans

If you are just starting out its easier to add a nic that does not have the vswif or vmk interfaces configured on it. Use that nic for VM data traffic and then when you more comfortable with the N1KV migrate your vswif and vmk interfaces.

Let us know if you still have problems with the installation.

louis

bedobash · ‎08-31-2009

No, VUM is not configured as/with a Proxy. It has a direct internet connection.

I went ahead and made the port-profile changes you recommended.

added system vlan 2 and 3 to system-uplink (2 is vmk/service console for management, 3 is vmk for vmotion). I'm using 12 and 13 for packet and control. (vswif/vmk refers to management vlan only? or does it include vmotion?).

I'm still getting strange and random results with the DV config dropping off the ESXi host after the host is rebooted.

Also, after adding the extra patch download source for the VEM, I still only see VEM400-200906002-BG, and VEM400-200904001-BG in the patch repository. I do see the ESX400-200907401-BG and ESXi400-200907401-BG however.

The VEM patches I downloaded manually are labeled differently: ESX400-VEM-200907001, ESXi400-VEM-200907001, and ESXi400-VEM-200906001. I'm just wondering why the latest VEM patches aren't downloading properly.

I'm going to do another scratch ESXi install, and follow your instructions regarding matching VEM VIB versions with ESXi version, and see if I can get it working this time.

-Ed

bedobash · ‎08-31-2009

I'm able to get it working with having 1 NIC attached to the standard switch, with the management port group, and the other attached to the DV, and having all other services run through it (public/vmotion/iscsi/nfs etc)

Just curious, but is there a way to easily move the management vmk between standard and distributed switches?

-Ed

lwatta · ‎08-31-2009

Ed,

Assuming you have port-profiles created for the VMK interfaces, the easiest way to migrate is from the configuration tab for the ESX host under vCenter Server. Choose networking then choose the Distributed Virutal Switch button. Select the "manage virtual adapters " link at the top of the page and select "add" in the new pop-up. The next screen will allow you to migrate existing interfaces from the vSwitch to the DVS.

louis

jpraveleau · ‎09-18-2009

"Those builds are patched and there is an issue where VUM does not recognize that the VEM module is compatibile with those builds. If you use the FCS build of 164009 it works no problem. There is a lot of confusion around what is the right procedure to get VUM working again. I don't have a good answer right now other than install manually.

I'm going to work on it today and hopefully have a procedure for fixing VUM to work with the patched ESX and ESXi images."

****************

Hello,

I have exactly the same problem than Ed. I tried all your tests (install via vCLI, vMA, VUM) but the VEM doesn't want to be install. The esx build that i use now is 186700. So the two VEM patch doesn't to run unless the core VEM is not installed. If you have some news concerning a new release of the Nexus 1000v, inform me. Thanks.

jp

sdog · ‎09-30-2009

Im stumped. Im trying to push this to the host using the RCLI on the VMA, but I keep getting the same error. Anyone have any insight on how I can fix this?

This is from the RCLI on the VMA login as: vi-admin vi-admin@10.12.10.52's password: Last login: Tue Sep 29 17:33:32 2009

Welcome to vMA run 'vma-help' or see for more details.

~~vi-admin@il-chi-vma bundles~~$ vihostupdate -i -b ./cisco-vem-v100-4.0.4.1.1.27-0.4.2.zip --server il-chi-colovmh09.qwestcolo.local Enter username: root Enter password: There was an error resolving dependencies. No VIB provides 'vmknexus1kvapi-0-4' (required by cross_cisco-vem-v100-esx_4.0.4.1.1.27-0.4.2) Requested VIB cross_cisco-vem-v100-esx_4.0.4.1.1.27-0.4.2 conflicts with the host ~~vi-admin@il-chi-vma bundles~~$

This is from the ESX host esxupdate.log

~~root@il-chi-colovmh09 vmware~~# grep cisco esxupdate.log DEBUG: vibcache: No DB entry matching vib ID=cross_cisco-vem-v100-esx_4.0.4.1.1.28-0.5.2 remoteLocation=http://il-chi-vmvctr01.na1.ad.group:9084/vci/hostupdates/hostupdate/csco/metadata/VEM/4.0.0/VEM-4.0.0-patch01/cross_cisco-vem-v100-4.0.4.1.1.28-0.5.2-release.vib found DEBUG: vibcache: No DB entry matching vib ID=cross_cisco-vem-v100-esx_4.0.4.1.1.28-0.5.2 remoteLocation=http://il-chi-vmvctr01.na1.ad.group:9084/vci/hostupdates/hostupdate/csco/metadata/VEM/4.0.0/VEM-4.0.0-patch01/cross_cisco-vem-v100-4.0.4.1.1.28-0.5.2-release.vib found DEBUG: vibcache: No DB entry matching vib ID=cross_cisco-vem-v100-esx_4.0.4.1.1.28-0.5.2 remoteLocation=http://il-chi-vmvctr01.na1.ad.group:9084/vci/hostupdates/hostupdate/csco/metadata/VEM/4.0.0/VEM-4.0.0-patch01/cross_cisco-vem-v100-4.0.4.1.1.28-0.5.2-release.vib found DEBUG: vibcache: No DB entry matching vib ID=cross_cisco-vem-v100-esx_4.0.4.1.1.28-0.5.2 remoteLocation=http://il-chi-vmvctr01.na1.ad.group:9084/vci/hostupdates/hostupdate/csco/metadata/VEM/4.0.0/VEM-4.0.0-patch01/cross_cisco-vem-v100-4.0.4.1.1.28-0.5.2-release.vib found DEBUG: cmdline: Combined metadata: rpm_krb5-libs_1.6.1-31.el5_3.3@x86_64:uninstalled, rpm_vmware-esx-drivers-vmklinux-vmklinux_4.0.0-0.6.181792@x86_64:uninstalled, <"ESX400-200906407-SG">, rpm_curl_7.15.5-2.1.el5_3.4@x86_64:uninstalled, <"ESX400-200906408-BG">, rpm_vmware-esx-vmx_4.0.0-0.5.175625@i386:uninstalled, rpm_vmware-esx-esxupdate_4.0.0-0.5.175625@i386:uninstalled, <"ESX400-200906412-BG">, rpm_krb5-workstation_1.6.1-31.el5_3.3@x86_64:uninstalled, <"ESX400-200906404-BG">, rpm_vmware-esx-tools_4.0.0-0.5.175625@i386:uninstalled, rpm_curl_7.15.5-2.1.el5_3.4@i386:uninstalled, <"ESX400-200907401-BG">, <"ESX400-200906406-SG">, rpm_sudo_1.6.9p17-3.el5_3.1@x86_64:uninstalled, cross_cisco-vem-v100-esx_4.0.4.1.1.28-0.5.2:uninstalled, <"ESX400-200906402-BG">, <"ESX400-200906403-BG">, rpm_udev_095-14.20.el5_3@x86_64:uninstalled, rpm_vmware-hostd-esx_4.0.0-0.5.175625@i386:uninstalled, rpm_pam_krb5_2.2.14-10@i386:uninstalled, rpm_pam_krb5_2.2.14-10@x86_64:uninstalled, rpm_vmware-esx-lsi_4.0.0-0.5.175625@i386:uninstalled, <"ESX400-200906409-BG">, rpm_vmware-esx-scripts_4.0.0-0.5.175625@i386:uninstalled, <"ESX400-200906401-BG">, <"ESX400-200906411-SG">, <"ESX400-200906405-SG">, rpm_krb5-libs_1.6.1-31.el5_3.3@i386:uninstalled, rpm_vmware-esx-drivers-scsi-qla2xxx_400.821.k1.39vmw-1.0.5.175625@x86_64:uninstalled, rpm_vmware-esx-cim_4.0.0-0.5.175625@i386:uninstalled, <"ESX400-200906410-BG">, rpm_vmware-esx-vmkernel64_4.0.0-0.6.181792@i386:uninstalled, rpm_vmware-esx-iscsi_4.0.0-0.5.175625@i386:uninstalled, <"VEM400-200906002-BG">, <"ESX400-200906413-BG"> DEBUG: vibcache: No DB entry matching vib ID=cross_cisco-vem-v100-esx_4.0.4.1.1.28-0.5.2 remoteLocation=http://il-chi-vmvctr01.na1.ad.group:9084/vci/hostupdates/hostupdate/csco/metadata/VEM/4.0.0/VEM-4.0.0-patch01/cross_cisco-vem-v100-4.0.4.1.1.28-0.5.2-release.vib found Options: {'nodeps': None, 'all': None, 'retry': 5, 'vibview': None, 'nocache': None, 'loglevel': 'DEBUG', 'cleancache': None, 'bundles': None, 'nosigcheck': None, 'bundlezips': , 'olderversion': None, 'proxyurl': None, 'meta': None, 'timeout': 30.0, 'cachesize': None, 'HA': True, 'maintenancemode': None} DEBUG: downloader: Using /var/cache/esxupdate/bundle3501591510409625895 for zip /tmp/updatecache/cisco-vem-v100-4.0.4.1.1.27-0.4.2.zip DEBUG: downloader: Skipping download of /tmp/updatecache/cisco-vem-v100-4.0.4.1.1.27-0.4.2.zip DEBUG: cmdline: Combined metadata: ~~cross_cisco-vem-v100-esx_4.0.4.1.1.27-0.4.2:uninstalled, <"VEM400-200904001-BG">~~ DEBUG: cmdline: Vibs from bulletins: set(~~cross_cisco-vem-v100-esx_4.0.4.1.1.27-0.4.2:uninstalled~~) DependencyError: No VIB provides 'vmknexus1kvapi-0-4' (required by cross_cisco-vem-v100-esx_4.0.4.1.1.27-0.4.2) Requested VIB cross_cisco-vem-v100-esx_4.0.4.1.1.27-0.4.2 conflicts with the host Options: {'nodeps': None, 'all': None, 'retry': 5, 'vibview': None, 'nocache': None, 'loglevel': 'DEBUG', 'cleancache': None, 'bundles': None, 'nosigcheck': None, 'bundlezips': , 'olderversion': None, 'proxyurl': None, 'meta': None, 'timeout': 30.0, 'cachesize': None, 'HA': True, 'maintenancemode': None} DEBUG: downloader: Using /var/cache/esxupdate/bundle3501591510409625895 for zip /tmp/updatecache/cisco-vem-v100-4.0.4.1.1.27-0.4.2.zip DEBUG: downloader: Skipping download of /tmp/updatecache/cisco-vem-v100-4.0.4.1.1.27-0.4.2.zip DEBUG: cmdline: Combined metadata: ~~cross_cisco-vem-v100-esx_4.0.4.1.1.27-0.4.2:uninstalled, <"VEM400-200904001-BG">~~ DEBUG: cmdline: Vibs from bulletins: set(~~cross_cisco-vem-v100-esx_4.0.4.1.1.27-0.4.2:uninstalled~~) DependencyError: No VIB provides 'vmknexus1kvapi-0-4' (required by cross_cisco-vem-v100-esx_4.0.4.1.1.27-0.4.2) Requested VIB cross_cisco-vem-v100-esx_4.0.4.1.1.27-0.4.2 conflicts with the host DEBUG: vibcache: No DB entry matching vib ID=cross_cisco-vem-v100-esx_4.0.4.1.1.28-0.5.2 remoteLocation=http://10.12.10.153:9084/vci/hostupdates/hostupdate/csco/metadata/VEM/4.0.0/VEM-4.0.0-patch01/cross_cisco-vem-v100-4.0.4.1.1.28-0.5.2-release.vib found DEBUG: vibcache: No DB entry matching vib ID=cross_cisco-vem-v100-esx_4.0.4.1.1.28-0.7.3 remoteLocation=http://10.12.10.153:9084/vci/hostupdates/hostupdate/csco/metadata/VEM/4.0.0/VEM-4.0.0-patch02/cross_cisco-vem-v100-4.0.4.1.1.28-0.7.3-release.vib found DEBUG: vibcache: No DB entry matching vib ID=cross_cisco-vem-v100-esx_4.0.4.1.1.28-0.6.2 remoteLocation=http://10.12.10.153:9084/vci/hostupdates/hostupdate/csco/metadata/VEM/4.0.0/VEM-4.0.0-EP1/cross_cisco-vem-v100-4.0.4.1.1.28-0.6.2-release.vib found DEBUG: vibcache: No DB entry matching vib ID=cross_cisco-vem-v100-esx_4.0.4.1.1.27-0.4.2 remoteLocation=http://10.12.10.153:9084/vci/hostupdates/hostupdate/csco/metadata/VEM/4.0.0/VEM-4.0.0-GA/cross_cisco-vem-v100-4.0.4.1.1.27-0.4.2-release.vib found Options: {'nodeps': None, 'all': None, 'retry': 5, 'vibview': None, 'nocache': None, 'loglevel': 'DEBUG', 'cleancache': None, 'bundles': None, 'nosigcheck': None, 'bundlezips': , 'olderversion': None, 'proxyurl': None, 'meta': None, 'timeout': 30.0, 'cachesize': None, 'HA': True, 'maintenancemode': None} DEBUG: downloader: Using /var/cache/esxupdate/bundle3501591510409625895 for zip /tmp/updatecache/cisco-vem-v100-4.0.4.1.1.27-0.4.2.zip DEBUG: downloader: Skipping download of /tmp/updatecache/cisco-vem-v100-4.0.4.1.1.27-0.4.2.zip DEBUG: cmdline: Combined metadata: ~~cross_cisco-vem-v100-esx_4.0.4.1.1.27-0.4.2:uninstalled, <"VEM400-200904001-BG">~~ DEBUG: cmdline: Vibs from bulletins: set(~~cross_cisco-vem-v100-esx_4.0.4.1.1.27-0.4.2:uninstalled~~) DependencyError: No VIB provides 'vmknexus1kvapi-0-4' (required by cross_cisco-vem-v100-esx_4.0.4.1.1.27-0.4.2) Requested VIB cross_cisco-vem-v100-esx_4.0.4.1.1.27-0.4.2 conflicts with the host Options: {'nodeps': None, 'all': None, 'retry': 5, 'vibview': None, 'nocache': None, 'loglevel': 'DEBUG', 'cleancache': None, 'bundles': None, 'nosigcheck': None, 'bundlezips': , 'olderversion': None, 'proxyurl': None, 'meta': None, 'timeout': 30.0, 'cachesize': None, 'HA': True, 'maintenancemode': None} DEBUG: downloader: Using /var/cache/esxupdate/bundle3501591510409625895 for zip /tmp/updatecache/cisco-vem-v100-4.0.4.1.1.27-0.4.2.zip DEBUG: downloader: Skipping download of /tmp/updatecache/cisco-vem-v100-4.0.4.1.1.27-0.4.2.zip DEBUG: cmdline: Combined metadata: ~~cross_cisco-vem-v100-esx_4.0.4.1.1.27-0.4.2:uninstalled, <"VEM400-200904001-BG">~~ DEBUG: cmdline: Vibs from bulletins: set(~~cross_cisco-vem-v100-esx_4.0.4.1.1.27-0.4.2:uninstalled~~) DependencyError: No VIB provides 'vmknexus1kvapi-0-4' (required by cross_cisco-vem-v100-esx_4.0.4.1.1.27-0.4.2) Requested VIB cross_cisco-vem-v100-esx_4.0.4.1.1.27-0.4.2 conflicts with the host

Any help is greatly appreciated.

lwatta · ‎09-30-2009

JP,

Sorry I have been sick and missed this. I beleive build 186700 is the 4u1 candidate. This build will not work with Cisco N1000V. You will need to use the 4.0 code for the VEM module to work.

When 4u1 is officially released we will have a VEM build that will work. Right now we are in beta for the latest code for VEM and VSM.

louis

lwatta · ‎09-30-2009

It looks like you are using a patched version of ESX 4.0. It looks like you are running build number 175625.

You'll need to use a different VEM module if you are running that build. You need VEM module cross_cisco-vem-v100-4.0.4.1.1.28-0.5.2-release.vib

You can download the patch from VMware via

For build 175625 you want patch VEM400-200906002-BG

louis

vpert · ‎10-06-2009

Hi Iwatta,

thx for the information.

Where can we get the cross_cisco-vem-v100-4.0.4.1.1.28-0.5.2-release.vib?

rgds

..we found the vib in the VMware patch ESX400-VEM-200909001.zip

sdog · ‎10-07-2009

The link appears broken.

Thanks

SD