All the VMs lost network connections with CISCO Nexus 1000 v installed

[jameszhu]

We frequently loss network connections for all the VMS, which are running on Nexus 1000 v, almost every one week.

We are running Nexus 1000 v on a UNISYS ES7000 Model 7600R partition with two cells,

We are keeping control, mgmt and packet traffic on a local standard vSwitch.

We are keeping service console and vMotion on the local standard vSwitch as well.

We are keeping VM network on CISCO Nexus 1000 v Switch.

We have to reboot all the hosts in the cluster to get network connection.

It doesn't fix the issue to simply restart VEM agent on the host.

I was told by CISCO that this issue is related to CDP potocol.

any ideas?

Please feel free to ask questions, I do appreciate your help.

JamesZhu

[lwatta]

James,

There was another issue recently with a Unisys system in which there was a misconfiguration on a NUMA BIOS setting that was causing the Nexus 1000V to drop. There can also be an issue with igmp snooping that will cause similar problems to what you describe. I would open a Cisco TAC case if you haven't already.

I'd be surprised if it was related to CDP. We only need CDP to setup Service Group memberships for vPC-HM during VEM install.

louis

[jameszhu]

Can you elaborate your recommendations, such as UNISYS BIOS, IGMP snooping, etc. ?

Our network team is working with CISCO now.

I am thinking to avoid rebooting ESX host, which is such a pain, any suggestions?

log on service console of hosts, which are added on Nexus 1000v switch as root:

1. vem-support

2. hotswap.sh -u

3. vem load

4. vem start

5. service mgmt-vmware restart

6. vem status

7. start checking network connection for VMs.

My co-worker has tried to restart dpa only, which seemed to be a failure to resume network connection.

JamesZhu

[RBurns-WIS]

As Louis pointed out, there is a known issue with IGMP, which is fixed in the next release due out in a couple weeks. This normally affects VEM hosts not reconnecting to the VSM followingreboots. May not impact your issue, but if you're not using IGMP you may want to disable it.

As for the CDP protocol, this only comes into play when you have more than one pnic in the same uplink Port Profile. If you do you should being using a auto port channel command such as "channel-group auto mode on sub-group cdp". In cases where your upstream switch doesn't support CDP, you still use the command above then manually set the subgroup IDs on each Ethernet interfaces for each VEM.

Ex.

int eth3/1

sub-group ID 0

int eth3/2

sub-group ID 1

**This method will be changing in the next release 4.0.4.SV1.2. In this release rather than only the sub-group "cdp" option, there will also be "manual" and "mac pinning".

If you're still having issues, can you post your VSM config?

Thanks,

Robert

[jameszhu]

Robert,

We are using auto mode for CDP, configuring Channel Group.

We are using IGMP as well. We will wait for the new release for fixing the problem related to IGMP.

As you mentioned in your reply, if we reload VEM modules and restart VEM agent, we will loss the connection between VSM and VEM. VEM is shown absent, not active, but still licensed in VSM. You are not able to execute a command in VEM, issued from VSM.

I have to remove then add the host back to Nexus 1000v switch through VMware VirtualCenter to reconnect VEM to VSM. I believe rebooting VSM should also be able to connect VEM to VSM.

checking, https://www.myciscocommunity.com/message/23190#23190

My question:

In this case, do we have a simple way to connect VEM to VSM? such as executing a command through the service console of ESX host, or executing a command at VSM, etc.

thanks

James

JamesZhu

[RBurns-WIS]

Sounds like you're hitting the IGMP bug.

At this point, the only workaround is to reboot your VSM (or disable IGMP). That should reset the connectivity and reconnect all your VEM hosts.

Robert

[jameszhu]

How can I disable IGMP in VSM?

1. config t

2. no ip igmp snooping

3. copy running-config startup-config

Do you know any side effects if we disable IGMP?

thanks, and have a good weekend,

JamesZhu

[lwatta]

There is one last step you need to run and that is to reboot the VSM. The "no igmp snooping" will not take until you reboot the VSM.

IGMP snooping is used for multicast traffic. So if you are not multicasting to your VMs you should be ok.

louis