VMware Cloud Community
robertl30
Contributor
Contributor

One vmnic won't trunk vlans

Here's the situation. I don't speak Cisco. And my Cisco Admin doesn't speak VMware. So we're both looking at each other like the other guy is nuts.

I've got a vSphere 4 cluster of two machines. HostA and HostB. They each have 6 nics (vmnic0, 1...5)

I've got 3 Distributed Virtual Switches: dvsManagement (runs Service Console and vMotion vmk), dvsStorage (NFS) and dvsVMNetwork.

dvsVMNetwork has 3 portgroups: dvpgVLAN92, etc.

dvsVMNetwork has 2 uplinks: vmnic3 & vmnic4 or each host.

Initially I was seeing random VMs lose network connectivity. I isolated the problem by removing one uplink and then the other.

When a host is using just vmnic3 everything is good for VMs on that host

When a host is using just vmnic4 no VMs can communicate on that host.

ifconfig shows:

vmnic3 Link encap:Ethernet HWaddr 00:24:81:F9:1A:CA

UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1

RX packets:1556 errors:0 dropped:0 overruns:0 frame:0

TX packets:507 errors:0 dropped:0 overruns:0 carrier:0

collisions:0 txqueuelen:1000

RX bytes:134384 (131.2 KiB) TX bytes:58437 (57.0 KiB)

Interrupt:58

vmnic4 Link encap:Ethernet HWaddr 00:24:81:F9:1A:CB

UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1

RX packets:239 errors:0 dropped:0 overruns:0 frame:0

TX packets:0 errors:0 dropped:0 overruns:0 carrier:0

collisions:0 txqueuelen:1000

RX bytes:19125 (18.6 KiB) TX bytes:0 (0.0 b)

Interrupt:66

Note the 0 TX bytes on vmnic4.

Now, these are HP blades so figuring out what's connected where is slightly maddening, but I'm farily confident we have that diagramed out correctly. It's like this:

Host A vmnic3: switch 1 port 3

Host A vmnic4: switch 2 port 3

Host B vmnic3: switch 1 port 4

Host B vmnic4: switch 2 port 4

So, the Cisco switch admin guy says he sees traffic properly trunking on switch 1 ports 3 & 4, but sees no "trunk" on switch 2 port 3 & 4.

I asked him to compare the configuration of all the ports and he says they're all identical. I remain skeptical, but what can I do? I don't speak Cisco. He showed me some screen shots of the port configs and they do look comparable to the naked eye. But I don't know what else could be lurking at higher levels (some setting on the switch)? These are Cisco 3020 blade switches.

How do I troubleshoot this further? I've twiddled with failover and beacon probing and every other switch I can flip and none seem to help. I'm back to default settings now. I'm fairly convinced this has to be a switch problem. Cisco guy tells me I'm not "trunking" correctly. Is there some other lever I have to pull in vSphere?

Thanks for any guidance!

Reply
0 Kudos
6 Replies
Josh26
Virtuoso
Virtuoso

Are these virtual switches running Nexus vSwitch modules?

If they are, well, is your Cisco guy across them, or are they too "vmware"?

Reply
0 Kudos
DennisBray
Enthusiast
Enthusiast

Have you configured andy load balancing on the vSwitch other than the default "Originating Port ID"?

Has the Cisco 3020 admin configured etherchannels? Has the Cisco admin configured load balancing on their side?

I have seen situations where the combination of IP hash and etherchannel configurations have led to the behavior your are describing.

The document referenced with the link below explains ESX 3.x virtual networking in language that Cisco folks recognize. The concepts, features and configurations all reference standard vSwitches, but translate easily to the vDistributed switches. It may also help you to understand the Cisco terminology.

VMware infrastructure in a Cisco Networking Environment

Dennis Bray, VCI, VCP

Reply
0 Kudos
robertl30
Contributor
Contributor

@Josh

No, these are VMware's virtual distributed switches.

Reply
0 Kudos
robertl30
Contributor
Contributor

@Dennis

Thd DVUplinks Load Balancing is Route based on originating port. Network Failover is Link Status only. Notify Switches is Yes, Failback is Yes.

I've tried different combinations of settings but no change in behavior.

He had etherchannels setup. I didn't know if they were needed so I asked him to remove and he did. Currently they're out. Do they need to be on or off? I only vaguely understand what this feature is for. My network guy said they're for bonding two ports together to increase bandwidth. That didn't sound like something VMware could take advantage of so I asked for that to go off.

Thanks for the doc link. I'll give it a read and pass it along.

Reply
0 Kudos
DennisBray
Enthusiast
Enthusiast

In a situation like you are describing, I have found that using the default load balancing without the etherchannels is more reliable.

Dennis Bray VCI, VCP

Reply
0 Kudos
robertl30
Contributor
Contributor

yes, that's what I'm doing. default load balancing without etherchannel.

still no luck getting some NICs to communicate.

Reply
0 Kudos