VMware Cloud Community
hbertz
Enthusiast
Enthusiast

Distributed Switch Network Issue

Hello All,

i have a very perplexing issue we just had with the Distributed switch. We have an internal port group that has not uplink access. We put some VMs on this and we found that they still had access to VMs that were in other port groups. The vlans are the same (vlan 1) but still didn't think they would talk to each other as they are entirely in two separate port groups. One nic each vm and again one in internal port group and other in production port group.  We are running 5.1. anyone witness something like this before?

thanks!

Reply
0 Kudos
8 Replies
a_p_
Leadership
Leadership

As I understand this, that's the expected behavior. Systems on the same VLAN, and in the same IP subnet can talk to each other.

André

Reply
0 Kudos
MKguy
Virtuoso
Virtuoso

This is normal. As long two VMs are on the same physical host and in the same VLAN (even if they are on different port groups), they will always be able to talk to each other. The physical uplink is only relevant for communication outside of a particular host. You will have to define a different VLAN ID on your internal port group to really isolate theese VMs.

-- http://alpacapowered.wordpress.com
Reply
0 Kudos
hbertz
Enthusiast
Enthusiast

i was suspecting they might be able to since they are in the same switch (dvSwitch). The servers where not on the same host though. we did put the internal port group into our black hole vlan which didn't resolve the issue. I didn't think the distributed switche kept a Mac table though

Reply
0 Kudos
MKguy
Virtuoso
Virtuoso

The dvSwitch does not keep MAC tables like a traditional switch. The whole forwarding logic is essentially identical to the standard vSwitch and independent from host to host. It's better to view the dvS as just one logical management plane or vSwitch template that is applied to each host. What's happening outside of it's own local physical uplinks is unknown to each host. The forwarding logic is explained in more detail here: http://blog.ipspace.net/2010/11/vmware-virtual-switch-no-need-for-stp.html

That sounds weird, a vNIC attached to a Port Group that has all (d)vSwitch uplinks defined as unused in the teaming options should never be able to send or receive traffic to/from outside the host. Do you VMs have multiple vNICs on different Port Groups? Just a hunch, but are you using LACP/etherchannel by chance?

And as long as two Port Groups of the type VLAN have a different VLAN ID, the (d)vSwitch segregates them into different broadcast domains locally just like a physical switch. Unless you route traffic on layer 3 or bridge the two VLANs somewhere on your physical network or with a VM with two vNICs again, the systems should never be able to communicate with each other just like physical systems would not be able to. Have you tried assigning a random VLAN ID for your internal blackhole?

-- http://alpacapowered.wordpress.com
hbertz
Enthusiast
Enthusiast

both VMs only have one vNIC. When i put the port groups in separate vlans then yes the VMs could not communicate with each other anymore. But what still perplexes me is that when they were in the same vlan they were in different port groups on two different host and could still talk to each other. The VM in the production port group does have uplinks attached to that port group and we are using LACP. The VM in the Internal port group does not have uplinks attached though so they should not have been able to communicate over the uplinks. I like the way you are approaching this as it is the same approach i have had. Helps me to confirm i have been on right track for troubleshooting this. I am wondering though how this would all act if we migrate to using the Nexus 1000v over the distributed switch? any experience with the difference between how these two switches functions?

Reply
0 Kudos
MKguy
Virtuoso
Virtuoso

Unfortunately I can't comment on the Nexus 1000v as I've never used it myself.

I suspect that this could also be related to the etherchannel/LACP configuration, because typically all uplinks for a vSwitch should be always be "active". Maybe the host applies that implicitly for all Port Groups or something. Take a look at the following KB articles which both state that standby or unused configurations are unsupported for link aggregation:

VMware KB: Host requirements for link aggregation for ESXi and ESX

>Do not configure standby or unused uplinks with IP HASH load balancing.

VMware KB: Sample configuration of EtherChannel / Link Aggregation Control Protocol (LACP) with ESXi...

>Do not configure standby or unused uplinks with IP HASH load balancing.

Something else you could try is setting the teaming and failover policy of the internal Port Group to explicit failover order besides putting all uplinks to unused.

-- http://alpacapowered.wordpress.com
Reply
0 Kudos
hbertz
Enthusiast
Enthusiast

so if a port group doesn't have any uplinks associated to it, it can still get to the network?

Reply
0 Kudos
MKguy
Virtuoso
Virtuoso

As long as there are absolutely no active or standby uplinks configured, it *should not* be possible to reach the outside network. But because of the above statements I'm not sure whether this is still the case with configurations involving link aggregation.

You should probably raise a case with VMware support to get clarification on this.

-- http://alpacapowered.wordpress.com
Reply
0 Kudos