New VLAN, Guest Can't Ping GW

[tuscani]

This is driving me nuts.. Added a new VLAN which we have done several times. Switches, routers and firewall all see new VLAN and can ping GW without issue. The host can also ping the nVLAN GW, but the guests cannot. However, if I add another guest to the same port group they can ping each other (as long as they are on the same host), but nothing on the port group can ping the GW. So far I have:

- Again, verified full connectivity to the new VLAN between all physical devices (trunks, etherchannel, etc are all good)

- Verified VLAN number on port group

- Verified host sees the VLAN

- Deleted and readded port group from vSwitch

- Rebooted host

- Removed and readded nic from guest (tried E1000 and VMXNET3)

- Deleted and recreated the VLAN

Other ideas?

[a_p_ · Accepted solution]

Is the new VLAN in the list of allowed VLANs on the physical switch port(s) used as uplinks for the vSwitch?

André

[tuscani]

Yep!

vSwitch config is as follows:

120 ports
Promis mode reject
MAC addr changes accept
Forged trans accept
Traffic shaping disabled
Route on PortID
Link Status only
Notify switch yes
Failback yes
All four vmnics are active

[a_p_]

Ok, the VLAN is allowed on all physical uplink/trunk ports and the guests on the port group are able to ping each other. In this case I could think of two possible issues. Either you are using wrong IP settings (typo, ...) in the guests or the native/default VLAN for the uplink ports match the VLAN-ID on the port group.

If this isn't the case either, then please provide some more details about your configuration. Like vSwitch and port group settings (e.g. policies), physical port configuration, ...

André

PS: Just saw you added the vSwitch settings to your latest reply...

[tuscani]

Makes no sense.. I even tried a different IP scheme thinking maybe there was an issue there, but made no difference. Below is config on the physcial switch. I am only including info for one of the physical ports for the port channel, vmnics, etc as the ports are setup the same.

The new VLAN is 110

Etherchannel between two switches:

interface Port-channel1
switchport trunk native vlan 200
switchport trunk allowed vlan 99-110,200,201,800,850,864,865,950
switchport mode trunk

interface GigabitEthernet1/0/36
switchport trunk native vlan 200
switchport trunk allowed vlan 99-110,200,201,800,850,864,865,950
switchport mode trunk
channel-group 1 mode on

Physical nic on host:

interface GigabitEthernet1/0/13
switchport trunk allowed vlan 100-110,200
switchport mode trunk
spanning-tree portfast trunk

Port to our firewall:

interface GigabitEthernet1/0/40
switchport trunk allowed vlan 100-110,200,800,850,864,865,950
switchport mode trunk

[a_p_]

I'm not a networking guru, but this basically looks ok to me. I compared your settings with my switches and found a few settings which - according to http://kb.vmware.com/kb/1004074 - should be configured. Not sure whether tey make a difference though.

interface GigabitEthernet0/34
switchport trunk encapsulation dot1q
switchport trunk allowed vlan 502,503
switchport mode trunk
switchport nonegotiate
spanning-tree portfast trunk

André

[chriswahl]

Just because the host can ping the IP doesn't mean the VM can or should be able to. The host has the luxury of using a default gateway on the mgmt vmkernel port, while VMs can only use what they are given on the port group.

To start with - Have you verified Layer 2 connectivity = Does your first hop switch see the MAC address of the VM? If it does, where do you stop seeing the MAC address - a further upstream switch?

If no switch can see the VM MAC address, I'd verify that the port group the VMs have been placed on is properly configured with a VM running wireshark (or similar) and make it promiscious to sniff for frames destined to/from the gateway. At the very least you have isolated the problem to something between the first hop switch and the VM.

From the host, try

esxcli network ip neighbor list

And make sure you see the MAC of your VMs and VLAN gateway are in the list on the right vmknic.

Then make sure the guest NIC config is proper - make sure the IP / subnet / GW all match what is necessary to reach your intra VLAN IP (assuming an SVI).

[vGuy]

Since you are using static etherchannel, have you configured the port group load balancing to ip hash?

[tuscani]

Thanks for sticking with me guys.

My 2960-S switches only support dot1q which is why there is not an explicitly defined command. The nonegotiate shouldn't make a difference.

When I check neighbors, I am only seeing the following which is my management network on vSwitch0..

Neighbor       Mac Address        vmknic  Expiry(sec)  State
192.168.0.1    00:1a:8c:15:f8:20  vmk0    1149
192.168.0.11   00:50:56:89:00:04  vmk0    1198
192.168.0.22   00:0c:29:37:83:e0  vmk0    1192
192.168.0.208  00:50:56:80:00:e8  vmk0    1189
~ #

From VIC I can see that the vSwitch is getting all the appropiate info from the Cisco switch via CDP. However, when I run show mac address-table I am not seeing the 110 VLAN or the MAC of the VM. I see all my other VLANs and VM MACs there.. Argh! So therein lies the issue.

I tried created a new port group on vSwitch1 with VLAN 125 and no difference (thinking maybe VLAN ID was the issue). Even tried creating a new port group on vSwitch0 and nothing. It's like the hosts simply stopped talking via L2 for new port groups.

I installed wireshark on the VM and enabled promiscious mode on the port group. All I see are ARP "who is" commands for the default gateway. See attached.

vGuy, no my vSwitches are route on port ID only which has never been an issue.

[chriswahl]

vGuy has a valid point. If you are using Etherchannel from your upstream switch to the host, you are required to set the switch to "route by IP hash". Failure to do so can cause traffic to traverse improperly. If you aren't using an Etherchannel to the host, you are fine with "route by virtual port ID".

The vSwitch does not perform any MAC address learning. It either forwards traffic to a vNIC of a known VM (locally) or sends to the uplink - no other decisions are made. In your case it should just fill in the 802.1Q portion of the header with the port group's VLAN and toss the packet onto an uplink.

I'd check the mac table for VLAN 110 to see what is going on. Ensure that the VLAN exists in the switch's VLAN DB and nothing goofy with spanning-tree is blocking the port(s) for that VLAN. Make sure you are not pruning the port (show int X switchport in IOS).

If all else fails you might copy the configuration of your interface going to the ESXi host to a different interface and plug in another device to ensure your configuration works.

[tuscani]

Well I am embarrased to say the VLAN creation didn't take on the switches for whatever reason. I should have checked that first. I recreated it and all is well!!

Glad it was that because I was out of ideas.. I will also change my vSwitches to IP Hash.. will this cause an outage to the guests?

[chriswahl]

Glad you found it. Changing the routing method will not cause an outage. I typically change it on one host as a test first and ensure the switch is configured to "mode on" (prior to 5.1) or if using 5.1 you can use "mode active" to form an LACP port channel.

Example:

http://wahlnetwork.com/2012/10/30/configuring-lacp-on-a-vsphere-5-1-distributed-switch-video/

[a_p_]

I will also change my vSwitches to IP Hash...

Wait a second! This is only an option if the uplinks for the vSwitch are connected to physical ports in an EtherChannel group! If I understand your configuration correctly this is not the case (interface GigabitEthernet1/0/13) and you only have EtherChannel configured for the trunk ports between your physical switches!? In this case the current policy "Route based on originating port ID" is correct.

André

[tuscani]

Yeah good catch... the etherchannel is only configured between the switches not the vmnics so I will no change to IP hash.