vSwitch issue not forwarding VLANS VMWare ESXi 5.0 Vsphere 5.0

[jd_morecraft]

With little VMware experiance I got assigned the task of trying to see why one of our VLANs is not working on one of our ESXi 5.0 servers. I will try to be as descriptive as possible.

The issue just started after working fine for months, when this first started to startup and running config had not been updated for several weeks on the router and switch.

Our server has 4 physical NICS 0 1 2 3

0 and 1 are teamed together on VLAN1 and are also used for the management network

NIC 2 is used for VLAN 111 (network our wireless users connect to so they cant interact with VLAN1)

NIC 3 is connected to VLAN 66 (a network for video and deployment services)

These 4 physical NICs are connected to three different vswitches as follows (I am not the one who set these up so don't ask why they are like they are)

vSwitch0 has vmnic0 and vmnic1

vSwitch1 has vmnic3

vSwitch2 has vmnic2

All these ports are connected physically to a Dell PowerConnect 5548 switch

vmnic0 to:

interface gigabitethernet1/0/24

description "VM Server 1"

switchport mode general

switchport general allowed vlan add 111 tagged

switchport general allowed vlan add 1 untagged

switchport general forbidden vlan add 5,66,98

vmnic1 to:

interface gigabitethernet1/0/17

description "VM Server 2"

switchport mode general

switchport general allowed vlan add 111 tagged

switchport general allowed vlan add 1 untagged

switchport general forbidden vlan add 5,66,98

vmnic2 to:

interface gigabitethernet1/0/6

description "VM Server 3"

switchport general allowed vlan add 111 tagged

switchport general allowed vlan add 1 untagged

switchport general forbidden vlan add 5,66,98

switchport access vlan 111 (Have tried many different configurations on this port including Trunk with 1 untagged and 111 tagged as VLAN 1 and 111 come through from the router on the same port. Also just a basic switchport mode access vlan 111)

vmnic3 to:

interface gigabitethernet1/0/43

description "VM Server 4"

switchport access vlan 66

If I plug a laptop into int gi 1/0/6 I get an IP address from DHCP and can get online. When i go into vSwitch2 properties and go to VMKernel and try to set it to obtain an IP address automatically I get a 169.254*.* address. If I statically set the address to one I know is available there still is no connection. I will add more information later if needed, I have a meeting to get to in regards to another project. Anyone that can help, thank you ahead of time. If you have specific information needed, I will respond ASAP.

On the configuration page under network adapters vmnic2 does not have any observed IP ranges and I am not sure how to edit the nic to maybe add the addresses

[homerzzz]

Is the VLAN ID for vswitch 2 set to 111? If so, I have had luck in the past just by removing vmnic2 from the vswitch, then re-add it to the vswitch.

[tomtom901]

If you configure the physical switch as an access port on VLAN 111, and remove (and check if the port group does VLAN tagging on VLAN ID 111). For an example, see the attached screenshot.

Can you post a screenshot of the Configuration -> Networking? That way we can further explore and examine your setup.

[jd_morecraft]

Homerzzz - Yeah sometimes the simplest idea is the best one, but no dice.

TomTom, please see the attached photo of my conf > network settings.  Do you need any photos of the properties screen?

As stated I am not the one who set this up initially, but got assigned the task to fix their setup.  fortunately or unfortunately, they no longer work here.

My latest config on the switchport is:

int gi 1/0/6

no negotiation

description "VM Server 3"

switchport access vlan 111

[tomtom901]

I see the issue already. The VM's attached to vSwitch 2 are configured to do VGT, VM Guest Tagging. Because you configured the VLAN ID as 4095, the guest OS needs to do VLAN tagging as VMware just passes through network packets with the VLAN ID header intact. To fix this, try the following:

1) Proporties of the vswitch

2) Edit the VMkernel and / or VM Network 1 .111

3) Set VLAN ID to 111

See attached for an example. Then DHCP should work on the VMkernel port group. You can test this first:

[jd_morecraft]

Still nothing.  I did look again at the vSwitch 1 settings that are working correctly and they are VMKernel2 VLAN ID none (0) and VM Network2 VLAN ID All (4095).  I also tried setting vSwitch2 up this way again and still nothing.  Here are two screen shots, anything else that may be helpful for you to see?

[tomtom901]

Try configuring the physical switch like this:

interface gigabitethernet1/0/6

description "Vmnic2"

switchport mode general

switchport general allowed vlan add 111 tagged

switchport general allowed vlan add 1 untagged

switchport general forbidden vlan add 5,66,98

Then configure the VMkernel to VLAN ID none, and leave the VM Network 1 .111 as is. Also check the VM's running here, I don't know if you have the possibillity to spin up a test VM in this network?

[homerzzz]

Is this host part of a cluster? Do other hosts have the same networking configuration?

[jd_morecraft]

tomtom - the network has reset with your configs, with DHCP still looking at a 169 address.  The config you gave is what it pretty much started as and is how the vlan1 ports are setup.  The computer on this switch called Ping Wireless is an XP machine that I intend to delete as we have a full server that runs real software for monitoring all of our network equipment and this is not needed anymore, so I can do whatever may help to this machine.

homerzzz - the host here is standalone, we do have two other hosts at another location but they are not configured as a cluster but two standalone boxes (this was against what I would have done).  They are on a different network 10.201.1.* and they only use the one network.

Message was edited by: jd_morecraft

[tomtom901]

Are you 100% sure that port gigabitethernet1/0/6 is connected to vmnic2? I'm beginning to think more and more that that isn't the case.

You could try grabbing the MAC address of Ping Wireless' NIC and tracing that one through the mac tables on the physical switch.

[homerzzz]

Have you eliminated the physical nic or cable being bad? The switchport seems to be good since you can plug a laptop in and get an IP.

[jd_morecraft]

Yes, when I edit the vswitch, I see the message come across the switch that the connection state of the port has changed.

Physically went to the server room and checked, it is connected to gi 1/0/6.

I really thought I had lost my mind in trying to figure this out of the past few days.  I had even tried to create a new vswitch.  We don't have any ports left on the router to send vlan111 to or I would do that.  Here is a slightly omitted version of the router config if it helps.

interface FastEthernet0/0

description connected to EthernetLAN

no ip address ip

flow ingress

duplex auto

speed auto

no cdp

enable

!

interface FastEthernet0/0.1

description Data VLAN

encapsulation dot1Q 1 native

ip address 10.100.1.1 255.255.255.0

ip helper-address 10.201.1.24

no ip redirects

no ip unreachables

no ip proxy-arp

ip flow ingress

ip policy route-map temp_pix

!

interface FastEthernet0/0.66

description Deployment VLAN

encapsulation dot1Q 66

ip address 10.100.66.1 255.255.255.0

ip helper-address 10.201.1.24

no ip redirects

no ip unreachables

no ip proxy-arp

ip flow ingress

no cdp enable

!

interface FastEthernet0/0.111

description Wireless VLAN

encapsulation dot1Q 111

ip address 10.1.111.1 255.255.255.0

ip access-group 110 in

ip helper-address 10.201.1.24

no ip redirects

no ip unreachables

no ip proxy-arp

ip flow ingress

!

interface FastEthernet0/0.120

description VRegistration

VLAN encapsulation dot1Q 120

ip address 10.100.120.1 255.255.255.0

ip helper-address 10.201.1.24

no ip redirects

no ip unreachables

no ip proxy-arp

ip flow ingress

no cdp enable

!

interface FastEthernet0/0.121

description Isolation VLAN

encapsulation dot1Q 121

ip address 10.100.121.1 255.255.255.0

ip helper-address 10.201.1.24

no ip redirects

no ip unreachables

no ip proxy-arp

ip flow ingress

no cdp enable

!

interface FastEthernet0/0.122

description MAC Detection VLAN

encapsulation dot1Q 122

ip address 10.100.122.1 255.255.255.0

ip helper-address 10.201.1.24

no ip redirects

no ip unreachables

no ip proxy-arp

ip flow ingress

no cdp enable

!

interface FastEthernet0/1

description Connection to Metro Ethernet

ip address 10.201.254.131 255.255.255.224

no ip redirects

no ip unreachables

no ip proxy-arp ip

nbar protocol-discovery

ip authentication mode eigrp 100 md5

ip authentication key-chain eigrp 100 ***

ip flow ingress

ip summary-address eigrp 100 10.1.0.0 255.255.0.0

ip summary-address eigrp 100 10.5.0.0 255.255.0.0

duplex full speed 100

!

interface FastEthernet0/1/0

description Connection to Sales Network

switchport access vlan 5

no ip address

duplex full speed 100

no cdp enable

!

interface FastEthernet0/1/1

description HomeLine

switchport access vlan 98

no ip address

!

interface FastEthernet0/1/2

description IT extra Subnet 66

switchport access vlan 119

switchport mode trunk

no ip address !

interface Vlan1 no ip address

!

[tomtom901]

Can you create a new port group on vSwitch 2 with no VLAN (0) and give the Wireless XP VM a vNIC in this network? Does it get an IP address in the 10.1.1.0/24 subnet? If not, can you plug your laptop into this port, and does this get an IP address?

Same goes for VLAN 111? Can you put an empty port on the switch (if any) into VLAN 111 and plug your laptop into it? Do you get an IP address in the 10.1.111.0/24 subnet? Do any other ports on the physical switch use VLAN 111 and does that work?

Hope this helps,

Tom

[jd_morecraft]

I will look at both of your suggestions in the morning. Had to leave a little early today.

thank you both for your time. Expect a response early tomorrow.

[tomtom901]

Great! Let us know.

[jd_morecraft]

Changed the cable on gi1/0/6 to vmnic2, no change - laptop got 10.100.1.x from dhcp - could ping outside network, static set 10.1.111.x no network access

Configured gi1/0/8 the same as gi1/0/6, no change - laptop got 10.100.1.x from dhcp - could ping outside network, static set 10.1.111.x no network access

Configured gi1/0/8 with only switchport access vlan 111, no change - laptop got 10.1.111.x from dhcp - can get to network and outside world

     vswitch2 could not contact dhcp, static set switch address, no access from pcs on that switch

     made test network on vswitch2 VLAN ID None (0) - security, traffic shaping, nic teaming tabs all have no options set

     put a nic on an xp and win 7 vm box from "test network", no network access using adapter type E1000

[tomtom901]

Have you perhaps got the possibility to remove vmnic2 from the vSwitch and add another? Remove one vmnic from vSwitch 0 in example, configure the physical switch port as gi1/0/6 and add that uplink to vSwitch2?

Perhaps the port is just broken.. VLAN is not the issue here.

[homerzzz · Accepted solution]

I would try to eliminate the physical NIC on the host being the issue. Silly question, but is there a link light? Drop a sniffer in there and see if there is any traffic passing if trying a different physical nic does not help.

[tomtom901]

Perhaps ESXtop also shows some packets being dropped. But first try and eliminate the pNIC als I (and homerzzz) pointed out.

[jd_morecraft]

I would have rather it been anything else but a hardware failure.  Yes taking vmnic3 off of vswitch1 and putting it on vswitch2 and moving the cable from vmnic2 to vmnic3 works.  I really don't want to call dell and tell them I need the mobo replaced that the nics are attached to, nor do I think they will come out here after hours to do it while it is ok to take the system down.

Oh well.  Sorry for dragging you guys through this for the hardware issue.  But thanks for the help.

Is there any way to go into esxi and try to repair the nic with a driver or just disable and re-enable it to see if I can get it working that way?