VMware Cloud Community
TwincaAdmin
Enthusiast
Enthusiast
Jump to solution

ESXi Standard switch blocks the network if all Nic's is marked as active in "Failover order" In switch settings

Hi I'm kind of new in this ESXi.... on this level.

ESXi Standard switch blocks the network if all Nic's is marked as active in "Failover order" In switch settings. Why is this the case?

I Have 2 Switches on an ESXi 6.7.U3 With 4 physical NIC's.

There is assigned 2 NIC's to each Switch.

Will Load balancing work if there are not active both NIC's?

And why does it not work with both NIC's Active in the "Failover order"?

By the way... This Esxi Host is connected to an Ubiquiti EdgeSwitch ES-48-Lite

pastedImage_3.png

pastedImage_4.png

pastedImage_1.png

pastedImage_2.png

interface 0/29

description 'Server-2 NIC-1 VLAN 1 -10-15'

vlan participation exclude 20,30,100

vlan participation include 10,15

vlan tagging 10,15

exit

interface 0/30

description 'Server-2 NIC-2 VLAN 1 -10-15'

vlan participation exclude 20,30,100

vlan participation include 10,15

vlan tagging 10,15

exit

interface 0/31

description 'Server-2 NIC-3 VLAN 1 -10-15'

vlan participation exclude 20,30,100

vlan participation include 10,15

vlan tagging 10,15

exit

interface 0/32

description 'Server-2 NIC-4 VLAN 1-10-15'

vlan participation exclude 20,30,100

vlan participation include 10,15

vlan tagging 10,15

exit

Tags (1)
1 Solution

Accepted Solutions
TwincaAdmin
Enthusiast
Enthusiast
Jump to solution

I've now added the Intel Quad Port Ethernet Server Adapter (I350T4).

Is has been working for almost a week now... so i think it is solved now... It seem like it was a hardware error

image_2020_04_20T04_43_56_976Z.pngimage_2020_04_20T04_44_38_820Z.pngimage_2020_04_20T04_43_02_061Z.png

View solution in original post

Reply
0 Kudos
24 Replies
a_p_
Leadership
Leadership
Jump to solution

Please explain "ESXi Standard switch blocks the network".

It's not uncommon to have all vmnics active on the vSwitch as well as on the port groups. In fact that's actually the default configuration for most environments.

Can you confirm that the physical switch ports used by the vmnics are all configured as tagged (802.1Q) ports.


André

NicolasAlauzet
Jump to solution

To have "Load Balancing" you must have both pNICs as Active.

If you change the settings and there is an issue with traffic. Please check that the configuration for the pNICs is the same (if using trunk, same vlans should be allowed)

Can you provide the SW configuration for all four (4) pNICs?

-------------------------------------------------------------------
Triple VCIX (CMA-NV-DCV) | vExpert | MCSE | CCNA
TwincaAdmin
Enthusiast
Enthusiast
Jump to solution

The machines on the network loses connectivity to the internet, both the ones on the ESXi and the physical computer... I'm no sure but I could be  broadcast storm... But i'm not sure... The network stops responding. As soon as I deactivate the one of the two nic's in the "Failover order" (The Last one) machines begins to appear as online again in e.g. Teamviewer.

Please see the new attached pictures in the first post...

All 4 NIC's on the server is Truk ports with VLAN 1 Untagged, VLAN 10 and 15 Tagged.

Reply
0 Kudos
a_p_
Leadership
Leadership
Jump to solution

Does this happen on both vSwitches, or only on vSwitch1 with the port group for the default VLAN?

You don't have these ports in any channel configuration, do you?

André

Reply
0 Kudos
TwincaAdmin
Enthusiast
Enthusiast
Jump to solution

Please see the trade... I think The answer for you question is answered, in the answer to a.p.

If not... Please let me know.

Reply
0 Kudos
TwincaAdmin
Enthusiast
Enthusiast
Jump to solution

interface 0/29

description 'Server-2 NIC-1 VLAN 1 -10-15'

vlan participation exclude 20,30,100

vlan participation include 10,15

vlan tagging 10,15

exit

interface 0/30

description 'Server-2 NIC-2 VLAN 1 -10-15'

vlan participation exclude 20,30,100

vlan participation include 10,15

vlan tagging 10,15

exit

interface 0/31

description 'Server-2 NIC-3 VLAN 1 -10-15'

vlan participation exclude 20,30,100

vlan participation include 10,15

vlan tagging 10,15

exit

interface 0/32

description 'Server-2 NIC-4 VLAN 1-10-15'

vlan participation exclude 20,30,100

vlan participation include 10,15

vlan tagging 10,15

exit

Reply
0 Kudos
NicolasAlauzet
Jump to solution

All seems to be ok. But if not working you can do this test to try to isolate de problem.

Create a new portgroup. Put NIC 3 as ACTIVE, NIC1 unnused adapter.

Put a VM in that new portgroup, test connectivity.

Repeate the same, but changing NIC3 from ACTIVE to unnused and biceversa.

The objective is to isolate traffic for each adapter and see exactly whats not working

-------------------------------------------------------------------
Triple VCIX (CMA-NV-DCV) | vExpert | MCSE | CCNA
Reply
0 Kudos
TwincaAdmin
Enthusiast
Enthusiast
Jump to solution

Hi Andre

IT seems like it is only on the switch 1... Because I can access bother the esxi and the center server... When this happens.... So I think you are rigth it is somehow related to the untagged VLAN1.

But I've tried to put VLAN1 in the switch as tagged... But I did not manage to make it work... Should I be able to tag VLAN1 on the ubiquitous switch and then put the old production network to VLAN1.... Should that work? And would it take a reboot of the esxi, to make itwork, if yes?

(I did not reboot last I tried)

Jens ole

Reply
0 Kudos
TwincaAdmin
Enthusiast
Enthusiast
Jump to solution

Hi Nicolas...


That's an option... I'll try it if I don't manage to make it work, but to be honest, I don't think it is the adapter it self.. more likely it is related to the untagged VLAN1... I was wondering when I took the configuration from the switch CLI .... I did not see the VLAN1 on the ports for the server, even then it is not tagged... But shouldn't it appears in the configuration on the port anyway, as VLAN1 untagged?

Reply
0 Kudos
NicolasAlauzet
Jump to solution

In other switches you need to set as native vlan in the trunk to have the vlan 1 included. And propagate it as untagged.

In Cisco sw for example is: switchport trunk native vlan

I went through edgemax documentation and wasn't able to find if there is a way to do that.  Maybe you can try adding vlan 1 to your trunk, and first test without changes in esxi (vlan 0) and if doesn't work, move it to vlan 1 (tagged) maybe the sw will move traffic anyways.

-------------------------------------------------------------------
Triple VCIX (CMA-NV-DCV) | vExpert | MCSE | CCNA
Reply
0 Kudos
TwincaAdmin
Enthusiast
Enthusiast
Jump to solution

It is working with only one active nic... And if you see the previous attached pictures from the UI in the ubiquiti switch... You will see the the VLAN1 is on the trunk as untagged. ... So I really don't get why it does not work with 2 active NIC's.... I have suspicion that is behaving bit like if you put at patch cable as a loop between 2 ports in a normal physical switch. (Which you of cause never should do)

Reply
0 Kudos
TwincaAdmin
Enthusiast
Enthusiast
Jump to solution

I've now tagged the Default VLAN1. .... And that works With one Active adapter.

I'll try out this config... And If it stabile for some time, I'll try to add one more active NIC, and if it crashes again.

Until now the problem always starts every time I add both NIC's as active in the "Failover order"

pastedImage_1.png

pastedImage_2.png

pastedImage_5.png

pastedImage_0.png

pastedImage_3.png

Reply
0 Kudos
a_p_
Leadership
Leadership
Jump to solution

But I've tried to put VLAN1 in the switch as tagged...

There should actually be no reason to do this. Using the default/native VLAN is also not unusual, and just requires that the virtual port group doesn't have a VLAN-ID configured.

The issue is indeed strange, because ESXi itself doesn't create loops. The default setting "Route based on originating port ID" works in a round robin manner, where one of the vmnics is assigned to a VM's virtual network adapter at power on. Network changes btw. do not require a host reboot.

Do you have VMs with multiple virtual network adapters, which are assigned to different port groups, e.g. a virtual router?

Would it be possible to provide (attach) the complete physical switch configuration (s.th. like show running-config) to see whether it contains something that could be related?

André

Reply
0 Kudos
TwincaAdmin
Enthusiast
Enthusiast
Jump to solution

I've to or more machines on the physical network there is member of several VLAN...

But All VM's i only member of one VLAN .

Config:

login as: **********

TwAdmin@192.168.10.2's password:

  _____    _

| ____|__| | __ _  ___          (c) 2010-2018

|  _| / _  |/ _  |/ _ \         Ubiquiti Networks, Inc.

| |__| (_| | (_| |  __/

|_____\__._|\__. |\___|         https://www.ubnt.com

             |___/

Welcome to EdgeSwitch

By logging in, accessing or using the Ubiquiti (UBNT) product, you

acknowledge that you have read and understood the Ubiquiti Licence

Agreement (available in the GUI and at https://www.ubnt.com/eula/)

and agree to be bound by its terms.

(ES-48-Lite) >enable

Password:************

(ES-48-Lite) #show running-config

!Current Configuration:

!

!System Description "EdgeSwitch 48-Port Lite, 1.9.0-lite, Linux 3.6.5-03329b4a, 1.1.0.5102011"

!System Software Version "1.9.0-lite"

!System Up Time          "1 days 0 hrs 31 mins 43 secs"

!Additional Packages     QOS,IPv6 Management,Routing

!Current SNTP Synchronized Time: Mar 29 10:22:11 2020 UTC

!

hostname "ES-48-Lite"

network protocol none

network parms 192.168.10.2 255.255.255.0 192.168.10.1

vlan database

vlan 10,15,20,30,100

vlan name 10 "VLAN10_Management"

vlan name 15 "VLAN15_Local_LAN"

vlan name 20 "VLAN20_Guest"

vlan name 30 "VLAN30_IoT"

vlan name 100 "VLAN100_Test"

exit

network mgmt_vlan 10

ip http session soft-timeout 60

ip http secure-session soft-timeout 60

sshcon timeout 60

configure

ip name server 8.8.8.8 8.8.4.4

username "**************" password ***************** Removed ****************************************************************************************************************************** Removed *********** level 15 encrypted

username "*************" password ***************** Removed ****************************************************************************************************************************** Removed ***********  level 15 encrypted

line console

exit

line telnet

exit

line ssh

exit

snmp-server sysname "ES-48-Lite"

snmp-server location "Twinca"

snmp-server contact "JOM"

!

interface 0/1

description 'Trunk To ER-6P'

switchport mode trunk

vlan participation auto 1

exit

interface 0/2

vlan participation exclude 1,10,15,20,30,100

exit

interface 0/3

description 'MeetingRoomAP (Trunk)'

vlan participation exclude 10,15,30

vlan participation include 20,100

vlan tagging 20,100

exit

interface 0/4

description 'KontorAP (Trunk)'

vlan participation exclude 10,15,30

vlan participation include 20,100

vlan tagging 20,100

exit

interface 0/5

vlan participation exclude 1,10,15,20,30,100

exit

interface 0/6

vlan participation exclude 1,10,15,20,30,100

exit

interface 0/7

vlan participation exclude 1,10,15,20,30,100

exit

interface 0/8

vlan participation exclude 1,10,15,20,30,100

exit

interface 0/9

vlan participation exclude 1,10,15,20,30,100

exit

interface 0/10

vlan participation exclude 1,10,15,20,30,100

exit

interface 0/11

vlan participation exclude 10,15,20,30,100

exit

interface 0/12

vlan participation exclude 10,15,20,30,100

exit

interface 0/13

vlan participation exclude 10,15,20,30,100

exit

interface 0/14

vlan participation exclude 10,15,20,30,100

exit

interface 0/15

vlan participation exclude 10,15,20,30,100

exit

interface 0/16

description 'Konstruktion-02'

vlan participation exclude 10,15,20,30,100

exit

interface 0/17

vlan participation exclude 10,15,20,30,100

exit

interface 0/18

vlan participation exclude 10,15,20,30,100

exit

interface 0/19

description 'Twinca-PC01'

vlan participation exclude 10,15,20,30,100

exit

interface 0/20

vlan participation exclude 10,15,20,30,100

exit

interface 0/21

vlan participation exclude 10,15,20,30,100

vlan tagging 30

exit

interface 0/22

vlan participation exclude 10,15,20,30,100

vlan tagging 30

exit

interface 0/23

description 'Kontor-PC2'

vlan participation exclude 10,15,20,30,100

exit

interface 0/24

description 'Lager-PC01'

vlan participation exclude 10,15,20,30,100

exit

interface 0/25

description 'APC UPS-1 SmartConnect VLAN30'

vlan pvid 30

vlan participation exclude 1,10,15,20,100

vlan participation include 30

exit

interface 0/26

description 'Server-1 NIC1 VLAN 1 -10-15'

vlan participation exclude 20,30,100

vlan participation include 10,15

vlan tagging 10,15

exit

interface 0/27

description 'APC UPS-2 SmartConnect VLAN30'

vlan pvid 30

vlan participation exclude 1,10,15,20,100

vlan participation include 30

exit

interface 0/28

description 'Server-1 NIC2 VLAN 1 -10-15'

vlan participation exclude 20,30,100

vlan participation include 10,15

vlan tagging 10,15

exit

interface 0/29

description 'Server-2 NIC-1 VLAN 1 -10-15'

vlan participation exclude 20,30,100

vlan participation include 10,15

vlan tagging 1,10,15

exit

interface 0/30

description 'Server-2 NIC-2 VLAN 1 -10-15'

vlan participation exclude 20,30,100

vlan participation include 10,15

vlan tagging 1,10,15

exit

interface 0/31

description 'Server-2 NIC-3 VLAN 1 -10-15'

vlan participation exclude 20,30,100

vlan participation include 10,15

vlan tagging 1,10,15

exit

interface 0/32

description 'Server-2 NIC-4 VLAN 1-10-15'

vlan participation exclude 20,30,100

vlan participation include 10,15

vlan tagging 1,10,15

exit

interface 0/33

description 'NAS01 - LAN1'

vlan participation exclude 10,15,20,30,100

exit

interface 0/34

description 'NAS01 - LAN2'

vlan pvid 15

vlan participation exclude 1,10,20,30,100

vlan participation include 15

vlan tagging 15

exit

interface 0/35

description 'NAS02 - LAN1'

vlan participation exclude 10,15,20,30,100

exit

interface 0/36

description 'NAS02 - LAN2'

vlan pvid 15

vlan participation exclude 1,10,20,30,100

vlan participation include 15

vlan tagging 15

exit

interface 0/37

description 'Zebra ZM400'

vlan participation exclude 10,15,20,30,100

exit

interface 0/38

description 'Fronti Alarm VLAN10'

vlan pvid 10

vlan participation exclude 1,15,20,30,100

vlan participation include 10

vlan tagging 15

exit

interface 0/39

description 'KontorKopiPrint'

vlan participation exclude 10,15,20,30,100

exit

interface 0/40

description 'IBM UPS AP9630 VLAN10'

vlan pvid 10

vlan participation exclude 1,15,20,30,100

vlan participation include 10

exit

interface 0/41

description 'Server- 2 RMM4 Lite BMC VLAN10'

vlan pvid 10

vlan participation exclude 15,20,30,100

vlan participation include 10

vlan tagging 1,10,15

exit

interface 0/42

description 'Server-1 RMM3 VLAN10'

vlan pvid 10

vlan participation exclude 1,15,20,30,100

vlan participation include 10

vlan tagging 1,15

exit

interface 0/43

description 'APC UPS-1 AP9630 VLAN10'

vlan pvid 10

vlan participation exclude 1,15,20,30,100

vlan participation include 10

vlan tagging 15

exit

interface 0/44

description 'APC UPS-2 AP9630 VLAN10'

vlan pvid 10

vlan participation exclude 1,15,20,30,100

vlan participation include 10

vlan tagging 15

exit

interface 0/45

vlan participation exclude 1,10,15,20,30,100

exit

interface 0/46

description 'UniFi CloudKey'

vlan participation exclude 10,15,20,30,100

exit

interface 0/47

vlan participation exclude 1,10,15,20,30,100

exit

interface 0/48

description 'Trunk to ES-24-Lite'

switchport mode trunk

vlan participation auto 1

exit

interface 0/49

vlan participation exclude 10,15,20,30,100

exit

interface 0/50

vlan participation exclude 10,15,20,30,100

exit

interface 0/51

vlan participation exclude 10,15,20,30,100

exit

interface 0/52

vlan participation exclude 10,15,20,30,100

exit

interface lag 1

vlan participation exclude 10,15,20,30,100

exit

interface lag 2

vlan participation exclude 10,15,20,30,100

exit

interface lag 3

vlan participation exclude 10,15,20,30,100

exit

interface lag 4

vlan participation exclude 10,15,20,30,100

exit

interface lag 5

vlan participation exclude 10,15,20,30,100

exit

interface lag 6

vlan participation exclude 10,15,20,30,100

exit

service ubnt-discovery-active-scan

exit

(ES-48-Lite) #

pastedImage_3.png

Reply
0 Kudos
a_p_
Leadership
Leadership
Jump to solution

I'm not familiar with these specific switches, but other than the LAG interface configuration (which I have to admit, I don't understand), it looks just ok to me.

Anyway, I assume that the switch has a logging functionality, so it may be worth taking a look at the log once that issue occurs.

André

Reply
0 Kudos
TwincaAdmin
Enthusiast
Enthusiast
Jump to solution

I'v put the Standby Nic's as active on both Switches before I went to bed last night.... And The issue did not occur during the night, so maybe it did work to put the VLAN1 as Tagged on boot ESXi Switch and the Ubiquiti Switch.

I'll post the final solution here when I find out if that did make it to work. but for now it seems promising.

Reply
0 Kudos
NicolasAlauzet
Jump to solution

Thats good.

Try to find the exact change and If that was it i'm glad to hear it 😃

-------------------------------------------------------------------
Triple VCIX (CMA-NV-DCV) | vExpert | MCSE | CCNA
Reply
0 Kudos
TwincaAdmin
Enthusiast
Enthusiast
Jump to solution

It is back again... It comes after 1 -2 days and I've found out that, by change NIC 3 on Switch 1 to Standby the connectivity comes back on the network ... see the mark 2.

I think somehow this issue is related to the "Old VM Production network" marked with 1. All the VM's on the network is moved  / migrated by Veeam for an old Esxi host to this new new host.

pastedImage_0.png

pastedImage_0.png

Reply
0 Kudos
a_p_
Leadership
Leadership
Jump to solution

Does it make a difference which vmnic is active, and which is standby?

Did you already try to replace the network cable on vmnic3 just to rule out a hardware issue?

André

Reply
0 Kudos