Hi I'm kind of new in this ESXi.... on this level.
ESXi Standard switch blocks the network if all Nic's is marked as active in "Failover order" In switch settings. Why is this the case?
I Have 2 Switches on an ESXi 6.7.U3 With 4 physical NIC's.
There is assigned 2 NIC's to each Switch.
Will Load balancing work if there are not active both NIC's?
And why does it not work with both NIC's Active in the "Failover order"?
By the way... This Esxi Host is connected to an Ubiquiti EdgeSwitch ES-48-Lite
interface 0/29
description 'Server-2 NIC-1 VLAN 1 -10-15'
vlan participation exclude 20,30,100
vlan participation include 10,15
vlan tagging 10,15
exit
interface 0/30
description 'Server-2 NIC-2 VLAN 1 -10-15'
vlan participation exclude 20,30,100
vlan participation include 10,15
vlan tagging 10,15
exit
interface 0/31
description 'Server-2 NIC-3 VLAN 1 -10-15'
vlan participation exclude 20,30,100
vlan participation include 10,15
vlan tagging 10,15
exit
interface 0/32
description 'Server-2 NIC-4 VLAN 1-10-15'
vlan participation exclude 20,30,100
vlan participation include 10,15
vlan tagging 10,15
exit
I've now added the Intel Quad Port Ethernet Server Adapter (I350T4).
Is has been working for almost a week now... so i think it is solved now... It seem like it was a hardware error
Please explain "ESXi Standard switch blocks the network".
It's not uncommon to have all vmnics active on the vSwitch as well as on the port groups. In fact that's actually the default configuration for most environments.
Can you confirm that the physical switch ports used by the vmnics are all configured as tagged (802.1Q) ports.
André
To have "Load Balancing" you must have both pNICs as Active.
If you change the settings and there is an issue with traffic. Please check that the configuration for the pNICs is the same (if using trunk, same vlans should be allowed)
Can you provide the SW configuration for all four (4) pNICs?
The machines on the network loses connectivity to the internet, both the ones on the ESXi and the physical computer... I'm no sure but I could be broadcast storm... But i'm not sure... The network stops responding. As soon as I deactivate the one of the two nic's in the "Failover order" (The Last one) machines begins to appear as online again in e.g. Teamviewer.
Please see the new attached pictures in the first post...
All 4 NIC's on the server is Truk ports with VLAN 1 Untagged, VLAN 10 and 15 Tagged.
Does this happen on both vSwitches, or only on vSwitch1 with the port group for the default VLAN?
You don't have these ports in any channel configuration, do you?
André
Please see the trade... I think The answer for you question is answered, in the answer to a.p.
If not... Please let me know.
interface 0/29
description 'Server-2 NIC-1 VLAN 1 -10-15'
vlan participation exclude 20,30,100
vlan participation include 10,15
vlan tagging 10,15
exit
interface 0/30
description 'Server-2 NIC-2 VLAN 1 -10-15'
vlan participation exclude 20,30,100
vlan participation include 10,15
vlan tagging 10,15
exit
interface 0/31
description 'Server-2 NIC-3 VLAN 1 -10-15'
vlan participation exclude 20,30,100
vlan participation include 10,15
vlan tagging 10,15
exit
interface 0/32
description 'Server-2 NIC-4 VLAN 1-10-15'
vlan participation exclude 20,30,100
vlan participation include 10,15
vlan tagging 10,15
exit
All seems to be ok. But if not working you can do this test to try to isolate de problem.
Create a new portgroup. Put NIC 3 as ACTIVE, NIC1 unnused adapter.
Put a VM in that new portgroup, test connectivity.
Repeate the same, but changing NIC3 from ACTIVE to unnused and biceversa.
The objective is to isolate traffic for each adapter and see exactly whats not working
Hi Andre
IT seems like it is only on the switch 1... Because I can access bother the esxi and the center server... When this happens.... So I think you are rigth it is somehow related to the untagged VLAN1.
But I've tried to put VLAN1 in the switch as tagged... But I did not manage to make it work... Should I be able to tag VLAN1 on the ubiquitous switch and then put the old production network to VLAN1.... Should that work? And would it take a reboot of the esxi, to make itwork, if yes?
(I did not reboot last I tried)
Jens ole
Hi Nicolas...
That's an option... I'll try it if I don't manage to make it work, but to be honest, I don't think it is the adapter it self.. more likely it is related to the untagged VLAN1... I was wondering when I took the configuration from the switch CLI .... I did not see the VLAN1 on the ports for the server, even then it is not tagged... But shouldn't it appears in the configuration on the port anyway, as VLAN1 untagged?
In other switches you need to set as native vlan in the trunk to have the vlan 1 included. And propagate it as untagged.
In Cisco sw for example is: switchport trunk native vlan
I went through edgemax documentation and wasn't able to find if there is a way to do that. Maybe you can try adding vlan 1 to your trunk, and first test without changes in esxi (vlan 0) and if doesn't work, move it to vlan 1 (tagged) maybe the sw will move traffic anyways.
It is working with only one active nic... And if you see the previous attached pictures from the UI in the ubiquiti switch... You will see the the VLAN1 is on the trunk as untagged. ... So I really don't get why it does not work with 2 active NIC's.... I have suspicion that is behaving bit like if you put at patch cable as a loop between 2 ports in a normal physical switch. (Which you of cause never should do)
I've now tagged the Default VLAN1. .... And that works With one Active adapter.
I'll try out this config... And If it stabile for some time, I'll try to add one more active NIC, and if it crashes again.
Until now the problem always starts every time I add both NIC's as active in the "Failover order"
But I've tried to put VLAN1 in the switch as tagged...
There should actually be no reason to do this. Using the default/native VLAN is also not unusual, and just requires that the virtual port group doesn't have a VLAN-ID configured.
The issue is indeed strange, because ESXi itself doesn't create loops. The default setting "Route based on originating port ID" works in a round robin manner, where one of the vmnics is assigned to a VM's virtual network adapter at power on. Network changes btw. do not require a host reboot.
Do you have VMs with multiple virtual network adapters, which are assigned to different port groups, e.g. a virtual router?
Would it be possible to provide (attach) the complete physical switch configuration (s.th. like show running-config) to see whether it contains something that could be related?
André
I've to or more machines on the physical network there is member of several VLAN...
But All VM's i only member of one VLAN .
Config:
login as: **********
TwAdmin@192.168.10.2's password:
_____ _
| ____|__| | __ _ ___ (c) 2010-2018
| _| / _ |/ _ |/ _ \ Ubiquiti Networks, Inc.
| |__| (_| | (_| | __/
|_____\__._|\__. |\___| https://www.ubnt.com
|___/
Welcome to EdgeSwitch
By logging in, accessing or using the Ubiquiti (UBNT) product, you
acknowledge that you have read and understood the Ubiquiti Licence
Agreement (available in the GUI and at https://www.ubnt.com/eula/)
and agree to be bound by its terms.
(ES-48-Lite) >enable
Password:************
(ES-48-Lite) #show running-config
!Current Configuration:
!
!System Description "EdgeSwitch 48-Port Lite, 1.9.0-lite, Linux 3.6.5-03329b4a, 1.1.0.5102011"
!System Software Version "1.9.0-lite"
!System Up Time "1 days 0 hrs 31 mins 43 secs"
!Additional Packages QOS,IPv6 Management,Routing
!Current SNTP Synchronized Time: Mar 29 10:22:11 2020 UTC
!
hostname "ES-48-Lite"
network protocol none
network parms 192.168.10.2 255.255.255.0 192.168.10.1
vlan database
vlan 10,15,20,30,100
vlan name 10 "VLAN10_Management"
vlan name 15 "VLAN15_Local_LAN"
vlan name 20 "VLAN20_Guest"
vlan name 30 "VLAN30_IoT"
vlan name 100 "VLAN100_Test"
exit
network mgmt_vlan 10
ip http session soft-timeout 60
ip http secure-session soft-timeout 60
sshcon timeout 60
configure
ip name server 8.8.8.8 8.8.4.4
username "**************" password ***************** Removed ****************************************************************************************************************************** Removed *********** level 15 encrypted
username "*************" password ***************** Removed ****************************************************************************************************************************** Removed *********** level 15 encrypted
line console
exit
line telnet
exit
line ssh
exit
snmp-server sysname "ES-48-Lite"
snmp-server location "Twinca"
snmp-server contact "JOM"
!
interface 0/1
description 'Trunk To ER-6P'
switchport mode trunk
vlan participation auto 1
exit
interface 0/2
vlan participation exclude 1,10,15,20,30,100
exit
interface 0/3
description 'MeetingRoomAP (Trunk)'
vlan participation exclude 10,15,30
vlan participation include 20,100
vlan tagging 20,100
exit
interface 0/4
description 'KontorAP (Trunk)'
vlan participation exclude 10,15,30
vlan participation include 20,100
vlan tagging 20,100
exit
interface 0/5
vlan participation exclude 1,10,15,20,30,100
exit
interface 0/6
vlan participation exclude 1,10,15,20,30,100
exit
interface 0/7
vlan participation exclude 1,10,15,20,30,100
exit
interface 0/8
vlan participation exclude 1,10,15,20,30,100
exit
interface 0/9
vlan participation exclude 1,10,15,20,30,100
exit
interface 0/10
vlan participation exclude 1,10,15,20,30,100
exit
interface 0/11
vlan participation exclude 10,15,20,30,100
exit
interface 0/12
vlan participation exclude 10,15,20,30,100
exit
interface 0/13
vlan participation exclude 10,15,20,30,100
exit
interface 0/14
vlan participation exclude 10,15,20,30,100
exit
interface 0/15
vlan participation exclude 10,15,20,30,100
exit
interface 0/16
description 'Konstruktion-02'
vlan participation exclude 10,15,20,30,100
exit
interface 0/17
vlan participation exclude 10,15,20,30,100
exit
interface 0/18
vlan participation exclude 10,15,20,30,100
exit
interface 0/19
description 'Twinca-PC01'
vlan participation exclude 10,15,20,30,100
exit
interface 0/20
vlan participation exclude 10,15,20,30,100
exit
interface 0/21
vlan participation exclude 10,15,20,30,100
vlan tagging 30
exit
interface 0/22
vlan participation exclude 10,15,20,30,100
vlan tagging 30
exit
interface 0/23
description 'Kontor-PC2'
vlan participation exclude 10,15,20,30,100
exit
interface 0/24
description 'Lager-PC01'
vlan participation exclude 10,15,20,30,100
exit
interface 0/25
description 'APC UPS-1 SmartConnect VLAN30'
vlan pvid 30
vlan participation exclude 1,10,15,20,100
vlan participation include 30
exit
interface 0/26
description 'Server-1 NIC1 VLAN 1 -10-15'
vlan participation exclude 20,30,100
vlan participation include 10,15
vlan tagging 10,15
exit
interface 0/27
description 'APC UPS-2 SmartConnect VLAN30'
vlan pvid 30
vlan participation exclude 1,10,15,20,100
vlan participation include 30
exit
interface 0/28
description 'Server-1 NIC2 VLAN 1 -10-15'
vlan participation exclude 20,30,100
vlan participation include 10,15
vlan tagging 10,15
exit
interface 0/29
description 'Server-2 NIC-1 VLAN 1 -10-15'
vlan participation exclude 20,30,100
vlan participation include 10,15
vlan tagging 1,10,15
exit
interface 0/30
description 'Server-2 NIC-2 VLAN 1 -10-15'
vlan participation exclude 20,30,100
vlan participation include 10,15
vlan tagging 1,10,15
exit
interface 0/31
description 'Server-2 NIC-3 VLAN 1 -10-15'
vlan participation exclude 20,30,100
vlan participation include 10,15
vlan tagging 1,10,15
exit
interface 0/32
description 'Server-2 NIC-4 VLAN 1-10-15'
vlan participation exclude 20,30,100
vlan participation include 10,15
vlan tagging 1,10,15
exit
interface 0/33
description 'NAS01 - LAN1'
vlan participation exclude 10,15,20,30,100
exit
interface 0/34
description 'NAS01 - LAN2'
vlan pvid 15
vlan participation exclude 1,10,20,30,100
vlan participation include 15
vlan tagging 15
exit
interface 0/35
description 'NAS02 - LAN1'
vlan participation exclude 10,15,20,30,100
exit
interface 0/36
description 'NAS02 - LAN2'
vlan pvid 15
vlan participation exclude 1,10,20,30,100
vlan participation include 15
vlan tagging 15
exit
interface 0/37
description 'Zebra ZM400'
vlan participation exclude 10,15,20,30,100
exit
interface 0/38
description 'Fronti Alarm VLAN10'
vlan pvid 10
vlan participation exclude 1,15,20,30,100
vlan participation include 10
vlan tagging 15
exit
interface 0/39
description 'KontorKopiPrint'
vlan participation exclude 10,15,20,30,100
exit
interface 0/40
description 'IBM UPS AP9630 VLAN10'
vlan pvid 10
vlan participation exclude 1,15,20,30,100
vlan participation include 10
exit
interface 0/41
description 'Server- 2 RMM4 Lite BMC VLAN10'
vlan pvid 10
vlan participation exclude 15,20,30,100
vlan participation include 10
vlan tagging 1,10,15
exit
interface 0/42
description 'Server-1 RMM3 VLAN10'
vlan pvid 10
vlan participation exclude 1,15,20,30,100
vlan participation include 10
vlan tagging 1,15
exit
interface 0/43
description 'APC UPS-1 AP9630 VLAN10'
vlan pvid 10
vlan participation exclude 1,15,20,30,100
vlan participation include 10
vlan tagging 15
exit
interface 0/44
description 'APC UPS-2 AP9630 VLAN10'
vlan pvid 10
vlan participation exclude 1,15,20,30,100
vlan participation include 10
vlan tagging 15
exit
interface 0/45
vlan participation exclude 1,10,15,20,30,100
exit
interface 0/46
description 'UniFi CloudKey'
vlan participation exclude 10,15,20,30,100
exit
interface 0/47
vlan participation exclude 1,10,15,20,30,100
exit
interface 0/48
description 'Trunk to ES-24-Lite'
switchport mode trunk
vlan participation auto 1
exit
interface 0/49
vlan participation exclude 10,15,20,30,100
exit
interface 0/50
vlan participation exclude 10,15,20,30,100
exit
interface 0/51
vlan participation exclude 10,15,20,30,100
exit
interface 0/52
vlan participation exclude 10,15,20,30,100
exit
interface lag 1
vlan participation exclude 10,15,20,30,100
exit
interface lag 2
vlan participation exclude 10,15,20,30,100
exit
interface lag 3
vlan participation exclude 10,15,20,30,100
exit
interface lag 4
vlan participation exclude 10,15,20,30,100
exit
interface lag 5
vlan participation exclude 10,15,20,30,100
exit
interface lag 6
vlan participation exclude 10,15,20,30,100
exit
service ubnt-discovery-active-scan
exit
(ES-48-Lite) #
I'm not familiar with these specific switches, but other than the LAG interface configuration (which I have to admit, I don't understand), it looks just ok to me.
Anyway, I assume that the switch has a logging functionality, so it may be worth taking a look at the log once that issue occurs.
André
I'v put the Standby Nic's as active on both Switches before I went to bed last night.... And The issue did not occur during the night, so maybe it did work to put the VLAN1 as Tagged on boot ESXi Switch and the Ubiquiti Switch.
I'll post the final solution here when I find out if that did make it to work. but for now it seems promising.
Thats good.
Try to find the exact change and If that was it i'm glad to hear it 😃
It is back again... It comes after 1 -2 days and I've found out that, by change NIC 3 on Switch 1 to Standby the connectivity comes back on the network ... see the mark 2.
I think somehow this issue is related to the "Old VM Production network" marked with 1. All the VM's on the network is moved / migrated by Veeam for an old Esxi host to this new new host.
Does it make a difference which vmnic is active, and which is standby?
Did you already try to replace the network cable on vmnic3 just to rule out a hardware issue?
André