Even after enabling BGP on UI on Tire0 Gateway, still on the CLI, it shows as BGP as "Administratively shut down".
I am able to ping neighbor address from nsxt-edge1(tier0_sr). Only BGP shows as "Administratively shut down"
Can anyone pls let me know, how to fix this?
Tire-1 gateway is working fine.
nsxt-edge01(tier0_sr)> get bgp neighbor
BGP neighbor is 192.168.100.1, remote AS 65100, local AS 65000, external link
Administratively shut down
BGP version 4, remote router ID 0.0.0.0, local router ID 192.168.100.2
BGP state = Idle
Last read 19:53:42, Last write never
Hold time is 180, keepalive interval is 60 seconds
Configured hold time is 180, keepalive interval is 60 seconds
Graceful restart information:
Local GR Mode : Helper*
Remote GR Mode : NotApplicable
R bit : False
Timers :
Configured Restart Time(sec) : 180
Received Restart Time(sec) : 0
Message statistics:
Inq depth is 0
Outq depth is 0
Sent Rcvd
Opens: 0 0
Notifications: 0 0
Updates: 0 0
Keepalives: 0 0
Route Refresh: 0 0
Capability: 0 0
Total: 0 0
Minimum time between advertisement runs is 0 seconds
Update source is 192.168.100.2
For address family: IPv4 Unicast
Not part of any update group
Community attribute sent to this neighbor(all)
0 accepted prefixes
Connections established 0; dropped 0
Last reset never
BGP Connect Retry Timer in Seconds: 10
Read thread: off Write thread: off
nsxt-edge01(tier0_sr)> ping 192.168.100.1 source 192.168.100.2 repeat 6
PING 192.168.100.1 (192.168.100.1) from 192.168.100.2: 56 data bytes
64 bytes from 192.168.100.1: icmp_seq=0 ttl=64 time=2.020 ms
64 bytes from 192.168.100.1: icmp_seq=1 ttl=64 time=2.382 ms
64 bytes from 192.168.100.1: icmp_seq=2 ttl=64 time=2.396 ms
64 bytes from 192.168.100.1: icmp_seq=3 ttl=64 time=1.961 ms
64 bytes from 192.168.100.1: icmp_seq=4 ttl=64 time=1.687 ms
64 bytes from 192.168.100.1: icmp_seq=5 ttl=64 time=1.891 ms
--- 192.168.100.1 ping statistics ---
6 packets transmitted, 6 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 1.687/2.056/2.396/0.257 ms
Can you perform a BGP debug on both the routers ?
Also please provide get interfaces output from tier-0
Thanks for the reply. Pls find the details of get interfaces....
On the other end (router), i did the debug, actually, it is establishing TCP connection and the other end, sends the BGP-OPEN message, then NSX sends TCP reset message...
Other end (router) is trying to establishing the TCP and sending BGP OPEN message, but NSX-T send TCP reset message, and not sending any BGP OPEN message...
its is like there is not BGP configured on NSX-T tire-0 gateway... On tire-0 it shows as "Administratively shutdown" , even with BGP enabled on UI...
On NSX-T, Snippet of the uplink interface, where BGP connection trying to establish:
=================================================================
Interface : 5093c7c7-b3ad-400e-9782-583a4047eae2
Ifuid : 288
Name : uplink01
Fwd-mode : IPV4_ONLY
Internal name : uplink-288
Mode : lif
Port-type : uplink
IP/Mask : 192.168.100.2/24
MAC : 00:50:56:b6:c2:38
VLAN : None
Access-VLAN : None
LS port : de6659c2-b0af-4862-ba71-d5e1d02af763
Urpf-mode : STRICT_MODE
DAD-mode : LOOSE
RA-mode : SLAAC_DNS_TRHOUGH_RA(M=0, O=0)
Admin : up
Op_state : up
MTU : 9000
Complete Output on vrf:
-----------------------------
nsxt-edge01(tier0_sr)> get interfaces
Logical Router
UUID VRF LR-ID Name Type
f2dbfc36-b3df-4b9f-8c50-95107e4fea02 5 2052 DR-Tire0-GW DISTRIBUTED_ROUTER_TIER0
Interfaces (IPv6 DAD Status A-Assigned, D-Duplicate, T-Tentative)
Interface : 1eb95b92-3457-442a-861d-1919ba8b8d77
Ifuid : 301
Name : Tire0-GW-Tire1-GW-t0_lrp
Fwd-mode : IPV4_ONLY
Internal name : downlink-301
Mode : lif
Port-type : downlink
IP/Mask : 100.64.112.0/31;fc7c:29ed:e1ac:d000::1/64(NA);fe80::50:56ff:fe56:4452/64(NA)
MAC : 02:50:56:56:44:52
VNI : 71689
Access-VLAN : None
LS port : d895832e-c514-48bf-820b-d3bb875c1cc6
Urpf-mode : PORT_CHECK
DAD-mode : LOOSE
RA-mode : SLAAC_DNS_TRHOUGH_RA(M=0, O=0)
Admin : up
Op_state : up
MTU : 1500
Interface : 41a82333-39b2-45e1-96aa-9c749805ca88
Ifuid : 296
Name : bp-dr-port
Fwd-mode : IPV4_ONLY
Mode : lif
Port-type : backplane
IP/Mask : 169.254.0.1/25;fe80::50:56ff:fe56:4452/64(NA)
MAC : 02:50:56:56:44:52
VNI : 71690
Access-VLAN : None
LS port : 8c064dea-e15b-43a5-b789-bd437eb0c7b9
Urpf-mode : PORT_CHECK
DAD-mode : LOOSE
RA-mode : RA_INVALID
Admin : up
Op_state : up
MTU : 1500
Interface : 1c535e28-1211-5c61-af7c-7f01c834d3d4
Ifuid : 293
Mode : cpu
Port-type : cpu
Interface : 57306016-9058-5171-82fc-0559d6e6e108
Ifuid : 294
Mode : blackhole
Port-type : blackhole
Logical Router
UUID VRF LR-ID Name Type
e8aae4b9-d9bf-4966-9721-4ba970bdc53f 3 2053 SR-Tire0-GW SERVICE_ROUTER_TIER0
Interfaces (IPv6 DAD Status A-Assigned, D-Duplicate, T-Tentative)
Interface : 56d780ea-31b0-501a-9030-310b4b5fd645
Ifuid : 281
Mode : cpu
Port-type : cpu
Interface : 610111d2-abb6-57c1-a8ef-31418ac71cc4
Ifuid : 282
Mode : blackhole
Port-type : blackhole
Interface : 24420827-5a60-424d-ae7b-4240ad08849f
Ifuid : 286
Name : sr0-internal-routing-port
Fwd-mode : IPV4_ONLY
Internal name : inter-sr-286
Mode : lif
Port-type : internal-routing
IP/Mask : 169.254.0.130/25;fe80::50:56ff:fe56:5201/64(NA)
MAC : 02:50:56:56:52:00
VNI : 71687
Access-VLAN : None
LS port : 42fd04d2-90a3-4194-a595-88f0432db89a
Urpf-mode : PORT_CHECK
DAD-mode : LOOSE
RA-mode : RA_INVALID
Admin : up
Op_state : up
MTU : 1500
Interface : a3f81c59-d705-4524-b0a6-78890102abfd
Ifuid : 290
Name : bp-sr0-port
Fwd-mode : IPV4_ONLY
Internal name : downlink-290
Mode : lif
Port-type : backplane
IP/Mask :
MAC : 02:50:56:56:53:00
VNI : 71690
Access-VLAN : None
LS port : a73cbb1c-986d-4258-86b3-c04eff9afac0
Urpf-mode : NONE
DAD-mode : LOOSE
RA-mode : RA_INVALID
Admin : up
Op_state : down
MTU : 1500
Interface : 5093c7c7-b3ad-400e-9782-583a4047eae2
Ifuid : 288
Name : uplink01
Fwd-mode : IPV4_ONLY
Internal name : uplink-288
Mode : lif
Port-type : uplink
IP/Mask : 192.168.100.2/24
MAC : 00:50:56:b6:c2:38
VLAN : None
Access-VLAN : None
LS port : de6659c2-b0af-4862-ba71-d5e1d02af763
Urpf-mode : STRICT_MODE
DAD-mode : LOOSE
RA-mode : SLAAC_DNS_TRHOUGH_RA(M=0, O=0)
Admin : up
Op_state : up
MTU : 9000
Interface : a812a72c-27a6-4152-9904-f90a0f5d2272
Ifuid : 291
Mode : loopback
Port-type : loopback
IP/Mask : 127.0.0.1/8;::1/128(NA)
Output on tire-0 box: (removed eth0 IP details from the output, rest of them, are all correct)
--------------------------------------------------------------------------------------------------------------
nsxt-edge01> get interfaces
Interface: bond0
Address: unknown
MAC address: be:7c:bb:9a:76:d4
MTU: 1500
Broadcast address: None
KNI: False
Bond mode: ROUND_ROBIN
Bond slaves:
Link status: down
Admin status: down
RX packets: 0
RX bytes: 0
RX errors: 0
RX dropped: 0
TX packets: 0
TX bytes: 0
TX errors: 0
TX dropped: 0
TX collisions: 0
Interface: eth0
Address: [REMOVED FROM OUTPUT]
MAC address: 00:50:56:b6:0f:ca
MTU: 1500
Default gateway: [REMOVED FROM OUTPUT]
Broadcast address: [REMOVED FROM OUTPUT]
KNI: False
Link status: up
Admin status: up
RX packets: 2498917
RX bytes: 176892161
RX errors: 0
RX dropped: 0
TX packets: 80014
TX bytes: 18575173
TX errors: 0
TX dropped: 0
TX collisions: 0
Interface: fp-eth0
ID: 0
Link status: up
MAC address: 00:50:56:b6:dc:b5
MTU: 1600
PCI: 0000:0b:00:00
Offload Capabilities: TX_VLAN_INSERT TX_UDP_CKSUM TX_TCP_CKSUM TX_TCP_TSO RX_VLAN_STRIP RX_IPV4_CKSUM RX_UDP_CKSUM RX_TCP_CKSUM RX_TCP_LRO
Polling Status: active
Driver: net_vmxnet3
Rx queue: 2
Tx queue: 2
Socket: 0
RX packets: 9555
RX bytes: 859890
RX errors: 0
RX badcrc: unknown
RX badlen: unknown
RX misses: 0
RX nombufs: 0
RX pause xoff: unknown
RX pause xon: unknown
TX packets: 139993
TX bytes: 5879706
TX errors: 0
TX pause xoff: unknown
TX pause xon: unknown
Interface: fp-eth1
ID: 1
Link status: up
MAC address: 00:50:56:b6:c2:38
MTU: 1600
PCI: 0000:13:00:00
Offload Capabilities: TX_VLAN_INSERT TX_UDP_CKSUM TX_TCP_CKSUM TX_TCP_TSO RX_VLAN_STRIP RX_IPV4_CKSUM RX_UDP_CKSUM RX_TCP_CKSUM RX_TCP_LRO
Polling Status: active
Driver: net_vmxnet3
Rx queue: 2
Tx queue: 2
Socket: 0
RX packets: 2820
RX bytes: 247591
RX errors: 0
RX badcrc: unknown
RX badlen: unknown
RX misses: 0
RX nombufs: 0
RX pause xoff: unknown
RX pause xon: unknown
TX packets: 8192
TX bytes: 539708
TX errors: 0
TX pause xoff: unknown
TX pause xon: unknown
Interface: fp-eth2
ID: 2
Link status: up
MAC address: 00:50:56:b6:2c:cc
MTU: 1500
PCI: 0000:1b:00:00
Offload Capabilities: TX_VLAN_INSERT TX_UDP_CKSUM TX_TCP_CKSUM TX_TCP_TSO RX_VLAN_STRIP RX_IPV4_CKSUM RX_UDP_CKSUM RX_TCP_CKSUM RX_TCP_LRO
Polling Status: active
Driver: net_vmxnet3
Rx queue: 2
Tx queue: 2
Socket: 0
RX packets: 3059847
RX bytes: 205469196
RX errors: 0
RX badcrc: unknown
RX badlen: unknown
RX misses: 0
RX nombufs: 0
RX pause xoff: unknown
RX pause xon: unknown
TX packets: 0
TX bytes: 0
TX errors: 0
TX pause xoff: unknown
TX pause xon: unknown
Interface config looks fine. Well i believe its AS number issue . You should cross check if BPG peering config AS is correct on both the sides
As per your config remote AS is 65100 and local AS is 65000
I cross-checked, configs on router and its correct....
Router side:
------------------
local-as: 65100
peer-as: 65000
tier-0 (NSX-T):
--------------------
local-as: 65000
peer-as: 65100
I am puzzled, why on tier-0 it shows BGP peer as "Administratively shutdown", even with UI shows as BGP enabled...
Even the ping between two peers are successful on both the side...
Any idea, under what all the scenarios, on nsx-t (tire-0 GW), it shows BGP peer as "Administratively shutdown"?
If possible please share the screenshots of Interface and BGP config from U.I as well.
Thanks for looking into this issue... Please find the details below:
Note: I tried with BFD enabled and disabled, in both case, BGP neighbor shows as "Administratively shut down"
With different VIew:
------------------------------
Do you have a single Edge node in the Edge cluster where this T0 is on?
Yes, currently I have single Edge Node in the Edge cluster. Will having two edge-node in the same cluster will solve the BGP - "Administratively shut down" issue?
Thanks.
It should work even with a single edge node in the cluster.
There is an option to administratively disable a BGP neighbor and it seems that it was switched on. This is the only way I found to reproduce what you have. This switch is only available in the Advanced Networking and Security (NSX 2.5 and below) or on Manager mode on NSX-T 3.0+.
The steps below were tested on NSX-T 3.0:
If you do not have the Policy/Manager button in the upper right corner go to System > User Interface Settings (Last option) and Toggle Visibility so that it appears.
Then go to Networking (click and make sure the Manager option in the upper right corner is selected)
Click on Tier-0 Logical Routers > Click on the T0
Under Routing > BGP > select the neighbor in the lower part of the screen and click on EDIT
There will be an option to change de Admin Status > Change to enable
Hope this helps
Thanks for looking into this issue... yeah, i have verified the same on UI and the BGP neighbor admin-status is "Enabled:... Still no luck... on the CLI it shows BGP "Administratively shutdown"...
Please find the details below:
Does this related to any physical NIC supportablility? I tried with both 10G and 1G physical NIC too...
But, i am able to successfully ping between the neighbor router and Tire-0 interface.
With different VIew:
------------------------------
Does your edge node have any alarms?
Sorry for the late reply. I am seeing the below behavior:
Please let me know, if there is anything I am missing during my config? I am not sure, why my BGP is "Administratively shut down", as soon as I attach VLANs (either to Tire-0 or Tire-1 or VRF)
Note: I have VMs attached to VLAN segments.
Case-1) If I do not attach any VLAN segments to either Tire-0 or Tire-1 or VRF, then BGP session is up between Tire-0 and external-router.
Case-2) If I attach VLAN segment to any node (either Tire-0 or Tire-1 or VRF), then the BGP on Tire-0 goes to "Administratively shut down"
Note: I have NOT configured BFD...
Reg Alarms, I see the below in the NSX-Manager:
==========================================
1)
Routing Routing Down nsxt-edge1
Open
Description
All BGP/BFD sessions are down.
Recommended Action
Invoke the NSX CLI command `get logical-routers` to get the tier0 service router and switch to this vrf, then invoke the following NSX CLI commands. 1. `ping <BFD peer IP address>` to verifyconnectivity. 2. `get bfd-config` and `get bfd-sessions` to check if BFD is running well. 3. `get bgp neighbor summary` to check if BGP is running well. Also check /var/log/syslog to see if there are any errors related to BGP connectivity.
2)
Infrastructure Communication Edge Tunnels Down nsxt-edge1 nsxt-edge1
Open
Description
The overall tunnel status of Edge node 01054703-43cc-4348-93b8-be2c9d38aded is down.
Recommended Action
Invoke the NSX CLI command `get tunnel-ports` to get all tunnel ports, then check each tunnel's stats by invoking NSX CLI command `get tunnel-port <UUID> stats` to check if there are any drops. Also check /var/log/syslog if there are tunnel related errors.
3)
Routing BGP Down nsxt-edge1
Description
In Router 66e152c1-606c-49e8-a89a-6c25e46fea9a, BGP neighbor ip_address:192.168.100.1 is down, reason: Network or config error.
Recommended Action
1. Invoke the NSX CLI command `get logical-routers`. 2. Switch to service-router ed9bf441-e57f-4ce7-a69a-26cec15fa5cf. 3. Invoke the NSX CLI command `get bgp neighbor summary` to check the BGP neighbor status. 4. Check /var/log/syslog to see if there are any errors related to BGP connectivity.
Are you attaching VMs to the same VLAN as the uplinks of the Tier0? Please send us some screenshots of your Edge node config.
When I create VM on vcenter, I use the segment (overlay-segment). Pls see the below screen-shot for "app90" name...
As soon as I create VM with "app90" as network adaptor (and attach to VRF), then BGP goes to "Administratively shut down".
I am attaching the VM to the segment (overlay-segment).
vlan-segment is attached to the Tier-0 interface (and NOT creating any VM on this vlan-sgment). This is just for uplink (connection to router for BGP session)...
Please find the config below and let me know, if you need more details:
==========================================================
overlay-segment:
---------------------
vlan-segment:
------------------
Tire-1:
------------
Tire-0:
------------
VRF:
-----------
overlay-segment:
---------------------
vlan-segment:
------------------
There are some problems with the screenshots that are not showing.
Please don't forget to send the screenshots of the Edge node configuration and status (System > Fabric > Nodes > Edge Transport Nodes).
What was the resolution to this problem? We have the same issue in one environment (of 5 built).
For us the issue got resolved by making sure all the Tunnels for Edges are in UP state in NSX-T Manager