I can't find a principle design how to integrate service provider managed private IP/MPLS VPN via Tier0 VRF with EVPN type5 routings and NSX own NAT/Firewall. All examples i've googled so far is to pass Tier0 VRF traffic to a standalone Firewall. Like scenario of using a NSX own firewall has not been thought through? Can anyone share their experience or some docus with this subject?
The basic requirements would be:
1) to get all MPLS routes into Tier0 gateway via single BGP session (no ugly outside vlan per tenant stuff)
2) firewall instance should see all dynamic MPLS routes and have configurable Internet routes (a 0.0.0.0 or more specifics)
3) firewall instance should do NAT and have its own public routable prefix that can be further distributed between servers/services behind it.
4) firewall instance should have ipsec tunnel capability with dynamic routing (aws and azure etc integration), having BGP makes life so much simpler.
The first part was amazingly easy, configure BGP with EVPN family between T0 and PE router. VRF needs to have matching VNI and route-targets with PE and you are done. No more bring every VRF into NSX via separate vlan and BGP session.
Now comes hard part, how to get internet through NSX from that VRF.
--- Try1 ---
With two tier model the most logical would be use a Tier1 for firewall for NAT and firewalling. Attaching Tier1 to T0 nsx autocreates default route, so i have to attach it to internet side. Basic topology looks like:
internet --- [T0-inet] [T0-vpn-vrf1] --- evpn_type5 remote offices
(auto segment) (separate service segment)
\........ [T1-inet-vrf1] ......./
This setup will not work:
- it is unsupported? Officially T1 cannot be attached to multiple T0. Well you can configure it, i can get basic internet from remote office via T1-inet-vrf NAT, but NSX strangely crashes on routing changes.
- static routing on T1 and T0-vpn-vrf1 side - i can route private supernets there, but there are lots of clients who have some public ip addresses inside office vpn. It is a management nightmare to make static routings for all public ip addresses inside private vpn!
--- Try2 ---
Do nat and firewalling directly on the T0-vpn-vrf
internet --- [T0-inet] ----- (separate service segment) ----- [T0-vpn-vrf1] --- evpn_type5 remote offices
- T0-vpn-vrf1 has dynamic routing from "remote offices" side and if needed you can configure BGP on ptp between T0-inet and T0-vpn-vrf1 or some other segment.
- i can route public subnet from T0-inet towards T0-vpn-vrf1 for tenant services. No this auto distribute ip/subnet stuff that T1 does, that allows T0-vpn-vrf administrators mistakes to propage into global routing. Whatever subnet i give out to that tenant it is statically routed from T0-inet, if T0-vpn-vrf1 admin configures wrong public ip, it will mess up only that particular tenant. I guess with outside tools and my own provisioning via API i could mitigate this with normal T1 setup as well.
- T0-vpn-vrf has a interface where you can make separate "internet in" rules, sort of like "internet zone". With T1 gateway firewalling you have to know exactly what side a ip resides, internet or office and make source/destination rules accordingly. With "internet in/internet zone" you can generalize rules with "any" in most places. Also i can make "internet in" default drop rule and stop that stupid "icmp unreachable" answers from NSX to the port scanners in internet. Its like nr1 thing that every security expert says, harden internet side, with default config NSX is happily reporting everything back to internet.
Minuses i've found thus far:
- separate T0-inet and T0-vpn. Packets to outside network have 2 hops, all cannot be done inside one T0. I tried to route "T0-vpn-vrf1" back to "T0-vpn" global table. It worked, sort of, i could get internet if i used NAT ip from connected service segment ptp, but routing additional public prefix made T0-vpn continuously crash. That's why i chose separate T0-inet.
- i cannot place IPSec into T0-vpn-vrf1, like why this limitation? I can place it on T1-vpn-vrf1 though, it messes up the design logic that all outside stuff is configured on T0-vpn-vrf1. Now i cannot give T1-vpn-vrf1 control fully to a servers administrator. Full control as to say to server admin that you do all service firewalling on T1 as you see fit and network guys will do all outside connectivity stuff for you on T0-vpn-vrf.
Hi Rainnomm. I kinda have the same issue in my hands.. 2 VRF's, one for Internet and one for MPLS/Remote Office.
And since i'm going to be needing NAT on one of the T0, i cannot use the VRF Lite Architecture because it does not support NAT between VRF's.
What i'm planning/designing is a BGP Peering between both T0's using an internal overlay segment.
One of the T0's is going to be in Active/Standby for NAT purposes and the other one will be Active/Active.
I think i'm matching your "Try-2".
I ran some testes in a Lab environment and so far i think it is both supported and fully functional.
The next phase will be performing some tests with the customer to assure the NAT is working properly.
How did your deployment work out?
It's good to see someone else exploring NSX-T in other complex scenarios.