PKS version 1.6.1, NSX-T version 2.5.0
Problem:
Policy API Categories and Section gerated by NCP by Manager API don't fit the right, recommended, needed rulebase strategy and the order is wrong.
NSX-T Distributed Firewall – Policy API versus Manager API
Implementation PKS with NSX-T
I had already written many distributed firewall policies and rules in Simplified UI (Policy API) as Infrastructure rules, Environment rules,... .
And I started to apply K8s Network policies and vice versa the PKS applies ("translate") them over NCP as FW sections and rules into Advance UI (Manager API).
The desired state of rulebase is following:
Ref. Kubernetes Network Policy in Kubernetes and VMware Enterprise PKS Networking & Security Operations with NSX-T Data Center post
Problem details:
The problem is with the order (sequence) of rules installed into resultant firewall rulebase consisting of FW configuration of both APIs. All is seen in the linear FW rulebase in advanced UI.
The problem is actually with the order of the Categories and Policies of Simplified UI and Sections of Manager API (Advance UI).
Found out order is:
PolicyAPI.Emergency Category.Policies with Rules
ManagerAPI.Sections with Inter+Intra Application Rules (generated by PKS-K8s-NetworkPolicies)
PolicyAPI.Infrastructure Category.Policies with Rules
PolicyAPI.Environment Category.Policies with Rules
PolicyAPI.Application Category.Policies with Rules
( PolicyAPI.Default Layer3 Policy Section – If Distributed Firewall Strategy <> None(my case) )
ManagerAPI.Sections with Cleanup (Default) Inter+Intra Application Rules (generated by PKS-K8s-NetworkPolicies)
ManagerAPI.Default Layer3 Section
But the desired rulebase design assumed and needs that all ManagerAPI.Sections are bellow at least Environment PolicyAPI Policies which contains:
- Mainly : Shared services and infrastructure rules, Dev-Test-Prod environment isolation block rules
- Optionally: PKS environments (DEV, TEST, PROD) infrastructure matrix
And now I see several ways or workarrounds how to use NSX-T with PKS and also manage Standard virtualized datacenter (VMs in vSphere).
But which is right and which is dead end?
A) Configure and manage all objects and firewalls in „legacy“ Manager Policy (Advanced UI)
B) Reinstall PKS into Simplify UI
C) Use mix of Policy and Manager API
D) Some how force Section sequence or Category priority
Moreover.
Now we use NSX-T SDN only under the PKS platform. But soon we must migrate current virtualized datacenter (VMs in vSphere) networked by NSX-V into NSX-T.
I cannot imagine that I will still have to stay in Advanced UI (Manager (API)), and besides other things, for example write automation by legacy imperative Manager API.
What is the right way or solution?
and many sub-questions and hopefully useful answers …
I would go with letter "c". Since Policy API is where VMware is going with since NSX-T 2.4, this way you use it with everything else except PKS, which still uses MP API. After PKS releases a version that supports Policy API you could then migrate so your whole environment is in Policy API.
Ok, so here's an important piece of information which is likely going to rain on your parade and thereby potentially alleviate your questions.
NSX-T as part of Enterprise PKS does not support you consuming it from other things outside of PKS itself and the workloads deployed by it. That means you're not technically allowed to migrate existing VMs into that environment and join them to NSX-T resources. Here's the source: https://www.vmware.com/content/dam/digitalmarketing/vmware/en/pdf/product/vmware-product-guide.pdf and important extract:
I would go with letter "c". Since Policy API is where VMware is going with since NSX-T 2.4, this way you use it with everything else except PKS, which still uses MP API. After PKS releases a version that supports Policy API you could then migrate so your whole environment is in Policy API.
Thank you for help. I guess everything fits.
We have purchased VMware Enterprise PKS with NSX-T Datacenter Advanced Edition.
Now the systems and servers are dedicated only for that.
And the issue is about priority(order) issue of Distributed Firewall Policy API Policies and Manager API Sections.
Note: The recommendation and I hope it make sense is to have only one SDN NSX-T in the datacenter (maybe over more datacenters).
And I am sure we will solve everything to be compliant we VMware licensing model. But it is not question now.
Thank you for your understanding and advise. This make sense.
But two related sub-questions:
- Do you think PKS will ever know Policy API?
- What is actually technical reason, differences of Simplified DFW Categories? Are they only cosmetic folders or do they have some useful features?
I can't say for sure, but since policy API is where work is concentrated I would say PKS should support it at some point.
The DFW categories are more of way of organizing the rules for an optimized and best practice way of using the DFW. This is part of the microseg strategy explained in VMware® NSX-T Reference Design, in section 5.4.
The C way has been selected.
And I wait for PKS and Polici API compatibility and then we will have to use rebuild all the NSX-T environment for PKS infrastructure, reconfigure PKS and redeploy clusters and all apps.