Help - NSX-V + Storage - (VLAN) Netapp/PureStorage - OCP 4 - DHCP/IP Pool - Port Group/LogicalSwitch

[felipecampos]

Hello,

I'm following the community for a while, it's my first question, so I'm sorry if it's not the right place to ask.

I installed OCP 4 (Openshift 4) with IPI (automated install) on VSphere.

It worked like a charm, I used DHCP 'cause OCP 4 IPI install uses DHCP (less problem, complex with RHCOS).

Now, we're trying to integrate our VLAN Storage to use in OCP 4. We have Netapp and also PureStorage.

Virtualization and Storage team suggested to not handle Storage traffic through NSX-V, they said that it has some data performance degradation, so they suggested to create a new VLAN and attach to our VMs, and our problem just starts.

Our concern is about NSX-V, we'd like to use ISCSI + NAS, but we think that DLR + ESG routing will degrade data performance between VMs and Storage.
So, because of that, do we need a VLAN routing external from NSX-V, I mean, our VM need to have some Storage data IP and same Storage gateway to have a directly route?

In OCP 4 with RHCOS they use Ignition to configure VMs generated dynamically, so it's hard to define static IP per VM.

So we'd like to use DHCP or something like that to connect to Storage network.

I don't know how IP Pools handle that, I suggest it to my teams (I don't even know if I'm right, but I tried some workaround), but I don't know yet how can I attach a network in our VM using IP Pools.

Some topics for my questions:

1 - Can I use DHCP with port group (VLAN X) and use it as network in my VMs do get dynamic IPs? Could it be a best practice to have more performance and not route through NSX-V (DLR + ESG), because it's Storage Appliance placed outside our virtualization (VSphere)? 

2 - If I define a IP Pool, do I need to create a port group to use in our VMs? Does it gonna works like DHCP (I need dynamic IP)?

3 - What should be a best approach and best practice?

I read everything in docs about IP Pool and DHCP.
I'm using DHCP + DHCP Relay with Logical Switch to deliver dynamic IP to my OCP 4 Nodes, it works great with Nodes communication and we install or reinstall fast an OCP 4 Cluster with that solution.

I looked for some docs in internet about how to use IP Pool in VMs (to my VMs get IP dynamically), I tried a lot, without success until now.

Thank you for your time.

Any suggestion is welcome.

Thank's,

Felipe Campos.

[Sreec]

Are you mounting the NFS/ISCSI to ESXi host and later using OCP Native Storage clustering?  This is one approach usually I have seen and it works flawlessly. Regarding the IP POOLS terminology in NSX-V, it's used for VXLAN VTEP assignment which is not required in your case as you are leveraging native CNI. IP pools in NSX-V  are not intended for Virtual Machine IP assignment. That being said you can use DHCP or Relay functionality which you have already mentioned in this thread.  The easiest option is to use the IP in the ignition file and ensure respective DNS entries are configured and inject into VM advanced settings.

https://access.redhat.com/solutions/5499911 

[felipecampos]

Are you mounting the NFS/ISCSI to ESXi host and later using OCP Native Storage clustering?

• We're trying to use Trident (CSI to Netapp Metrocluster) and also Portworx with PureArray.
• https://netapp-trident.readthedocs.io/
• https://docs.portworx.com/cloud-references/auto-disk-provisioning/pure-flash-array/
• Both needs IP configuration to route data traffic.

https://access.redhat.com/solutions/5499911 -> Thanks for the link, I took look, but our VMs are created dynamically in IPI installation and I don't generate ignition files before installation, I'm using RHACM.

After installation, it's a MachineConfig that I need to create (but it's for all my workers / any node that I need).

I'm aware that I can use ESXI host to use OCP ODF (or OCS), but until now we would like to explore better Trident and Portworx.

Portworx has a kind of limits, if I use VMDK or RDM or something similar, it's limited to 5 nodes / 5TB.

If I use ISCSI with Pure FlashArray, I don't have limit.

That's because I'm trying to attach ISCSI.

Also, Trident doesn't work with VMDK / RDM, it's CSI.

My DHCP Server / Relay are from NSX-V, Can I use NSX-V DHCP with Port Group?
Let me try to explain.

I have a port group (DVPG ? Distributed Virtual Port Group ?), it has a VMKernel Ports (ESXI Host route/connect to storage/Netapp/PureStorage).

If I create a DHCP Server to those IPs, can I use in my VMs? Is it going to work?

Example:

VLAN Storage: 10.0.1.0/24

VLAN Gateway: 10.0.1.1

ESXI Host with Storage IP: 10.0.1.2

Create a DHCP using IPs from this Port Group: 10.0.1.0/24 -> 10.0.1.10-10.0.1.120

My VM/Node 1: DHCP deliver an IP from the list, example: 10.0.1.30

My VM/Node 2: DHCP deliver an IP from the list, example: 10.0.1.31

Is it going to route through VM/Node -> DLR -> ESG and then to physical storage or is it route directly from VM/Node -> ESXI Host to our physical storage?

Thank you in advance.

[acchomearc]

thank sir !

[felipecampos]

Hello,

Just to update about the topic.

I'm trying with multus + IPAM (whereabouts).

I'm asking also at code.purestorage in slack if they can help.

Thanks,

Felipe.

[felipecampos]

Hi,

Just to update about the topic.

I left multus + IPAM (whereabouts) for now.
I tried with nmstate, it worked with static IP.

I think I'm going to keep the nmstate configuration.

If anyone needs some info, I searched about it throughout OCP 4.8 docs.

https://docs.openshift.com/container-platform/4.8/networking/multiple_networks/understanding-multipl...

About nmstate: https://docs.openshift.com/container-platform/4.8/networking/k8s_nmstate/k8s-nmstate-about-the-k8s-n... 

Some example https://itnext.io/security-zones-in-openshift-worker-nodes-part-iii-network-configuration-3a887854a4...:

apiVersion: nmstate.io/v1alpha1
kind: NodeNetworkConfigurationPolicy
metadata:
  name: worker2-extaccess-static
spec:
  nodeSelector:
    kubernetes.io/hostname: worker2.ocp.136.243.40.222.**bleep**.io
  desiredState:
    interfaces:
    - name: ens10
      type: ethernet
      state: up
      ipv4:
        address:
        - ip: 192.168.210.62
          prefix-length: 24
        dhcp: false
        enabled: true

Thank you.