Will communication with VTEP and NSX controller (such as VNI-VTEP report) flows management vmkernel network?

[pcparts001]

Hello

I have question regarding to communication with VTEP and NSX controller.

Will communication with VTEP and NSX controller flows within management vmkernel network?

Or will it flows within VTEP network?(VXLAN network?)

I believe it is via management vmkernel network, because ESXi host with NSX controller do not always need to have

VTEP(or logical switches) segment.

I thought it might flows within NSX controller segment if it is independent from normal(vCenter and ESXi host communication)

management vmkernel, but I think there is restrictions which controller network must be in same layer 2 segment.

I tried to find document but none of document clearly states which network port group will transmit VTEP and NSX controller flows.

I'll be very happy if you experts give me answer.

BR

[Richard__R · Accepted solution]

As far as I'm aware you're correct - this traffic goes over the host Management network and a routed connection between Controllers is not currently supported with NSX-v.

[pcparts001]

Thank you for your response.

I want to have environment where I can packet trace and see the flow, but unfortunately I don't have it.

[grosas]

Hi pcparts001,

This is actually covered by a note in the documentation.  "The IP address of the controller must be reachable from the NSX Manager and the management network of the vSphere hosts communicating with the controller."

Reference the NSX 6 Documentation Center

NSX Installation and Upgrade Guide > Installing NSX Components > Prepare and Enable Clusters for Logical Switches

The VTEP is a component of the VXLAN data plane.  There is absolutely no requirement for the network used by the VTEP VMKernel interfaces to be able to reach the NSX Controllers.  For a flow from ESXi Management network to NSX-Controller network, you need IP reachability.

" I think there is restrictions which controller network must be in same layer 2 segment. "

#correction

For communications from NSX Controller to NSX Controller, they do not have to be in the same L2 segment.  It's probably not a bad idea for the cluster to have adjacent members, but the configuration UI does not impose restriction on the Port Group or IP Pool selection.

[rbudavari]

That is not correct - NSX Controllers do not need to be in the same L2 segment and can use different dvPortgroups and IP Pools if required.

[pcparts001]

grosas

Thank you for your response.

OK.  So, VTP-VTP flow includes "only" dataplane data.

And control plane data flows between controller/manager/vsphere host flow.

I'm going for VCIX-NV and your post helped me alot.

[pcparts001]

rbudavari

Well the document "NSXvSphereDesignGuidev2.1.pdf" page 72 states

The ESXi hosts part of the management cluster do not normally require to be provisioned for VXLAN. If they are

deployed across two racks (to survive a rack failure scenario) they usually require extending L2 connectivity

(VLANs) across those racks for management workloads such as vCenter Server, NSX Controllers, NSX Manager

and IP Storage. Figure 87 highlights the recommended (even if not the only) way to provide L2 connectivity across

the management racks.

[rbudavari]

The section you have quoted from the NSX Design Guide is making a different point. As the VMs in the management cluster are on VLAN backed dvPortgroups, if this management cluster is extended across two racks for greater resiliency then L2 needs to span those racks. This is because VMs could vMotion between ESXi hosts in different racks, or vSphere HA can recover VMs in a different rack so the same VLAN needs to be available.

It is not the same as all 3 NSX Controller nodes needing to be in the same L2 segment, which is not a requirement (particularly as it is possible to deploy NSX Controllers in the edge cluster also).

regards,

Ray

[pcparts001]

rbudavari and grosas

Thank you for your great support.

But, I'm still confused.. (Last month I got CCIE certification and currently I'm aiming for VCIX, and my current hurdle is that thre is so many management segment that I need to be aware...)

I don't want to make thing complex, so I'd like to divide the discussion based on IP segment as below.

1.Controller management IP addresses

Where I'm reffereng:[Network & Security] -> [Installation] -> [NSX Controller (+ sign)] -> IP Pool

All 3 controller should be in same L2 segment?(should use same IP Pool?)

->No, it can be in different L2 segment, and can use different IP Pool.

Is this IP segment must be in same segment as NSX Manager IP?

->No, but it must have IP reachablilty with NSX Manager IP.  If I have different requirement other than Controller<->Controller communication

    such as vMotion between the rack, then I may need same L2 segment design, but if I focus just for Controller<->Controller communication

    Contrllers do not need to be in same IP subnet.

2.VTEP IP addresses

Where I'm reffereng:[Network & Security] -> [Installation] ->[Host Preparation] ->[Cluster Configure] ->[Configure VXLAN networking]

All VTEP should be in same L2 segment?(should use same IP Pool?)

->Ofcourse no, that is the main reason why we use VXLAN.

Is this IP segment must be in same segment as NSX Manager IP?

->No, there is no relation between VTEP IP segment and NSX Manager IP or Controller IP.  Only dataplane will flow betwen VTEP<->VTEP.

3.NSX Manager IP address

Where I'm reffereng:[While installing NSX Manager OVF] -> [Network 1 IPV4 address]

All NSX Manager should be in same L2 segment?(should use same IP Pool?)

->Only one NSX Manager is supported so this discussion is not relevant.

Is this IP segment must be in same segment as NSX Manager IP?

->No, there is no relation between VTEP IP segment and NSX Manager IP or Controller IP.  Only dataplane will flow.

If I want to, I can make [Controller management IP addresses],[NSX Manger IP address],[vCenter IP address],[ESXi host management vkernel IP address] in same IP segment.

is above information correct?

(I didn't mention Edge IP address and DLR IP address because it might make confusion even more...)

[Richard__R]

That looks like a good summary to me but would be interested to hear others input. Apologies for confusing the issue previously as I wasn't aware that it was supported to place the different Controller nodes in separate subnets.

[grosas]

pcparts001pcparts001

That reads pretty accurate to me.  Sounds like you're understanding it more than most. :smileysilly:

[pcparts001]

Thank you for your response!  Now I'm pretty confident.

[pcparts001]

No need for apologies.  In fact,,,,, to be honest,,, my confusion and misunderstanding started exactly from the site you mentioned.

"His" explanation certainly lead misunderstanding to newfaces of NSX...

Well, my concern have been solved so I would like to close this issue.

Again, thank you guys for helping me out for long time.

[rbudavari]

Actually you can do exactly this - individual Controller nodes from the same NSX Controller cluster can be in different L2 Domains. Hope this is clearer

[scottmwa]

This is perfect, thank you!