VMware Networking Community
IvarHome
Hot Shot
Hot Shot

Two bridges with two ESXi and with one logical switch?

Hi, can I have 2 ESXi, in each Edge with bridge. And both bridges connect to the same logical switch. I just want all VM-s in ESXi-1 get output through ESXi-1 and all VMs in ESXi-2 get output through ESXi-2.

Reply
0 Kudos
36 Replies
noonchris
Contributor
Contributor

If I understand correctly.

You have VM's on ESXi 01 and you have VMs on ESXi 02.  These VM's all connect to 1 logical switch.  You want to deploy 2 ESG's, 1 on ESXi 01 and 1 on EXi 02.  You then want VM's on ESXi 01 to leave via ESG 01 and the same concept for the 02 nodes.

I can only think of one way this would be possible.

You deploy 2 ESGs and use affinity rules to attach each ESG to the respective ESXi host.  Configure 1 gateway on ESG 01 (e.g. 192.168.1.1) and ESG 02 (e.g. 192.168.1.2).  Then for the VM's that sit on ESXi 01, set the gateway to 192.168.1.1 and .2 for those that sit on ESXi 02.

The issue with this is if the VM's move around using DRS, then the traffic will float between the 2 ESXi hosts as you have manually set the gateway.  This has issues with redundancy as you will map the VM's and Edges to an ESXi host.

Another method would be using 2 Edges in ECMP mode.  Some of the traffic will leave via the local Edge and some will leave via the Edge on the other host.  You could set the affinity rules to have the Edges on separate hosts for redundancy and allow DRS for VM's if and we required.  Although it won't guarantee the traffic leaves out of the same host's Edge.

The question is, why do you need to have the traffic leave the local ESXi host?

Reply
0 Kudos
IvarHome
Hot Shot
Hot Shot

But I dont use routing. I have local L2 subnets only everywhere. I use L2 bridge.

Reply
0 Kudos
noonchris
Contributor
Contributor

When you say layer 2 bridge, do you mean DLR bridge?  Or something else?

Reply
0 Kudos
IvarHome
Hot Shot
Hot Shot

Yes, Edge in bridge mode, processing in ESXi level.

Reply
0 Kudos
chrisgnoon
Enthusiast
Enthusiast

Are you using NSX-v or NSX-T?

Chris Noon | CCDP | CCNP | VCDX 289
Don't forget to mark as solved if your questions are answered.
Reply
0 Kudos
IvarHome
Hot Shot
Hot Shot

I think its V, its NSX manager v6.4.1.

Reply
0 Kudos
chrisgnoon
Enthusiast
Enthusiast

Ok.  NSX-V 6.4.1

You must be using a DLR (this is also seen in the Edge section of NSX). 

For bridging, the traffic is switch from VXLAN (Logical Switch) to VLAN by the DLR control VM.  So where ever this DLR VM is placed is where the traffic is bridged.  It is not possible to have 2 of these bridging VMs for the same VXLAN/VLAN conversion as you risk causing a layer 2 loop.

You can find the DLR control VM position in the host and clusters are of vSphere.

If you are doing this for 10/20/30 or more VLANs you could create 2 DLRs for bridging, have a DLR on each ESXi host and have half the bridges on one and the other half on the other. It's not perfectly balanced by might help your situation.

Chris Noon | CCDP | CCNP | VCDX 289
Don't forget to mark as solved if your questions are answered.
Reply
0 Kudos
IvarHome
Hot Shot
Hot Shot

I wasnt get, can I put DLR bridge in every ESXi? And let local VM output through local DLR? I understand other ESXi VM-s in the same logical switch can also use all DLR-s and then of course becomes loop. But I can make filters and block traffic by MAC-s. When I allow only MAC-s from VM-s local to this ESXi and DLR, then there must not happen loops. But do NSX at all allows to do this? I want every VM can communicate with any other VM in the same logical switch, but go out to vDS and VLAN only through DLR local to this ESXi. 

Reply
0 Kudos
chrisgnoon
Enthusiast
Enthusiast

This may be technically possible, but is dangerous (loops) and not recommended.  You will have lots of problems if VMs move to different ESXi hosts and if any links or DLRs fail.  

The recommendation would be to create multiple DLRs and bridge different VLANs using different DLRs.

E.g.

DLR 01 -

Bridges logical switch 10000 to VLAN 100

Bridges logical switch 10001 to VLAN 101

DLR 02 -

Bridges logical switch 10002 to VLAN 102

Bridges logical switch 10003 to VLAN 103

Then place the DLRs on different ESXi hosts.

What problem do you currently have? Bandwidth issues?

Chris Noon | CCDP | CCNP | VCDX 289
Don't forget to mark as solved if your questions are answered.
Reply
0 Kudos
chrisgnoon
Enthusiast
Enthusiast

You have asked a similar question here:

How to create multiple Control VM for multiple L2 Bridging

The same concept applies

Chris Noon | CCDP | CCNP | VCDX 289
Don't forget to mark as solved if your questions are answered.
Reply
0 Kudos
IvarHome
Hot Shot
Hot Shot

Cant connect to different VLAN-s, because they must reach the same destination. I dont usually move VM-s to different host. And example when I move, then nothing more happens that this VM still goes out through previous ESXi DLR. Im using now vDS-es. Some VM-s ports communicate only between themselves (firewalls in chain), some VM-s ports must reach common physical devices (hardware firewalls and servers), and some VM-s ports accessed through VLAN translation by Mikrotik SwOS. And then I have 24 port central managed L2 physical switch, that communicates all this VLAN-s army. Also, things go harder because not all ESXi-s are the same versions (hardware dont allow upgrade) and this means vDS-s are also different version. This all goes here little too much complicated. I have NSX ready and installed, only right now its turned off. I have plan to go to NSX and make traffic things little clear, let VM-s communicate without central L2 switch and all this VLAN-s army. Only I have one restriction - all VMs must be able to connect to outside world only through the same ESXi in which they located. Because I have only 1Gbit network and some ESXi-s with only one nic-s. When traffic starts flowing out not directly from local ESXi, but through next ESXi, then this puts my network into too pressure. Also its not safe when VM-s cant communicate with outsaid world locally - maybe next ESXi is down, maybe network is down. 

Reply
0 Kudos
noonchris
Contributor
Contributor

It sounds like you have a number of issues which should be addressed.

Going back to the original question, everything must leave the local ESXi host via bridging.

It seems in your environment, with all your caveats, issues and requirements, it would be best to place all VM's with the same VXLAN/VLAN on one ESXi host and create a DLR bridge for that VXLAN/VLAN.  Repeat this for the second ESXi host and continue to use the same DLR's for bridging and manually move the VM's on the same VXLAN/VLAN to one host or another.  However, this doesn't create a very robust design and I'm sure with any failure you will have big problems.

If you kept a more open design, with VM's floating between the ESXi hosts, you could use traffic shares to ensure no 1 type of traffic would dominate the entire link.

Are your 1Gbps links currently 100% utilised?

Reply
0 Kudos
IvarHome
Hot Shot
Hot Shot

But then why I at all need to use NSX? I can connect VM-s also directly to portgroups. I understand NSX point is only for connecting VM in different ESXi-s together without portgroups and VLAN-s. But yes, probably better is not to use NSX at all when I still need lots of connections between VM-s and outside world. To begin making MAC filters is also little tedious work. I still hope vmware in future itself allows choose what DLR each VM is allowed to use. Some rules. I dont understand why such rules already dont exist in NSX.

Reply
0 Kudos
chrisgnoon
Enthusiast
Enthusiast

If you are not using NSX routing (Edge/DLR) or more (Edge Load Balancing/VPN), then in your situation, perhaps it's not good to use NSX.

You seem to still be using layer 2 functionality, so stick to port groups until you are ready to move to the next stage.

Chris Noon | CCDP | CCNP | VCDX 289
Don't forget to mark as solved if your questions are answered.
Reply
0 Kudos
IvarHome
Hot Shot
Hot Shot

What is next stage? Do you want to say L3 is next stage to L2? Really? For routing I have Palo-Alto PanOS 9. For VPN I have Sophos UTM. DLR firewall is no any competitor for Palo-Alto. Smiley Happy

Reply
0 Kudos
chrisgnoon
Enthusiast
Enthusiast

Haha! L3 in NSX would be the next stage (microsegmenation, distributed routing etc), but you don't seem to be able to move to that so just keep using your physical devices and use NSX once you have upgraded your servers and network equipment.

You could save a lot of traffic going from the ESXi hosts to the physical devices if you did the routing using DLRs but that goes into a wider discussion of how you should be utilising NSX.

Chris Noon | CCDP | CCNP | VCDX 289
Don't forget to mark as solved if your questions are answered.
Reply
0 Kudos
IvarHome
Hot Shot
Hot Shot

Palo-Alto firewall and Sophos UTM are both VM-s (not physical). But yes, NSX advantage is very hard to understand. And Im not the only person. I like it, the whole idea of VXLAN virtualization, but still I see its not fully developed product jet. Example lack of rules to route VM-s L2 traffic to special bridge. Also 1:1 (logical switch to portgroup) bridge restriction is not justified. I hope some engineer from VMWare also read this thread and then think twice and next version becomes just great product........So, its simple. There must be done one simple automatic in L2 level. Each DLR bridge must have MAC-s filter, to allow or block VM-s. But it must be automatic. Admin only choose VM and NSX automatically makes MAC filters. When VM MAC changes, then it must automatically reflect in MAC filters table.

Reply
0 Kudos
chrisgnoon
Enthusiast
Enthusiast

I think there are a lot of advantages to NSX,  but it's difficult to take advantage of them if the infrastructure is not right for it.  Good products don't fit every situation.

Good luck fixing your design/situation.

Chris Noon | CCDP | CCNP | VCDX 289
Don't forget to mark as solved if your questions are answered.
Reply
0 Kudos
IvarHome
Hot Shot
Hot Shot

Advantages are never too much, with no any kind of product. Othewise there never happens next versions and developing becomes dead.

Reply
0 Kudos