Two bridges with two ESXi and with one logical switch?

[IvarHome]

Hi, can I have 2 ESXi, in each Edge with bridge. And both bridges connect to the same logical switch. I just want all VM-s in ESXi-1 get output through ESXi-1 and all VMs in ESXi-2 get output through ESXi-2.

[IvarHome]

And how the infrasture must be then?

[chrisgnoon]

I think we need to start with, what infrastructure do you have, because anyone can give you a good design with an unlimited budget.

How many ESXi hosts do you have?

Are they currently all in 1 cluster?

How much compute (RAM/CPU) do they have? (roughly or an average)

What is the utilisation of the ESXi hosts compute?

How many NIC's does each ESXi server have?

What switches do you have?

How many switches do you have?

What is the maximum port speed?

What is the utilisation of the network links?

What type of storage do you have?

[IvarHome]

>>>How many ESXi hosts do you have?

8 hosts

>>>Are they currently all in 1 cluster?

For NFS I have one host in cluster and soon I put also other (to other cluster, also for VXLAN). I just need Edge full featured firewall for one special operation, additionally for Sophos and Palo VMs. But DRS and HA I dont need. Instead I use Zerto replication (actually it dont need cluster).

>>>How much compute (RAM/CPU) do they have? (roughly or an average)

Intersestin question. I now calculate zum. CPU - 57 GHz  RAM - 124 GB.

>>>What is the utilisation of the ESXi hosts compute?

I dont know. I think they usu regularly half of the sesources, not more, I have reserv.  Mostly they run firewalls, different kind of enterprise backup software products, some UPS automation and mailserver.

>>How many NIC's does each ESXi server have?

Almost all with 1 NIC, only one host have 2 NIC. Most of non compatible NIC-s with 3rd party made drivers for ESXi, some with compatible Intel NIC-s. I can buy Intel nic-s, but I dont see much need for this. Also some are barebons, with integrated NIC and no place to put more. Altough I heared in market there exist some USB-NIC adapters with extra drivers made to ESXi. But I dont see requirements for more NIC-s. I have all 1GBit network. And 10Gbit cost too much.

>>>What switches do you have?

I have 4 L2 managed TP-Link switches and one Mikrotik L2 managed SwOS switch.

>>>What is the maximum port speed?

1Gbit/s

>>>What is the utilisation of the network links?

It depends. Most of time there is no utilization at all. But when I make or delete snapshots, then iSCSI utilizes all 1Gbit/s

>>>What type of storage do you have?

Internal SSD, internal HDD-s, 2 x Qnap raid storages with about total 14 TB plus all internal storages (some 10 TB maybe).

[chrisgnoon]

So, given that information I would do the following.

Regarding NSX:

Prepare all hosts for VXLAN.

Deploy logical switches.

Deploy DLRs for routing.

Peer the DLR control VM with your FW VM used for breakout.

Able microsegmenation if applicable.

Migrate VLANs to VNI's/VXLAN and route accordingly on the FW.

Regarding the network interfaces and throughput:

Ensure shares are enabled on the 1Gbps ESXi host interfaces.

In this design you have local routing with the DLR's so this will save on East/West traffic.  The traffic that would originally route between VLANs via the FW, now only goes to the FW if it is leaving the data center.

Microsegmenation can provide you with layer 4 access rules if you need restrictions between the internal VLAN's/VNI's or between VM's in the same VLAN/VNI.

Enabling the shares means that while storage could use all the bandwidth, if production data trust to transmit, it will be allowed as part of the sharing algorithm (NSX not needed for this).

[IvarHome]

Thanx. I find out, maybe with "distributed firewall" I can block-allow VMs to traffic out from choosen bridge in choosen ESXi. But I wasnt tested it and have no idea. Example, in L2 rules, for outbound, "applies to" I choose edge or distributed portgroup. Source I choose vNIC or VM. And destination not important. For inbound traffic source-destination viceversa. Altough I have no idea do it works. Second problem is 1600 MTU. And this is big problem. I dont want to change my physical L2 switches configuration. This can upside down all other connections, including internet connection.  Also I dont like the L2VPN idea - it makes whole system too complicated, as I plan to install in most host only edge bridges, not edge firewalls. Bridge works in ESXi kernal level, the edge VM is some dummy vm. But edge FW need real ressources. This 1600 MTU requirement is unbelievable, its just isnt normal infrastructure. I have readed articles where are written that jumbo frames in most situations dont speed up traffic at all, instead can make problems. And of course I cant set this in all appliances, some hardware probably dont allow it at all, and also all VM-s too. Its crazy idea from vmware. I say - NO, stop this madness and instead develop normal network stack.

[chrisgnoon]

With the setup I proposed, you do not need DLR Bridges, unless you need the VMs to talk to physical devices.  If this is the case, surely not every VLAN needs VNI/VXLAN to VLAN translation, so just do it for the ones you need.

Position the DLR Bridge and the Virtual FW on different ESXI hosts to separate the virtual to physical traffic bandwidth.

The MTU of 1600 is only for VXLAN traffic, not anything else. It is only needed so the ESXi hosts can send VXLAN packets to each other (it needs this for the extra packet headers).  When the traffic moves from VNI/VXLAN to VLAN through the FW VM or the DLR Bridge the packet size will reduce to 1500 MTU again.  So internet access will be unaffected.

VMware aren't making it up as they go along, they are follow IETF frame work:

RFC 7348 - Virtual eXtensible Local Area Network (VXLAN): A Framework for Overlaying Virtualized Lay...

You can see other examples of the "mandatory" MTU of 1600 in Cisco ACI or EVPN utilising BGP (aka non vendor specific VXLAN).  Any vendor with a VXLAN product will recommend the same MTU and usually 9k.

[IvarHome]

Of course almost all VMs I have need connection to outside world. Example internet connection, connection with my desktop computer etc. And to allow this connection only through one host is too dangerous. Example host is down or physical network switch is down. Then I have problems. VMs must connect to outside world only through its own host.

Yes, of course 1600 MTU is between VXLANs in different hosts. But this is whole logical switch concept. There is no point to set up logical switches only in one host. VMs can in this case connect directly with Edge (when its FW edge and VMs can connect with it through portgroup) or with portgroups without logical switches.

btw I dont like Cisco-s at all, I also dont have any Cisco and never becomes until I alive . Cisco is maistream crap, not security brand like Palo or CheckPoint. For L3 and up firewalls I like PaloAlto and Sophos (I have those). For L2 firewalls I like Mikrotik RouterOS (also I have). For L2 managed switches I like Mikrotic SwOS and new RouterOS with switch capability and also TP-Link is enough good.

[chrisgnoon]

I thought you used FW VM's for your breakout and not Edges?  In either case, you can use HA or ECMP to have 2 FW breakouts for the DC, sat on different hosts for redundancy.

One of the main advantages of VXLAN is preservation (saving of VLAN's) as only 1 VLAN is used to transport all the VXLAN traffic.  Not to mention the 1000's of VNI's you can create in VXLAN (10k max in VMware).  I suspect this will grow with future versions.

I said before, not every software/tool is for everybody.  If you aren't convinced it is good or you believe you have a better way, then I suggest you design your infrastructure in the way you are most comfortable.  However, I believe NSX would benefit you, but I don't know the in's and out's of your infrastructure.

[IvarHome]

I can tell what was long time ago. When I first set up my network, usual VLAN-s usage wasnt enough for me. Example I have 3 points, A, B and C. A must communicate with B and C, but B and C must not communicate each other. Therefore I set for every devide personal VLAN number. So, ingress and egress goes through different VLAN-s. But ESXi. ESXi dont allow this. To set personal VLAN to every VM. Distributed switch dont allow to send out with one VLAN, but get traffic from selectively different VLAN-s. Its area where VMWare can develop in future versions. I tried it with help to port mirroring and portgroup rules, but unsuccessfully. Also NSX logical switches dont fix this A<-->B, A<-->C, B<-x->C. Maybe NSX global L2 firewall can (not Edge), but I havent tested it jet. But some times ago I find one other solution. Its little extreme. I installed new Mikrotik RouterOS (with integrated switch with full featured managed switch) VM. Used one portgroup as "uplink" (allow all tags). And other portgroups for connecting Mikrotik interfaces with VMs. And it works. It was missing layer in vSphere. Mikrotik works as managed switch. In Mikrotik Im able to send any input vlan tag to any choosen VM untaged. Also choose what VMs can communicate with each other (also ingress and egress in different vlans) and also let each VM to go out with its own VLAN. But there was limitation - ESXi allows only maximum 10 interfaces per VM (for Mikrotik example). Of course Mikrotik allows to make virtual interfaces with VLANs and still work as managed switch (Mikrotik is the most powerful L2 device), but from VM side ( Mikrotik-LAN<--->portgroup internal side<--->VM ) there is no possibility to set VLAN. 

[chrisgnoon]

NSX micro-segmentation could help with this, using either layer 2 or layer 3 firewall rules.  The distributed firewall places 1 small firewall on each VM vNIC and this restricts access in and out of the VM, this can be either layer 2 or layer 3 rules.  I recommend using layer 3 rules as they are easy to track and update.

[IvarHome]

I operate in L2 as VLAN itselt is L2. My L2 managed physical switch cant do nothing in L3 (altough its with some L3 features), thats why I operate in L2. Also L2 dont need IPs, routings, its more natural than L3 in communication or marsrute operations. But as I see this distributed L2 firewall cant operate with VLANS at all. I see it can put also not only into vNIC-s, but also into portgroups, I mean "applies to". Ok, I can use it to demand VM go output only through fixed host (in case of logical switches), but for VLANs it seems dont give any help. In theory, I can make different portgroups with different VLAN-s, with distributed firewall or by portgroups traffic rules itself, I can restrict outgoing only by one of those portgroups, inbound allow to all. But now I must connect all those portgroups with the same VM or logical switch. Outside NSX environment VM can connect only with one portgoup. And in NSX environment, the same, edge bridge can connect 1:1 relationship. Only edge full firewall can connect one VM or logical switch with many portgroups. But, its not L2, but instead L3. And I have nothing to do with L3. But all this refer to one missing layer in vSphere - its fully managed L2 and vlans capable switch. Switch must do: 1) PVID-s marking to untaged input 2) ports selection for every VLAN 3) output traffic to port taged or untaged by choice .....and....for even better switches 4) VLAN-s translation by rules. Example input is VLAN 5 --> output VLAN 6. I have Mikrotik physical switch "CSS106-5G-1S" ,it can do this by very dynamic rules conditions. New RouterOS have also integrated fully managed switch component (physical and virtual versions both), but without VLANs translation capability. But RouterOS can do additionally one very unique operation - it can do SNAT and DNAT for MAC-addresses. So, thats why I dont need Cisco, Mikrotik can do all this and more. 

[chrisgnoon]

I think the discussion is becoming quite circular, we keep getting back to a layer 2 bridges when there are other options.

Even though your network is mainly layer 2, the distributed firewall can help with access restrictions using layer 3 address:

Distributed Firewall

[IvarHome]

What you can do with L3? Regulate traffic between IP-s. What IP-s? I dont have IPs before I set them up. But before this, L2 loop kills everything in network, everything, every single device. The only solution is take out all cables and you dont know exactly what cables, because you even dont see nothing, you even cant access switches. Second, when you control only L3, this dont means there is no traffic. There is no traffic only for you, but all unwanted traffic still exist, even network viruses and DDoS. Third, VLANs  -  you even dont have connections between devices, L3 even dont have chance to control traffic.

[chrisgnoon]

This discussion is covering a number of topics.  I suggest if you want to discuss other topics like L2 loops, then you open more threads, so that more people get involved in the specific topic.

I think I have answered the original question of layer 2 bridges within NSX-V and provided a number of other solutions which you can use.  If you are not satisfied with the advice given, I suggest you contact your VMware account manager or support to discuss further options for your specific scenario.

[IvarHome]

But I dont have VMware account (I have only trial user account). Im pensioner, I dont work, I dont have company. But I write software reviews, to magazines, paper and internet ones. Im independent journalist, I dont depend from no any brand, Im incorruptible, but I write only about tested experiences without emotions. Compare different brands enterprise products. Mostly vmware, firewalls and backup-replication software. I just want someone from VMWare read this thread and think about this......And maybe in next version there it is.

[chrisgnoon]

They won't address this in NSX-V as it is end of development in 2021 (i believe).  Try looking at NSX-T.

Good luck with writing your article(s).

[IvarHome]

But NSX-T cost more. Companies in whole world dont have enough money. To develop only expensive stuff is not very trustworth business. Bankrupt is easy to come. I surely believe NSX-V stay.