It is not only throughput that might make for a choice of a bare metal Edge. Some services like load balancing and VPN also impact the choice of the Edge. If you have high demands in these services bare metal edges might be necessary. Take a look at config max and you will see that the numbers for these services are quite higher on a bare metal.
The other design reason for using bare-metal edges is convergence time. If one edge fails, the other will take over in under a second (in the neighborhood of 900ms). If these edges are virtual machines the best they can do is sub-3 seconds.
Another good aspect is that I might be easier to integrate two BM edge node than a dedicated vSphere cluster to run VM-based edge nodes, in case you prefer purpose built vSphere clusters instead a collapsed cluster approach.
A dedicated vSphere cluster typically required some kind of shared datastore and when you deal with vSAN, your vSphere cluster requires 4 hosts (3 is the requirement, but 4 hosts provide you some operational flexibility). On the top, you need networks for vMotion, vSAN or IP storage etc.
In the case of BM edge node, you can easily use the local SDD in Raid 1.
And when you deal with BM edge nodes, it might provide you most likely an additional level of design flexibility to deploy these BM edge node as part of the Tier-0 Gateway in A/S mode, as your BM edge nodes typically deal with 40Gbps interfaces; this single 40Gbps interface might be enough for all North-South traffic. When your Tier-0 Gateway is run in A/S mode, you can enable Tier-0 stateful services as example an edge firewall. In case with VM-based edge nodes, I see very often these edge nodes are deployed in A/A respective leveraging ECMP to provide the required North-South bandwidth. As example to provide 40Gbps North-South bandwidth you might need 2 or more VM-based edge node in A/A mode. That design choice with A/A respective ECMP mode let you not run stateful service on the Tier-0 Gateway.
Hope this helps
thanks for the detailed reply! it helped a lot!
after you mentioned about the A/A mode of VM edge nodes which run ECMP and therefore do not run stateful services,
I started to check the document again and found that the installation guide (2.2 or 2.4) said "in active-standby mode the gateway can also provide stateful services".
is there any differences between my understanding and your experiences? or the installation guide is basically correct but the throughput would be affected by stateful services?
Hi niceguy001, when the T0 Gateway is running in A/A (or ECMP mode) for the Service Router (SR), you can run stateful services on the Service Router. We are able to run stateless services in A/A mode, as stateless Edge Firewall. Please add the link with the information about stateful. I assume it is a documentation bug. I will open a documentation bug.
NSX-T doesn't have a hardware VTEP support as of now. So for use cases like Overlay to VLAN bridging where the overlay workloads demands higher data transfers to an external storage on VLAN network, its good to do the bridging on baremetal edges leveraging DPDK acceleration.
Secondly, when u use NSX loadbalancers with SSL Offloading, its good to have baremetal edges as it supports higher TPS.
I suggest you read VMWare NFV 3.0 for a full picture:
Copied from this document:
Table 8‑1. Edge Node Options
Edge Node Type Use
VM form-factor - Production deployment with centralized services like NAT, Edge firewall, and load balancer. n
- Workloads that can tolerate acceptable performance degradation loss with virtual edges.
- Can tolerate lower failure convergence by using BFD (3 seconds).
- Lower cost options instead of dedicated bare-metal nodes
- Test proof of concept and trial setups.
Bare metal form factor
- Production deployment with centralized services like NAT, Edge firewall, and load balancer.
- Higher throughput more than 10Gbps.
- Faster failure convergence using BFD (less than 1 second).
NFV workloads mainly, mobile networks with real time high availability requirements.
I published a blog on NSX-T Edges Form factor comparison - Baremetal vs VM. I covered all the details I had, let me know if this helps.