a little bit confused about the correct MTU if both the overlay and vlan uplink TZs are using same uplink profile for the edge VM.
In my experience I always use separate uplink profiles for overlay TZ and vlan uplink TZ respectively therefore the MTU will be different, such as 1700 and 1500; now I discovered that there's a design which is the overlay and uplink TZs using same uplink profile on the same hostswitch for the edge VM, and using all active fp-ethx NICs .
My question is, if the underlying infra(physical switch, VDS...etc) using MTU 9000 jumbo frame, then should the edge VM uplink profile MTU use 8900 or 9000 in this single-uplinkprofile-for-overlay-and-uplink-tz design?
I have been using 9000, but then we have had slightly larger MTU in the physical fabric, like 9216 bytes. You need at least 1600, but 9000 is recommended for future proofing.
The uplink profile should match or be less than the physical network. For Edge you should take care of having the MTU of the T0/T1 gateways 100 bytes less. The same is valid for VMs in the overlay.
So in the NSX-T 3.0 environment in vSphere, i can simply set VDS MTU to 9000 and just make the ESXi and edge VM nodes use this MTU 9000 globally, and there won't be issues?(assume that the physical network had MTU 9000 configured.)
I had experience on using max. MTU 1700 for NSX-T successfully and worked just fine but not sure about whether setting edge VM's vlan TZ uplink profile(for N-S external traffic) MTU to 1700(same as overlay TZ's) cloud impact the whole NSX-T network or not.
With NSX-T 3.0, you are able to create on edge profile for the TEP and Uplinks (Single N-VDS deployment). And when we said TEP so it is required more than 1500.
So in your case, you are using 9k on your physical switches. so the edge profile will be ok with 9k.