1) Are there any security concerns with having the VM's on the same L2 overlay even though they are protected by the DFW?
From my experience the way people discuss and think about security is from different angle - Physical/Virtual/Application are most common entities and based on the packet flow - Inter-Site, Intra-Site , DMZ etc security measures differs . So i'm not sure from which angle you are thinking . However checking the packet flow for ingress/egress at vnic level is certainly a granular level option and nsx micro segmentation does it extremely well irrespective of the workload characteristics. MY Feedback is understand the workload mobility and traffic flow and later configure DFW accordingly.
2) Are there any limitations by putting many applications in the same overlay? does this affect the edge services in any way?
There are no hard limitations per overlay network(VNID). If isolation is the key criteria traditionally we go with VLAN and here VXLAN is the best bet . I didn't get your second query - impact on edge ? What services are we referring here ? Ideally i don't recommend two many services(DHCP/VPN/LB/Routing) on same device and deployment option will be based on tenancy(Single tenant/Multi tenant etc) - In the both the cases there are advantages and disadvantages. If you look at below design , you can see two tenants connected to unique edge and establishing a routing pair with the upstream router while it fits for overlapping IP address and multi tenancy from a compute/storage perspective design demands more edges -so that way cluster sizing matters while the environment scale .
3) Concerned about overlay sprawl (1VM in its own network multiplied by hundreds ), should I be? best practices for server design are one one app per server, would similar follow with NSX, one app per overlay?
There is a significant difference in the way in which BUM (Broadcast, Unknown unicast, and Multicast)traffic is handled by NSX-v comparing with traditional flood&learn mechanism. Kindly read https://www.vmware.com/content/dam/digitalmarketing/vmware/en/pdf/products/nsx/vmw-nsx-network-virtualization-design-gui… page no:33 to understand network requirement and packet flow in each replication mode. My preference is Unicast & hybrid mode which is fine for small/medium/large data centers. Again i want to emphasize on one main point- overall cluster design(Number of cluster- separate/collapsed etc) , total number of site is also a deciding factor apart from considering the size of the environment,