These articles explain very similar scenario, i.e. how does VM1 on H1 learns the MAC Address of VM2 on H2:
https://telecomoccasionally.wordpress.com/2014/10/27/nsx-v-under-the-hood-vxlan-arp-suppression/
http://www.routetocloud.com/2014/12/nsx-v-ip-discovery/
Step1: If Silent VM2 has not sent any Arp Query requests for its default gateway, or any DHCP requests, then the SwSec dvFilter cannot learn and send the IP Address to add the Controllers ARP Table. For this reason the first arp request of H1 to Controller gets ff:ff:ff:ff:ff:ff (I don't know) response. But during this request Controllers ARP table is populated with VM1 Arp Entry through a VM IP Update Packet.
(If the replication mode is Multicast, this step is already bypassed since thhere are no controllers, and the ARP Request is sent through a Multicast IP destination address that all VTEPs with members on that VXLAN has registered to)
Step2:In that case H1 has no other choice to consider the ARP as an ordinary BUM packet, Replicating this Broadcast as Unicast Packets to all other VTEPs that has members on the VXLAN. (Hybrid Mode offloads to Pysicala as Multicast similar to Multicast Mode). Since H2 VTEP receives this Packet and it knows that the MAC Address belongs to VM2 locally, the packet is forwarded to VM2. VM2 sees the Arp Query to itself as well as IP and MAC Address of VM1 already in the packet, and sends a Unicast Arp Reply to VM1. H2 looks for the MAC Address of VM1 and learns from the Controller that VM1 is behind VTEP1 on H1. During the Arp Reply, SwSec of H2 sends VM IP Update packet to the Controller. At this phase the ARP Table of the contoller has both VM1 and VM2 Entries. So if VM3 on H3 needs to learn MAC address or VM1 or VM2 may learn it from Controller directly, reducing the broadcast, thus the CPU Load on Hosts. By this way the Scalability of the VXLAN increases, very Efficient mechanism for Small to Large scale environments.
Don't know when it may be useful to disable ARP Suppression, but is enabled by checking "Enable IP Discovery" during the Logical Switch VXLAN creation. If not enabled there will be more ARP broadcasts in addition to other broadcast traffic:
One Important point may be the DLR Arp Resolution Process. Could not find how it works on the documentation, but Since DLR is connected to vdrPort which has no SwSec Filter, so Controllers can't learn about LIF Interfaces of DLR Instances and may return ff:ff:ff:ff:ff:ff as in Step1. But since on every host there is a DLR instance running this ARP Response should be from this Local LIF interface, and Controllers may not have a Role about Resolving the MAC Address of the DLR LIF by VMs.
But if a packet has reached to the DLR on H1 from VM4 (again on H1, but on another VNI such as VXLAN-5002) which is destined to VM1 on the same host (which is on VXLAN-5001 VNI) then DLR may resolve ARP locally. But if the same VM4 needs to send a Packet to VM2 on VXLAN-5001 on H2, then it may ask the Controller to Suppress the Broadcast, but not sure if it uses this mechanism or sends as BUM traffic without asking the Controller first. Not having a SwSec dvFilter prevents registering the Mac of DLR LIF to Controller through VM IP Update, but it may or may not prevent learning the VM2 Mac address directly from the Controller without asking every host for this MAC address that is already on the Arp cache of the Controller.