I like the explanation available here: VMware NSX-V - Replication Modes , but there are many good resources about this on the Internet.
Replication in overlay networks come into play whenever we have traffic that needs to be flooded. In traditional networks this is native and occurs with BUM (Broadcast, Unknown Unicast and Multicast) traffic. For an overlay there needs to be some replication mechanism to achieve the same behavior.
Unicast is the simplest and scales well in most environments. In this mode NSX Controllers maintain and distribute the information necessary to reach endpoints in the overlay and whenever flooding needs to occur packets are replicated to each host participating in the overlay using unicast packets. In this mode no multicast addresses are necessary and there is no dependancy on the physical network.
In multicast the flooding is done through the physical network using multicast. This requires that the network be configured to support this, which includes protocols like IGMP and PIM, which is not simple and few customer attempt this.
Hybrid is a mix of the 2 modes that tries to simplify some of requirements of the physical network while leveraging some help from the Controllers. In this mode local traffic (between vTEPs on the same subnet) is flooded using multicast. A unicast replication occurs to a single host on every other subnet so the same local flooding occurs.
Generally speaking unicast is the only recommended mode. One important thing to note is that these different modes don't even exist in NSX-T, which scales to a thousand hosts using only unicast replication.