I have NSX environment, version 6.4.3, Unicast set on the transport zone. Is it possible to allow multicast only for VMs connected to the same logical switch? So these VMs should be using multicast between each other.
Transport Zones must have a multicast address range configured. See Assign a Segment ID Pool and Multicast Address Range in the NSX Installation Guide.
IGMP configuration must be the same across the Edge Services Gateway and the Logical (Distributed) Router.
Enable IGMP snooping on the L2 switches to which VXLAN participating hosts are attached. If IGMP snooping is enabled on L2, IGMP querier must be enabled on the router or L3 switch with connectivity to multicast enabled networks
Does it mean I can enable it only on the Transport zone level?
You could change the replication mode per logical switch as well and it will work as long underlying network requirements are met based on the design. But understand that replication mode benefit is primarily for head end replication and if you leverage multicast, controllers are not used for logical switching.
Layer 2 multicast,--->BUM Replication to VTEPs on the Same Subnet,
Layer 3 multicast--> BUM Replication to VTEPs on a Different Subnet