In NSX DFW all rules are published to ESXi hots based on source/destination IP. If you write a rule where source/destination is VM name/cluster/port-group etc then NSX manager needs to figure out the IP/IPs to which this rule has to be applied. Without Vmware tools the vCenter cannot figure out IP and that info cannot be passed to NSX manager, hence NSX manager doesnt publish that rule to esxi host.
To fix it you can use DHCP snooping and ARP snooping to detect VM IP.
Refer to below article for more details.