I have tried it here and it works fine. Make sure the fqdn is getting correctly translated to an IP address in the DFW dataplane. Also enable logging for the rule to see if it is correctly matching the desired rule.
Useful commands in the ESXi host the VM you are using to test:
summarize-dvfilter | grep -A 2 VM_NAME
This will give you the filter name that should look something like nic-xxxxx-ethx-vmware-sfw.2
With this you can check rules applied to the VMs filter with:
vsipioctl getfwconfig -f nic-xxxxx-ethx-vmware-sfw.2
vsipioctl getfqdnentries -f nic-xxxxx-ethx-vmware-sfw.2
And check logs at /var/log/dfwpktlogs.log
A good way to see if your test is not matching the right rule is to log all FQDN rules you are testing and under those create a permit any rule for that test VM. This way if the rule does not match the reject rule it should probably log in the permit rule.
The most probable reason for FQDN rule not working is that the DNS snooping did not find out the IP address of that FQDN. This usually happens if the host caches the DNS entry and thus there is nothing to snoop. It is good to flush dns cache to make sure that all DNS requests can be snooped and the firewall can learn the IPs for each FQDN.
