I think you both should use scopes in these rules.
First create a policy with VDI VM scope. The scope applies firewall rules under the policy to VDI VM's only. Use a segment or a vm name prefix as a filter for the group used in scoping. This policy is meant exclusively for egress traffic for the VDI VM filters.
Under this first policy just add one/two rules. Two for a blacklist or one for a whitelist. I'll use a blacklist as an example.
Then create a second policy with server VM scope. You can use group servers Y as a scope here. This policy is meant exclusively for ingress traffic for the server VM filters.
With the help of scope you can divide the DFW into different sets of rules, which is quite helpful in the IDFW case.