Did I get it right that you are directly connecting the devices to physical ports on the server? Like there is no switch between your devices? (not sure if the virtual switch let it through - I'm really thinking if I have seen smt like that, but no luck).
If the answer is yes, then the question would be why you need VLANs, when you have the devices directly connected? The devices connected directly would have to understand 802.1Q (VLANs) too.