Hello Judgewooden,
I'm no ESXi expert but I'm currently investigating the optimal server/gaming setting for me.
I've got two Fury X in my system and I'm able to do PCI Passthrough for both of them.
I don't think, that you need a dedicated ESXi Graphic to get the host running.
I would remove the R5 230 and place the Nano in the first PCIe Slot.
Setup your ESXi and activate Passthrough for the Nano GPU (did you activate any virtualization option in BIOS?)
Create new VM for Windows 10, first without any additional hardware.
Install Windows -> do all Windows updates -> install VMwareTools -> add PCI devices (GPU/USB/etc.)
After installing AMD driver, the system should work flawless.
For me, no crimson 16.x driver is working, while installation the display goes black and the whole PC needs to be hard reseted.
I'm currentlyy going with crimson 15.12.
Best regards