Fixed it. In case anyone comes across this, the solution was to do the following:
- Remove the host from the distributed vSwitch
- Disable the host in vCloud Director
- Put in maintenance mode and remove from the DRS cluster
- Remove from vCenter entirely
- Unprepare host in vCloud Director
- Add host back to vCenter, the DRS Cluster and the distributed vSwitch
- Take out of maintenance mode
- Prepare host in vCloud Director
Basically burned it all the way down from vcd and vCenter. The step I had missed was doing the unprepare when the host wasnt in the vCenter inventory.