I have run into the same issue I have even tried to disconnect physical connections and virtual connections then try to remove the Distributed Switch but get same error. Have you been able to resolve this?
