Make sure first the PSC is powerdown and try by proper FQDN name.Mostly below error will come only if the IP or FQDN name failed to search..

or try this command with proper FQDN name /usr/lib/vmware-vmdir/bin/vdcleavefed -h -u [-w ] ( Refer VMware KB 2114233 )

VMware KB:    Attempting to join an Appliance-based Platform Services Controller or vCenter Server t...