There is indeed a problem in the Get-EsxCli when trying to use the 'remove' method.
But Alan shows the way around this in his Removing the vCD Agent from hosts post.
You have connect to the ESX(i) server instead of the vCenter.
Blog: lucd.info Twitter: @LucD22 Co-author PowerCLI Reference