change that line to be:

Connect-VIServer -server $server -NotDefault -WarningAction SilentlyContinue | Out-Null

That's what got it working for me.