How do I troubleshoot a slow connection with Connect-VIserver?

[shawn149]

Hello,

When I connect to my vSphere server with the PowerCLI cmdlet Connect-VIserver, the connection takes anywhere between 10 to 25 seconds. To me, that feels like it takes too long.  What is an acceptable time for the initial connection?

I investigated if I could switch to the RestAPI, but I can't seem to find any of the performance metrics.  Are they even available via RestAPI?

In any case, does anyone know what I could do to speed up the Connect-VIserver cmdlet or how to track down what might be the problem?

Thanks!

[LucD]

Which PowerShell and PowerCLI versions are you using?
To which authentication domain (local, AD, ...) does the account you are using to connect belong?
Are you bypassing invalid certificate verification?

You could try to check what is happening during the connection with a network monitoring tool like WireShark.

[shawn149]

Hello LucD,

Sorry, it didn't occur to me to include those details.  My apologies for such a noob post.  

• Attempting to connect to Active Directory
• PowerShell version 7.2.6
• VMware.PowerCLI version 12.2.0.17538434
• Yes, I have InvalidCertificateAction set to Ignore.

Hmm... WireShark...  That's not something I have much experience with, but I will check it out.  Is there anything I should pay particular attention to?  I.e., are there any actions that I could time to isolate where the problem might be?

Thanks!

[shawn149]

I'm still researching, but it looks like a latency issue between VMware and AD.

I have submitted a request to the VMware admins to review the following KB articles:

• Using the lw-measure tool to detect latency between vCenter Server and Active Directory domain controllers (79317)
  • https://kb.vmware.com/s/article/79317
• Using checkADConfig to detect connectivity and DNS issues between vCenter Server and Active Directory (79323)
  • https://kb.vmware.com/s/article/79323 

Unfortunately, I don't know if or when they will get around to checking.

So, I'm back to looking at if I can get some performance data via the RestAPI. 

My issue is this, I have a Lamda in AWS that lets me call a PowerShell script, but it is limited to 7 seconds of wall time.  That means I must ensure the script finishes in 7 seconds or less.  Hence, the desire to reduce how long it takes to authenticate.

In any case, assuming that the same vSphere metrics are available in the vStats Rest API, I attempted the following using this PowerCLI script as a reference:

$results = Get-Stat -Entity "VM-85" -Start $start -Finish $end -Stat "cpu.usage.average"

But when I attempted the same thing with Rest API, I got no results:

curl -X GET 'https://vcenter.lcl/api/stats/data/dp?cid=cpu.usage.PCPU&metric=cpu.usage&resources=type.VM=vm-85' -H 'vmware-api-session-id: token'

I guess I am asking if my assumption is valid that I can get the same metrics via RestAPI that I can get with PowerCLI?  And if so, am I pursuing the correct API resource?  

Am I out in left field on this?

[LucD]

Afaik, the Get-Stat cmdlet is using the vSphere PerformanceManager to get metric data.
The REST API you mention queries the vStats service running on the VCSA.

These are not producing the same data.

[shawn149]

Ha!  This morning has been a morning of surprises...  

So, Authentication in our vShpere is LDAP because the AD integration is being deprecated in vSphere 7.0.

I was also told that the account I'm using to test with is a vsphere.local account.  So, AD wasn't in use at all.

I'm not sure what options I have left...  Any suggestions?  😅

[LucD]

Is that the AD LDAP implementation?
Or a standalone LDAP?

Do all the LDAP servers react with no packet loss on a ping?
You could try using one of the LDAP servers instead of the LDAP domain in the URI you use for the Connect-VIServer.
Does that make a difference?

[shawn149]

It is an AD LDAP implementation.

And the LDAP server has been confirmed to react without packet loss.

That said, I now believe that the slow authentication is a symptom and not the real cause of the slowness.  Your suggestion to use WireShark caused me to consider executing a Process Monitor trace.  I've noticed that it takes a while for PowerShell to step through a bunch of modules when I execute the "Connect-VIServer" command. 

I found that the initial file activity almost matched the delay time.  For an initial session that took ~7 seconds, PowerShell spent 6.63 seconds accessing modules.  I have attached a screen capture of the File Summary from Process Monitor to show the time elapsed for each file.

If I maintain the session, the next connect/disconnect takes about 3 seconds and is isolated to file system activity with the VMware.VimAutomatin.Common (~1.2 seconds) and VMware.Vim (~2.5 seconds) modules.  The first connect/disconnect was always the slowest each time I restarted the session.  (Because of the nature of our automation, I didn't initially consider maintaining the session to see if it was a first-time loading issue.)

I've found so far that if I start pwsh.exe with the -noprofile startup parameter, I can get consistent initial connection times under 7 seconds.  

I have not had to dig into how PowerShell starts up before, so any suggestions on how to set up the environment for a fast initial load?  

[LucD]

PowerShell uses cmdlet caching which enables module auto-loading.
The 1st time a cmdlet from a specific module is used, the auto-loading will load that module.

You can of course already do an Import-Module for specific modules, when you know which cmdlets you are primarily using.

What you should not do is

Import-Module -Name VMware.PowerCLI

at the start of your script.
That will cause an import of all PowerCLI modules which will take a while.

Some useful cmdlets to pre-load modules

# Lists all cmdlets inside the module
Get-Command -Module VMware.VimAutomation.Core

# Find the module for a specific cmdlet
Get-Command -Name Connect-VIServer

[likeahoss]

This -Credential parameter sped up my login.  There are numerous ways to restructure this.  I have LDAPS as my Identity Source so I'm able to use the $env variables.  If you don't have that, just replace it with your info.

Connect-VIServer vCenterName.lab -Force -Protocol https -Credential (Get-Credential -UserName $env:USERDOMAIN\$env:USERNAME -Message "Password")