Connect-VIServer occurs error intermittently

[handover]

Hello.

When I deploy VM using Ansible + PowerCLI, usually it works well.

But Connect-VIServer occurs error rarely. (about once or twice in a hundred chances)

Following is the stdout.

stdout

Connect-VIServer : 9/19/18 1:39:20 AM Connect-VIServer An error occurred while sending the request. At /var/lib/awx/projects/vmware_dev/roles/1_common_vmware/files/gatherVMInfo_v1.1.ps1:21 char:1 + Connect-VIServer -Server $server -User $user -Password $password| O ... + ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + CategoryInfo : NotSpecified: (:) [Connect-VIServer], ViError + FullyQualifiedErrorId : Client20_ConnectivityServiceImpl_Reconnect_SoapException,VMware.VimAutomation.ViCore.Cmdlets.Commands. ConnectVIServer get-vm : 9/19/18 1:39:20 AM Get-VM You are not currently connected to any servers. Please connect first using a Connect cmdlet. At /var/lib/awx/projects/vmware_dev/roles/1_common_vmware/files/gatherVMInfo_v1.1.ps1:23 char:7 + $vm = get-vm | where { $_.ExtensionData.config.uuid -eq $uuid } + ~~~~~~ + CategoryInfo : ResourceUnavailable: (:) [Get-VM], ViServerConnectionException + FullyQualifiedErrorId : Core_BaseCmdlet_NotConnectedError,VMware.VimAutomation.ViCore.Cmdlets.Commands.GetVM Disconnect-VIServer : 9/19/18 1:39:20 AM Disconnect-VIServer PowerCLI is currently not connected to a server. To create a new connection use Connect-VIServer. /var/lib/awx/projects/vmware_dev/roles/1_common_vmware/files/gatherVMInfo_v1.1.ps1:41 char:1 + Disconnect-VIServer -Confirm:$false + ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + CategoryInfo : InvalidOperation: (:) [Disconnect-VIServer], ViServerConnectionException + FullyQualifiedErrorId : Core_DisconnectVIServer_NotConnectedError,VMware.VimAutomation.ViCore.Cmdlets.Commands.DisconnectVIServer

I checked it and vCenter was in normal. (cpu usage, mem usage, network bandwidth, max sessions..)

There are no proxy or firewall. Ansible server and vCenter is on same network.

and InvalidCertificateAction option is set Ignore in profile.ps1 file.

What would make this error?

Could anyone please help me?

/opt/microsoft/powershell/6.0.1/profile.ps1
Set-PowerCLIConfiguration -InvalidCerficateAction ignore -confirm:$false | Out-Null

[LucD]

It's of course hard to analyse what goes wrong from a distance.

I personally would try to collect as much info as possible before doing the Connect-VIServer:

• Is there anything in $global:defaultviservers
• Can the FQDN of the VCSA be resolved
• Is the VCSA reachable
• What route is followed to the VCSA
• ...

And I would check the vpxd log when it goes wrong.

If it is an intermittent issue, I would also see if the connection between your station and the VCSA might have packet drops.

The first step is trying to find a clue of what goes wrong.

[handover]

Thank you for your quick reply.

Actually, I am not familiar with vmware or vCenter. I am developer of automation script for vCenter using Ansible.

In our environments, ansible server (linux with powershell and powercli) calls powershell script which uses connect-VIserver.

99% of these actions is successful. But in very rare case, Connect-VIServer was failed like bellow.

Here is my answers.

• Is there anything in $global:defaultviservers

       -> No. we don't use  $global:defaultviservers. Instead of it, ansible server passed the necessary parameters as ps scripts arguements such as -sever, -user, -password

• Can the FQDN of the VCSA be resolved

       -> I am not sure I understand this clearly. but ansible server has a vCenter server name in /etc/hosts with it's IP address. And we use only IP address to call Connect-VIServer.

• Is the VCSA reachable

       -> You mean vCenter? It is reachable. ansible server and vCenter are in the same network. No firewall or security things between them.

• What route is followed to the VCSA

        -> Like I said, no routing info is needed. They are in the same NW.

In addition, this occurs very rarely and temporarilly as well. The very next call within a seconds was always successful.

I checked vCenter server resource utilization. but it was very low.

So, according to your advise, I will monitor the communication with ping or http ping and log it.

Could you tell me how to check vpxd log? I have an account with only read permission.

Thank you in advance

[LucD]

The $global:defaultviservers is an automatic variable, which shows which vCenter connections are open.

See Connect-VIServer

The vCenter relies on DSN, name resolution is a pre-req for a proper functioning vSphere setup.

And that needs to work forward and reverse on the vCenter.

Even if you have a local hosts file, it's on the vCenter that name resolution has to work.

The vpxd log is present on the vCenter. You can use the Get-Log cmdlet to retrieve it.

Get-Log -Key vpxd:vpxd.log | select -ExpandProperty Entries |

Out-File -FilePath /tmp/vpxd.log