Getting random SSL/TLS errors in our deployment script

[BenConrad]

We have a complex PowerCli deployment script in use and we've been getting these errors on and off:

The request was aborted: Could not create SSL/TLS secure channel

Our C# client is not having any issues staying connected to vCenter, the server where we run the deployment script is on the same VLAN as the vCenter server so no firewall involved.  We get this error after the script connects to one of the vCenter servers, the initial connection always establishes itself ok.  The script only runs for about 15-20 minutes, I don't think we are hitting any standard timeout on the vCenter connection.

Has anybody else experience this issue?  We see this on about 10% of all deployments.

PowerCLI Version

----------------

   VMware vSphere PowerCLI 5.5 Release 2 Patch 1 build 1931983

Ben

[LucD]

No real answer, but you could check a couple of things.

• Do you always get this, even if you run the script from another PC
• Is the Windows FW active on the station where you run the script ? Try to switch it off.
• Is the vCenter certificate valid ? Did you try to set InvalidCertificateAction on the Set-PowerCLICOnfiguration cmdlet to 'ignore'
• Did you try setting the WebOperationTimeoutSeconds to more than the default 300 seconds (Set-PowerCLICOnfiguration)
• Any additional clues in the vpxd.log ?

[stacycarter]

I'm seeing this intermittently when running a PowerCLI script against multiple clusters/hosts  (specifically during a get-datastore loop for each VM). Running ESXi 5.5 U2 Build 2718055.  Have run the script from two different Windows machines, both with firewalls off, and the same thing happens on both.  I set InvalidCertificateAction on the Set-PowerCLIConfiguration cmdlet to 'ignore', but this did not resolve it.  This typically happens within a couple of minutes after starting the script, so extending WebOperationTimeoutSeconds doesn't help.  I checked the vpxd.log file, and I see the following SSL related warnings around the time this occurs:

Error reading from client while waiting for header: class Vmacore::SystemException(An existing connection was forcibly closed by the remote host)

SSL Handshake failed for stream <>, <>, error: class Vmacore::SystemException(An existing connection was forcibly closed by the remote host)

Strange how random/intermittent it is - I'm querying hundreds of hosts/VMs/datastores, and I typically only see this error occur 1-3 times when running the script.  However, it is enough to cause it to miss data and therefore make the output unreliable.

[stacycarter]

BenConrad‌ were you able to figure out the cause of your intermittent SSL/TLS errors?

[BenConrad]

I implemented the timeouts in power-cli but my deployment script continues to fail.  I've even had a non-deployment script related action (I think I was getting a datastore object) have this issue.  I've got an SR open with VMware and I have Trivia logging turned on in vCenter.  VMware support has asked me to try to reproduce this locally on the vCenter server, I have not got this setup yet.  If I connect to vCenter with PowerCli the connection normally lasts days (or weeks) with no issues.

Most (not all) of our deployments fail when in this function.  I have a simple message bus written so the deployment script can communicate with another script running on the VM and I do that all though the VirtualMachineConfigSpec properties.  This prevents me from having to rely on the front end LAN communications and authentication:

[ fails frequently in this block]

$vmConfigSpec = New-Object VMware.Vim.VirtualMachineConfigSpec

$extra = New-Object VMware.Vim.optionvalue

$extra.Key = $key

$extra.Value = $value

$vmConfigSpec.extraconfig += $extra

$vm = Get-View -ViewType VirtualMachine -Filter @{ "Name" = $newComputerName } -Verbose:$false

$vm.ReconfigVM($vmConfigSpec)

I'll try to get a loop going on my vCenter to see if I can reproduce locally.

Ben

[stacycarter]

Thanks BenConrad ‌, much appreciated.  Please continue to share how your SR/troubleshooting efforts go...