PowerCLI 6.5.1 connect-viserver returns nothing

[piercj2]

I recently installed PowerCLI 6.5.1 on a Windows 2012 R2 server.

From this Server is I run connect-viserver 'MyVCenterServerName',

• nothing is returned when attempting to connect to a v5.5 U3b vCenter. the vCenter does not connect
• I can connect to v6.0 vCenters

From a Server running powerCLI 6.3, if I run connect-viserver 'MyVCenterName' for the same v5.5 U3b vCenter, it does connect

I've checked the compatibility matrix for powerCLI 6.5 and, it looks like it is compatible with vCenter 5.5

Anyone else encountered this ?

[LucD · Accepted solution]

Well vCenter 5.5U3 is not in the PowerCLI compatibility matrix.

But I'm not sure of that is intended or an oversight.

But then there are the vCenter 5.5U3 Release Notes, that state "...vSphere PowerCLI might fail to connect to vCenter Server 5.5, with a handshake failure."

But then it doesn't mention W2K12R2 at all.

Perhaps worth to check the vpxd logs to see if the "No matching cipher suite" message is in there.

Worst case do a network trace, and watch the handshake exchange

[piercj2]

Thanks Luc,

I checked the vpxd.log during a powerCLI 6.5.1 to VC 6.0 authentication.

I was able to see "Authenticated Successfully" messages for my login.

I then checked the vpxd.log during a PowerCLI 6.5.1 to VC 5.5 authentication, I couldn't see any login attempts at all.

Using wireshark to capture during a PowerCLI 6.5.1 to VC 6.0 authentication, I see

• client hello
• server hello
• encrypted handshake run from start to finish

Using wireshark to capture during a PowerCLI 6.5.1 to VC 5.5 authentication, I see

• client hello
• server hello
• RST, ACK
  • Connection Reset, "Severity Level: Warning"

As usual, looks like you're correct, the handshake could not complete.

Given that I've a combination of vc 5.5 and 6.0 vCenters in production, how would you suggest I proceed.

Is it possible to remove PowerCLI 6.5.1 and reinstall 6.5.0 R1 ?

I attempted this but it seems 6.5.1 remained

Thanks,

Jason

[LucD]

Did you already compare the PowerCLI configuration settings (Get-PowerCLIConfiguration) between the two versions?

Do they differ? Perhaps on the Invalid Certificate action?

Uninstalling PowerCLI 6.5.1, which are only modules, no msi involved, is just a matter of deleting the folders.

These folders will most probably be in one of the standard module paths. The folder is listed when you do

Get-Module -Name VMware* -ListAvailable

Once 6.5.1 is removed, you can install the other version through the MSI.

[piercj2]

I've uninstalled the PowerCLI 6.5.1 modules by deleting the 18 folders identified using

get-module -name VMware* -ListAvailable | select name,path

I installed powercli 6.5.0 R1

get-powercliversion now returns

PowerCLI Version

----------------

   VMware PowerCLI 6.5 Release 1 build 4624819

---------------

Component Versions

---------------

   VMware Cis Core PowerCLI Component 6.5 build 4624453

   VMware VimAutomation Core PowerCLI Component 6.5 build 4624450

   VMWare ImageBuilder PowerCLI Component 6.5 build 4561891

   VMWare AutoDeploy PowerCLI Component 6.5 build 4561891

   VMware Vds PowerCLI Component 6.5 build 4624695

   VMware Cloud PowerCLI Component 6.5 build 4624821

   VMware HA PowerCLI Component 6.0 build 4525225

   VMware HorizonView PowerCLI Component 7.0.2 build 4596620

   VMware Licensing PowerCLI Component 6.5 build 4624822

   VMware PCloud PowerCLI Component 6.5 build 4624825

   VMware Storage PowerCLI Component 6.5 build 4624820

   VMware vROps PowerCLI Component 6.5 build 4624824

   VMware vSphere Update Manager PowerCLI 6.5 build 4540462

the get-PowerCLIConfiguration matches the config on my Server which was not upgraded to powerCLI 6.5.1

Unfortunately, it looks like some part of PowerCLI 6.5.1 remains as I'm still unable to login to v5.5 U3b vCenters from this Server.

I've double checked the Compatibility Matrix and it should be ok.

I've also installed a fresh PowerCLI 6.5.0 R1 on a different Server and it will connect to the v5.5U3b vCenters.

I'm guessing it's a Certificate/SSO issue from me installing 6.5.1 remains on my server ?

[LucD]

What do these return after you installed PowerCLI 6.5R1?

Get-Module -Name VMware* -ListAvailable

Get-PSSnapin -Registered

Did you update the $env:PSModulePath content?

Besides the module folders and the environment variable, PowerCLI 6.5.1 normally doesn't change anything else.

You did a reboot before installing PowerCLI 6.5R1?

Perhaps not really necessary, but better safe than sorry :smileygrin:

[piercj2]

here you are Luc

C:\> Get-Module -Name VMware* -ListAvailable | ft -AutoSize

    Directory: C:\Program Files (x86)\VMware\Infrastructure\PowerCLI\Modules

ModuleType Version       Name                             ExportedCommands
---------- -------       ----                             ----------------
Binary     6.0.0.0       VMware.DeployAutomation
Binary     6.0.0.0       VMware.ImageBuilder
Binary     6.5.0.4624453 VMware.VimAutomation.Cis.Core
Binary     6.5.0.4624821 VMware.VimAutomation.Cloud
Manifest   6.5.0.4624451 VMware.VimAutomation.Common
Binary     6.5.0.2604913 VMware.VimAutomation.Core        HookGetViewAutoCompleter
Binary     6.0.0.0       VMware.VimAutomation.HA
Binary     7.0.2.4596620 VMware.VimAutomation.HorizonView
Binary     6.5.0.4624822 VMware.VimAutomation.License
Binary     6.5.0.4624825 VMware.VimAutomation.PCloud
Manifest   6.5.0.4624452 VMware.VimAutomation.Sdk         Get-PSVersion
Binary     6.5.0.4624820 VMware.VimAutomation.Storage
Binary     6.5.0.4624695 VMware.VimAutomation.Vds
Binary     6.5.0.4624824 VMware.VimAutomation.vROps
Binary     6.0.0.0       VMware.VumAutomation

Get-PSSnapin -registered returns nothing

In my $Profile I've the following

Get-Module -Name VMware* -ListAvailable | Import-Module -Force

I didn't reboot between uninstalling 6.5.1 and installing 6.5.0 R1

I could completely remove all PowerCLI components and reboot, then do a fresh install of 6.5.0 R1 ?

[LucD]

Yes, I would suggest a complete removal, reboot and then install PowerCLI 6.5R1

[piercj2]

Hi Luc,

I removed all traces of PowerCLI on the Server and rebooted (twice).

After verifying that all PowerCLI folders, etc... had been removed, I reinstalled PowerCLI 6.5.0 R1

• VMware PowerCLI 6.5 Release 1 build 4624819

I then changed the PowerCLIConfiguration to the following (just in case)

Scope    ProxyPolicy     DefaultVIServerMode InvalidCertificateAction  DisplayDeprecationWarnings WebOperationTimeout

                                                                                                  Seconds

-----    -----------     ------------------- ------------------------  -------------------------- -------------------

Session  UseSystemProxy  Single              Prompt                    True                       300

User                     Single              Prompt

AllUsers                 Single              Prompt

The available modules are

ModuleType Version       Name                             ExportedCommands

---------- -------       ----                             ----------------

Binary     6.0.0.0       VMware.DeployAutomation

Binary     6.0.0.0       VMware.ImageBuilder

Binary     6.5.0.4624453 VMware.VimAutomation.Cis.Core

Binary     6.5.0.4624821 VMware.VimAutomation.Cloud

Manifest   6.5.0.4624451 VMware.VimAutomation.Common

Binary     6.5.0.2604913 VMware.VimAutomation.Core        HookGetViewAutoCompleter

Binary     6.0.0.0       VMware.VimAutomation.HA

Binary     7.0.2.4596620 VMware.VimAutomation.HorizonView

Binary     6.5.0.4624822 VMware.VimAutomation.License

Binary     6.5.0.4624825 VMware.VimAutomation.PCloud

Manifest   6.5.0.4624452 VMware.VimAutomation.Sdk         Get-PSVersion

Binary     6.5.0.4624820 VMware.VimAutomation.Storage

Binary     6.5.0.4624695 VMware.VimAutomation.Vds

Binary     6.5.0.4624824 VMware.VimAutomation.vROps

Binary     6.0.0.0       VMware.VumAutomation

after verifying the above, I tried again to connect to my v5.5u3b vCenter

Connect-VIServer MyVCServer -Credential (Get-Credential) -verbose

The following warning was returned

VERBOSE: Could not establish secure channel for SSL/TLS with authority 'MyVCServer'.

I attempted a connecting again with Wireshark running in the background and the following was returned

src_IP vc_IP TCP 54 50574 → 443 [RST, ACK] Seq=158 Ack=1047 Win=0 Len=0

Transmission Control Protocol, Src Port: 50574, Dst Port: 443, Seq: 158, Ack: 1047, Len: 0

    Source Port: 50574

    Destination Port: 443

    [Stream index: 0]

    [TCP Segment Len: 0]

    Sequence number: 158    (relative sequence number)

    Acknowledgment number: 1047    (relative ack number)

    Header Length: 20 bytes

    Flags: 0x014 (RST, ACK)

        000. .... .... = Reserved: Not set

        ...0 .... .... = Nonce: Not set

        .... 0... .... = Congestion Window Reduced (CWR): Not set

        .... .0.. .... = ECN-Echo: Not set

        .... ..0. .... = Urgent: Not set

        .... ...1 .... = Acknowledgment: Set

        .... .... 0... = Push: Not set

        .... .... .1.. = Reset: Set

            [Expert Info (Warning/Sequence): Connection reset (RST)]

                [Connection reset (RST)]

                [Severity level: Warning]

                [Group: Sequence]

        .... .... ..0. = Syn: Not set

        .... .... ...0 = Fin: Not set

        [TCP Flags: ·······A·R··]

    Window size value: 0

    [Calculated window size: 0]

    [Window size scaling factor: 256]

    Checksum: 0xf816 [unverified]

    [Checksum Status: Unverified]

    Urgent pointer: 0

    [SEQ/ACK analysis]

The handshake doesn't seem to be getting to the SSL/TLS part.

(successfully connecting to a v6.0 vCenter, I see from Wireshark that TLS1.2 should be used)

Does PowerShell / PowerCLI locally store Certificates ?

could these be deleted (or am I going in the wrong direction with this ?)

Thanks again,

Jason

[LucD]

Did you already setting the InvalidCertifcateAction to Ignore?

And yes, you can delete those certificates on the machine where you run the Connect-VIServer.

Start "Manage User Certificates" and/or "Manage Computer Certificates", then find the certificate(s) ( I always search for 'vmware').

From there you can remove a certificate.

[piercj2]

So, I'm admitting defeat on this one.

After following all of the excellent advice provided by Luc, I'm still unable to connect from a Windows 2012 R2 Server to v5.5 vCenter Servers or v5.5 ESXi Hosts.

This happens for both PowerCLI Client and, for vSphere Client (note: I can connect using the Web Client).

Note also, I can connect from Windows 2012 R2 to v6.0 vCenter Servers and v6.0 ESXi Hosts using both the PowerCLI Client and C# Clients.

Eventually, I rebuilt the 2012 R2 from scratch.

The following were installed

• PowerCLI 6.5.0 R1
• C# Client (by connecting to the v5.5 ESXi Host)
• Client Integration Plug-In's

I was still unable to connect to v5.5 infrastructure using PowerCLI or C# Clients.

Telnetting to a host on port 902 gave the following message

220 VMware Authentication Daemon Version 1.10: SSL Required, ServerDaemonProtocol:SOAP, MKSDisplayProtocol:VNC , VMXARGS supported, NFCSSL supported/t

Using Wireshark to sniff the packets, I could see that the handshake stopped before SSL Comms started/completed

I found the following on Chris Wahl's site

Windows 2003 and vSphere Refuse To Handshake - Wahl Network

This in turn let me to

TLS Cipher Suites in Windows 8.1 (Windows)

I enabled everything on my Server, SSL2.0, SSL3.0, TLS1.0, TLS1.1 & TLS1.2

Unfortunately nothing changed

I have to conclude that there's an issue between Windows 2012 R2 and v5.5 infrastructure.

Given that Microsoft / VMware had 5 years to fix this, it's disappointing that the issue remains in place today

If anyone has a workaround/fix for this I'd love to hear it.

Otherwise i'm looking at one of 2 very large upgrades

• VMware infrastructure from 5.5 to 6.0 (this has started in the company but it is a HUGE environment and there are many dependencies that heed to be met)
• Microsoft Server 2012 R2 to 2016, again a HUGE environment with many dependencies

Thanks

[LucD]

Did you already see KB2146255?

[piercj2]

Sorry it took so long to reply Luc,

I had seen the KB alright but it didn't help. I did manage to fix the issue though.

In the end, it turned out to be a TLS issue.

The issue started around the same time as the Wannacry virus was released.

It turns out our Global Security office disabled TLS on all endpoints as a preventative measure (of course they told nobody).

Telnetting to 902 and the Wireshark capture were both pointing at TLS.

I was discussing the issue with a colleague who was having issues with another Application and together we discovered that despite TLS showing as enabled in Internet Explorer, in the Registry it had been disabled. Enabling TLS in the registery resolved the issues we were both seeing

Thanks for your time on this.

Jason

[LucD]

Great that you found the solution.

And yes, those TLS issues seem to be popping up all over the place after that virus outbreak :smileygrin: