Connect-VIServer to Host - Incorrect ID or Password if passed as secure string

[Jaren_Haywood]

Hi All, We have several PowerCLI scripts which have served us well for a few years. We recently patched the hosts to vSphere 7.0U3i (Build 20842708) and have noticed that since then, we cannot establish connections to the hosts using the "Connect-VIServer" cmdlet if passing the password via a secure string.

Below is the code snippet for collecting the credentials:

$HostUsername = Read-Host "Enter username"
$HostPassword = Read-Host "Enter password" -AsSecureString

Below is the code snippet for connecting to the host using the credentials obtained above:

$ESXiHost = "myhost"
$null = Connect-VIServer $ESXiHost -User $HostUsername -Password $HostPassword

After providing known-good credentials via the first snippet, executing the second snippet results in "Connect-VIServer Cannot complete login due to an incorrect user name or password.".

If we remove the -User and -Password arguments and variables, the same credentials can be entered via the credential pop-up window and the connection is successful. I suspect something has changed between patch versions, possibly in the way vSphere hosts interpret the secure string.

I have ensured that PowerCLI is configured to "ignore" invalid certificate warnings for all users, and the credentials have been validated. The currently installed version of PowerCLI is 12.7.0.20091289.

Lastly, I realize there are multiple ways to pass credentials securely via PowerShell and PowerCLI. We would prefer to type or manually copy/paste credentials rather than storing encrypted values within an XML file or similar. Ideally, I would like to adjust the snippet(s) above as needed to get us back on track.

Has anyone seen this before? We are grateful for any ideas you may provide!

[LucD]

Are you using a user/password of an account defined on the ESXi node?
Can you for example open an SSH session with those credentials (you might have to start the SSH service temporarily)

Anything in the /var/log/auth.log or /var/log/vobd.log files on that ESXi node?

[Jaren_Haywood]

Hi @LucD and thank you for the response.

Yes, we are authenticating as the "root" user with the password defined during kernel installation.

Yes, I can successfully launch SSH sessions to the host using the same credentials, including authentication to the vSphere Web UI.

After reviewing the /var/log/auth.log or /var/log/vobd.log files stored on the host:

vodb.log : I see the below:

2023-01-03T15:25:06.226Z: [UserLevelCorrelator] 1035987034708us: [vob.user.ssh.session.opened] SSH session was opened for 'root@(removed workstation IP)'.
2023-01-03T15:25:06.226Z: [GenericCorrelator] 1035987034708us: [vob.user.ssh.session.opened] SSH session was opened for 'root@(removed workstation IP)'.
2023-01-03T15:25:06.226Z: [UserLevelCorrelator] 1035987035122us: [esx.audit.ssh.session.opened] SSH session was opened for 'root@(removed workstation IP)'.
2023-01-03T15:25:07.434Z: [GenericCorrelator] 1035988242962us: [vob.user.ssh.session.closed] SSH session was closed for 'root@(removed workstation IP)'.
2023-01-03T15:25:07.434Z: [UserLevelCorrelator] 1035988242962us: [vob.user.ssh.session.closed] SSH session was closed for 'root@(removed workstation IP)'.
2023-01-03T15:25:07.434Z: [UserLevelCorrelator] 1035988243139us: [esx.audit.ssh.session.closed] SSH session was closed for 'root@(removed workstation IP)'.
2023-01-03T15:25:18.814Z: [GenericCorrelator] 1035999622432us: [vob.user.ssh.session.opened] SSH session was opened for 'root@(removed workstation IP)'.
2023-01-03T15:25:18.814Z: [UserLevelCorrelator] 1035999622432us: [vob.user.ssh.session.opened] SSH session was opened for 'root@(removed workstation IP)'.
2023-01-03T15:25:18.814Z: [UserLevelCorrelator] 1035999622752us: [esx.audit.ssh.session.opened] SSH session was opened for 'root@(removed workstation IP)'.

auth.log : I see the below:

2023-01-03T15:25:04.318Z sshd[2203431]: FIPS mode initialized
2023-01-03T15:25:04.318Z sshd[2203431]: Connection from (removed workstation IP) port 49981
2023-01-03T15:25:04.495Z sshd[2203431]: Using arbitrary primes is not allowed in FIPS mode. Falling back to known groups.
2023-01-03T15:25:06.210Z sshd[2203431]: Accepted keyboard-interactive/pam for root from (removed workstation IP) port 49981 ssh2
2023-01-03T15:25:06.233Z sshd[2203431]: pam_unix(sshd:session): session opened for user root by (uid=0)
2023-01-03T15:25:06.421Z sshd[2203431]: User 'root' running command '/usr/lib/vmware/openssh/bin/sftp-server -f LOCAL5 -l INFO'

Unless I am missing something obvious, I don't see anything interesting in the logs, unfortunately.

[LucD]

Indeed, those seem to be normal entries for the SSH session.

A couple of things to try before we dive in deeper.

You did stop/start your PowerCLI session and received the same error in a fresh session?

You are getting the same error from multiple stations where you try the Connect-VIServer?
And also to different ESXi nodes?

As a long shot, you might want to try and upgrade PowerCLI to 13.0.0.
It was released about 1 month ago.

Update: just to make sure this is not an ESXi node with a "Free" license?

[Jaren_Haywood]

Hi @LucD ,

Yes, I have disconnected from all sessions (remote connections and PowerShell sessions).

Yes, I am able to reproduce the issue on another workstation with the same version of PowerCLI installed.

Yes, I am able to reproduce the issue among other vSphere hosts/ESXi nodes.

Great point about the license: this node is being staged and was indeed running on the "evaluation" license at first, as our build workflow applies the Enterprise license in a later step. That said, I tried applying the vSphere 7 Enterprise Plus license this morning, but the behavior has not changed. Of worthy mention, the other hosts I have reproduced the issue with have been operating with the Enterprise Plus license for several months.

I upgraded PowerCLI on both workstations to version 13.0.0.20829139, but the issue persists.

I tested the same code snippets this morning without any change in behavior. However, I tried omitting the password variable and instead typing the password in plain-text, wrapped in single quotes. The connection was successful using this method. The issue appears to be relocated to the way the password string is converted (or perhaps later interpreted by the host) using the "-AsSecureString" argument.

Thanks again for your insight!

[LucD]

Do you have any "special" characters in the password?
It is in any case always preferable to pass such strings in single quotes, that way you avoid any substitution or interpretation by PowerShell

[Jaren_Haywood]

Hi @LucD ,

Yes, there are special characters inside the password. As a test, I changed the password to something less complex. Although I may authenticate successfully via SSH (Putty), web UI, and by allowing PowerShell to prompt for the password (not storing in variable as a secure string via the snippet above), I still cannot authenticate when sending the password as a secure string.

I tried typing the password wrapped in single quotes when prompted by the Read-Host cmdlet above, but the password is still reported to be invalid.

Here's something amusing... When authenticating to a vCenter Server Appliance using the Connect-VIServer cmdlet, I have always transmitted the password a bit differently. I took that same code and applied it to this scenario where I must authenticate to the host directly. See below:

$HostUsername = Read-Host "Enter username"
$HostPassword = Read-Host "Enter password" -AsSecureString
$ESXiHost = "myhost"
$null = Connect-VIServer $ESXiHost -User $HostUsername -Password ([Runtime.InteropServices.Marshal]::PtrToStringAuto([Runtime.InteropServices.Marshal]::SecureStringToBSTR($HostPassword)))

Much to my surprise (although I'm sure it will make sense to you), authentication is now successful! It seems that although it was not necessary in the past to convert the secure string to plain text (I believe that is what it is doing but I'll admit I'm uncertain as to how it works), it is now required - at least through my observation. I'm very interested to hear your thoughts on this! I'm also receptive to feedback if you have a more sophisticated approach, keeping in mind we would prefer to avoid storing the password (encrypted or not) locally.

Thank you again for your assistance!

[LucD]

Can you try this snippet?

$HostUsername = Read-Host "Enter username"
$HostPassword = Read-Host "Enter password" -MaskInput
$ESXiHost = "myhost"

$cred = New-Object System.Management.Automation.PSCredential($HostUsername, (ConvertTo-SecureString -String $HostPassword -AsPlainText -Force))

$null = Connect-VIServer $ESXiHost -User $HostUsername -Password $cred

[Jaren_Haywood]

Hi @LucD ,

I don't think the -MaskInput argument functioned as intended, as the password was visible in plain-text as I was typing it. Do you want me to proceed anyway? Please see the attached screenshot! Thank you

[LucD]

Strange, it works for me in PSv5.1 and PSv7 in Windows.
What platform are you running PowerShell on?

[LucD]

Ah, it seems to be a known bug in the earlier PSv7 releases.
See Issue#13256
Are you perhaps running on of the impacted PS versions?

[Jaren_Haywood]

Hi @LucD ,

Perhaps the installed version of PowerShell is indeed the issue then. I am running PowerShell version 5.1.19041.2364 on Windows 10 21H2. As you may imagine, we rely on a separate group to deploy most of the OS, PS, and .NET updates to our workstations, although I can manually upgrade if needed in most cases. Specifics below:

Name                           Value
----                           -----
PSVersion                      5.1.19041.2364
PSEdition                      Desktop
PSCompatibleVersions           {1.0, 2.0, 3.0, 4.0...}
BuildVersion                   10.0.19041.2364
CLRVersion                     4.0.30319.42000
WSManStackVersion              3.0
PSRemotingProtocolVersion      2.3
SerializationVersion           1.1.0.1

I don't have any reservations with upgrading PowerShell editions if you feel that would be beneficial.

Update: Based on some quick research, it appears PowerShell 7 is actually PowerShell Core, which would be installed in parallel to PowerShell 5.1, not necessarily supersede it altogether.

[LucD · Accepted solution]

No, I didn't pay attention to your screenshot and didn't test correctly in PSv5.1.
That MaskInput parameter is only available in PSv7.

In fact the snippet is even simpler with the AsSecureString switch

$HostUsername = Read-Host "Enter username"
$HostPassword = Read-Host "Enter password" -AsSecureString
$ESXiHost = "myhost"

$cred = New-Object System.Management.Automation.PSCredential($HostUsername, $HostPassword)

$null = Connect-VIServer $ESXiHost -Credential $cred

[Jaren_Haywood]

Hi @LucD ,

That works very well, thank you!

With respect to how the secure string is interpreted by the host, could you please explain the difference between the working snippet I posted and the one you provided? Understood that your snippet is combining the user ID in concert with the password, but aside from that, what is going on differently "under the hood"? I just want to make sure I understand the differences so I can improve my code moving forward (and more importantly, know why).

Thanks again!

[LucD]

Both snippets as you discovered do practically the same.

The System.Runtime.InteropServices.Marshal class is a rather low-level class that allows you to interact with (managed and unmanaged) memory.
A drawback IMHO is that it is so low-level, that your code has to take care of "freeing" the memory that was allocated via the SecureStringToBSTR method. You will need to call the ZeroFreeBSTR method to do that.
It is not as much the memory that is lost in this way, but the fact one is handling passwords makes it a possible vector of attack.

With the PSCredential constructor, the garbage collection is built in, and the memory content is removed when you do a Remove-Variable of the $cred variable or when you leave the scope where $cred was created for a parent scope.

Yes, I know nobody does the Remove-Variable thing, but we should (to please our Security Officer).
Or at least do these credential things in a function, so the scope is limited.

Using the Set-StrictMode cmdlet will show you a bunch of best practices most of us tend to ignore.

On a side note, I see more and more cmdlets that only accept PSCredential arguments, and not the trusted User/Password anymore.

But I agree, it is a trade-off between quick results from a script and following best practices.

[Jaren_Haywood]

Hi @LucD ,

Thank you for the detailed response! I'll move forward with this snippet and evolve other scripts with the same logic.

Always appreciated, thank you!