VMware Cloud Community
LabMasterBeta
Enthusiast
Enthusiast
Jump to solution

CANNOT backup vSphere 7.0 U3 Native Key Provider NKP as required to activate it per KB 84068

Testing and failing on CLEAN INSTALL.

 

I can create the new VMware Native Key Provider (NKP) in vSphere 7.0 U3g Web UI, but CANNOT back it up using IP-Only setup (that does not use FQDN or DNS for secured Air-Gap environments).

 

KB article 84068 says VMware has Fixed the IP-Only use-case in vSphere 7.0 Update 3, and NO LONGER requires FQDN for Backups of NKP to work, but it is NOT fixed (or, its broken again, or, I'm missing an undocumented required step..??).

NOTE: Backups are CRITICAL to not lose encrypted VM or vTPM, and the 1st Backup is REQUIRED to enable the NKP service.

 

Official VMware KB Article 84068 says, "This issue is caused because of the Browser security. The browser is checking the origin of the code that generates the backup file and compares it with the URL. This does not match because one uses FQDN, and the other uses an IP. This is a known issue affecting VMware vSphere 7.0 U2.  Resolved in 7.0 U3."

https://kb.vmware.com/s/article/84068

 

But, this does NOT seem to be resolved in 7.0 U3??

Or... Are there some other requirements?

We're just doing a clean-install of ESXi 7.0u3f and vCenter 7.0u3g (latest as of 7/23/22)?

 

ATTEMPT #1: WEB UI

I've tried downloading and installing the IP-Only root CA certs to my Firefox web browser (which show in keystore as "localhost" of the vCenter), and tested disabling in Firefox and Chrome as much security as I could find, but none of that helped. 

Error is always the same in the vCenter Web UI (pops-up error, instead of prompting me where to save the p12 file..):

 

 

Back up of Native Key Provider has failed. 

 

 

 

ATTEMPT #2: POWERCLI

I tried working-around the whole Web UI browser issue, by doing the following:

Installed PowerCLI version 12.7 (latest as of 7/23/22), and using the new cmdlet called Export-KeyProvider added in PowerCLI v12.3, but no luck:

1. Disable all certificate (set to IGNORE)

2. Disable the Proxy (set to NOproxy)

3. Disable all PowerShell signing requirements.

4. PLUGGED THE VCENTER HOST DIRECTLY INTO SAME VLAN AS ESXI AND MY WORKSTATION, THERE IS NO FIREWALL.

 

No matter what I try, I get an error and a 0-byte file, following this guide:

https://blogs.vmware.com/PowerCLI/2021/04/new-release-powercli-12-3.html

 

Example:
In the Web UI we create a Native Key Provider called "NKPNAME" and have an empty folder c:\vcsa-keystore-backup, after installing PowerCLI, and disabling scripted security or signing the ps1 etc etc, this PowerShell as logged in to vCenter as Admin (using IP-Address), it is supposed to backup/save "mykeyfile.p12" but I get an error and the mykeyfile is 0 bytes:

 

 

Export-KeyProvider -KeyProvider NKPNAME -FilePath c:\vcsa-keystore-backup\mykeyfile -Force

 

 

 

Here is the full error:

 

 

Export-KeyProvider : 7/23/2022 6:10:02 PM Export-KeyProvider An error occurred while sending the request.
At line:1 char:1
+ Export-KeyProvider -KeyProvider NKPNAME -FilePath c:\...
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : NotSpecified: (:) [Export-KeyProvider], VimException
+ FullyQualifiedErrorId : Core_BaseCmdlet_UnknownError,VMware.VimAutomation.Security.Commands.Cmdlets.KeyProvider.ExportKeyProvider

 

 

 

This confounds me, because after trying 2 methods (and skipping the web browser issue entirely), the VMware KB Article 84068 says this is supposed to work now in 7.0 U3.

But, maybe I'm missing some kind of OTHER digital signature procedure besides just root CA's, because in our IP-only for the vCenter and ESXi hosts, we are TESTING WITH A CLEAN-INSTALL????

 

Basically, has anyone gotten this to work and HOW, and are there pre-requsites for this??

EDIT/UPDATE 9/5/22: Patched both ESXi and vCenter to 7.0.3g (aka Update-3G), still does not work.

Any suggestions welcome!!

1 Solution

Accepted Solutions
LabMasterBeta
Enthusiast
Enthusiast
Jump to solution

@shawnh_ 

I'd tried all of this before, but NOW it seems fixed with latest ESXI/VCSA patches!!

Just worked for me on vSphere 7.0.3 (latest patch to ESXi and VCSA), with my doing less than HALF the work I did before trying to fix this(!):

This was the procedure that worked for MY specific setup; with vCenter Server VM in SSO domain "homenet.local", with IP of 192.168.1.202:

1.) In VCSA Console; enable SSH, and Shell.

SSH as root. Login as root. Enable Bash by typing "shell".

Edit on the VCSA VM (using VI):
/etc/hosts

NOTE: During install, I'd changed default SSO with IP-ONLY from "vsphere.local" to "homenet.local" - Change this to match your own config!

Then, in two places:

a.) Within VCSA VM:  /etc/hosts
--And--
b.) Within my OS:  c:\Windows\System32\drivers\etc

Add one line:

192.168.1.202 vcsa.homenet.local vcsa

NOTE on syntax: IP <space> SSO-FQDN <space> HOSTNAME

Save, exit the VCSA and Local Workstation's HOSTS files.

2.) Login web to VCSA VM (VAMI) at:  https://192.168.1.202:5480

Navigate to:

https://192.168.1.202:5480/#/ui/networking

I changed my Networking name to (reminder: Change "homenet" to use whatever SSO domain you used at install):

vcsa.homenet.local

Must WAIT an extra 5 min or so, after GUI looks like it's done (as mentioned above it's actually still regenerating certificates etc).

NOTE:  This scripted process ALSO edits the VCSA's  /etc/hosts

Your VCSA's HOSTS file will then look like this (even though I have ipv6 disabled):

root@vcsa [ /etc ]# cat hosts
# Begin /etc/hosts (network card version)

192.168.1.202 vcsa.homenet.local vcsa
# VAMI_EDIT_BEGIN
# Generated by Studio VAMI service. Do not modify manually.
127.0.0.1 vcsa.homenet.local vcsa localhost
::1 vcsa.homenet.local vcsa localhost ipv6-localhost ipv6-loopback
# VAMI_EDIT_END


3.) Third, it auto-redirects to:

https://vcsa.homenet.local:5480/

NOTES:
* VCSA Web will NOT auto-refresh/reload, you must WAIT a few min longer, THEN reboot..
* It worked for me, but not until after patience, and from ESXI host did a VCSA VM reboot.
* I left my ESXi host as-is; still using IP-only, and did NOT edit its HOSTS file in ESXi to add any name.

Finally then, in VCSA Web, I was able to backup (REQUIRED to enable the Native Key Provider), saved its p12 crypto key/file, tested delete, tested restore, etc - For the very FIRST TIME it all worked fine!

Only drawback (normal for no DNS), just to login to VCSA at all:
** In every Workstation I use (in my case, yours differs), I must now add to its HOSTS file: "192.168.1.202 vcsa.homenet.local vcsa"

To reiterate: I'd tried this long ago and it did not work, so it must be fixed now in latest ESXi/VCSA patches.

View solution in original post

32 Replies
rdeadwyler
Contributor
Contributor
Jump to solution

I am experiencing the same issue specifically using the UI, running 7.0.3 and via the FQDN.  Were you able to find a solution?

Reply
0 Kudos
acartwright
Contributor
Contributor
Jump to solution

If you're connecting via FQDN check whether you have a CNAME for vCenter as this can lead to some cross-site request issues in the browser.

If you connect to vCenter on the CNAME FQDN you may (as I did) hit an issue due to CORS which prevents the XMLHttpRequest to allow the certificate backup to download which causes this error message. You can view this error in the web inspector console.

Workaround - connect to vCenter via IP to backup and activate the Native KMS (vmware addresses direct IP connections in 7u2 I believe)

Reply
0 Kudos
Kinnison
Commander
Commander
Jump to solution

Comment edited to remove its contents, it is reasonably no longer of any interest.

Reply
0 Kudos
rdeadwyler
Contributor
Contributor
Jump to solution

I will verify CNAME records, but I didn't have any luck using an IP rather than the FQDN. 

Reply
0 Kudos
LabMasterBeta
Enthusiast
Enthusiast
Jump to solution

To clarify a few posts:

1. No I'm not using CNAME or any DNS at all, but I did try a few common work-arounds to no avail (still fails to backup or activate):

-(a) Tried the built-in Alias Name feature in vCenter (which is totally different than CNAME) - Didnt help.

-(b) Tried using Hosts file in ESXi *AND* My local admin workstation - Didnt help.

-(c) Tried to download certificate into Web Browser so dont get the warning when login - Didnt help.

-(d) Tried disabling all security and cross-site blocking etc etc in both FIrefox and Chrome - Didnt help.

2. Native Key Provider for IP-Only (no DNS or FQDN at all), was not EVER supported in earlier builds of vSphere 7.0...  However, there were so many customers who complained, VMware claims to have added support for this in 7.0 U3 (as per the official KB article I cited - but I cannot make it work...).

3. I've now patched ESXi and vCenter both to 7.0 Update-3G, but the problem persists (it simply will NOT backup the Native Key Provider in my IP-Only configuration as advertised by VMware that now it should).

 

Any best-guess ideas would be really appreciated.

Reply
0 Kudos
LabMasterBeta
Enthusiast
Enthusiast
Jump to solution

@acartwright  regarding, "...hit an issue due to CORS which prevents the XMLHttpRequest to allow the certificate backup to download which causes this error message..." - Can you please elaborate in detail?

I did find a VMware KB article that blames the web browser security on inability to do the NKP backup, but I was also unable to do a NKP backup with PowerCLI using its new NKP commands that I cited in my original post.

 

I'd be grateful for your details on what you're referring to and how to try it with the browser.

Note:  I am not using DNS or CNAMES. Just basic "IP-Only" now supported in U3.

Thanks for any suggestions!!

Reply
0 Kudos
acartwright
Contributor
Contributor
Jump to solution

My issue may have been different to yours but I can let you know how I diagnosed it in my case on the off chance it helps.

We access our vCenter via a FQDN CNAME for the actual FQDN of the vCenter server. This doesn't normally cause any issues with management however was causing an issue with the p12 backup download of the native key data.

I opened the web inspector in the browser I was using (Firefox) and in the Console noticed that there was a failed CORS request to the A record for our vCenter server (i.e. not the CNAME) which was resulting in a CORS violation and preventing the download. My error could also been seen in the Network tab as a blocked (failed) xhr request due to CORS.

I was able to work around the issue by logging into vCenter via IP instead of FQDN but I don't think this is working for you. I see you've already tried the powercli method (I'm not familiar with powercli), however you may be able to try copying the failed CORS request from the Network tab of the inspector and use this with curl or Postman to workaround your issue and download a backup of your native key provider data. Given you have had difficulties with powercli this workaround method may not work either.

Andrew

Reply
0 Kudos
LabMasterBeta
Enthusiast
Enthusiast
Jump to solution

@acartwright 

Thank you for the suggestions!

I think you're on the right track, because VMware specifically mentions the KB's web browsers blocking receiving what it thinks is an invalid security scenario. And, when I try to backup the keystore, it hangs for an abnormally long time before throwing the errors.

Do you have specific guidance on using curl and Postman for this? I'm unclear on your thinking.  Using this as a solution or work-around, would also need to accommodate doing the Restore test successfully as well. Do you have thoughts on how to do the Upload for the Restore?

Once you backup successfully, it enables NKP, true - but if you remove it - then it wipes out your VM's and vTPM etc... so the restore is critical.

I now have this setup on a dedicate micro server for lab testing it - So, I'm willing to try anything in this lab environment, that anyone can think of..

 

A few more things I've tried (to no avail), just to test removing security-

FireFox (easy method without ADMX template) -> Change non-fqdn to true:

 

about:config
network.negotiate-auth.allow-non-fqdn

 

Add the vCenter IP to trusted list:

 

network.negotiate-auth.trusted-uris

 

VCSA-alias instead of DNS-cname alias, in vCenter (SSH -> BASH -> Shell)

 

service-control --stop vsphere-ui
cd /etc/vmware/vsphere-ui/
cp webclient.properties /var/tmp/webclient.properties.bak
vi webclient.properties 

 

Enabled "SSO Alias" in VCSA (by removing the "#" on this line, and adding the /etc/hosts name):

 

sso.serviceprovider.alias.whitelist=vcsa,vcsa.domain.local
service-control --start vsphere-ui

 

Added "vcsa.domain.local" to the workstation:

 

c:\windows\system32\drivers\etc\hosts
192.168.1.201 esxi1.domain.local esxi1
192.168.1.202 vcsa.domain.local vcsa

 

I've Succeeded in ONE thing: 

By doing all of this PLUS installing the VCSA certificate, I can now login to https // vcsa.domain.local, using the fake name, and get a green trusted bar - Without any DNS.

 

Unfortunately, None of that helped the goal of Backing up (or restoring..) the NKP - Same errors as before.

Any other thoughts (and how-to details), would be greatly appreciated - thanks again!

Reply
0 Kudos
Zewwy
Contributor
Contributor
Jump to solution

Thank you acartwright, I too had this issues and chkecing the console log showed the same CORS problem. In my case it was not due to accessing the server via a alternative name using a CNAME, but rather from testing out vCenter renaming I must have done it again and forgot to follow the steps of changing the hostname manually via a shell prompt on the vCenter Server, as well as correcting the localhost file.

Once those two things were fixed (no reboot required), back up of the NKP via the vCenter UI worked perfectly.

Thank you again. 🙂

Reply
0 Kudos
NMG-NTNX
Contributor
Contributor
Jump to solution

I am also encountering the same issue - CORS, on vCenter 7.0.3, I have tried with both vCenter IP and FQDN. Has the solution been found? It would be helpful if someone can point me out..

Reply
0 Kudos
LabMasterBeta
Enthusiast
Enthusiast
Jump to solution

 
Updated all core patches for January 2023, and NKP Backup is still not working for me, with my IP-Only requirement:
 
VCSA: 7.0U3j
** VCSA Version: 7.0.3.01200
** VCSA Build: 20990077
 
ESXi: Update 3i
** Imageprofile: ESXi-7.0U3i-20842708-standard
 
I am unfamiliar with CORS troubleshooting: What is happening during CORS troubleshooting, how to do it, and how to fix it?
I'd really like to try that CORS workaround for resolving this issue of NKP Backup Failing and inability to enable vTPM..
Please elaborate on how, exactly, to troubleshoot CORS to fix this Native Key Provider backup fail during setup issue?
 
In Chrome AND Firefox browsers:  I tried the following CORS Unblock extensions/plugins, but they did NOT help (backup still fails):
** "Allow CORS: Access-Control-Allow-origin"
** "CORS Unblock"
 
In vSphere: I looked for hours to no avail...
** I do NOT understand how to troubleshoot this "CORS" issue within vSphere 7 update3.
 
ANY Tips for how to fix this "CORS problem" would be fantastic and GREATLY appreciated!!
 
Thank you in advance!!
Reply
0 Kudos
shawnh_
Contributor
Contributor
Jump to solution

I had this same problem, but on vCenter/ESXi 8.0b

The CORS violation is just an artifact of the underlying issue, which is that vCenter thinks it's hostname is "localhost" (or whatever, in your case), and the certificate embeds that hostname.

To fix, I did this:

ssh into vCenter server and add entry in /etc/hosts mapping vCenter's LAN IP to the hostname you want it to use. e.g. mine was (there is nothing special about the name, I just wasn't creative):

192.168.2.58 vsphere.local

Make the same entry on your client machine that will be accessing the web ui (mine was also Windows).

Using the ip address, access e.g. https://192.168.2.58:5480/#/ui/networking and edit the hostname to match whatever you've picked out. Commit the change and wait for stuff to be regenerated (this can take much longer than the UI makes it appear - I added some vCPUs to the vCenter VM to speed it up a bit). I also rebooted the vCenter VM, although I'm not sure if it's required.

You should now be able to access the web ui via the new hostname and properly download the key provider backup.

This is probably a bug that should be fixed by vmware, though.

mruskys
Contributor
Contributor
Jump to solution

shawnh_  Thank you very much for your time to write this helpful instructions. I had vSphere8 and was struggling to backup key. But with your instructions managed to backup key now.   Again, much appreciated. 

Regards,

Mantas

Reply
0 Kudos
Claudio3
Contributor
Contributor
Jump to solution

@shawnh_ 
I'm struggling with exact the same.
I also have vSphere 8.
But HOW exactly have you changed the /etc/hosts via ssh?

I mean, I was able to do it on ESXi, with the vi command.
But in vSphere 8, the vi command is unknown.
(I have installed vSphere 8 with the tiny option).

Thank you,
Claudio

Reply
0 Kudos
Kinnison
Commander
Commander
Jump to solution

Comment removed...

Reply
0 Kudos
Claudio3
Contributor
Contributor
Jump to solution

Thank you @Kinnison 
I had to enable BASH in vSphere Server Management first.

Now I was able to edit the /etc/hosts.
But I'm still unable to back-up my native key provider (I don't have a DNS). 😞

Reply
0 Kudos
Kinnison
Commander
Commander
Jump to solution

Comment removed...

Reply
0 Kudos
Claudio3
Contributor
Contributor
Jump to solution

@Kinnison 
The only reason, why I set-up a vSpere Server is, because I need to deploy Windows 11 machines.
I only set-up an ESXi and I can deploy VMs perfectly. Except for Windows 11...
This is the only reason why I have to fiddle with vSphere.

And there is no need to set-up a DNS for all the clients, because these are test machines, which are reinstalled every 2-3 days, just to test some of our internal software.....

So it's - unfortunately - in our case a huge effort to only deploy a Windows 11 machine 😞

I try to follow the instructions by @shawnh_ , but I currently struggle, with changing the hostname and DNS for the vSphere Server. It won't let me do this right now.

Reply
0 Kudos
Kinnison
Commander
Commander
Jump to solution

Comment removed...

Reply
0 Kudos