CANNOT backup vSphere 7.0 U3 Native Key Provider NKP as required to activate it per KB 84068

[LabMasterBeta]

Testing and failing on CLEAN INSTALL.

I can create the new VMware Native Key Provider (NKP) in vSphere 7.0 U3g Web UI, but CANNOT back it up using IP-Only setup (that does not use FQDN or DNS for secured Air-Gap environments).

KB article 84068 says VMware has Fixed the IP-Only use-case in vSphere 7.0 Update 3, and NO LONGER requires FQDN for Backups of NKP to work, but it is NOT fixed (or, its broken again, or, I'm missing an undocumented required step..??).

NOTE: Backups are CRITICAL to not lose encrypted VM or vTPM, and the 1st Backup is REQUIRED to enable the NKP service.

Official VMware KB Article 84068 says, "This issue is caused because of the Browser security. The browser is checking the origin of the code that generates the backup file and compares it with the URL. This does not match because one uses FQDN, and the other uses an IP. This is a known issue affecting VMware vSphere 7.0 U2.  Resolved in 7.0 U3."

https://kb.vmware.com/s/article/84068

But, this does NOT seem to be resolved in 7.0 U3??

Or... Are there some other requirements?

We're just doing a clean-install of ESXi 7.0u3f and vCenter 7.0u3g (latest as of 7/23/22)?

ATTEMPT #1: WEB UI

I've tried downloading and installing the IP-Only root CA certs to my Firefox web browser (which show in keystore as "localhost" of the vCenter), and tested disabling in Firefox and Chrome as much security as I could find, but none of that helped. 

Error is always the same in the vCenter Web UI (pops-up error, instead of prompting me where to save the p12 file..):

Back up of Native Key Provider has failed. 

ATTEMPT #2: POWERCLI

I tried working-around the whole Web UI browser issue, by doing the following:

Installed PowerCLI version 12.7 (latest as of 7/23/22), and using the new cmdlet called Export-KeyProvider added in PowerCLI v12.3, but no luck:

1. Disable all certificate (set to IGNORE)

2. Disable the Proxy (set to NOproxy)

3. Disable all PowerShell signing requirements.

4. PLUGGED THE VCENTER HOST DIRECTLY INTO SAME VLAN AS ESXI AND MY WORKSTATION, THERE IS NO FIREWALL.

No matter what I try, I get an error and a 0-byte file, following this guide:

https://blogs.vmware.com/PowerCLI/2021/04/new-release-powercli-12-3.html

Example:
In the Web UI we create a Native Key Provider called "NKPNAME" and have an empty folder c:\vcsa-keystore-backup, after installing PowerCLI, and disabling scripted security or signing the ps1 etc etc, this PowerShell as logged in to vCenter as Admin (using IP-Address), it is supposed to backup/save "mykeyfile.p12" but I get an error and the mykeyfile is 0 bytes:

Export-KeyProvider -KeyProvider NKPNAME -FilePath c:\vcsa-keystore-backup\mykeyfile -Force

Here is the full error:

Export-KeyProvider : 7/23/2022 6:10:02 PM Export-KeyProvider An error occurred while sending the request.
At line:1 char:1
+ Export-KeyProvider -KeyProvider NKPNAME -FilePath c:\...
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : NotSpecified: (:) [Export-KeyProvider], VimException
+ FullyQualifiedErrorId : Core_BaseCmdlet_UnknownError,VMware.VimAutomation.Security.Commands.Cmdlets.KeyProvider.ExportKeyProvider

This confounds me, because after trying 2 methods (and skipping the web browser issue entirely), the VMware KB Article 84068 says this is supposed to work now in 7.0 U3.

But, maybe I'm missing some kind of OTHER digital signature procedure besides just root CA's, because in our IP-only for the vCenter and ESXi hosts, we are TESTING WITH A CLEAN-INSTALL????

Basically, has anyone gotten this to work and HOW, and are there pre-requsites for this??

EDIT/UPDATE 9/5/22: Patched both ESXi and vCenter to 7.0.3g (aka Update-3G), still does not work.

Any suggestions welcome!!

[Claudio3]

@Kinnison 
It's more complicated, than you think.
The whole VMware is in a dedicated VLAN with a dedicated DHCP Server and a "light" DNS Server.

However, vSphere does not recognize this DNS Server to be valid.
And our Server does not have enough ressources to set-up an additional Server with DNS for the vSphere infrastructure.

I think, the only problem is, that this "light" DNS Server does not support reverse entries.....

[LabMasterBeta]

Any discussion at all of DNS built-in to vSphere or external or otherwise, is OFF-Topic for this thread.

Please do not derail this unique thread with off-topics like DNS.

Again, this topic's thread is explicitly for IP-Only configs, WITHOUT any DNS at all, NONE whatsoever (e.g. for Air-Gapped secured environments that use Static IP's-only for 100% of everything).

[LabMasterBeta · Accepted solution]

@shawnh_ 

I'd tried all of this before, but NOW it seems fixed with latest ESXI/VCSA patches!!

Just worked for me on vSphere 7.0.3 (latest patch to ESXi and VCSA), with my doing less than HALF the work I did before trying to fix this(!):

This was the procedure that worked for MY specific setup; with vCenter Server VM in SSO domain "homenet.local", with IP of 192.168.1.202:

1.) In VCSA Console; enable SSH, and Shell.

SSH as root. Login as root. Enable Bash by typing "shell".

Edit on the VCSA VM (using VI):
/etc/hosts

NOTE: During install, I'd changed default SSO with IP-ONLY from "vsphere.local" to "homenet.local" - Change this to match your own config!

Then, in two places:

a.) Within VCSA VM:  /etc/hosts
--And--
b.) Within my OS:  c:\Windows\System32\drivers\etc

Add one line:

192.168.1.202 vcsa.homenet.local vcsa

NOTE on syntax: IP <space> SSO-FQDN <space> HOSTNAME

Save, exit the VCSA and Local Workstation's HOSTS files.

2.) Login web to VCSA VM (VAMI) at:  https://192.168.1.202:5480

Navigate to:

https://192.168.1.202:5480/#/ui/networking

I changed my Networking name to (reminder: Change "homenet" to use whatever SSO domain you used at install):

vcsa.homenet.local

Must WAIT an extra 5 min or so, after GUI looks like it's done (as mentioned above it's actually still regenerating certificates etc).

NOTE:  This scripted process ALSO edits the VCSA's  /etc/hosts

Your VCSA's HOSTS file will then look like this (even though I have ipv6 disabled):

root@vcsa [ /etc ]# cat hosts
# Begin /etc/hosts (network card version)

192.168.1.202 vcsa.homenet.local vcsa
# VAMI_EDIT_BEGIN
# Generated by Studio VAMI service. Do not modify manually.
127.0.0.1 vcsa.homenet.local vcsa localhost
::1 vcsa.homenet.local vcsa localhost ipv6-localhost ipv6-loopback
# VAMI_EDIT_END

3.) Third, it auto-redirects to:

https://vcsa.homenet.local:5480/

NOTES:
* VCSA Web will NOT auto-refresh/reload, you must WAIT a few min longer, THEN reboot..
* It worked for me, but not until after patience, and from ESXI host did a VCSA VM reboot.
* I left my ESXi host as-is; still using IP-only, and did NOT edit its HOSTS file in ESXi to add any name.

Finally then, in VCSA Web, I was able to backup (REQUIRED to enable the Native Key Provider), saved its p12 crypto key/file, tested delete, tested restore, etc - For the very FIRST TIME it all worked fine!

Only drawback (normal for no DNS), just to login to VCSA at all:
** In every Workstation I use (in my case, yours differs), I must now add to its HOSTS file: "192.168.1.202 vcsa.homenet.local vcsa"

To reiterate: I'd tried this long ago and it did not work, so it must be fixed now in latest ESXi/VCSA patches.

[Kinnison]

Comment removed...

[LabMasterBeta]

Ferdinando, I have no idea what you're talking about - the above IS the solution.....

As for FQDN "notation", finally now I understand this incredibly vague comment located as official VMware KB:

https://kb.vmware.com/s/article/84068

That KB says, "To work around this issue, access vCenter using a fully qualified domain name instead of IP address."

It appears what they MEANT and SHOULD have said, was:

1.) the solution above, to use the HOSTS file.

2.) yes, you still must use "FQDN notation" referenced in HOSTS file (to access the Static IP... with NO DNS)....

However, I want to reiterate this was attempted before posting, but this strategy had also failed before a few more recent patches seems to have finally fixed it.........

[Kinnison]

Comment removed...

[jbbarttech]

There is another option that has not come up here.  You CAN backup the native key provider using an ip address if you use the api to back it up directly.   This can be done using a number of different tools, like curl or postman, i used curl on a Linux server.

1. The first step is to create an api session.  I used basic authentication by hashing "username:password" with the command 'echo "username:password" | base64', however there are many commands and websites to get a basic base64 hash of a string.  That hash is then used with the following command to get a session id:

     curl -k -X POST -H 'Authorization: Basic <base64 hash of username:password>' https://<vcenter_ip_address>/api/session

     The output of this command will be a session ID that you use in the next command:
     "3f40c0c2c61ff60af8d6e985b6739ecc"

2. The next step is to get a CryptoManagerKmsProvidersExportResult which contains the CryptoManagerKmsProvidersLocation. This json data will give you everything you need to download the backup.  To get this data:
     curl -k -X POST -H 'vmware-api-session-id: 3f40c0c2c61ff60af8d6e985b6739ecc' -H 'Content-type: application/json' -d '{ "password": "<password_for_NKP_backup>", "provider": "<Key_Provider_name>" }' https://<vcenter_ip_address>/api/vcenter/crypto-manager/kms/providers?action=export 

    The output of this command is the CryptoManagerKmsProvidersExportResult in json:
    {
        "location": {
            "download_token": {
                "expiry": "2023-06-09T22:04:59.000Z",
                "token": "eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJleHAiOjE2ODYzNDgyOTksInBhc3N3ZCI6IipQbklEdSsrcVA5NkcydUlSVkRJWHhmeGpvSlpOMzF0MFNJc3lRQjBsanFsUW1WTGg3RlgvUGxYdGxsbjB5VHhVIiwidXJsIjoiaHR0cHM6Ly9sb2NhbGhvc3QvY3J5cHRvbWFuYWdlci9rbXMvQUlQX1ZNQ0EifQ.QlRfXS-jbnw3TfVOib2dy3ejgkI_L6SFPffC3upYBbE"
            },
            "url": "https://localhost/cryptomanager/kms/VC_NKP_Name"
        },
        "type": "LOCATION"
    }

3. As you can see in the returned data the download url contains the host name of vcenter.  "https://localhost/cryptomanager/kms/VC_NKP_Name".  This is what causes the web ui to fail, it tries to access that location.  In order to backup the key you need to modify this location to be the ip address of your vcenter server then run the command:
    curl -k -X GET -H 'Authorization: Bearer <token_from_CryptoManagerKmsProvidersExportResult >' https://<vcenter_ip_address>/cryptomanager/kms/VC_NKP_Name  > VC_NKP_Name.bak

The output of this final command will be the raw backup data for the key provider.  I piped this into a local file then copied it to my cloud backup location.

[Ben_Z99]

@LabMasterBeta This works for me perfectly. Thank you so much!👍👍

[LabMasterBeta]

@jbbarttech  thank you, that's handy for automation!

I'd falsely assumed the API would fail in the same way PowerCLI did...

Did you test doing a Restore with the API method?

[ObPS]

I have recently upgraded to vSphere 8.0U2 and performed clean install of vCenter8 and this issue still happens.

I have tried the workaround, forgotten to add local hosts in Windows, applied and really long time required to wait for restarting all VCSA services.

And then backup worked with this workaround. I have also tried to specify FQDN at the install but no success.

It is interesting that this issue happens even with fresh v8.0U2 install. 

Thanks for the workaround.

[LabMasterBeta]

vSphere 8 should be the same as 7 for this issue resolution.

Did you precisely follow all steps in my consolidated solution post dated 04-15-2023 11:41 AM

[mozzie]

I followed the instruction to backup the key and it works perfectly, however, it has broken my ability to backup the VCSA!

I now get "Backup PNID 'vcsa.vsphere.local' is not resolved on the network. Configure the network DNS service accordingly."

My DNS IP is my router IP. What can I do to get backups working again?

[LabMasterBeta]

Since you do not have a real DNS server, you'll have to keep adding hostnames to the hosts file instead.

So, in my instructions above, you'd simply keep repeating these steps:

1.) In VCSA Console; enable SSH, and Shell.

SSH as root. Login as root. Enable Bash by typing "shell".

Edit on the VCSA VM (using VI):
/etc/hosts

NOTE: During install, I'd changed default SSO with IP-ONLY from "vsphere.local" to "homenet.local" - Change this to match your own config!

Then, in two places:

a.) Within VCSA VM:  /etc/hosts
--And--
b.) Within my OS:  c:\Windows\System32\drivers\etc

Add one line:

192.168.1.202 vcsa.homenet.local vcsa

NOTE on syntax: IP <space> SSO-FQDN <space> HOSTNAME

Save, exit the VCSA and Local Workstation's HOSTS files.

However... in your case, for the 192.168.1.202 vcsa.homenet.local vcsa:

Change it to:

10.10.10.10 vcsa.vsphere.local vcsa

(Where "10.10.10.10" is your actual local IP to use instead).