Can't create Windows customization after upgrade to 6.7

[a1exp]

After an upgrade from 6.5 to 6.7U3 we're unable to create any customizations, they all fail in the HTML5 UI with the following error:

vim.fault.NotFound

Looking in /var/log/vmware/vpxd/vpxd.log there's the following error:

Method vim.CustomizationSpecManager.GetEncryptionKey threw undeclared fault of type vim.fault.NotFound

Comparing the MOB entries for CustomizationSpecManager with another linked vSphere server I can see that the working server has an entry for encryptionKey, but the broken server just has "unset" against it.

Does anybody known if having encryptionKey under CustomizationSpecManager unset is normal or not?

If it's not normal, any idea how to fix it? (I've tried multiple reboots of VCSA and cert regeneration via UI and certificate-manager).

If it is normal, any ideas how I can fix the issue of creating Windows Customizations?

I'm assuming that the reason this only affects Windows customizations is that there's no stored password for a Linux customization so no need to access the missing public key.

[a1exp]

Anybody?

[idontneedaname]

Same error, completely reinstalled vCenter 6.7 Update 3

[daphnissov]

Open support request.

[a1exp]

Did you backup and restore the config or just rebuild from scratch?

[idontneedaname]

Completely new ESX host and vCenter server

[johncol]

Did you change the PNID by any chance here?

[a1exp]

Yes I did, as part of the upgrade wizard (Which I thought was supported).

I've had a quick look at the /var/log/vmware/applmgmt/pnid_change.log that looks to be dated from the upgrade and it shows as completing successfully.

Do you think it's worth running through swapping to a new hostname and then back?

The blog here seems to cover the steps and it's now supported in 6.7U3:

Changing your vCenter Server's FQDN - VMware vSphere Blog

[johncol]

No, dont change the hostname back, no point  - Can you have a look in /var/log/vmware/vsphere-ui/logs/vsphere_client_virgo.log when you repro the error, do you see the (unknown Source)

VmCustomizationMutationProvider Error occurred while creating vm customization spec.

|

|  -----> further down the backtrace

|

        at com.vmware.vim.vmomi.client.common.impl.MethodInvocationHandlerImpl.invoke(MethodInvocationHandlerImpl.java:191)

        at com.sun.proxy.$Proxy946.getEncryptionKey(Unknown Source)

[a1exp]

Yep:

2019-10-03T16:48:19.274+01:00] [ERROR] http-nio-5090-exec-4         70030074 101690 200073 c.v.v.c.vm.customizations.impl.VmCustomizationMutationProvider    Error occurred while creating vm customization spec. com.vmware.vim.binding.vmodl.fault.SystemError: A general system error occurred: unexpected error reading property

        at sun.reflect.GeneratedConstructorAccessor876.newInstance(Unknown Source)

        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)

        at java.lang.reflect.Constructor.newInstance(Constructor.java:423)

        at java.lang.Class.newInstance(Class.java:442)

        at com.vmware.vim.vmomi.core.types.impl.ComplexTypeImpl.newInstance(ComplexTypeImpl.java:174)

        at com.vmware.vim.vmomi.core.types.impl.DefaultDataObjectFactory.newDataObject(DefaultDataObjectFactory.java:25)

        at com.vmware.vim.vmomi.core.soap.impl.unmarshaller.ComplexStackContext.<init>(ComplexStackContext.java:30)

        at com.vmware.vim.vmomi.core.soap.impl.unmarshaller.UnmarshallerImpl$UnmarshallSoapFaultContext.parse(UnmarshallerImpl.java:150)

        at com.vmware.vim.vmomi.core.soap.impl.unmarshaller.UnmarshallerImpl$UnmarshallSoapFaultContext.unmarshall(UnmarshallerImpl.java:101)

        at com.vmware.vim.vmomi.core.soap.impl.unmarshaller.UnmarshallerImpl.unmarshalSoapFault(UnmarshallerImpl.java:88)

        at com.vmware.vim.vmomi.core.soap.impl.unmarshaller.UnmarshallerImpl.unmarshalSoapFault(UnmarshallerImpl.java:83)

        at com.vmware.vim.vmomi.client.common.impl.SoapFaultStackContext.setValue(SoapFaultStackContext.java:40)

        at com.vmware.vim.vmomi.client.common.impl.ResponseUnmarshaller.processNextElement(ResponseUnmarshaller.java:127)

        at com.vmware.vim.vmomi.client.common.impl.ResponseUnmarshaller.unmarshal(ResponseUnmarshaller.java:70)

        at com.vmware.vim.vmomi.client.common.impl.ResponseImpl.unmarshalResponse(ResponseImpl.java:279)

        at com.vmware.vim.vmomi.client.common.impl.ResponseImpl.setResponse(ResponseImpl.java:234)

        at com.vmware.vim.vmomi.client.http.impl.HttpExchangeBase.parseResponse(HttpExchangeBase.java:156)

        at com.vmware.vim.vmomi.client.http.impl.HttpAsyncExchange$1$2.run(HttpAsyncExchange.java:128)

        at com.vmware.vim.vmomi.core.impl.BlockingFuture.executeRunnable(BlockingFuture.java:150)

        at com.vmware.vim.vmomi.core.impl.BlockingFuture.get(BlockingFuture.java:75)

        at com.vmware.vim.vmomi.client.common.impl.MethodInvocationHandlerImpl$RetryingFuture.get(MethodInvocationHandlerImpl.java:548)

        at com.vmware.vim.vmomi.client.common.impl.MethodInvocationHandlerImpl$CallExecutor.executeCall(MethodInvocationHandlerImpl.java:618)

        at com.vmware.vim.vmomi.client.common.impl.MethodInvocationHandlerImpl.completeCall(MethodInvocationHandlerImpl.java:360)

        at com.vmware.vim.vmomi.client.common.impl.MethodInvocationHandlerImpl.invokeFetch(MethodInvocationHandlerImpl.java:336)

        at com.vmware.vim.vmomi.client.common.impl.MethodInvocationHandlerImpl.invoke(MethodInvocationHandlerImpl.java:180)

        at com.sun.proxy.$Proxy797.getEncryptionKey(Unknown Source)

        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)

        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

        at java.lang.reflect.Method.invoke(Method.java:498)

        at com.vmware.vise.vim.commons.vcservice.impl.VcServiceImpl$ManagedObjectHandler.invoke(VcServiceImpl.java:2363)

        at com.sun.proxy.$Proxy798.getEncryptionKey(Unknown Source)

[johncol]

did you open an SR for this?

[a1exp]

Not yet, will get one raised next week and see what they say

[johncol · Accepted solution]

Try this, I just published it, that should fix it - if not open an SR VMware Knowledge Base

[a1exp]

Thanks very much, following the steps in the KB has resolved for me.

[johncol]

Good to know