vCenter 5.5 > 6.0 U1 upgrade fails "error attempting restore authz data"

[mikeb0101]

Long time lurker first time poster.

So we were running through some dry run upgrades of vCenter 5.5 to 6.0 (U1). I have the latest installer downloaded. Our install goes great up until the point of importing data from 5.x. I keep getting the error of:

Error attempting Restore Authz data

Please check InvSvc upgrade logs for details

So I submitted a VMware ticket, and they told me this is due to having custom role definitions in vCenter, which we have. They said there is currently no workaround for this but it is a known issue, and we need to remove all of our custom roles, and go again. I did this in our dev environment (where we essentially clone our existing vCenter infrastructure and test the ugprade), and it seemed to work fine. So I go to do prod today, and it's failing with the same error, even though we've removed all of our custom role definitions. I've reopened my ticket with VMware, but in the meantime has anyone else seen this at all? I can't find any mention of this online anywhere, can I be the first person to experience this issue?

The specific line I find in the logs is here:

2015-12-03T16:33:27.433Z INFO upgrade.states.component_states is:Import: Caught exception while adding permission java.util.concurrent.ExecutionException: (vmodl.fault.InvalidArgument) {

2015-12-03T16:33:27.433Z INFO upgrade.states.component_states is:Import:    faultCause = null,

2015-12-03T16:33:27.433Z INFO upgrade.states.component_states is:Import:    faultMessage = null,

2015-12-03T16:33:27.434Z INFO upgrade.states.component_states is:Import:    invalidProperty = Invalid role Ids [33554635]

2015-12-03T16:33:27.434Z INFO upgrade.states.component_states is:Import: }

2015-12-03T16:33:27.434Z INFO upgrade.states.component_states is:Import: Restore Assignment failed java.util.concurrent.ExecutionException: (vmodl.fault.InvalidArgument) {

2015-12-03T16:33:27.434Z INFO upgrade.states.component_states is:Import:    faultCause = null,

2015-12-03T16:33:27.434Z INFO upgrade.states.component_states is:Import:    faultMessage = null,

2015-12-03T16:33:27.436Z INFO upgrade.states.component_states is:Import:    invalidProperty = Invalid role Ids [33554635]

2015-12-03T16:33:27.436Z INFO upgrade.states.component_states is:Import: }

2015-12-03T16:33:27.436Z INFO upgrade.states.component_states is:Import: Restore Authz failed

2015-12-03T16:33:27.437Z INFO upgrade.states.component_states is:Import: java.util.concurrent.ExecutionException: (vmodl.fault.InvalidArgument) {

2015-12-03T16:33:27.437Z INFO upgrade.states.component_states is:Import:    faultCause = null,

2015-12-03T16:33:27.437Z INFO upgrade.states.component_states is:Import:    faultMessage = null,

2015-12-03T16:33:27.437Z INFO upgrade.states.component_states is:Import:    invalidProperty = Invalid role Ids [33554635]

2015-12-03T16:33:27.437Z INFO upgrade.states.component_states is:Import: }

2015-12-03T16:33:27.438Z INFO upgrade.states.component_states is:Import:     at com.vmware.vim.vmomi.core.impl.BlockingFuture.get(BlockingFuture.java:70)

2015-12-03T16:33:27.438Z INFO upgrade.states.component_states is:Import:     at com.vmware.vim.dataservices.RestoreAuthz.processItem(RestoreAuthz.java:322)

2015-12-03T16:33:27.438Z INFO upgrade.states.component_states is:Import:     at com.vmware.vim.dataservices.RestoreAuthz.restoreAssignment(RestoreAuthz.java:247)

2015-12-03T16:33:27.440Z INFO upgrade.states.component_states is:Import:     at com.vmware.vim.dataservices.RestoreAuthz.main(RestoreAuthz.java:695)

2015-12-03T16:33:27.440Z INFO upgrade.states.component_states is:Import: Caused by: (vmodl.fault.InvalidArgument) {

2015-12-03T16:33:27.440Z INFO upgrade.states.component_states is:Import:    faultCause = null,

2015-12-03T16:33:27.440Z INFO upgrade.states.component_states is:Import:    faultMessage = null,

2015-12-03T16:33:27.440Z INFO upgrade.states.component_states is:Import:    invalidProperty = Invalid role Ids [33554635]

2015-12-03T16:33:27.441Z INFO upgrade.states.component_states is:Import: }

Thanks in advance for any help you can provide.

[kurimargo]

Hi mikeb0101

Today I encountered exactly the same problem during upgrade my test environment from 5.5 to 6.0U1

And I also have some custom roles in my environment.

Did VMware solve your problem? Or I have to also open support case.

BR,

margo

[MarcelMack]

Hey all,

we had the exact same problem when updating our VC 5.5 U2 to any 6.X Version.

In our enviroment we use alot of custom roles and permissions but in our scenario the error was caused by a courrupted Inventory Service Database.

So we followed VMware KB 2042200 and reset the Inventory Service Database (the old one was 8GB small?/big?).

After that the update worked perfectly fine!

[JonathanThorpe]

Has anyone else had a resolution to this problem?

I've been banging my head against a wall for the past few days trying to resolve exactly the same error going to VC 6.0 Update 2 from 5.5. Resetting the Inventory Service is not an option because we're running vCloud Director, Zerto and Veeam - all of these components will fail in some way as a result of Refs being updated.

Like the original poster, prior to the upgrade, I cloned the VC, SQL and a Domain Controller. The upgrade completes successfully.

Trying this again after having to rollback also works fine.

Upgrading to Update 2a directly is not an option as it fails in our environment when parsing server.xml for the SSO/STS Service. This is something that happened in a similar environment (also has vCloud Director, Veeam and Zerto - although Zerto isn't doing any work) and did upgrade successfully.

I've had a case open with VMware for almost a week and haven't got anywhere just yet.

This is what I see in the logs

---

Restore Assignment failed java.util.concurrent.ExecutionException: (vmodl.fault.InvalidArgument) {

   faultCause = null,

   faultMessage = null,

   invalidProperty = Invalid role Ids [838861204]

}

Restore Authz failed

java.util.concurrent.ExecutionException: (vmodl.fault.InvalidArgument) {

   faultCause = null,

   faultMessage = null,

   invalidProperty = Invalid role Ids [838861204]

}

  at com.vmware.vim.vmomi.core.impl.BlockingFuture.get(BlockingFuture.java:70)

  at com.vmware.vim.dataservices.RestoreAuthz.processItem(RestoreAuthz.java:322)

  at com.vmware.vim.dataservices.RestoreAuthz.restoreAssignment(RestoreAuthz.java:247)

  at com.vmware.vim.dataservices.RestoreAuthz.main(RestoreAuthz.java:695)

Caused by: (vmodl.fault.InvalidArgument) {

   faultCause = null,

   faultMessage = null,

   invalidProperty = Invalid role Ids [838861204]

}

  at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)

  at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)

  at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)

  at java.lang.reflect.Constructor.newInstance(Constructor.java:526)

  at java.lang.Class.newInstance(Class.java:383)

  at com.vmware.vim.vmomi.core.types.impl.ComplexTypeImpl.newInstance(ComplexTypeImpl.java:173)

  at com.vmware.vim.vmomi.core.types.impl.DefaultDataObjectFactory.newDataObject(DefaultDataObjectFactory.java:26)

  at com.vmware.vim.vmomi.core.soap.impl.unmarshaller.ComplexStackContext.<init>(ComplexStackContext.java:31)

  at com.vmware.vim.vmomi.core.soap.impl.unmarshaller.UnmarshallerImpl$UnmarshallSoapFaultContext.parse(UnmarshallerImpl.java:141)

  at com.vmware.vim.vmomi.core.soap.impl.unmarshaller.UnmarshallerImpl$UnmarshallSoapFaultContext.unmarshall(UnmarshallerImpl.java:102)

  at com.vmware.vim.vmomi.core.soap.impl.unmarshaller.UnmarshallerImpl.unmarshalSoapFault(UnmarshallerImpl.java:89)

  at com.vmware.vim.vmomi.core.soap.impl.unmarshaller.UnmarshallerImpl.unmarshalSoapFault(UnmarshallerImpl.java:84)

  at com.vmware.vim.vmomi.client.common.impl.SoapFaultStackContext.setValue(SoapFaultStackContext.java:41)

  at com.vmware.vim.vmomi.client.common.impl.ResponseUnmarshaller.unmarshal(ResponseUnmarshaller.java:112)

  at com.vmware.vim.vmomi.client.common.impl.ResponseImpl.unmarshalResponse(ResponseImpl.java:273)

  at com.vmware.vim.vmomi.client.common.impl.ResponseImpl.setResponse(ResponseImpl.java:230)

  at com.vmware.vim.vmomi.client.http.impl.HttpExchangeBase.parseResponse(HttpExchangeBase.java:144)

  at com.vmware.vim.vmomi.client.http.impl.HttpExchange.run(HttpExchange.java:51)

  at com.vmware.vim.vmomi.client.http.impl.HttpProtocolBindingBase.executeRunnable(HttpProtocolBindingBase.java:186)

  at com.vmware.vim.vmomi.client.http.impl.HttpProtocolBindingImpl.send(HttpProtocolBindingImpl.java:115)

  at com.vmware.vim.vmomi.client.common.impl.MethodInvocationHandlerImpl$CallExecutor.sendCall(MethodInvocationHandlerImpl.java:581)

  at com.vmware.vim.vmomi.client.common.impl.MethodInvocationHandlerImpl$CallExecutor.executeCall(MethodInvocationHandlerImpl.java:562)

  at com.vmware.vim.vmomi.client.common.impl.MethodInvocationHandlerImpl.completeCall(MethodInvocationHandlerImpl.java:348)

  at com.vmware.vim.vmomi.client.common.impl.MethodInvocationHandlerImpl.invokeOperation(MethodInvocationHandlerImpl.java:308)

  at com.vmware.vim.vmomi.client.common.impl.MethodInvocationHandlerImpl.invoke(MethodInvocationHandlerImpl.java:182)

  at com.sun.proxy.$Proxy70.addAccessControlList(Unknown Source)

  at com.vmware.vim.dataservices.RestoreAuthz.processItem(RestoreAuthz.java:319)

  ... 2 more

Removing Client@1511370100 reference from CompiledHttpConfiguration@1477403302, 0 active clients left.

Shutting down CompiledHttpConfiguration@1477403302 as there are no more clients.

Shutting down the sync ClientConnectionManager

Shutting down connection monitor thread

Shutting down the connection monitor.

Total Memory: 491

Max Memory: 491

Free Memory: 443

Used Memory: 48

Exiting with code 1 after Restore Authz.

2017-01-23T10:55:55.42Z INFO is Restore Authz data StdErr:

2017-01-23T10:55:55.42Z INFO is

2017-01-23T10:55:55.43Z ERROR __main__ Upgrade Phase 'is:Import' failed. Exception: 'instancemethod' object has no attribute '__getitem__'

Traceback (most recent call last):

  File "Z:\PFiles\VMware\CIS\cis_upgrade_runner\payload\componentPhaseLauncher.py", line 379, in main

    executionResult = systemExtension(exeContext)

  File "Z:\PFiles\VMware\CIS\cis_upgrade_runner\libs\sdk\extensions.py", line 94, in __call__

    result = self.extension(*args)

  File "Z:\PFiles\VMware\CIS\cis_upgrade_runner\libs\sdk\extensions.py", line 110, in _func

    return func(*args)

  File "Z:\PFiles\VMware\CIS\cis_upgrade_runner\payload\component-scripts\is\__init__.py", line 686, in importData

    execIsCmd(reporter, restoreAuthz, "Restore Authz data", 50, destComponentAssistant)

  File "Z:\PFiles\VMware\CIS\cis_upgrade_runner\payload\component-scripts\is\__init__.py", line 242, in execIsCmd

    raiseUpgradeFailed(opStr)

  File "Z:\PFiles\VMware\CIS\cis_upgrade_runner\payload\component-scripts\is\__init__.py", line 218, in raiseUpgradeFailed

    if os.environ.has_key['VLSI_CLIENT_PROTOCOLS']:

TypeError: 'instancemethod' object has no attribute '__getitem__'

---

Interestingly enough, the Role ID doesn't appear to exist:

PowerCLI C:\> Get-VIRole -Id 838861204

PowerCLI C:\>

Thoughts anyone?

[JonathanThorpe]

After my previous attempt, I did a clone of the VC, SQL and DCs and did a successful upgrade, so I thought that whatever it was, it must have been something transient within the environment.

A couple of days ago, I reattempted another upgrade in production, this time trying to mimic the behaviour in the clone. The following was done:

1. Shut down everything that could potentially interact with VC.

2. Disconnect (not remove) the hosts from VC - this was after some discussion with VMware support.

3. Went further to actually isolate the VC, SQL and DC from the main network - this mimics what we had in the clone.

4. Upgrade failed at the same point with reference to the same Role ID.

Still working with VMware support on trying to find a solution to this. Their advice was to try an intermediate upgrade to 5.5 u3e which did appear to complete with a few issues which we managed to work around (presumably).

VC 6u2  then installed successfully, however we found that:

1. Global Permissions for anything within the domain (<domain>.local) were no longer present.

2. Groups pertaining to <domain>.local were missing, but users still appeared.

Interestingly enough, VPX_ACCESS which contains all of these was still in tact, but nothing shows up. I'm beginning to suspect that roles are assigned to objects within the inventory service in which 5.5 u3e might have dropped during the upgrade.

[ANSServiceDesk]

I experienced this issue and after raising an SR with VMware they provided the below fix which has worked for me:

1. Take a snapshot of the vCenter Server and backup of VCDB of the existing setup.

2. On the vCenter Server, stop these services in this order:

       VMware VirtualCenter Server service

       VMware Inventory service

3. Connect to the vCenter Server database.

Note:  Ensure to take a backup of the database before proceeding.

4. Run this query to truncate the VPX_PROPERTY_BULLETIN table:

truncate table VPX_PROPERTY_BULLETIN;

5. Run this query to ensure that the VPX_PROPERTY_BULLETIN table is empty:

select * from VPX_PROPERTY_BULLETIN;

6. Start the VMware VirtualCenter Server service.

Note: Allow 2-3 minutes before proceeding. This will repopulate 'VPX_PROPERTY_BULLETIN'

7. Start the VMware Inventory Service. Wait for 10-15 minutes depending on inventory size.

Note: If there are other products that use the VMware Inventory Service, they may require a restart to function properly.

Above step will re-set xdb documents. If the role id '603979981' is not present in VCDB. This invalid id will not be part of any XDB document.

8. Proceed to upgrade to 6.0. Should be successful.

[JonathanThorpe]

Hi,

These are exactly the same steps that I ended up doing as well - apologies for not posting back here as this may have helped.

It took Engineering a while to sort it out, but I managed to find out what was causing this in the end:

1. There were RBAC permissions stored within the inventory service to Roles (as noted by the Role ID) that didn't exist in the VMware LDAP/ADAM.

2. When the Inventory Service data is exported during the upgrade, it retains these permissions in the inventory service.

3. When attempting to restore the permissions, there was no corresponding role within the LDAP/ADAM which resulted in the error we saw.

It did take VMware a while to sort this out, however I did come up with a couple of alternative solutions - this one was by far the safest.

1. There's a file called "assignments" that's generated during the export process of the upgrade. If you delete any permission references to the erroring Role ID seemed to address the problem.

2. Manually adding the Role ID back into LDAP/ADSM using ADSI Edit seemed to work as well.

Each of these had its shortcomings, so I was glad to find a VMware blessed solution in the end.