VMware Cloud Community
actyler1001
Enthusiast
Enthusiast

Can't patch beyond VCSA 7.0.1.00300: Invalid Type,expected String, instead got NoneType

Hi everyone, I've tried to patch my lab instance of the VCSA multiple times since originally upgrading to 7.0.1.00300 and it's been rough. I must say that the VMware 6.5 and 6.7 builds were much easier to maintain thus far.

Anyway, the latest issue occurred while using the VAMI to stage the 7.0.2.00200 update first, then ran install after stage was completed.  I got this error and after about an hour of waiting, no change.

actyler1001_0-1623612770196.png

Revert snapshot, disable HA for my single cluster, disable DRS, apply "config.vcls.clusters" hack to get vCLS VMs redeployed.  Enable HA, Enable DRS, all normal again.  That was fun... NOT.

It is worth noting that my original upgrade to this version of the VSCA 7 was really rough too, I have a post about there here:
https://tylercrew.com/?p=428

Regards,
Adam Tyler

Tags (2)
27 Replies
actyler1001
Enthusiast
Enthusiast

Tried to update via the CLI just now, still failed.  Love how it asks you to go grab the log file but gives you no information about the file name or where it is stored.  Garbage...

---

Command> software-packages install --staged
[2021-06-13T20:09:12.164] : For the first instance of the identity domain, this is the password given to the Administrator account. Otherwise, this is the password of the Administrator account of the replication partner.
Enter Single Sign-On administrator password:

[2021-06-13T20:09:19.164] : Validating software update payload
[2021-06-13T20:09:19.164] : UpdateInfo: Using product version 7.0.1.00300 and build 17491101
[2021-06-13 20:09:19,718] : Running validate script.....
[2021-06-13T20:09:26.164] : Validation successful
[2021-06-13 20:09:26,748] : Copying software packages [2021-06-13T20:09:26.164] : ISO mounted successfully
152/152
[2021-06-13T20:10:55.164] : ISO unmounted successfully
[2021-06-13 20:10:55,368] : Running system-prepare script.....
[2021-06-13 20:11:04,402] : Running test transaction ....
[2021-06-13 20:11:13,443] : Running prepatch script.....
[2021-06-13 20:14:23,781] : Upgrading software packages ....
[2021-06-13T20:18:37.164] : Setting appliance version to 7.0.2.00200 build 17958471
[2021-06-13 20:18:37,424] : Running patch script.....
[2021-06-13T20:28:22.164] : Patch command patch failed
[2021-06-13T20:28:22.164] :
Mismatch:
summary: Internal error occurs during execution of update process .
resolution: Send upgrade log files to VMware technical support team for further assistance.

Reply
0 Kudos
VMAKS
VMware Employee
VMware Employee

@actyler1001


Thank you for posting on VMware Communities.

There are many ways through which vCenter Server support bundle can be collected which are discussed in the below KB:
https://kb.vmware.com/s/article/1011641

 

Arun Kumar

Install Upgrade Specialist

"Did you find this helpful? Let us know by completing this survey (takes 1 minute!)

Regards,
Arun Kumar
Install Upgrade Specialist
"Did you find this helpful? Let us know by completing this survey (takes 1 minute!)
Reply
0 Kudos
actyler1001
Enthusiast
Enthusiast

I created a support package after the failed install using the bash cli.  Downloaded via WinSCP.  Can't find anything helpful.  Tried deploying an older build of vCenter 7.0.2u2a, failed with same error.  This is going well...

Reply
0 Kudos
actyler1001
Enthusiast
Enthusiast

software-packages.log entries to note....

---------------------

2021-06-15T21:03:09.166 [60667]DEBUG:vmware.vherd.base.software_update:Restored software update
Staged status = True
Staged packages:

{'libsolv': {'type': 'rpm', 'relativepath': 'libsolv-0.6.35-5.ph3.x86_64.rpm', 'epoch': '(none)', 'version': '0.6.35', 'sha1val':

Installed packages:
{}
Payload packages:
{}
MetaData:
{'category': 'Bugfix', 'kb': 'https://docs.vmware.com/en/VMware-vSphere/7.0/rn/vsphere-vcenter-server-70u2b-release-notes.html', 'leaf_services': ['vmware-pod', 'vsphere-ui', 'wcp', 'vlcm'], 'vendor': 'VMware, Inc.', 'name': 'VC-7.0U2b', 'tags': '[]', 'type': 'Update', 'version_supported': '[]', 'size in MB': 5142, 'releasedate': 'May 25, 2021', 'executeurl': 'https://my.vmware.com/group/vmware/get-download?downloadGroup=VC70U2B', 'version': '7.0.2.00200', 'updateversion': True, 'allowedSourceVersions': '[7.0.0.0,]', 'buildnumber': '17958471', 'rebootrequired': False, 'productname': 'VMware vCenter Server', 'changesetnumber': '8919831', 'summary': {'id': 'patch.summary', 'translatable': 'In-place upgrade for vCenter appliances.', 'localized': 'In-place upgrade for vCenter appliances.'}, 'severity': 'Critical', 'TPP_ISO': False, 'thirdPartyInstallation': False, 'timeToInstall': 0, 'requiredDiskSpace': {'/storage/core': 13.506058044433594, '/storage/seat': 58.03203125}, 'eulaAcceptTime': '2021-06-15 20:40:21 UTC', 'installdate': '2021-06-15T20:51:51.677Z'}
Script results:
CollectRequirementsHook {'requirements': {'requiredDiskSpace': {'/storage/core': 13.506058044433594, '/storage/seat': 58.03203125}, 'rpmIgnoreList': [], 'questions': [{'userDataId': 'vmdir.password', 'text': {'id': 'vmdir.password.text', 'translatable': 'Single Sign-On administrator password', 'localized': 'Single Sign-On administrator password'}, 'description': {'id': 'vmdir.password.desc', 'translatable': 'For the first instance of the identity domain, this is the password given to the Administrator account. Otherwise, this is the password of the Administrator account of the replication partner.', 'localized': 'For the first instance of the identity domain, this is the password given to the Administrator account. Otherwise, this is the password of the Administrator account of the replication partner.'}, 'kind': 'password', 'allowedValues': [], 'defaultAnswer': None}], 'rebootRequired': False}, 'patchInfo': {'timeToInstall': 0, 'patchSummary': {'id': 'patch.summary', 'translatable': 'In-place upgrade for vCenter appliances.', 'localized': 'In-place upgrade for vCenter appliances.'}}, 'mismatches': []}
ValidationHook {'mismatches': []}
PrepareProductInstallationHook {}
PrepatchHook {}
PatchHook {'mismatches': [{'severity': 'error', 'text': {'id': 'patch.internal.error.text', 'translatable': 'Internal error occurs during execution of update process %(0)s.', 'args': [''], 'localized': 'Internal error occurs during execution of update process .'}, 'description': None, 'resolution': {'id': 'patch.internal.error.resolution', 'translatable': 'Send upgrade log files to VMware technical support team for further assistance.', 'localized': 'Send upgrade log files to VMware technical support team for further assistance.'}}]}

2021-06-15T21:03:09.166 [60667]ERROR:root:
Mismatch:
summary: Internal error occurs during execution of update process .
resolution: Send upgrade log files to VMware technical support team for further assistance.

2021-06-15T21:03:09.166 [60667]DEBUG:vmware.vherd.base.software_update:Destroyed software update state

2021-06-15T21:03:09.166 [60667]DEBUG:vmware.vherd.base.software_update:Destroyed software update state

2021-06-15T21:03:09.166 [60667]DEBUG:vmware.vherd.base.software_update:Initialized software update state

Reply
0 Kudos
actyler1001
Enthusiast
Enthusiast

Never could get this figured out, ended up deploying a new vCenter instance on the latest build.

Regards,

Adam Tyler

mholloway1
Contributor
Contributor

I get same error after I had to spend 3 days with vmWare support to fix the problems with the previous version upgrade mess.

Still no resolution. And just in time a vmaware rep called to "check up" and got an earful. What does she ask? Did you send a support request? VMware is on the slide.

Reply
0 Kudos
actyler1001
Enthusiast
Enthusiast

Bro, you said it.  I'm not amused with VMware as of late at all.  Tons of problems with the vSphere 7 upgrade.  Nothing like administering 5.x or 6.x.  Support has taken a real turn too.  No one in support that I've interacted with can really assist.  They just ask you for logs and sit on the issue for days/weeks until you figure it out yourself.  Yikes.

Regards,

Adam Tyler

andrey09
Contributor
Contributor

Ha-ha ... I wondered if I only one who always sending tons of logs to helpless tech support from VMware last few years. Got that error too.

Reply
0 Kudos
actyler1001
Enthusiast
Enthusiast

It's so bad.  I can't believe I am saying it, but Hyper-V is looking better all the time.

Latest outrage is this: https://www.provirtualzone.com/vsphere-7-update-2-loses-connection-with-sd-cards-workaround/

Tried upgrading some hosts to vSphere 7u2 and the USB/SD cards used for the OS just go offline randomly.  Opened a case with support and they want to start with logs.  Many customers experiencing this across many different server brand/models.  Crazy.  We're unpaid beta testers now.

Regards,

Adam Tyler

Reply
0 Kudos
mholloway1
Contributor
Contributor

Latest error: 7.02 patch! "Unexpected error occurred while fetching the updates". I'm afraid to even touch the platform anymore. We run mission critical apps for our business on this software. Should not have to call a technical support that has no answers to the problems of applying patches for vulnerabilites that increase risk and cost.

Did DELL do this to VMware? I think so. 

  • 7.0.2 Updates Available
  • Build:17958471
Reply
0 Kudos
VMAKS
VMware Employee
VMware Employee

@mholloway1

 

Regarding the "Unexpected error occurred while fetching the updates"

Please refer to the VMware vCenter Server 7.0 Update 2b Release Notes

https://docs.vmware.com/en/VMware-vSphere/7.0/rn/vsphere-vcenter-server-70u2b-release-notes.html

 

Which discusses the below information:

 

  • When you use the Update Planner, in the vSphere Client you might see Unexpected error occurred while fetching the updates

When you use Update Planner, which is part of vSphere Lifecycle Manager, used to facilitate vCenter Server updates, you might see the following error in the vSphere Client:
Unexpected error occurred while fetching the updates


The issue occurs when you use a custom HTTPS port that prevents you from running interoperability reports by using the vSphere Client.

Workaround: Manually invoke the API. For more details, see the vSphere Automation API.

 

Regards,
Arun Kumar Sharma
Install Upgrade Specialist
"Did you find this helpful? Let us know by completing this survey (takes 1 minute!)

 

Regards,
Arun Kumar
Install Upgrade Specialist
"Did you find this helpful? Let us know by completing this survey (takes 1 minute!)
Reply
0 Kudos
showkwave
Contributor
Contributor

I ran into the same issue trying to upgrade from 7.0.1.0020 to 7.0.2.00400.  We were doing the upgrade from the appliance shell with the ISO mounted from a datastore to the vcenter vm.  The error message was less than helpful

showkwave_1-1631112342410.png

 

In our case, this vcenter had been upgraded from 6.7 to 7.0 and were originally part of an external PSC and then converted to embedded  as part of the upgrade.  Clues to the issue were found near the end of  /storage/log/vmware/support/applmgt/PatchRunner.log file on the vcenter appliance.

vmdir: Patch ERROR faled to delete the legacy schema tombstone entry on DC testdc.mydomain.com.

vmdir .Patch ERROR ldap_delete: No such object (32) 

additional info : (9703) ( MDB_NOTFOUND: No Matching key/data pair found

There were remnants of and old sso domain (testdc.mydomain.com) members remaining in the PSC and we were able to identify and delete from sections

'Domain Controllers', DomainDNSZones'  using  jxplorer tool before attempting the upgrade to 7.0.2.00400.  

After reverting the vcenter snapshot, using jxplorer to clean up the orphaned entries, the next upgrade attempt was successful.

 

Hopefully, this is helpful .

 

Reply
0 Kudos
ksl281
Contributor
Contributor

We are facing the exact same issue - and this is from an freshly installed 7.0.2.00200!

The amount of issues and bugs introduced in 7.0 (both vCenter and ESXi), is massive!

Really not impressed!

Reply
0 Kudos
andrey09
Contributor
Contributor

In my case it were problems with trust for some solution user certificates.

I've been guided to Lsdoctor KB https://kb.vmware.com/s/article/80469 , so it fixed some errors and update processed smoothly.

Reply
0 Kudos
Priiiit
Contributor
Contributor

Hi, In the VCF solution we got the same VCSA upgrade error in the "Converting data as part of post install" around 93% mark. The issue according to support was missing version.txt file in /etc/vmware-vlcm folder. The file must contain the expected version number. We were upgrading from VCSA 7.0.1.00301-17956102 so in our case the file had to contain 0.0.2.

Reply
0 Kudos
AnyykeyNL
Contributor
Contributor

Having upgrade issues here as well.

 

Trying from a clean install 7.0.2.00500 to 7.0.3.00000 build 1870043

[2021-10-09T23:01:11.282] : Validating software update payload
[2021-10-09T23:01:11.282] : UpdateInfo: Using product version 7.0.2.00500 and build 18455184
[2021-10-09 23:01:11,115] : Running validate script.....
[2021-10-09T23:01:15.282] : Validation successful
[2021-10-09 23:01:15,136] : Copying software packages [2021-10-09T23:01:15.282] : ISO mounted successfully
133/133
[2021-10-09T23:04:05.282] : ISO unmounted successfully
[2021-10-09 23:04:05,236] : Running system-prepare script.....
[2021-10-09 23:04:10,259] : Running test transaction ....
[2021-10-09 23:04:12,281] : Running prepatch script.....
[2021-10-09 23:05:46,435] : Upgrading software packages ....
[2021-10-09T23:07:57.282] : Setting appliance version to 7.0.3.00000 build 18700403
[2021-10-09 23:07:57,663] : Running patch script.....
[2021-10-09T23:09:53.282] : Patch command patch failed
[2021-10-09T23:09:53.282] :
Mismatch:
summary: Internal error occurs during execution of update process Traceback (most recent call last):
File "/storage/core/software-packages/scripts/patches/py/vmware_b2b/patching/phases/patcher.py", line 203, in patch
_patchComponents(ctx, userData, statusAggregator.reportingQueue)
File "/storage/core/software-packages/scripts/patches/py/vmware_b2b/patching/phases/patcher.py", line 84, in _patchCompo
_startDependentServices(c)
File "/storage/core/software-packages/scripts/patches/py/vmware_b2b/patching/phases/patcher.py", line 53, in _startDepen
serviceManager.start(depService)
File "/storage/core/software-packages/scripts/patches/libs/sdk/service_manager.py", line 901, in wrapper
return getattr(controller, attr)(*args, **kwargs)
File "/storage/core/software-packages/scripts/patches/libs/sdk/service_manager.py", line 794, in start
super(VMwareServiceController, self).start(serviceName)
File "/storage/core/software-packages/scripts/patches/libs/sdk/service_manager.py", line 665, in start
raise IllegalServiceOperation(errorText)
service_manager.IllegalServiceOperation: Service cannot be started. Error:

.
resolution: Send upgrade log files to VMware technical support team for further assistance.

 

Anyone any idea? I tried setting the start Timeout in the json files to 1500, but did not help

Reply
0 Kudos
ifescgit
Contributor
Contributor

Hi AnyykeNY, did you fixed this error?

I'm having the same issue when tried to update from 7 u2b to 7 u3a.

Did some tricks without success.

Reply
0 Kudos
MaciekTX
Contributor
Contributor

Did you find a solution for this one? I am running into same issue.

Reply
0 Kudos
AnyykeyNL
Contributor
Contributor

This is so annoying. I noticed there is a new update 7.0.3.00200 (November 11th) and with that I still get the same installhook error 😞

Reply
0 Kudos