VSCA Update failed again. (6.7.0.2xxx->6.7.0.32...)

[Alhazred242]

Hello,

same problem like some month before.

We habe here a VSCA (Deploy from a BareMetal with the Migration Manager). I try to Install the last Update (i try the last 3) and the VMWare vCenter-Server (and 2 other Services) crash at Start.

I check one Logfile and see only a exit 10.

I need some Tipps for Troubleshooting. I have the SupportBundle for Investigation and start a Rollback.

[NathanosBlightc]

Hi

Give us more information about your problem's situation please

Did you upgrade the VCSA to up3 completely and then issue has been occurred?

Did you check remaining spaces of your VCSA virtual Disks (VMDK)?? (df -h)

and one point: you said bare metal... do you means your old VCSA is a physical/Windows-Based vCenter server?

[Alhazred242]

Hi,

here some more Information:

We use VMWare ESXi since 8 years or so. The vCenter Server was long time a physical Server (2008R2). At the last big Upgrade we use the Chance and switch with the Migration Assistant to a VSCA.

The Appliance works fine. The Conditions are good. The Management Interface looks fine.

The Installed Version is 6.7.0.2100. I Update the Server over the Management Port.

First i try the update to .3000 and it fails. After the Restart, the Center Server crash direct at the Start. AFAIK ( in can check it tomorrow) i see in the vpxd.log Exit Code 1.

I have restore the Backup. Restart the Server and go back to the "daily doing".

I try then the next Update in May (.3100?) and the last time the Update mid July. Same Problem. The most Services Starts fine, Center Server crashes at start.

The Disks looks good (i check it tomorrow again). I think, there is a bad Config since long time (we Migrate the Server from 5->5.5->6.0->6.5). The only 3rd Party Plugin is IBM Spectrum Protect (v 8.4.1).

[IRIX201110141]

2 Questions:

- Is your Host based on a Intel Skylake or newer?

- If you have updated VCSA vHW to higher than 10/13?

Regards,
Joerg

[NathanosBlightc]

Did you check the firstboot.log? There may be some information about all problematic state of your VCSA.

Is your PSC embedded or external? did you check its services?

What's the status of license of vCenter? Did you check its correctness after upgrade?

If you migrate your VCSA's VM to another ESXi, is the problem still exist or not?

[Alhazred242]

Processor is Xeon E5-3650 (Sandy Bridge)

VSCA HW is 13.

The Hostsystem is a ESX 6.5 on IBM X Series.

I try at the First Tests a nother ESXi Server, and same Problem.

Licences are ok (but this Service also Crash)

I check now the firstboot log.

[Alhazred242]

On another Host it is the same.  PSC is embedded. I can try another Update today, and make a List which Services are crashed.

[IRIX201110141]

In April this year i updated a customer environment

- 6.5/6.7? -> VCSA 6.7latest

- Replace Dell Server with R740(Skylake)

All went very well and runs 2 days.... until i schedule the vHW upgrade of the VCSA to the latest/highest and pressing the restart button.  After that our VCSA reacts as same as hours which mean a lot of servcies starting very well but at the end some are crashing or not starting and already started services disapeers.

We opened a #SR and it took a long fight with GSS and after 3 days they found the solution

- As we knows within the vHW also its hardcoded which CPU features are presented to the VM(GuestOS)

- The VCSA contains libraries which acts differently when seeing a Skylake and the Features are enabled. Certains code contains bugs the the services crashed or never starts

The engineer de-register the VCSA VM, edit the *.vmx and change vHW to 10. Register and starts the VM and our VCSA was back into the game and runs perfectly.

Your CPUs are older so its another issure youre facing.  I suggest to open a #SR.

GSS als says they seen a lot of problem when the VCSA is jonied a Windows AD. The suggest to use the "LDAP" option to connect to AD rather than using the kerberos integration.

Regards,
Joerg

[Alhazred242]

i can check the HW. I upgrade the HW for a last change yesterday. The redeployment from the Backup with the old vHW ist easier.

I restore the VSCA from the Backup. Select Medium Type for more resources. vHW Type is 10.

Shutdown the VM for Snapshot.

Reboot and Install the Upgrade at the WebUI.

Same.

Failed Services:

Content Library

Image Builder (we don´t use it)

vCenter Server

Update Manager

Profile Storage Service

I don´t have access to Licence Server to check it. I try at Monday Backup and Restore a VSCA to a new ESXi with the .3200 Version.

I check the first logs, and its look like a fail from vmware-sps. I think a need disable the 3rd Party Plugin and Check the Configs.

[RajeevVCP4]

How you are trying to update

By command line or GUI.

What error you are getting , provide full name of ISO which you are using for upgrade.

[Alhazred242]

I use the WebUI (Port 5480)

Patch for VMWare VCSA 6.7.0.32000 from Jul 16.

[ChrisFD2]

Have you logged a support call?

[Alhazred242]

Yes i have.

I start later with the next try.

[Alhazred242]

So, some testing and clean up and found a Problem.

I can't delete one Datacenter. There are a datastore who has no Host and some old Networks (without any hosts).

Any hint how can I delete these? I can access the DB or shell if we need it.

[NathanosBlightc]

Use in CLI (Shell/SSH) following commands to show your datastore's attributes:

esxcli storage vmfs extent list

esxcli storage core device list

Then copy naa.xxx of considered for next command:

esxcli storage core device detached remove -d naa.xxx

[NathanosBlightc]

And Also if you cannot remove it with ESXCLI, you can try this if you connect to the VCSA DB by management tools like PgAdmin and remove the specified datastore object.

I explained how to do it in the following link:

Connect and Manage VCSA Database (PostgreSQL)

[Alhazred242]

I need to try it with the DB connect. The host with the Datastore are no exist anymore and I don't have esxcli on the VCSA right? I have found also some Networks, which are no more attached to anything and I cannot remove then.

After I remove the datastore, it is the same. I cannot remove the Datacenter and the Update failed.

I check at this time the Database. Also I check the Backupfiles, when I create a new one, without any stats. I think, it is the best way, to Backup all Settings and deploy a new clean VM, without any Config Errors.

[NathanosBlightc]

Yes you don't have ESXCLI in VCSA, it's just for ESXi host. So you have to only remove your ORPHANED objects (like that datastore) from vCenter Server database.  But before that, I guess you know, you can remove datastore with PowerCLI but you need to mentioned host's name (required syntax) , So if you can, please try this PowerCLI too:

Remove-Datastore -Datastore Datastore -VMHost ESXi_IPAddress -Confirm:$false

and then you can remove Datacenter object:

Remove-Datacenter Datacenter

Also in following link describe about how to remove datastore forcefully with PowerCLI

PowerCLI script to forcibly delete a Datastore that is reported as in use. – The Lowercase w

[Alhazred242]

esxcli I used in the past for a similar Problem.

Here its looks like a little bit more chaos.

The Datastore are not attached to any Host. In the Datacenter are no hosts attached. The hosts that I removed, don't attached to this datastore. The Networks are from an old Project. I cannot remove the Networks, the option is not active. I think, the other guys just shutdown the Hosts and don't cleanup the environment. No VMs are attached to the datastore/network (the lists are empty).

I check the folders with the apiexplorer for the Datastore. I think I try to remove the artifacts from the DB, but IMHO it is the best, I save the Configs, Keys, DRS Settings and Setup a new VCSA. The VCSA was in the past a Windows System (since v5) and we have so many changes since then.

[Alhazred242]

I give it up. I remove the datastore from the DB, and can't remove the Datacenter. Try to remove the Networks (also refresh the PG Views for entity and so), the Appliance are broken.

I setup my new VCSA (with the evaluation License), apply my ESXi Server License and move all active Hosts (create the Clusters, Resource Pools and so on) After then, I only change the IP.

I think now the Config is clean (I check the datastores and VM for old ISO mounts etc.).

I take my initial Backup. Tomorrow I only need add IBM Spectrum Protect. and then everything is work back again.

Thanks for your your help.