Hello all,
Just wanted to share: I was updating a HA VCSA environment and on the Witness appliance (the first one to be upgraded in an HA setup), the upgrade failed for no apparent reason. No useful messages. It just died after a few seconds saying "package update failed" and that was it.
In the file "/storage/log/vmware/applmgmt/software-packages.log" I could see it fail, but not due to anything wrong with packages or prerequisites or anything. All good and it still kept failing. Disk-space was not an issue. I could see no package-related problems what so ever.
Until the following lines caught my eye:
DEBUG:vmware.vherd.base.software_update:STDERR: You are required to change your password immediately (root enforced)
You are required to change your password immediately (root enforced)
error=You are required to change your password immediately (root enforced)
Hmmm. So out of curiosity, I entered the Shell and used the passwd command to simply re-enter the root password (the same as before). As passwd does not check if the password was already used etc. it accepted it.
I left the shell (went back to command mode) and tried to update again. This time it went through without a hitch. Problem solved.
Summary: if VCSA updates seem to fail and you see nothing that has anything to do with the packages etc. etc., look for the lines above. There is a good chance that the root password needs to be reset (even though you could login with it just fine). Appliances that have been running for a while are prone to this issue.
______________________________________________________________________________________________________________
As a reference, this is my workflow for updating HA VCSA Setup's:
Download the FP patch ISO from VMware and upload it to a datastore.
Example: the 6.5 U2 patch ISO is called "VMware-vCenter-Server-Appliance-6.5.0.20000-8307201-patch-FP.iso"
Put the VCSA HA Cluster in normal Maintenance mode
SSH into the active VCSA as root by using the public IP address.
We are not going to patch this appliance yet. It will actually be the 3rd and last one that gets patched. The Witness appliance is the first.
Enter/run the Shell using the "shell" command
From there, SSH into the witness appliance using it's "vCenter HA IP Address". Do not run the shell. You must stay in command mode.
Attach the Patch ISO to the witness appliance
Then, from the Command mode, enter the following commands:
Command> software-packages stage --iso --acceptEulas
Command> software-packages list --staged
Command> software-packages install --staged
Command> shutdown reboot -r "patch reboot"
(this throws you out of the SSH session and back into the Shell of the active appliance)
Detach the ISO
Wait for the witness Appliance to come back and under "HA Monitoring" in the Web GUI, wait for, and verify that things like replication etc. are all green and good.
You are still in the shell on the active appliance.
SSH into the passive appliance using it's "vCenter HA IP Address". Do not run the shell. You must stay in command mode.
Attach the Patch ISO to the passive appliance
Command> software-packages stage --iso --acceptEulas
Command> software-packages list --staged
Command> software-packages install --staged
Command> shutdown reboot -r "patch reboot"
(this throws you out of the SSH session and back into the Shell of the active appliance)
Detach the ISO
Wait for the passive Appliance to come back and under "HA Monitoring" in the Web GUI, wait for, and verify that things like replication etc. are all green and good.
Log out from the appliance shell of the Active node.
Initiate a vCenter HA failover manually (wait 5 to 10 minutes for it to complete).
Have patience. Ramming on refresh in the browser will not speed things up. There will be replication errors during the failover but they should clear after a couple of minutes.
Afterwards, you can see in the vSphere Web Client that the Passive node has become the Active node and the Active node became the Passive node.
Clear your browser's cache. It you don't, you could have issues with the version number under "about" still displaying the old version, graphical elements missing etc.
Really, clear your browsers cache...
Verify that you are on the "new" active node by checking the version (use the "about" function). If it still shows the old version, did you clear the browser cache before you logged in?
Log in as root to the appliance shell of the new Active node by using the public IP address.
SSH into the, now new, passive appliance (used to be the original active one) using it's "vCenter HA IP Address". Do not run the shell. You must stay in command mode.
To patch the new Passive node:
Attach the Patch ISO to the passive appliance
Command> software-packages stage --iso --acceptEulas
Command> software-packages list --staged
Command> software-packages install --staged
Command> shutdown reboot -r "patch reboot"
Do your checks. Replication etc. should all be green and good.
Manually Fail back (again, wait 5 to 10 minutes for it to complete). The roles switch again.
Do your checks. Replication etc. should all be green and good.
Exit VCSA HA Cluster Maintenance mode
Done, get a beer
