Highlighted
Enthusiast
Enthusiast

External PSC upgrade error 6.0 -> 6.5

Jump to solution

Been beating our heads against this for a day -

Upgrading an external PSC from 6.0 to 6.5.  Tried this via the CLI **and** the UI.  We manage to get past all the phase 1 export stuff - when we hit phase 2 (copying data over), the pre-upgrade checks run again and fail on a well-known issue in the README:

The upgrade of a vCenter Server Appliance might fail because a dependency shared library path is missing
The upgrade of a vCenter Server Appliance might fail before the export phase and the error log shows: /opt/vmware/share/vami/vami_get_network: error while loading shared libraries: libvami-common.so: cannot open shared object file: No such file or directory. This problem occurs due to missing dependency shared library path.

but this README item covers the fix for the FIRST phase on the original PSC you're upgrading, and does not anticipate the check failing on the NEW PSC you are deploying to.

The error surfaced to the installers is "Cannot collect upgrade requirements. For more details check out the server logs."

Not sure how/why these checks are even running again on phase 2, but I see no viable workaround.


Wondered if anyone has seen this, before I file a support ticket.

Tags (2)
1 Solution

Accepted Solutions
Highlighted
Contributor
Contributor

Hi eai,

First of all, sorry for my bad english.

Have you tried to put in your /etc/profile of your PSC source AND destination (its important on both ) ? Thi

I put in /etc/profile and not to bashrc because i can't execute it i don't know why.

My error logs was in this file : /var/log/vmware/upgrade/upgrade-requirements.log on PSC destination.

ERROR upgrade_commands Pre-upgrade checks failed. Check upgrade-source-requirements.log log for details.

2016-11-18T14:22:13.43Z ERROR transport.guestops Invalid command: "/bin/bash" --login -c '/opt/vmware/share/vami/vami_get_network eth0 1>/tmp/vmware-root/exec-vmware181-stdout 2>/tmp/vmware-root/exec-vmware45-stderr'

None

2016-11-18T14:22:13.43Z ERROR upgrade_commands Unable to execute pre-upgrade checks on "PSC IP Source"


And this one :

ERROR apply_networking Failed to get source system network configuration stdout: , stderr: /opt/vmware/share/vami/vami_get_network: error while loading shared libraries: libvami-common.so: cannot open shared object file: No such file or directory

, exit-code: 127.

I created a ticket on vmware technical support and the answer was :

"I validate this solution, this is a bug, I requested the publication of a KB "

So a KB will be born soon, i guess.

Thanks to PuliSukumar again !

View solution in original post

0 Kudos
20 Replies
Highlighted
Enthusiast
Enthusiast

Upgrade of VCSA done through CLI or UI have 2 stages.

stage 1 ovf deployment

stage 2 actual upgrade.

And Each stage have there own pre-checks

i.e usually stage 1 have installer related pre-checks like  verify source vc credentials as well as checking source and target container credentials. etc..

similarly stage 2 also have its own precheck like verifying source and newly deployed appliance are deployed on same network , verifiying the validity of ssl certificate of the  legacy vc etc..

Many of these prechecks are also overlapped.

>>Not sure how/why these checks are even running again on phase 2, but I see no viable workaround.

its expected and there is no work around for it.

regards,

Pradeep

0 Kudos
Highlighted
Enthusiast
Enthusiast

Thanks, so what's the fix?  I can't proceed.  My PSC is a completely valid 6.0 PSC appliance.  I also have another PSC in replicating mode and it results in exactly the same error.  How is it expected that I should get past this error; it's a fatal error.

Cheers

0 Kudos
Highlighted
Enthusiast
Enthusiast

whats the exact error message and where its failing .

if its failing at

The upgrade of a vCenter Server Appliance might fail because a dependency shared library path is missing
The upgrade of a vCenter Server Appliance might fail before the export phase and the error log shows: /opt/vmware/share/vami/vami_get_network: error while loading shared libraries: libvami-common.so: cannot open shared object file: No such file or directory.

This problem occurs due to missing dependency shared library path


have performed the workaround mentioned??

0 Kudos
Highlighted
Enthusiast
Enthusiast

I have performed the workaround on the original (6.0) PSC.  I verified that the call works.

The error is happening again in phase 2 not phase 1.  There is not really an opportunity to log into the NEW PSC which is being deployed to fix this.

After the first time it failed, I logged into the new PSC VM anyway, after the failure, and applied the workaround.  I then restarted the phase 2 (using the UI installer).  It did not fix the issue. Smiley Sad

0 Kudos
Highlighted
Enthusiast
Enthusiast

As upgrade has failed during the export stage, it would have rendered the new appliance useless , so u delete the new appliance and again deploy newly PSC and perform the upgrade as u have already performed the work around on the old PSC.

The  work around is associated with the old PSC not new appliance.

0 Kudos
Highlighted
Enthusiast
Enthusiast

I appreciate your assistance, and this is exactly what I have done several times.  I applied the workaround on my existing 6.0 PSC.  This allows phase 1 to complete.  After phase 1 completes, phase 2's pre-upgrade checks begin on the NEW appliance.  At this point, the message above is displayed and the upgrade fails.  There is no workaround I can find and I cannot get past this.

0 Kudos
Highlighted
VMware Employee
VMware Employee

Hi,

    Couple more questions.

1) Did pre-checks of Stage2 pass?

     a) If Yes, at what stage the upgrade failed.

     b) If No, can you upload the vc-support from 6.5 appliance to thread so that I can see where the issue is.

Thanks,

Sukumar.

0 Kudos
Highlighted
Contributor
Contributor

Exactly same problem for us.

Precheck sage 2 failed with internal error.

When i check /var/log/vmware/upgrade/upgrade-source-requirements.log i can find this error :

ERROR apply_networking Failed to get source system network configuration stdout: , stderr: /opt/vmware/share/vami/vami_get_network: error while loading shared libraries: libvami-common.so: cannot open shared object file: No such file or directory

, exit-code: 127.

0 Kudos
Highlighted
VMware Employee
VMware Employee

Can you please upload the vc-support from 6.5 appliance.

0 Kudos
Highlighted
Contributor
Contributor

Of course, have you receive my tar.gz ?

Thx for your help.

0 Kudos
Highlighted
VMware Employee
VMware Employee

Hi,

    I got your support and issue is same as mentioned in Known Issues sections of release notes.

1) Did you try the workaround mentioned?

SSH to the source appliance.

Run the following command

echo "LD_LIBRARY_PATH=${LD_LIBRARY_PATH:+$LD_LIBRARY_PATH:}/opt/vmware/lib/vami/" > /root/.bashrc echo 'export $LD_LIBRARY_PATH' >> /root/.bashrc

exit the shell

Start upgrade again.

2) Please post your results post trying the workaround.

0 Kudos
Highlighted
Contributor
Contributor

Great !

its seems ok for stage 2 prechecks.

Be carreful in the echo command some quote and other things are missing. But i understood the idea. So i put this line (export LD_LIBRARY_PATH=/opt/vmware/lib/vami/) in /etc/profile on the source and destination PSC.

thanks you.

0 Kudos
Highlighted
Enthusiast
Enthusiast

How and when did you log into the destination PSC to put this in /etc/profile?  How did you even know to do that?  I did this in the SOURCE PSC but not destination one.... because you are not supposed to log into destination PSC till phase 2 finished, right?

0 Kudos
Highlighted
Enthusiast
Enthusiast

Answers:

1) no, never passes stage 2 precheck

2) absolutely, please see attached.  This attachment one is from running UI version of the installer.

FYI, here are all the combinations of things I have tried:

- VC as deploy target for new PSC, VC as source system for original PSC
- ESXi as target, VC as source
- VC as target, ESXi as source
- ESXi as target, ESXi as source
- Using temporary network with 22 prefix (my actual network topology)
- Using temporary network with 24 prefix (just in case there's an issue there)
- setting reverse DNS on temporary network (PTR) to existing hostname of PSC appliance
- setting reverse DNS on temporary network (PTR) to temporary name
- Using UI installer with all above combos
- Using CLI installer with all above combos
- Setting SSH enabled setting in CLI JSON template to true
- Setting SSH enabled setting in CLI JSON template to false (why not, just was hoping)

For the record, I disabled Automated DRS per the warning/instructions... just in case you needed to be sure.

All of these from Windows, btw.

0 Kudos
Highlighted
VMware Employee
VMware Employee

Hi eai,

Is your problem solved after referring the workaround below?

The upgrade of a vCenter Server Appliance might fail because a dependency shared library path is missing
The upgrade of a vCenter Server Appliance might fail before the export phase and the error log shows: /opt/vmware/share/vami/vami_get_network: error while loading shared libraries: libvami-common.so: cannot open shared object file: No such file or directory. This problem occurs due to missing dependency shared library path.

Workaround:

  1. Log in to the appliance Bash shell of the vCenter Server Appliance.
  2. Run the following commands.
    echo "LD_LIBRARY_PATH=${LD_LIBRARY_PATH:+$LD_LIBRARY_PATH:}/opt/vmware/lib/vami/" >> /root/.bashrc
    echo 'export $LD_LIBRARY_PATH' >> /root/.bashrc
  3. Log out of the appliance shell.
  4. Retry the upgrade.

You mentioned there is a quote issue in this. Could you please give more details on that?

0 Kudos
Highlighted
Contributor
Contributor

Hi eai,

First of all, sorry for my bad english.

Have you tried to put in your /etc/profile of your PSC source AND destination (its important on both ) ? Thi

I put in /etc/profile and not to bashrc because i can't execute it i don't know why.

My error logs was in this file : /var/log/vmware/upgrade/upgrade-requirements.log on PSC destination.

ERROR upgrade_commands Pre-upgrade checks failed. Check upgrade-source-requirements.log log for details.

2016-11-18T14:22:13.43Z ERROR transport.guestops Invalid command: "/bin/bash" --login -c '/opt/vmware/share/vami/vami_get_network eth0 1>/tmp/vmware-root/exec-vmware181-stdout 2>/tmp/vmware-root/exec-vmware45-stderr'

None

2016-11-18T14:22:13.43Z ERROR upgrade_commands Unable to execute pre-upgrade checks on "PSC IP Source"


And this one :

ERROR apply_networking Failed to get source system network configuration stdout: , stderr: /opt/vmware/share/vami/vami_get_network: error while loading shared libraries: libvami-common.so: cannot open shared object file: No such file or directory

, exit-code: 127.

I created a ticket on vmware technical support and the answer was :

"I validate this solution, this is a bug, I requested the publication of a KB "

So a KB will be born soon, i guess.

Thanks to PuliSukumar again !

View solution in original post

0 Kudos
Highlighted
VMware Employee
VMware Employee

Could you please try below steps and let me know if your issue is solved or not.

  1. Log in to the appliance Bash shell of the Source vCenter Server Appliance.
  2. Run the following commands.

        echo "LD_LIBRARY_PATH=${LD_LIBRARY_PATH:+$LD_LIBRARY_PATH:}/opt/vmware/lib/vami/" >> /etc/profile

        echo 'export LD_LIBRARY_PATH' >> /etc/profile

  1. Log out of the appliance shell.
  2. Retry the upgrade.
Highlighted
Enthusiast
Enthusiast

Thanks to both of you, Neelei and PuliSukumar.

Here is what I did -

1) apply workaround with LD_LIBRARY_PATH to SOURCE PSC only

2) I did this in /etc/profile NOT .bashrc

3) it was not needed to do this on destination PSC, if you do this in /etc/profile on source PSC, it works

I will say, the Release Notes are wrong, .bashrc does not work.  In addition, the Release Notes have a bug with the export command, you do not put the $ sign in front of the variable when exporting.

So, yes, export LD_LIBRARY_PATH=/path is the correct solution as long as you do it in /etc/profile.

I hope this gets resolved so others have a better experience - this upgrade to 6.5 has by far been the most buggy and difficult of any VMware upgrade, and I have been a user since 5.0.

Highlighted
Contributor
Contributor

Your solution worked for me.

running echo "export LD_LIBRARY_PATH=/opt/vmware/lib/vami/" >> /etc/profile

did the trick.

Thank you!

0 Kudos