Tried to upgrade VIO from 2.0 to 2.5, now connection to OMS fails

[raflax]

I had a working VIO version 2.0 and tried to upgrade it to 2.5.

The upgrade steps succeeded through the viocli dbverify step.

I went to provision the new VIO deployment and vCenter can no longer connect to the OMS instance.  I can ssh to it just fine, I've shutdown and restarted the vApp that contains OMS to no avail.  I tried to connect directly to the web interface at https://<OMS_server>:8443/VIO and logged in with the vCenter credentials as it suggests and I get the resulting http 500 error page from the servlet.

Anyone else get this and get past it?

Thanks.

[raflax]

Same thing..

root@nsx11-vio-manager:/opt/vmware/vio/patches/downloads# apt-get --reinstall install vio-cli=2.5.0.3955000

Reading package lists... Done

Building dependency tree      

Reading state information... Done

The following packages were automatically installed and are no longer required:

  libevent-2.0-5 libgssglue1 libnfsidmap2 libtirpc1 rpcbind

Use 'apt-get autoremove' to remove them.

The following extra packages will be installed:

  vio-cli

0 upgraded, 0 newly installed, 0 to remove and 168 not upgraded.

[raflax]

root@nsx11-vio-manager:/opt/vmware/vio/patches/downloads# apt-get -V install vio-cli=2.5.0.3955000

Reading package lists... Done

Building dependency tree      

Reading state information... Done

The following extra packages will be installed:

   vio-cli (2.0.0.3037964)

0 upgraded, 0 newly installed, 0 to remove and 167 not upgraded.

root@nsx11-vio-manager:/opt/vmware/vio/patches/downloads#

root@nsx11-vio-manager:/opt/vmware/vio/patches/downloads# dpkg -l | grep vio-cli

hi  vio-cli                                    2.0.0.3037964                    all          VIO CLI component provides vio* cli commands

Looks like it wants to install the version that is already installed.

[raflax]

Does the repo not think there is a 2.5 version of vio-cli?

[ssurana]

lets do the following:

1. uninstall the cli:  "apt-get remove vio-cli"

2. install again: "apt-get install vio-cli=2.5.0.3955000"

hopefully this will do the trick

[raflax]

Ok, apt-get did remove vio-cli but it can't install the new one:

root@nsx11-vio-manager:~# apt-get install vio-cli=2.5.0.3955000

Reading package lists... Done

Building dependency tree      

Reading state information... Done

Some packages could not be installed. This may mean that you have

requested an impossible situation or if you are using the unstable

distribution that some required packages have not yet been created

or been moved out of Incoming.

The following information may help to resolve the situation:

The following packages have unmet dependencies:

vio-cli : Depends: python-openstackclient but it is not installable

           Depends: python-oslo-vmware but it is not installable

E: Unable to correct problems, you have held broken packages.

[ssurana]

okay, that's something..

Can you provide the output of:

1. apt-cache madison python-oslo-vmware

2. apt-get update

3. apt-get install python-oslo-vmware

[raflax]

root@nsx11-vio-manager:~# apt-cache madison python-oslo-vmware

root@nsx11-vio-manager:~# apt-get update

Ign file: trusty/kilo InRelease

Ign file:  InRelease                                 

Ign file:  InRelease                                 

Ign file: trusty/kilo Release.gpg                    

Ign file:  Release.gpg                               

Ign file:  Release.gpg                               

Ign file:  Release                                   

Ign file:  Release                                                   

Get:1 file: trusty/kilo Release [2097 B]                             

Ign file:  Translation-en                                                     

Ign file:  Translation-en                                            

Ign file: trusty/kilo/main Translation-en                                     

Ign file: trusty/kilo/universe Translation-en                                 

Ign http://ubuntu-cloud.archive.canonical.com trusty-updates/juno InRelease   

Get:2 http://ubuntu-cloud.archive.canonical.com trusty-updates/juno Release.gpg [543 B]

Hit http://ubuntu-cloud.archive.canonical.com trusty-updates/juno Release

Ign http://ubuntu-cloud.archive.canonical.com trusty-updates/juno Release

Ign http://ubuntu-cloud.archive.canonical.com trusty-updates/juno/main amd64 Packages/DiffIndex

Ign http://ubuntu-cloud.archive.canonical.com trusty-updates/juno/main i386 Packages/DiffIndex

Hit http://ubuntu-cloud.archive.canonical.com trusty-updates/juno/main amd64 Packages

Hit http://ubuntu-cloud.archive.canonical.com trusty-updates/juno/main i386 Packages

Ign http://ubuntu-cloud.archive.canonical.com trusty-updates/juno/main Translation-en

Fetched 543 B in 1s (365 B/s)

Reading package lists... Done

W: GPG error: http://ubuntu-cloud.archive.canonical.com trusty-updates/juno Release: The following signatures couldn't be verified because the public key is not available: NO_PUBKEY 5EDB1B62EC4926EA

root@nsx11-vio-manager:~# apt-get install python-oslo-vmware

Reading package lists... Done

Building dependency tree      

Reading state information... Done

Package python-oslo-vmware is not available, but is referred to by another package.

This may mean that the package is missing, has been obsoleted, or

is only available from another source

E: Package 'python-oslo-vmware' has no installation candidate

[ssurana]

did you add any additional sources to the apt-get sources.list in your environment?

specifically from ubuntu-cloud.archive.canonical.com

can you post the following files:

/etc/apt/ folder contents

[raflax]

No. I don't remember adding any additional sources.  At least not manually.

Here is the list of files / directories in /etc/apt/

root@nsx11-vio-manager:/etc/apt# ls -lR

.:

total 52

-rw-r--r-- 1 root root    76 Jun 17  2015 apt.conf

drwxr-xr-x 2 root root  4096 Jun  7 20:26 apt.conf.d

drwxr-xr-x 2 root root  4096 Apr 10  2014 preferences.d

-rw-r--r-- 1 root root   147 Sep 14  2015 sources.list

-rw-r--r-- 1 root root   151 Jul 31  2015 sources.list.2015-09-14@19:37~

drwxr-xr-x 2 root root  4096 Jul 31  2015 sources.list.d

-rw-r--r-- 1 root root   151 Jul 31  2015 sources.list.save

-rw-r--r-- 1 root root  2883 Jun 17  2015 sources.list.ubuntu

-rw-r--r-- 1 root root 12335 Apr 16  2014 trusted.gpg

drwxr-xr-x 2 root root  4096 Apr 10  2014 trusted.gpg.d

./apt.conf.d:

total 28

-rw-r--r-- 1 root root   40 Jun 17  2015 00trustcdrom

-rw-r--r-- 1 root root  643 Apr 10  2014 01autoremove

-rw-r--r-- 1 root root 1058 Jun 17  2015 01autoremove-kernels

-rw-r--r-- 1 root root  123 Apr 10  2014 20changelog

-rw-r--r-- 1 root root 2331 Mar 12  2015 50unattended-upgrades

-rw-r--r-- 1 root root  182 Feb 23  2014 70debconf

-rw-r--r-- 1 root root   41 Jun  7 20:26 vio-dpkg-overwrite

./preferences.d:

total 0

./sources.list.d:

total 12

-rw-r--r-- 1 root root 162 Jul 31  2015 cloudarchive-juno.list

-rw-r--r-- 1 root root  59 Jul 31  2015 vioapp.list

-rw-r--r-- 1 root root  59 Jul 31  2015 vioapp.list.save

./trusted.gpg.d:

total 0

root@nsx11-vio-manager:/etc/apt#

[ssurana]

So seems like there were additional sources added in attempt to may be install something from juno.

do you remember running the following any time?

sudo add-apt-repository cloud-archive:juno

did you by any chance tries to manually update your deployment to juno or something?

Anyways.. lets try to fix your environment.

lets remove the following file

/etc/apt/source.list.d/cloudarchive-juno.list

do an "apt-get update"

Also, can you attach the /etc/apt/source.list , /etc/apt/source.list.d/vioapp.list

Once I can take a look and make sure the required things are there, we can move to the next step

[raflax]

I've kept this system pretty pristine.  The intent was to only allow VIO through ansible scripting to make changes/updates.  So, no, I have not done any manual updates and did not try to upgrade or update or install juno.

I've attached the two files you've requested.

Really appreciate your help with this!!

[ssurana]

Well, the only think I can tell from the information is that, if not you, someone else who had access to your system tried to do something. Since within VIO we never add any external sources to the appliance.

continuing, seems like we are missing one entry in there

run the following:

1. echo "deb [arch=amd64] file:/opt/vmware/repository/apt/viostack /" > /etc/apt/sources.list.d/viostack.list

2. do an "apt-get update"

3. now run "apt-cache madison python-oslo-vmware". This time this should give you some output

4. if you get an output for #3 then run "apt-get install vio-cli=2.5.0.3955000" this should get the newer cli installed

~ Sidharth

[raflax]

Yes.  That worked! 

root@nsx11-vio-manager:~# dpkg -l | grep vio-cli

ii  vio-cli                                    2.5.0.3955000                    all          VIO CLI component provides vio* cli commands

When I run viocil I get:

root@nsx11-vio-manager:~# viocli

Traceback (most recent call last):

  File "/usr/local/bin/viocli", line 6, in <module>

    from viocli.cli import main

  File "/usr/local/lib/python2.7/dist-packages/viocli/cli.py", line 23, in <module>

    from viocli.bundle.inventory_admin import InventoryBundle

  File "/usr/local/lib/python2.7/dist-packages/viocli/bundle/inventory_admin.py", line 11, in <module>

    import viocli.inventory_ops as inv_ops

  File "/usr/local/lib/python2.7/dist-packages/viocli/inventory_ops/__init__.py", line 1, in <module>

    from nova_ops import NovaOps

  File "/usr/local/lib/python2.7/dist-packages/viocli/inventory_ops/nova_ops.py", line 7, in <module>

    from base_operations import BaseOperations

  File "/usr/local/lib/python2.7/dist-packages/viocli/inventory_ops/base_operations.py", line 9, in <module>

    from pyVmomi import vim

ImportError: No module named pyVmomi

[ssurana]

‌please run the following command.

pip install pyvmomi

Hopefully that should resolve it.

[raflax]

thanks.. that fixed viocli

Final problem seems to be that cinder-api can't communicate with its database.

I've attached the cinder-api.log.gz files from controller01, but it's the same on controller02 as well.

When I run vioctl services start, I get the following at the end:

TASK: [cinder-api | restart cinder-api] ***************************************

skipping: [10.127.11.56]

skipping: [10.127.11.57]

TASK: [cinder-api | wait for cinder-api to start] *****************************

failed: [10.127.11.56] => {"elapsed": 300, "failed": true}

msg: Timeout when waiting for 127.0.0.1:8776

failed: [10.127.11.57] => {"elapsed": 300, "failed": true}

msg: Timeout when waiting for 127.0.0.1:8776

FATAL: all hosts have already failed -- aborting

start-site failed on the following nodes: ['10.127.11.57', '10.127.11.56'].

Any ideas?

[raflax]

cinder-volume has the same problem.

[ssurana]

Seems like as you mentioned something wrong with the db. when you ran the "viocli services start" did you see any other failures in the beginning?

what do you see in the UI plugin for the deployment? See if you have an option to stop the deployment from the UI, if yes, then go through the UI to stop and then start the deployment again. Otherwise try running "viocli services stop" before you try the start.

Also, just to be sure you have not upgraded your deployment yet and that It is still 2.0 deployment?

[raflax]

I have tried vioctl services stop, waiting for everything to finish, then vioctl services start and it fails as I've indicated on waiting for cinder-api to respond.  But when I look at the cinder logs (both api and volume) neither can connect to the database.

I just tried the UI plugin shutdown function and it correctly stopped the existing 2.0 VIO deployment.

To confirm, I have not yet upgraded to 2.5.

The restart via the UI plugin failed with an Error message stating Failed to execute task: INNER

Task execution failed because the following nodes were unreachable: 10.127.11.48, 10.127.11.47 and 10.127.11.46.

These nodes represent the mysql database cluster nodes.

After checking some of the openstack infrastructure VMs, some have rebooted and not come back up.. I'm working on fixing that now.  More later.

[raflax]

I've now got all of the infrastructure VMs back up and running correctly, but the 'vioctl services start' still fails waiting for cinder-api to respond.  When I look at the log for cinder-api, I can see that it is failing to connect to its database.  I'm a little puzzled by this since all of the other services, nova, neutron, keystone, etc.. can successfully reach the database just fine.  I also don't understand how the connection tokenization works.  What I'm referring to is the fact that the connection setting for each and every service's database is an encrypted tokenized string.  I'm not sure how to decipher this string to make sure cinder (api and volume) is pointing at the correct database with the proper credentials.  But this should all be happening automatically by VIO anyways, so I shouldn't really need to care.

Regardless, I can't get cinder to behave and I'm still stuck at the moment.

[ssurana]

can you provide me the file "system-state-*.log" generated by running following command on your controller

alias viostat='log=system-state-$(date +'%Y%m%d%M%S').log; echo System state at $(date)>> $log; echo Packages>> $log;dpkg --list>> $log; echo "">>$log; echo Python libraries>>$log; pip freeze>>$log; echo "===============">> $log; ls -laR /etc >> $log;’ && viostat