I had a working VIO version 2.0 and tried to upgrade it to 2.5.
The upgrade steps succeeded through the viocli dbverify step.
I went to provision the new VIO deployment and vCenter can no longer connect to the OMS instance. I can ssh to it just fine, I've shutdown and restarted the vApp that contains OMS to no avail. I tried to connect directly to the web interface at https://<OMS_server>:8443/VIO and logged in with the vCenter credentials as it suggests and I get the resulting http 500 error page from the servlet.
Anyone else get this and get past it?
Thanks.
Same thing..
root@nsx11-vio-manager:/opt/vmware/vio/patches/downloads# apt-get --reinstall install vio-cli=2.5.0.3955000
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following packages were automatically installed and are no longer required:
libevent-2.0-5 libgssglue1 libnfsidmap2 libtirpc1 rpcbind
Use 'apt-get autoremove' to remove them.
The following extra packages will be installed:
vio-cli
0 upgraded, 0 newly installed, 0 to remove and 168 not upgraded.
root@nsx11-vio-manager:/opt/vmware/vio/patches/downloads# apt-get -V install vio-cli=2.5.0.3955000
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following extra packages will be installed:
vio-cli (2.0.0.3037964)
0 upgraded, 0 newly installed, 0 to remove and 167 not upgraded.
root@nsx11-vio-manager:/opt/vmware/vio/patches/downloads#
root@nsx11-vio-manager:/opt/vmware/vio/patches/downloads# dpkg -l | grep vio-cli
hi vio-cli 2.0.0.3037964 all VIO CLI component provides vio* cli commands
Looks like it wants to install the version that is already installed.
Does the repo not think there is a 2.5 version of vio-cli?
lets do the following:
1. uninstall the cli: "apt-get remove vio-cli"
2. install again: "apt-get install vio-cli=2.5.0.3955000"
hopefully this will do the trick
Ok, apt-get did remove vio-cli but it can't install the new one:
root@nsx11-vio-manager:~# apt-get install vio-cli=2.5.0.3955000
Reading package lists... Done
Building dependency tree
Reading state information... Done
Some packages could not be installed. This may mean that you have
requested an impossible situation or if you are using the unstable
distribution that some required packages have not yet been created
or been moved out of Incoming.
The following information may help to resolve the situation:
The following packages have unmet dependencies:
vio-cli : Depends: python-openstackclient but it is not installable
Depends: python-oslo-vmware but it is not installable
E: Unable to correct problems, you have held broken packages.
okay, that's something..
Can you provide the output of:
1. apt-cache madison python-oslo-vmware
2. apt-get update
3. apt-get install python-oslo-vmware
root@nsx11-vio-manager:~# apt-cache madison python-oslo-vmware
root@nsx11-vio-manager:~# apt-get update
Ign file: trusty/kilo InRelease
Ign file: InRelease
Ign file: InRelease
Ign file: trusty/kilo Release.gpg
Ign file: Release.gpg
Ign file: Release.gpg
Ign file: Release
Ign file: Release
Get:1 file: trusty/kilo Release [2097 B]
Ign file: Translation-en
Ign file: Translation-en
Ign file: trusty/kilo/main Translation-en
Ign file: trusty/kilo/universe Translation-en
Ign http://ubuntu-cloud.archive.canonical.com trusty-updates/juno InRelease
Get:2 http://ubuntu-cloud.archive.canonical.com trusty-updates/juno Release.gpg [543 B]
Hit http://ubuntu-cloud.archive.canonical.com trusty-updates/juno Release
Ign http://ubuntu-cloud.archive.canonical.com trusty-updates/juno Release
Ign http://ubuntu-cloud.archive.canonical.com trusty-updates/juno/main amd64 Packages/DiffIndex
Ign http://ubuntu-cloud.archive.canonical.com trusty-updates/juno/main i386 Packages/DiffIndex
Hit http://ubuntu-cloud.archive.canonical.com trusty-updates/juno/main amd64 Packages
Hit http://ubuntu-cloud.archive.canonical.com trusty-updates/juno/main i386 Packages
Ign http://ubuntu-cloud.archive.canonical.com trusty-updates/juno/main Translation-en
Fetched 543 B in 1s (365 B/s)
Reading package lists... Done
W: GPG error: http://ubuntu-cloud.archive.canonical.com trusty-updates/juno Release: The following signatures couldn't be verified because the public key is not available: NO_PUBKEY 5EDB1B62EC4926EA
root@nsx11-vio-manager:~# apt-get install python-oslo-vmware
Reading package lists... Done
Building dependency tree
Reading state information... Done
Package python-oslo-vmware is not available, but is referred to by another package.
This may mean that the package is missing, has been obsoleted, or
is only available from another source
E: Package 'python-oslo-vmware' has no installation candidate
did you add any additional sources to the apt-get sources.list in your environment?
specifically from ubuntu-cloud.archive.canonical.com
can you post the following files:
/etc/apt/ folder contents
No. I don't remember adding any additional sources. At least not manually.
Here is the list of files / directories in /etc/apt/
root@nsx11-vio-manager:/etc/apt# ls -lR
.:
total 52
-rw-r--r-- 1 root root 76 Jun 17 2015 apt.conf
drwxr-xr-x 2 root root 4096 Jun 7 20:26 apt.conf.d
drwxr-xr-x 2 root root 4096 Apr 10 2014 preferences.d
-rw-r--r-- 1 root root 147 Sep 14 2015 sources.list
-rw-r--r-- 1 root root 151 Jul 31 2015 sources.list.2015-09-14@19:37~
drwxr-xr-x 2 root root 4096 Jul 31 2015 sources.list.d
-rw-r--r-- 1 root root 151 Jul 31 2015 sources.list.save
-rw-r--r-- 1 root root 2883 Jun 17 2015 sources.list.ubuntu
-rw-r--r-- 1 root root 12335 Apr 16 2014 trusted.gpg
drwxr-xr-x 2 root root 4096 Apr 10 2014 trusted.gpg.d
./apt.conf.d:
total 28
-rw-r--r-- 1 root root 40 Jun 17 2015 00trustcdrom
-rw-r--r-- 1 root root 643 Apr 10 2014 01autoremove
-rw-r--r-- 1 root root 1058 Jun 17 2015 01autoremove-kernels
-rw-r--r-- 1 root root 123 Apr 10 2014 20changelog
-rw-r--r-- 1 root root 2331 Mar 12 2015 50unattended-upgrades
-rw-r--r-- 1 root root 182 Feb 23 2014 70debconf
-rw-r--r-- 1 root root 41 Jun 7 20:26 vio-dpkg-overwrite
./preferences.d:
total 0
./sources.list.d:
total 12
-rw-r--r-- 1 root root 162 Jul 31 2015 cloudarchive-juno.list
-rw-r--r-- 1 root root 59 Jul 31 2015 vioapp.list
-rw-r--r-- 1 root root 59 Jul 31 2015 vioapp.list.save
./trusted.gpg.d:
total 0
root@nsx11-vio-manager:/etc/apt#
So seems like there were additional sources added in attempt to may be install something from juno.
do you remember running the following any time?
sudo add-apt-repository cloud-archive:juno
did you by any chance tries to manually update your deployment to juno or something?
Anyways.. lets try to fix your environment.
lets remove the following file
/etc/apt/source.list.d/cloudarchive-juno.list
do an "apt-get update"
Also, can you attach the /etc/apt/source.list , /etc/apt/source.list.d/vioapp.list
Once I can take a look and make sure the required things are there, we can move to the next step
I've kept this system pretty pristine. The intent was to only allow VIO through ansible scripting to make changes/updates. So, no, I have not done any manual updates and did not try to upgrade or update or install juno.
I've attached the two files you've requested.
Really appreciate your help with this!!
Well, the only think I can tell from the information is that, if not you, someone else who had access to your system tried to do something. Since within VIO we never add any external sources to the appliance.
continuing, seems like we are missing one entry in there
run the following:
1. echo "deb [arch=amd64] file:/opt/vmware/repository/apt/viostack /" > /etc/apt/sources.list.d/viostack.list
2. do an "apt-get update"
3. now run "apt-cache madison python-oslo-vmware". This time this should give you some output
4. if you get an output for #3 then run "apt-get install vio-cli=2.5.0.3955000" this should get the newer cli installed
~ Sidharth
Yes. That worked!
root@nsx11-vio-manager:~# dpkg -l | grep vio-cli
ii vio-cli 2.5.0.3955000 all VIO CLI component provides vio* cli commands
When I run viocil I get:
root@nsx11-vio-manager:~# viocli
Traceback (most recent call last):
File "/usr/local/bin/viocli", line 6, in <module>
from viocli.cli import main
File "/usr/local/lib/python2.7/dist-packages/viocli/cli.py", line 23, in <module>
from viocli.bundle.inventory_admin import InventoryBundle
File "/usr/local/lib/python2.7/dist-packages/viocli/bundle/inventory_admin.py", line 11, in <module>
import viocli.inventory_ops as inv_ops
File "/usr/local/lib/python2.7/dist-packages/viocli/inventory_ops/__init__.py", line 1, in <module>
from nova_ops import NovaOps
File "/usr/local/lib/python2.7/dist-packages/viocli/inventory_ops/nova_ops.py", line 7, in <module>
from base_operations import BaseOperations
File "/usr/local/lib/python2.7/dist-packages/viocli/inventory_ops/base_operations.py", line 9, in <module>
from pyVmomi import vim
ImportError: No module named pyVmomi
please run the following command.
pip install pyvmomi
Hopefully that should resolve it.
thanks.. that fixed viocli
Final problem seems to be that cinder-api can't communicate with its database.
I've attached the cinder-api.log.gz files from controller01, but it's the same on controller02 as well.
When I run vioctl services start, I get the following at the end:
TASK: [cinder-api | restart cinder-api] ***************************************
skipping: [10.127.11.56]
skipping: [10.127.11.57]
TASK: [cinder-api | wait for cinder-api to start] *****************************
failed: [10.127.11.56] => {"elapsed": 300, "failed": true}
msg: Timeout when waiting for 127.0.0.1:8776
failed: [10.127.11.57] => {"elapsed": 300, "failed": true}
msg: Timeout when waiting for 127.0.0.1:8776
FATAL: all hosts have already failed -- aborting
start-site failed on the following nodes: ['10.127.11.57', '10.127.11.56'].
Any ideas?
cinder-volume has the same problem.
Seems like as you mentioned something wrong with the db. when you ran the "viocli services start" did you see any other failures in the beginning?
what do you see in the UI plugin for the deployment? See if you have an option to stop the deployment from the UI, if yes, then go through the UI to stop and then start the deployment again. Otherwise try running "viocli services stop" before you try the start.
Also, just to be sure you have not upgraded your deployment yet and that It is still 2.0 deployment?
I have tried vioctl services stop, waiting for everything to finish, then vioctl services start and it fails as I've indicated on waiting for cinder-api to respond. But when I look at the cinder logs (both api and volume) neither can connect to the database.
I just tried the UI plugin shutdown function and it correctly stopped the existing 2.0 VIO deployment.
To confirm, I have not yet upgraded to 2.5.
The restart via the UI plugin failed with an Error message stating Failed to execute task: INNER
Task execution failed because the following nodes were unreachable: 10.127.11.48, 10.127.11.47 and 10.127.11.46.
These nodes represent the mysql database cluster nodes.
After checking some of the openstack infrastructure VMs, some have rebooted and not come back up.. I'm working on fixing that now. More later.
I've now got all of the infrastructure VMs back up and running correctly, but the 'vioctl services start' still fails waiting for cinder-api to respond. When I look at the log for cinder-api, I can see that it is failing to connect to its database. I'm a little puzzled by this since all of the other services, nova, neutron, keystone, etc.. can successfully reach the database just fine. I also don't understand how the connection tokenization works. What I'm referring to is the fact that the connection setting for each and every service's database is an encrypted tokenized string. I'm not sure how to decipher this string to make sure cinder (api and volume) is pointing at the correct database with the proper credentials. But this should all be happening automatically by VIO anyways, so I shouldn't really need to care.
Regardless, I can't get cinder to behave and I'm still stuck at the moment.
can you provide me the file "system-state-*.log" generated by running following command on your controller
alias viostat='log=system-state-$(date +'%Y%m%d%M%S').log; echo System state at $(date)>> $log; echo Packages>> $log;dpkg --list>> $log; echo "">>$log; echo Python libraries>>$log; pip freeze>>$log; echo "===============">> $log; ls -laR /etc >> $log;’ && viostat