Hi. I am facing exactly the same issue with VIO 3.0 and then with 3.1 after upgrade, in compact mode. Really need a fix - in a POC lab environment it's hard to get tons of vCPUS and RAM hence I am using the compact deployment model.
I have adjusted the VIO management node config (/opt/vmware/vio/etc/omjs.properties) to relax the requirements and all deployment works fine up to 98% or so when the deployment script tries to create a VM under OpenStack/Project (f2d4ac6a851d4f4b902eb9eed718a60a)/Images (glance?). I have already wasted days with this bug and can't find a logical explanation or any hints in the logs
The adjusted settings relax requirements on minimum count if controllers, LB, etc as well as on vMotion, DRS, etc. The controller VM and the compute VM are created just fine, can login to them, however the deployment fails with "Bootstrap failed" on both VMs.
In the vSphere client task pane I can see a "Deploy OVF" task failed. I see HTTPS connect errors, not sure what they are but the management server, the controller and the compute VM are all on same management vlan on the VDS and checked connectivity among them just fine.
Name Target Status
-----------------------------------------------------------------------------------------------------------------
Deploy OVF template d0203844-2ee6-4fee-b93f-ecce7144e5c0 0%
after a while...
Deploy OVF template d0203844-2ee6-4fee-b93f-ecce7144e5c0 Operation timed out
Getting the logs with "viocli deployment getlogs" and searching for errors around the time the deployment was aborted I can see the errors in the nova and glance logs
>>>compute01\logs\nova-compute.log (4 hits)
Line 5: 2017-03-10 20:51:01.252 1273 ERROR oslo.messaging._drivers.impl_rabbit [req-94ec5e5c-190e-4315-a092-94dc4a49c48d - - - - -] AMQP server on 127.0.0.1:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds.
Line 6: 2017-03-10 20:51:02.259 1273 ERROR oslo.messaging._drivers.impl_rabbit [req-94ec5e5c-190e-4315-a092-94dc4a49c48d - - - - -] AMQP server on 127.0.0.1:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 2 seconds.
Line 180: 2017-03-10 20:51:12.177 1448 DEBUG oslo_service.service [req-fb976bc3-776e-4654-bd4c-2c33f67e4863 - - - - -] logging_exception_prefix = %(asctime)s.%(msecs)03d %(process)d ERROR %(name)s %(instance)s log_opt_values /usr/lib/python2.7/dist-packages/oslo_config/cfg.py:2519
Line 594: 2017-03-10 20:51:12.464 1448 ERROR nova.compute.manager [req-49950028-f747-4113-9fe0-3e5a5bbd7f37 - - - - -] No compute node record for host compute01
>>>compute01\logs\upstart\nova-compute.log (2 hits)
Line 10: 2017-03-10 20:51:01.252 1273 ERROR oslo.messaging._drivers.impl_rabbit [req-94ec5e5c-190e-4315-a092-94dc4a49c48d - - - - -] AMQP server on 127.0.0.1:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds.
Line 11: 2017-03-10 20:51:02.259 1273 ERROR oslo.messaging._drivers.impl_rabbit [req-94ec5e5c-190e-4315-a092-94dc4a49c48d - - - - -] AMQP server on 127.0.0.1:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 2 seconds.
>>> loadbalancer01\logs\glance-api.log (110 hits)
Line 282: 2017-03-10 20:51:54.897 10094 DEBUG oslo_db.sqlalchemy.engines [req-d9b1c15a-2a21-4737-a27c-41abaee4e192 30ad58c2a8c3434ca9c81f314a281991 f2d4ac6a851d4f4b902eb9eed718a60a - - -] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION _check_effective_sql_mode /usr/lib/python2.7/dist-packages/oslo_db/sqlalchemy/engines.py:256
Line 305: 2017-03-10 20:53:41.265 10093 DEBUG oslo_db.sqlalchemy.engines [req-4af23a9a-8e43-45a4-8e94-144bd3355ec5 30ad58c2a8c3434ca9c81f314a281991 f2d4ac6a851d4f4b902eb9eed718a60a - - -] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION _check_effective_sql_mode /usr/lib/python2.7/dist-packages/oslo_db/sqlalchemy/engines.py:256
Line 397: G2017-03-10 20:53:54.788 10093 ERROR glance.api.v2.image_data [req-bb2ca00e-2352-481b-98ea-25f34d119fed 30ad58c2a8c3434ca9c81f314a281991 f2d4ac6a851d4f4b902eb9eed718a60a - - -] Failed to upload image data due to internal error
Line 398: 2017-03-10 20:53:54.788 10093 ERROR glance.api.v2.image_data Traceback (most recent call last):
Line 399: 2017-03-10 20:53:54.788 10093 ERROR glance.api.v2.image_data File "/usr/lib/python2.7/dist-packages/glance/api/v2/image_data.py", line 114, in upload
Line 400: 2017-03-10 20:53:54.788 10093 ERROR glance.api.v2.image_data image.set_data(data, size)
Line 401: 2017-03-10 20:53:54.788 10093 ERROR glance.api.v2.image_data File "/usr/lib/python2.7/dist-packages/glance/domain/proxy.py", line 195, in set_data
Line 402: 2017-03-10 20:53:54.788 10093 ERROR glance.api.v2.image_data self.base.set_data(data, size)
... etc
Line 446: 2017-03-10 20:54:03.434 10093 ERROR glance.common.wsgi [req-bb2ca00e-2352-481b-98ea-25f34d119fed 30ad58c2a8c3434ca9c81f314a281991 f2d4ac6a851d4f4b902eb9eed718a60a - - -]
Caught error: <requests.packages.urllib3.connection.VerifiedHTTPSConnection object at 0x7f90cca71150>: Failed to establish a new connection: [Errno 113] EHOSTUNREACH
Line 447: 2017-03-10 20:54:03.434 10093 ERROR glance.common.wsgi Traceback (most recent call last):
Line 448: 2017-03-10 20:54:03.434 10093 ERROR glance.common.wsgi File "/usr/lib/python2.7/dist-packages/glance/common/wsgi.py", line 902, in __call__
etc...
Line 504: 2017-03-10 20:54:03.434 10093 ERROR glance.common.wsgi NewConnectionError: <requests.packages.urllib3.connection.VerifiedHTTPSConnection object at 0x7f90cca71150>: Failed to establish a new connection: [Errno 113] EHOSTUNREACH
>>>loadbalancer01\logs\haproxy.log (4 hits)
Line 34845: Mar 10 20:54:03 loadbalancer01 glance-api: 2017-03-10 20:53:54.788 10093 ERROR glance.api.v2.image_data [req-bb2ca00e-2352-481b-98ea-25f34d119fed 30ad58c2a8c3434ca9c81f314a281991 f2d4ac6a851d4f4b902eb9eed718a60a - - -] Failed to upload image data due to internal error
#0122017-03-10 20:53:54.788 10093 ERROR glance.api.v2.image_data Traceback (most recent call last):
#0122017-03-10 20:53:54.788 10093 ERROR glance.api.v2.image_data File "/usr/lib/python2.7/dist-packages/glance/api/v2/image_data.py", line 114, in upload
#0122017-03-10 20:53:54.788 10093 ERROR glance.api.v2.image_data image.set_data(data, size)
Line 34847: Mar 10 20:54:03 loadbalancer01 glance-api: 2017-03-10 20:54:03.434 10093 ERROR glance.common.wsgi [req-bb2ca00e-2352-481b-98ea-25f34d119fed 30ad58c2a8c3434ca9c81f314a281991 f2d4ac6a851d4f4b902eb9eed718a60a - - -]
Caught error: <requests.packages.urllib3.connection.VerifiedHTTPSConnection object at 0x7f90cca71150>: Failed to establish a new connection: [Errno 113] EHOSTUNREACH
#0122017-03-10 20:54:03.434 10093 ERROR glance.common.wsgi Traceback (most recent call last):
#0122017-03-10 20:54:03.434 10093 ERROR glance.common.wsgi File "/usr/lib/python2.7/dist-packages/glance/common/wsgi.py", line 902, in __call__
etc
>>>vio_mgmt\logs\jarvis\jarvis.log (1 hit)
Line 6054: 2017-03-10 20:54:05,581 ERROR [jarvis.ans.task][Thread-2] task-3f84b40a-a17d-4f43-aeab-f5c167dbcb9f failed. Task failed on the following nodes: ['10.100.20.180']. Refer logs for more details
Here is the info on the deployment.
root@localhost:~# viocli show -i
[all:vars]
multi_vc = False
deployment_version = 3.1.0
deployment_type = singlevm
region_name = nova
default_availability_zone = nova
cinder_volumes = "{\"nova:10.100.20.203\": {\"vcenter_ip\": \"10.100.20.203\", \"vcenter_insecure\": \"true\", \"vcenter_user\": \"administrator@vsphere.local\", \"cluster_name_list\": [\"MOS_cluster2\"], \"availability_zone_name\": \"nova\"}}"
internal_vip = 10.100.20.180
public_hostname = 10.20.0.179
public_vip = 10.20.0.179
glance_datastores = hsh:MOS_datastore:100
vcenter_ip = 10.100.20.203
vcenter_insecure = True
dvs_default_interface_name = eth2
token_expiration_time = 7200
admin_tenant_name = admin
cinder_folder = Volumes
vcenter_user = administrator@vsphere.local
glance_folder = /images
admin_user = admin
management_default_gateway = 10.100.20.202
neutron_backend = dvs
horizon_regions = "AVAILABLE_REGIONS = [('http://10.100.20.180:5000/v3','VIO'),]"
dvs_default_name = dvSwitch
keystone_backend = sql
dvs_integration_bridge = br-dvs
deployment_name = VIO
[compute]
10.100.20.181 datastore_regex="MOS\_datastore2" vcenter_ip=10.100.20.203 hostname=compute01 cluster_name=MOS_cluster2 vcenter_user=administrator@vsphere.local cluster_moid=domain-c818 vcenter_insecure=True vcenter_uuid=51C0A55D-3EF7-4753-8E0D-0E251A88DB57
[lb] 10.100.20.180 hostname=loadbalancer01
[controller] 10.100.20.180 hostname=loadbalancer01
[memcache] 10.100.20.180 hostname=loadbalancer01
[storage] 10.100.20.180 hostname=loadbalancer01
[db] 10.100.20.180 hostname=loadbalancer01
[mq] 10.100.20.180 hostname=loadbalancer01
[dhcp] 10.100.20.180 hostname=loadbalancer01
I have tried "viocli deployment configure" after it failed, it went well to the end when it failed on same OVF deploy template"
Same with "viocli deployment post-deploy"
Attaching the scrubbed-passwd "vioclo deployment getlogs"
