Solved: Re: vCenter reinstall

dalo · ‎10-22-2015

Because a upgrade from our vCenter 6.0 to 6.0u1 failed I had to completely reinstall our VC of the VIO2 environment.

Unfortunately the self-signed certificates of the VC are overwritten with new ones. Now it seems, that the VIO management server doesn't trust the VC anymore:

/var/log/oms/oms.log

[2015-10-22T11:06:44.754+0000] INFO tomcat-http--24| com.vmware.openstack.security.UserAuthenticationFilter: updated vc host null

[2015-10-22T11:06:44.755+0000] INFO tomcat-http--24| com.vmware.openstack.vc.AuthenticateVcUser: try orignal vc service: https://VC-URL:443/sdk

[2015-10-22T11:06:44.762+0000] ERROR tomcat-http--24| com.vmware.openstack.security.UserAuthenticationProvider: Authentication error: javax.net.ssl.SSLHandshakeException: java.security.cert.CertificateException: Server certificate chain is not trusted and thumbprint doesn't match

[2015-10-22T11:06:44.811+0000] INFO tomcat-http--16| com.vmware.openstack.security.UserAuthenticationFilter: updated vc host null

[2015-10-22T11:06:44.811+0000] INFO tomcat-http--16| com.vmware.openstack.vc.AuthenticateVcUser: try orignal vc service: https://VC-URL:443/sdk

[2015-10-22T11:06:44.813+0000] ERROR tomcat-http--16| com.vmware.openstack.security.UserAuthenticationProvider: Authentication error: javax.net.ssl.SSLHandshakeException: java.security.cert.CertificateException: Server certificate chain is not trusted and thumbprint doesn't match

[2015-10-22T11:06:44.834+0000] DEBUG tomcat-http--26| org.springframework.web.filter.CommonsRequestLoggingFilter: Before request [uri=/oms/api/connection/status?null;client=10.205.208.3]

[2015-10-22T11:06:44.836+0000] DEBUG tomcat-http--26| org.springframework.web.filter.CommonsRequestLoggingFilter: After request [uri=/oms/api/connection/status?null;client=10.205.208.3]

Is it possible to accept this new certificate from the management server?

JunW · ‎10-27-2015

For VC cert, if it is signed by another VC CA(I think this is true for your VC), please download the VC CA cert at https://vc_fqdn/certs/download , and add it into controller nodes and compute nodes' /etc/ssl/certs/ca-certificates.crt

View solution in original post

yjia · ‎10-22-2015

Can you please upload the follow items?

1> the certificate file of VC

2> run viogetlogs on the managment server and upload the tgz file.

JunW · ‎10-22-2015

A quick way worth to try is:

1. login into management server console, and run:

- sudo vi /opt/vmware/vio/etc/omjs.properties and change "oms.extension.registered = false"

2. power off/power on the VIO vApp.

If it is still not working, please follow Yixing's comment to collect logs.

Regards,

Jun

dalo · ‎10-26-2015

Thank you for your answers.

Unfortunately junW's hint doesn't work.

I set now "insecure = False" to true on a few config files on controller 1&2 and I can boot the VIO again but this is only a workaround.

Replacing the VC public key would be definitively the better option. If you can guide me with this, this would be helpful.

In my opinion, this should be documented, because this can happen in every installation.

Also I observed, that some of the VIO VMs sometime doesn't startup fully. I have to go to the console and send a ctrl-alt-del then the VM boots correctly.

Logs are here: [deleted]

certificate is attached [deleted]

Thank you, Daniel

dalo · ‎10-26-2015

I set vmware_insecure = true" in cinder.conf, but it seems, that cinder cannot connect, because of the wrong certificate:

2015-10-26 11:30:29.229 20958 TRACE cinder

2015-10-26 11:30:29.610 20973 DEBUG oslo_db.api [-] Loading backend 'sqlalchemy' from 'cinder.db.sqlalchemy.api' _load_backend /usr/lib/python2.7/dist-packages/oslo_db/api.py:214

2015-10-26 11:30:30.455 20973 DEBUG oslo_db.sqlalchemy.session [-] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION _check_effective_sql_mode /usr/lib/python2.7/dist-packages/oslo_db/sqlalchemy/session.py:513

2015-10-26 11:30:30.551 20973 DEBUG oslo_db.sqlalchemy.session [req-b080a2bb-e399-4403-99bb-5174d047d8db - - - - -] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION _check_effective_sql_mode /usr/lib/python2.7/dist-packages/oslo_db/sqlalchemy/session.py:513

2015-10-26 11:30:30.564 20973 INFO cinder.volume.manager [req-b080a2bb-e399-4403-99bb-5174d047d8db - - - - -] Determined volume DB was empty at startup.

2015-10-26 11:30:30.565 20973 DEBUG cinder.volume.manager [req-b080a2bb-e399-4403-99bb-5174d047d8db - - - - -] Cinder Volume DB check: vol_db_empty=True __init__ /usr/lib/python2.7/dist-packages/cinder/volume/manager.py:212

2015-10-26 11:30:30.606 20973 DEBUG oslo_vmware.api [req-b080a2bb-e399-4403-99bb-5174d047d8db - - - - -] Waiting for function _create_session to return. func /usr/lib/python2.7/dist-packages/oslo_vmware/api.py:121

2015-10-26 11:30:30.607 20973 DEBUG oslo_concurrency.lockutils [-] Lock "oslo_vmware_api_lock" acquired by "_create_session" :: waited 0.000s inner /usr/lib/python2.7/dist-packages/oslo_concurrency/lockutils.py:444

2015-10-26 11:30:30.608 20973 DEBUG oslo_vmware.service [-] Creating suds client with soap_url='https://VCENTER-HOST:443/sdk' and wsdl_url='https://VCENTER-HOST:443/sdk/vimService.wsdl' __init__ /usr/lib/python2.7/dist-packages/oslo_vmware/service.py:191

2015-10-26 11:30:30.627 20973 DEBUG oslo_concurrency.lockutils [-] Lock "oslo_vmware_api_lock" released by "_create_session" :: held 0.020s inner /usr/lib/python2.7/dist-packages/oslo_concurrency/lockutils.py:456

2015-10-26 11:30:30.628 20973 CRITICAL cinder [req-b080a2bb-e399-4403-99bb-5174d047d8db - - - - -] SSLError: [Errno 1] _ssl.c:510: error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed

2015-10-26 11:30:30.628 20973 TRACE cinder Traceback (most recent call last):

2015-10-26 11:30:30.628 20973 TRACE cinder File "/usr/bin/cinder-volume", line 10, in <module>

2015-10-26 11:30:30.628 20973 TRACE cinder sys.exit(main())

2015-10-26 11:30:30.628 20973 TRACE cinder File "/usr/lib/python2.7/dist-packages/cinder/cmd/volume.py", line 80, in main

2015-10-26 11:30:30.628 20973 TRACE cinder server = service.Service.create(binary='cinder-volume')

2015-10-26 11:30:30.628 20973 TRACE cinder File "/usr/lib/python2.7/dist-packages/cinder/service.py", line 249, in create

2015-10-26 11:30:30.628 20973 TRACE cinder service_name=service_name)

2015-10-26 11:30:30.628 20973 TRACE cinder File "/usr/lib/python2.7/dist-packages/cinder/service.py", line 129, in __init__

2015-10-26 11:30:30.628 20973 TRACE cinder *args, **kwargs)

2015-10-26 11:30:30.628 20973 TRACE cinder File "/usr/lib/python2.7/dist-packages/cinder/volume/manager.py", line 221, in __init__

2015-10-26 11:30:30.628 20973 TRACE cinder self.driver = profiler.trace_cls("driver")(self.driver)

2015-10-26 11:30:30.628 20973 TRACE cinder File "/usr/lib/python2.7/dist-packages/osprofiler/profiler.py", line 139, in decorator

2015-10-26 11:30:30.628 20973 TRACE cinder for attr_name, attr in inspect.getmembers(cls):

2015-10-26 11:30:30.628 20973 TRACE cinder File "/usr/lib/python2.7/inspect.py", line 253, in getmembers

2015-10-26 11:30:30.628 20973 TRACE cinder value = getattr(object, key)

2015-10-26 11:30:30.628 20973 TRACE cinder File "/usr/lib/python2.7/dist-packages/cinder/volume/drivers/vmware/vmdk.py", line 265, in ds_sel

2015-10-26 11:30:30.628 20973 TRACE cinder self._ds_sel = hub.DatastoreSelector(self.volumeops,

2015-10-26 11:30:30.628 20973 TRACE cinder File "/usr/lib/python2.7/dist-packages/cinder/volume/drivers/vmware/vmdk.py", line 258, in volumeops

2015-10-26 11:30:30.628 20973 TRACE cinder self._volumeops = volumeops.VMwareVolumeOps(self.session,

2015-10-26 11:30:30.628 20973 TRACE cinder File "/usr/lib/python2.7/dist-packages/cinder/volume/drivers/vmware/vmdk.py", line 1905, in session

2015-10-26 11:30:30.628 20973 TRACE cinder insecure=insecure)

2015-10-26 11:30:30.628 20973 TRACE cinder File "/usr/lib/python2.7/dist-packages/oslo_vmware/api.py", line 180, in __init__

2015-10-26 11:30:30.628 20973 TRACE cinder self._create_session()

2015-10-26 11:30:30.628 20973 TRACE cinder File "/usr/lib/python2.7/dist-packages/oslo_vmware/api.py", line 122, in func

2015-10-26 11:30:30.628 20973 TRACE cinder return evt.wait()

2015-10-26 11:30:30.628 20973 TRACE cinder File "/usr/lib/python2.7/dist-packages/eventlet/event.py", line 121, in wait

2015-10-26 11:30:30.628 20973 TRACE cinder return hubs.get_hub().switch()

2015-10-26 11:30:30.628 20973 TRACE cinder File "/usr/lib/python2.7/dist-packages/eventlet/hubs/hub.py", line 294, in switch

2015-10-26 11:30:30.628 20973 TRACE cinder return self.greenlet.switch()

2015-10-26 11:30:30.628 20973 TRACE cinder File "/usr/lib/python2.7/dist-packages/oslo_vmware/common/loopingcall.py", line 123, in _inner

2015-10-26 11:30:30.628 20973 TRACE cinder idle = self.f(*self.args, **self.kw)

2015-10-26 11:30:30.628 20973 TRACE cinder File "/usr/lib/python2.7/dist-packages/oslo_vmware/api.py", line 95, in _func

2015-10-26 11:30:30.628 20973 TRACE cinder result = f(*args, **kwargs)

2015-10-26 11:30:30.628 20973 TRACE cinder File "/usr/lib/python2.7/dist-packages/oslo_concurrency/lockutils.py", line 445, in inner

2015-10-26 11:30:30.628 20973 TRACE cinder return f(*args, **kwargs)

2015-10-26 11:30:30.628 20973 TRACE cinder File "/usr/lib/python2.7/dist-packages/oslo_vmware/api.py", line 225, in _create_session

2015-10-26 11:30:30.628 20973 TRACE cinder session_manager = self.vim.service_content.sessionManager

2015-10-26 11:30:30.628 20973 TRACE cinder File "/usr/lib/python2.7/dist-packages/oslo_vmware/api.py", line 195, in vim

2015-10-26 11:30:30.628 20973 TRACE cinder insecure=self._insecure)

2015-10-26 11:30:30.628 20973 TRACE cinder File "/usr/lib/python2.7/dist-packages/oslo_vmware/vim.py", line 41, in __init__

2015-10-26 11:30:30.628 20973 TRACE cinder super(Vim, self).__init__(wsdl_url, soap_url, cacert, insecure)

2015-10-26 11:30:30.628 20973 TRACE cinder File "/usr/lib/python2.7/dist-packages/oslo_vmware/service.py", line 197, in __init__

2015-10-26 11:30:30.628 20973 TRACE cinder cache=_CACHE)

2015-10-26 11:30:30.628 20973 TRACE cinder File "/usr/lib/python2.7/dist-packages/suds/client.py", line 112, in __init__

2015-10-26 11:30:30.628 20973 TRACE cinder self.wsdl = reader.open(url)

2015-10-26 11:30:30.628 20973 TRACE cinder File "/usr/lib/python2.7/dist-packages/suds/reader.py", line 152, in open

2015-10-26 11:30:30.628 20973 TRACE cinder d = self.fn(url, self.options)

2015-10-26 11:30:30.628 20973 TRACE cinder File "/usr/lib/python2.7/dist-packages/suds/wsdl.py", line 136, in __init__

2015-10-26 11:30:30.628 20973 TRACE cinder d = reader.open(url)

2015-10-26 11:30:30.628 20973 TRACE cinder File "/usr/lib/python2.7/dist-packages/suds/reader.py", line 79, in open

2015-10-26 11:30:30.628 20973 TRACE cinder d = self.download(url)

2015-10-26 11:30:30.628 20973 TRACE cinder File "/usr/lib/python2.7/dist-packages/suds/reader.py", line 95, in download

2015-10-26 11:30:30.628 20973 TRACE cinder fp = self.options.transport.open(Request(url))

2015-10-26 11:30:30.628 20973 TRACE cinder File "/usr/lib/python2.7/dist-packages/oslo_vmware/service.py", line 144, in open

2015-10-26 11:30:30.628 20973 TRACE cinder resp = self.session.get(request.url, verify=self.verify)

2015-10-26 11:30:30.628 20973 TRACE cinder File "/usr/lib/python2.7/dist-packages/requests/sessions.py", line 469, in get

2015-10-26 11:30:30.628 20973 TRACE cinder return self.request('GET', url, **kwargs)

2015-10-26 11:30:30.628 20973 TRACE cinder File "/usr/lib/python2.7/dist-packages/requests/sessions.py", line 457, in request

2015-10-26 11:30:30.628 20973 TRACE cinder resp = self.send(prep, **send_kwargs)

2015-10-26 11:30:30.628 20973 TRACE cinder File "/usr/lib/python2.7/dist-packages/requests/sessions.py", line 569, in send

2015-10-26 11:30:30.628 20973 TRACE cinder r = adapter.send(request, **kwargs)

2015-10-26 11:30:30.628 20973 TRACE cinder File "/usr/lib/python2.7/dist-packages/requests/adapters.py", line 420, in send

2015-10-26 11:30:30.628 20973 TRACE cinder raise SSLError(e, request=request)

2015-10-26 11:30:30.628 20973 TRACE cinder SSLError: [Errno 1] _ssl.c:510: error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed

2015-10-26 11:30:30.628 20973 TRACE cinder

v_bala · ‎10-26-2015

The option 'vmware_insecure' is ignored if the option 'vmware_ca_file' is set. To disable certificate verification, unset/remove/comment vmware_ca_file option and set vmware_insecure to True.

JunW · ‎10-27-2015

From the logs, I can see the quick hint actually works in terms of solving the management server's complain about vc cert thumbprint issue you mentioned before. Now the issue changes to how to update the vc certs information on the controller nodes easier. Thanks for this good question and we will improve in the next versions or patches. For now, unset vmware_ca_file from cinder.conf file will remove the cinder's complain. Another potential issue you mentioned is VIO VM does not startup sometimes, could you past a console screenshot when you meet it next time?

Thanks,

Jun

dalo · ‎10-27-2015

Thank you both, disable the vmware_ca_file in cinder.conf works now for cinder. Also I had to modify the nova.conf on the compute01 node.

But isn't it possible to add the new certificate to /etc/ssl/certs/ca-certificates.crt? I already tried this, but without success. I would prefer to use the secure mode instead of disabling the certificate check on all components.

JunW, I edit the thumbprint on the management server in the /opt/vmware/vio/etc/vc.properties and then I installed the plugin on the VC server, this helps me with the management server issue. I made this before I read your message about the oms.extension.registered = false

I will open a new thread about the startup issue, because this seems to be something other.

Thanks,

Daniel

JunW · ‎10-27-2015

For VC cert, if it is signed by another VC CA(I think this is true for your VC), please download the VC CA cert at https://vc_fqdn/certs/download , and add it into controller nodes and compute nodes' /etc/ssl/certs/ca-certificates.crt

dalo · ‎11-02-2015

I tried this already, but wasn't successful. But maybe I did something wrong, because now I added the certs again and it works: I could disable the "insecure" Options.

So to sum up, the following steps are required after a change of the VCenter certificate:

- replace Thumbprint in /opt/vmware/vio/etc/vc.properties on the management server

- download new certificate from https://vc_fqdn/certs/download, rename and unzip this file

- append the certificate to /etc/ssl/certs/ca-certificates.crt on controller01 and 02 and on computeXX

- reboot everything

All

vCenter reinstall (new ssl certificate)