dalo
Hot Shot
Hot Shot

vCenter reinstall (new ssl certificate)

Jump to solution

Because a upgrade from our vCenter 6.0 to 6.0u1 failed I had to completely reinstall our VC of the VIO2 environment.

Unfortunately the self-signed certificates of the VC are overwritten with new ones. Now it seems, that the VIO management server doesn't trust the VC anymore:

/var/log/oms/oms.log

[2015-10-22T11:06:44.754+0000] INFO  tomcat-http--24| com.vmware.openstack.security.UserAuthenticationFilter: updated vc host null

[2015-10-22T11:06:44.755+0000] INFO  tomcat-http--24| com.vmware.openstack.vc.AuthenticateVcUser: try orignal vc service: https://VC-URL:443/sdk

[2015-10-22T11:06:44.762+0000] ERROR tomcat-http--24| com.vmware.openstack.security.UserAuthenticationProvider: Authentication error: javax.net.ssl.SSLHandshakeException: java.security.cert.CertificateException: Server certificate chain is not trusted and thumbprint doesn't match

[2015-10-22T11:06:44.811+0000] INFO  tomcat-http--16| com.vmware.openstack.security.UserAuthenticationFilter: updated vc host null

[2015-10-22T11:06:44.811+0000] INFO  tomcat-http--16| com.vmware.openstack.vc.AuthenticateVcUser: try orignal vc service: https://VC-URL:443/sdk

[2015-10-22T11:06:44.813+0000] ERROR tomcat-http--16| com.vmware.openstack.security.UserAuthenticationProvider: Authentication error: javax.net.ssl.SSLHandshakeException: java.security.cert.CertificateException: Server certificate chain is not trusted and thumbprint doesn't match

[2015-10-22T11:06:44.834+0000] DEBUG tomcat-http--26| org.springframework.web.filter.CommonsRequestLoggingFilter: Before request [uri=/oms/api/connection/status?null;client=10.205.208.3]

[2015-10-22T11:06:44.836+0000] DEBUG tomcat-http--26| org.springframework.web.filter.CommonsRequestLoggingFilter: After request [uri=/oms/api/connection/status?null;client=10.205.208.3]

Is it possible to accept this new certificate from the management server?

0 Kudos
1 Solution

Accepted Solutions
JunW
Enthusiast
Enthusiast

For VC cert, if it is signed by another VC CA(I think this is true for your VC), please download the VC CA cert at https://vc_fqdn/certs/download , and add it into controller nodes and compute nodes' /etc/ssl/certs/ca-certificates.crt

View solution in original post

0 Kudos
9 Replies
yjia
VMware Employee
VMware Employee

Can you please upload the follow items?

1> the certificate file of VC

2> run viogetlogs on the managment server and upload the tgz file.

0 Kudos
JunW
Enthusiast
Enthusiast

A quick way worth to try is:

1. login into management server console, and run:

- sudo vi /opt/vmware/vio/etc/omjs.properties and change "oms.extension.registered = false"

2. power off/power on the VIO vApp.

If it is still not working, please follow Yixing's comment to collect logs.

Regards,

Jun

0 Kudos
dalo
Hot Shot
Hot Shot

Thank you for your answers.

Unfortunately junW's hint doesn't work.

I set now "insecure = False" to true on a few config files on controller 1&2 and I can boot the VIO again but this is only a workaround.

Replacing the VC public key would be definitively the better option. If you can guide me with this, this would be helpful.

In my opinion, this should be documented, because this can happen in every installation.

Also I observed, that some of the VIO VMs sometime doesn't startup fully. I have to go to the console and send a ctrl-alt-del then the VM boots correctly.

Logs are here: [deleted]

certificate is attached [deleted]

Thank you, Daniel

0 Kudos
dalo
Hot Shot
Hot Shot

I set vmware_insecure = true" in cinder.conf, but it seems, that cinder cannot connect, because of the wrong certificate:

2015-10-26 11:30:29.229 20958 TRACE cinder

2015-10-26 11:30:29.610 20973 DEBUG oslo_db.api [-] Loading backend 'sqlalchemy' from 'cinder.db.sqlalchemy.api' _load_backend /usr/lib/python2.7/dist-packages/oslo_db/api.py:214

2015-10-26 11:30:30.455 20973 DEBUG oslo_db.sqlalchemy.session [-] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION _check_effective_sql_mode /usr/lib/python2.7/dist-packages/oslo_db/sqlalchemy/session.py:513

2015-10-26 11:30:30.551 20973 DEBUG oslo_db.sqlalchemy.session [req-b080a2bb-e399-4403-99bb-5174d047d8db - - - - -] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION _check_effective_sql_mode /usr/lib/python2.7/dist-packages/oslo_db/sqlalchemy/session.py:513

2015-10-26 11:30:30.564 20973 INFO cinder.volume.manager [req-b080a2bb-e399-4403-99bb-5174d047d8db - - - - -] Determined volume DB was empty at startup.

2015-10-26 11:30:30.565 20973 DEBUG cinder.volume.manager [req-b080a2bb-e399-4403-99bb-5174d047d8db - - - - -] Cinder Volume DB check: vol_db_empty=True __init__ /usr/lib/python2.7/dist-packages/cinder/volume/manager.py:212

2015-10-26 11:30:30.606 20973 DEBUG oslo_vmware.api [req-b080a2bb-e399-4403-99bb-5174d047d8db - - - - -] Waiting for function _create_session to return. func /usr/lib/python2.7/dist-packages/oslo_vmware/api.py:121

2015-10-26 11:30:30.607 20973 DEBUG oslo_concurrency.lockutils [-] Lock "oslo_vmware_api_lock" acquired by "_create_session" :: waited 0.000s inner /usr/lib/python2.7/dist-packages/oslo_concurrency/lockutils.py:444

2015-10-26 11:30:30.608 20973 DEBUG oslo_vmware.service [-] Creating suds client with soap_url='https://VCENTER-HOST:443/sdk' and wsdl_url='https://VCENTER-HOST:443/sdk/vimService.wsdl' __init__ /usr/lib/python2.7/dist-packages/oslo_vmware/service.py:191

2015-10-26 11:30:30.627 20973 DEBUG oslo_concurrency.lockutils [-] Lock "oslo_vmware_api_lock" released by "_create_session" :: held 0.020s inner /usr/lib/python2.7/dist-packages/oslo_concurrency/lockutils.py:456

2015-10-26 11:30:30.628 20973 CRITICAL cinder [req-b080a2bb-e399-4403-99bb-5174d047d8db - - - - -] SSLError: [Errno 1] _ssl.c:510: error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed

2015-10-26 11:30:30.628 20973 TRACE cinder Traceback (most recent call last):

2015-10-26 11:30:30.628 20973 TRACE cinder   File "/usr/bin/cinder-volume", line 10, in <module>

2015-10-26 11:30:30.628 20973 TRACE cinder     sys.exit(main())

2015-10-26 11:30:30.628 20973 TRACE cinder   File "/usr/lib/python2.7/dist-packages/cinder/cmd/volume.py", line 80, in main

2015-10-26 11:30:30.628 20973 TRACE cinder     server = service.Service.create(binary='cinder-volume')

2015-10-26 11:30:30.628 20973 TRACE cinder   File "/usr/lib/python2.7/dist-packages/cinder/service.py", line 249, in create

2015-10-26 11:30:30.628 20973 TRACE cinder     service_name=service_name)

2015-10-26 11:30:30.628 20973 TRACE cinder   File "/usr/lib/python2.7/dist-packages/cinder/service.py", line 129, in __init__

2015-10-26 11:30:30.628 20973 TRACE cinder     *args, **kwargs)

2015-10-26 11:30:30.628 20973 TRACE cinder   File "/usr/lib/python2.7/dist-packages/cinder/volume/manager.py", line 221, in __init__

2015-10-26 11:30:30.628 20973 TRACE cinder     self.driver = profiler.trace_cls("driver")(self.driver)

2015-10-26 11:30:30.628 20973 TRACE cinder   File "/usr/lib/python2.7/dist-packages/osprofiler/profiler.py", line 139, in decorator

2015-10-26 11:30:30.628 20973 TRACE cinder     for attr_name, attr in inspect.getmembers(cls):

2015-10-26 11:30:30.628 20973 TRACE cinder   File "/usr/lib/python2.7/inspect.py", line 253, in getmembers

2015-10-26 11:30:30.628 20973 TRACE cinder     value = getattr(object, key)

2015-10-26 11:30:30.628 20973 TRACE cinder   File "/usr/lib/python2.7/dist-packages/cinder/volume/drivers/vmware/vmdk.py", line 265, in ds_sel

2015-10-26 11:30:30.628 20973 TRACE cinder     self._ds_sel = hub.DatastoreSelector(self.volumeops,

2015-10-26 11:30:30.628 20973 TRACE cinder   File "/usr/lib/python2.7/dist-packages/cinder/volume/drivers/vmware/vmdk.py", line 258, in volumeops

2015-10-26 11:30:30.628 20973 TRACE cinder     self._volumeops = volumeops.VMwareVolumeOps(self.session,

2015-10-26 11:30:30.628 20973 TRACE cinder   File "/usr/lib/python2.7/dist-packages/cinder/volume/drivers/vmware/vmdk.py", line 1905, in session

2015-10-26 11:30:30.628 20973 TRACE cinder     insecure=insecure)

2015-10-26 11:30:30.628 20973 TRACE cinder   File "/usr/lib/python2.7/dist-packages/oslo_vmware/api.py", line 180, in __init__

2015-10-26 11:30:30.628 20973 TRACE cinder     self._create_session()

2015-10-26 11:30:30.628 20973 TRACE cinder   File "/usr/lib/python2.7/dist-packages/oslo_vmware/api.py", line 122, in func

2015-10-26 11:30:30.628 20973 TRACE cinder     return evt.wait()

2015-10-26 11:30:30.628 20973 TRACE cinder   File "/usr/lib/python2.7/dist-packages/eventlet/event.py", line 121, in wait

2015-10-26 11:30:30.628 20973 TRACE cinder     return hubs.get_hub().switch()

2015-10-26 11:30:30.628 20973 TRACE cinder   File "/usr/lib/python2.7/dist-packages/eventlet/hubs/hub.py", line 294, in switch

2015-10-26 11:30:30.628 20973 TRACE cinder     return self.greenlet.switch()

2015-10-26 11:30:30.628 20973 TRACE cinder   File "/usr/lib/python2.7/dist-packages/oslo_vmware/common/loopingcall.py", line 123, in _inner

2015-10-26 11:30:30.628 20973 TRACE cinder     idle = self.f(*self.args, **self.kw)

2015-10-26 11:30:30.628 20973 TRACE cinder   File "/usr/lib/python2.7/dist-packages/oslo_vmware/api.py", line 95, in _func

2015-10-26 11:30:30.628 20973 TRACE cinder     result = f(*args, **kwargs)

2015-10-26 11:30:30.628 20973 TRACE cinder   File "/usr/lib/python2.7/dist-packages/oslo_concurrency/lockutils.py", line 445, in inner

2015-10-26 11:30:30.628 20973 TRACE cinder     return f(*args, **kwargs)

2015-10-26 11:30:30.628 20973 TRACE cinder   File "/usr/lib/python2.7/dist-packages/oslo_vmware/api.py", line 225, in _create_session

2015-10-26 11:30:30.628 20973 TRACE cinder     session_manager = self.vim.service_content.sessionManager

2015-10-26 11:30:30.628 20973 TRACE cinder   File "/usr/lib/python2.7/dist-packages/oslo_vmware/api.py", line 195, in vim

2015-10-26 11:30:30.628 20973 TRACE cinder     insecure=self._insecure)

2015-10-26 11:30:30.628 20973 TRACE cinder   File "/usr/lib/python2.7/dist-packages/oslo_vmware/vim.py", line 41, in __init__

2015-10-26 11:30:30.628 20973 TRACE cinder     super(Vim, self).__init__(wsdl_url, soap_url, cacert, insecure)

2015-10-26 11:30:30.628 20973 TRACE cinder   File "/usr/lib/python2.7/dist-packages/oslo_vmware/service.py", line 197, in __init__

2015-10-26 11:30:30.628 20973 TRACE cinder     cache=_CACHE)

2015-10-26 11:30:30.628 20973 TRACE cinder   File "/usr/lib/python2.7/dist-packages/suds/client.py", line 112, in __init__

2015-10-26 11:30:30.628 20973 TRACE cinder     self.wsdl = reader.open(url)

2015-10-26 11:30:30.628 20973 TRACE cinder   File "/usr/lib/python2.7/dist-packages/suds/reader.py", line 152, in open

2015-10-26 11:30:30.628 20973 TRACE cinder     d = self.fn(url, self.options)

2015-10-26 11:30:30.628 20973 TRACE cinder   File "/usr/lib/python2.7/dist-packages/suds/wsdl.py", line 136, in __init__

2015-10-26 11:30:30.628 20973 TRACE cinder     d = reader.open(url)

2015-10-26 11:30:30.628 20973 TRACE cinder   File "/usr/lib/python2.7/dist-packages/suds/reader.py", line 79, in open

2015-10-26 11:30:30.628 20973 TRACE cinder     d = self.download(url)

2015-10-26 11:30:30.628 20973 TRACE cinder   File "/usr/lib/python2.7/dist-packages/suds/reader.py", line 95, in download

2015-10-26 11:30:30.628 20973 TRACE cinder     fp = self.options.transport.open(Request(url))

2015-10-26 11:30:30.628 20973 TRACE cinder   File "/usr/lib/python2.7/dist-packages/oslo_vmware/service.py", line 144, in open

2015-10-26 11:30:30.628 20973 TRACE cinder     resp = self.session.get(request.url, verify=self.verify)

2015-10-26 11:30:30.628 20973 TRACE cinder   File "/usr/lib/python2.7/dist-packages/requests/sessions.py", line 469, in get

2015-10-26 11:30:30.628 20973 TRACE cinder     return self.request('GET', url, **kwargs)

2015-10-26 11:30:30.628 20973 TRACE cinder   File "/usr/lib/python2.7/dist-packages/requests/sessions.py", line 457, in request

2015-10-26 11:30:30.628 20973 TRACE cinder     resp = self.send(prep, **send_kwargs)

2015-10-26 11:30:30.628 20973 TRACE cinder   File "/usr/lib/python2.7/dist-packages/requests/sessions.py", line 569, in send

2015-10-26 11:30:30.628 20973 TRACE cinder     r = adapter.send(request, **kwargs)

2015-10-26 11:30:30.628 20973 TRACE cinder   File "/usr/lib/python2.7/dist-packages/requests/adapters.py", line 420, in send

2015-10-26 11:30:30.628 20973 TRACE cinder     raise SSLError(e, request=request)

2015-10-26 11:30:30.628 20973 TRACE cinder SSLError: [Errno 1] _ssl.c:510: error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed

2015-10-26 11:30:30.628 20973 TRACE cinder

0 Kudos
v_bala
VMware Employee
VMware Employee

The option 'vmware_insecure' is ignored if the option 'vmware_ca_file' is set. To disable certificate verification, unset/remove/comment vmware_ca_file option and set vmware_insecure to True.

JunW
Enthusiast
Enthusiast

From the logs, I can see the quick hint actually works in terms of solving the management server's complain about vc cert thumbprint issue you mentioned before. Now the issue changes to how to update the vc certs information on the controller nodes easier. Thanks for this good question and we will improve in the next versions or patches. For now, unset vmware_ca_file from cinder.conf file will remove the cinder's complain. Another potential issue you mentioned is VIO VM does not startup sometimes, could you past a console screenshot when you meet it next time?

Thanks,

Jun

dalo
Hot Shot
Hot Shot

Thank you both, disable the vmware_ca_file in cinder.conf works now for cinder. Also I had to modify the nova.conf on the compute01 node.

But isn't it possible to add the new certificate to /etc/ssl/certs/ca-certificates.crt? I already tried this, but without success. I would prefer to use the secure mode instead of disabling the certificate check on all components.

JunW, I edit the thumbprint on the management server in the /opt/vmware/vio/etc/vc.properties and then I installed the plugin on the VC server, this helps me with the management server issue. I made this before I read your message about the oms.extension.registered = false

I will open a new thread about the startup issue, because this seems to be something other.


Thanks,

Daniel

0 Kudos
JunW
Enthusiast
Enthusiast

For VC cert, if it is signed by another VC CA(I think this is true for your VC), please download the VC CA cert at https://vc_fqdn/certs/download , and add it into controller nodes and compute nodes' /etc/ssl/certs/ca-certificates.crt

0 Kudos
dalo
Hot Shot
Hot Shot

I tried this already, but wasn't successful. But maybe I did something wrong, because now I added the certs again and it works: I could disable the "insecure" Options.

So to sum up, the following steps are required after a change of the VCenter certificate:

- replace Thumbprint in /opt/vmware/vio/etc/vc.properties on the management server

- download new certificate  from https://vc_fqdn/certs/download, rename and unzip this file

- append the certificate to /etc/ssl/certs/ca-certificates.crt on controller01 and 02 and on computeXX

- reboot everything

0 Kudos