Hi,
I just changed our vCAs self-signed certificates, replaced them with certificates from our internal CA.
I followed VMware KB: Configuring Certificate Authority (CA) signed certificates for vCenter Server Appliance 5...exactly.
Our CA server runs CentOS 6.5, which ships with OpenSSL > 0.9.8, which means keys are saved in a new key format (PKCS#8 vs. the "traditional" format.)
I transformed my vCenter keys back to the old format using "openssl rsa -in server.key -out server.rsa.key", which was actually mentioned in one VMware KB, too.
Just to summarize: I created four 2048-bit certificates using SHA512, having IP, FQDN and hostname as SAN and different organizationalUnitNames as mentioned in the above KB.
Keys were created in PKCS#8 format which seems to be standard since OpenSSL > 0.9.8n.
Well, everything seemed to work fine until I tried to connect to my VDP via vSphere Web Client. I got the message that the SSO service couldn't be reached and was asked whether I wanted to be redirected to the VDPs management page.
I took a look into /usr/local/avamar/var/vdr/server_logs/vdr-server.log and found this:
2014-02-11 17:46:17,632 INFO [com.emc.vdp2.server.VDRServer$1]-server.ConnectionService: Trying to establish connection with vCenter.
2014-02-11 17:46:17,642 INFO [com.emc.vdp2.server.VDRServer$1]-service.AdapterUtils: MCS Web Services URL: https://server:9443/services/mcService MCUserId="MCUser" MCUserPswd="*****************************"
2014-02-11 17:46:18,162 INFO [com.emc.vdp2.server.VDRServer$1]-service.ServiceInstance: ServiceInstanceMoref desc=Service Id: urn:uuid:SOMEID name=urn:uuid:SOMEID value=SERVICE
2014-02-11 17:46:18,178 INFO [com.emc.vdp2.server.VDRServer$1]-vi.VCenterServiceImpl: Found VCenter 'server' in domain 'server' which has 'VirtualMachines' as subDomain
2014-02-11 17:46:18,190 ERROR [com.emc.vdp2.server.VDRServer$1]-server.ConnectionService: Unable to get the vi access
java.rmi.RemoteException: VI SDK invoke exception:javax.net.ssl.SSLProtocolException: Certificate contains invalid public key: Invalid RSA (1.2.840.113549.1.1.1) public key encoding.
at com.vmware.vim25.ws.WSClient.invoke(WSClient.java:213)
at com.vmware.vim25.ws.WSClient.invoke(WSClient.java:137)
at com.vmware.vim25.ws.VimStub.retrieveServiceContent(VimStub.java:1480)
at com.vmware.vim25.mo.ServiceInstance.<init>(ServiceInstance.java:99)
at com.vmware.vim25.mo.ServiceInstance.<init>(ServiceInstance.java:83)
at com.emc.vdp2.common.vi.VIAccess.getServiceInstance(VIAccess.java:200)
at com.emc.vdp2.server.ConnectionService.run(ConnectionService.java:55)
at java.lang.Thread.run(Unknown Source)
Thing is, I can't make backups right now. A certificate rollback is possible would only a short-term solution.
