Installing CA-signed SSL certificates on vCenter 5.1 (Server or Appliance)

[natedev]

I recently upgraded my ESXi 5.0 hosts to ESXi 5.1 and they all kept the CA-signed SSL certificates I previously installed. I did a fresh install of vCenter 5.1 Server where the same box ran SSO, Inventory Services, vCenter Server, and Update Manager. After install, everything was working perfectly except that none of the vCenter services were using my CA-signed SSL certificate - only the ESXi 5.1 hosts had these.

So I followed the directions in the Replacing Default vCenter 5.1 and ESXi Certificates PDF found at http://www.vmware.com/resources/techresources/10318. The document is terrible. For example, page 10 lists the three default locations for SSL certificates on Windows 2008. None of these paths are correct. The first has a typo of an extra space between "Program" and "Data" and the other two say "Program Files" when they should have been "ProgramData". This is just the beginning of the problems.

If you follow the directions to the letter, you'll break vCenter. I got frustrated and thought I'd give the vCenter 5.1 Appliance a shot. With respect to CA-signed SSL certificates, it was worse. The vCenter 5.1 Appliance can't even auto-generate a new SSL certificate if you change the hostname (turn on auto-certificate generation, change hostname and reboot). It gives a 653 error during the boot up process and keeps the original certificate. Don't even bother trying the steps on page 18 in the aforementioned guide - you'll just get the same 653 error.

It seems to me that VMware has not done any testing around CA-signed SSL certificate installation on vCenter 5.1. It's amazing to me that SSL certificate installation is so tedious for vCenter and ESXi when vShield Manager 5.1 has a very simple process that works well (and is similar to the SSL certificate installation process on DRACs, RSAs, iLOs, various firewalls, etc.).

I did a lot of Google searches and found various blogs on SSL certificate installation but many were based on pre-5.1 GA products. If you have had any success installing CA-signed SSL certificates with vCenter Server or Appliance 5.1 GA, please let me know how you got around some of these issues. Please indicate whether your vCenter Server or Appliance was running on an ESXi 5.1 GA host as well. Please don't respond regarding vCenter 5.0 - I didn't have any issues with it and SSL certificates (other than that it was more tedious that it had to be).

Thanks in advance,

Nate

[natedev]

Loren,

Thanks a lot for your contribution with the shell script. Wish a similar one could be built in PowerShell for the Windows side but I suspect there are far more permutations to deal with.

Take care,

Nate

[karipk]

Thanks Loren,

Edit:

I used the off the shelf appliance.

Works perfectly. All I had to do was to change following lines:

LDAP_PROTO=ldap         # ldaps or ldap

LDAP_PORT=389           # 389 or 3268 for ldap, 636 or 3269 for ldaps

DISABLE_ROOT_SSH=0      # 1 to disable ssh for root; 0 to leave it enabled

DISABLE_UNDESIRED_SERVICES=0 # 1 to disable undesired service; 0 to leave them enabled

I'm not sure if DISABLE_UNDESIRED_SERVICES=1 works, because I tried it and the vcenter server -service was not started after reboot.

here is how I created the certificates. (I use microsoft CA):

I used another linux machine to do this.

create new key:

openssl genrsa -out rui.key 2048

create new request:

openssl req -new -key rui.key -out rui.csr

cat rui.csr

request the certificate from CA (server.cer)

convert certificate :

openssl x509 -in server.cer -inform d -out rui.crt

Download CA  certificate (root.cer)

convert root certificate :

openssl x509 -in root.cer -inform d -out root.crt

copy rui.crt, rui.key, root.crt  and vcsa-config.sh to appliance.

in appliance run:

./vcsa-config.sh vserver.domain.local 'vCenterrootpassword' rui.crt,root.crt rui.key vserver.domain.local 'domainuserpassword' domainuser domain.local 'ssoserviceuserpassword' 'ssoadminpassword'

Kari

[lorengordon]

Cool, thanks for the feedback. The option DISABLE_UNDESIRED_SERVICES=1 will disable the embedded vCenter Server by design. It's intended for use with an external vCenter Server...

[alexmoore83]

Thanks very much for the script, it has been very helpful! I'm not using it as-is, but rather picking up clues from it about how to do some things that I hadn't figured out myself yet... I have been trying to achieve something very similar (ie retain vCenter on Windows for now, but move various other components over to VCSA appliances - in particular the Web Client and SSO - and try to achieve HA for them if possible).

What are your thoughts on getting SSO on VCSA working behind a load-balancer? My understanding is that your script results in each VCSA being configured to use its local embedded DB for SSO, but presumably they would either need to point to the same DB, or use replication of some kind? It looks to me like the Windows HA SSO setup takes the former approach (ie I think both SSO instances still depend upon a single database, and if you wanted to achieve HA for the database you'd need to address that separately, although I've not explicitly tested what happens to a Windows HA SSO setup if the DB goes down...).

So I tried setting up a pair of VCSA appliances running nothing but the vmware-vpostgres database (in a master-slave arrangement, using replication), and pointing a pair of VCSA appliances running SSO at them (ie as an external postgres SSO DB). Unfortunately I was unable to get the DB replication to work (I was trying to use postgresql's built-in WAL streaming with the slave configured as a hot-standby), seemingly because the vpostgres install on VCSA is missing some libraries (specifically: libpqwalreceiver.so, but perhaps others too). I guess it makes sense that VMware probably stripped out any unnecessary components from the vPostgres product when bundling it with VCSA.

Next attempt could be a pair of general purpose VMs running vanilla postgresql (rather than VCSA with bundled vpostgres), but at that stage the setup starts to lose some of the benefits of using the VMware supplied appliances. So I'm wondering whether you've had any other thoughts on that side of things...

[lorengordon]

Very good questions re: HA for SSO. At the moment, the approach I'm investigating is using the scripts in the Windows version of vCenter to repoint the Inventory and vCenter services at each SSO, then reconfiguring the services to use the load-balanced lookup service alias. Or something along those lines.

KB2033620 has the steps, but the VMware scripts and guidance have some issues (assuming default install paths, poorly quoted variables, etc). And then after fixing those minor things, they break something else that I haven't figure out yet. I keep getting an error in the vSphere Web Client saying that it couldn't authenticate to the Inventory Service. I think there might be some lingering certificate issues. Still investigating. Hoping to have better news later today.

[lorengordon]

I opened a case with VMware to look into the error message. Re-registering the vCenter Server and Inventory Service with a new SSO breaks something, somehow. Still seems like the cleanest way to get HA for the SSO service, so I'll keep pursuing it.

If we can get it working, it means there isn't any need to reconfigure the SSO urls. So with the script as it is, just pass the VCSA hostname.fqdn for the 5th parameter, instaed of the load-balancer alias.

Tomorrow, I'm going to rebuild it all with the default SSL certificates to see if re-registering works at that point.

P.S. I updated the script today to fix the logbrowser service after updating the SSL certificates. You can dl it from the earlier post. Derek Seaman's blog had the answer to that. http://derek858.blogspot.com/2012/10/vmware-vcenter-51-installation-part-14.html

-Loren

[Michaell2011101]

Hi all!

Did anyone try to install wildcard CA-signed SSL certificate to vCenter Virtual Appliance?

When I try to install I alway have error

VC_CFG_RESULT=659

I execute this command:

/usr/sbin/vpxd_servicecfg certificate change /root/ssl/rui.crt /root/ssl/rui.key

I tried 3 options:

rui.crt - only my CA-signed wildcard cert

rui.crt - wildcard cert + intermidiate cert + root CA cert. (cat server.crt intermidiate.crt ca.crt > rui.crt)

rui.crt - wildcard cert + root cert

Where I made a mistake?

[ealaqqad]

The good news is much of this pain will be now waived with the announcement of vCenter Certificate Automation Tool 1.0. You can download the tool at: https://my.vmware.com/web/vmware/info/slug/datacenter_cloud_infrastructure/vmware_vsphere/5_1#drive...

I have just posted a brief post about it at: How to replace vCenter 5.1, SSO, Web Client, vCO Certificates

vCenter Certificate Automation Tool 1.0 is worth your time if you are planning to replace certificates specially if you are using custom certificates.

Hope this help & Enjoy the new tool!

Regards,

Eiad Al-Aqqad

B: http://www.Virtualizationteam.com

B: http://www.TSMGuru.com

[bedobash]

ealaqqad,

your response isn't helpful as this entire thread is talking about the vCSA (or vCVA) and the documentation of the vCenter Certificate Automation Tool specifically states that you cannot use it to replace the certs on the vCenter Appliance.