Astro,

I'm attempting to do the same with my lab environment, by replacing all the ESX and the VirtualCenter default certificates with my own created from our Microsoft Cert. Services CA.  (I used the process outlined in the VMware Technical Note "VMware Infrastructure 3 - Replacing VirtualCenter Server Certificates" but it is basically the same thing here except for some of the syntax).  Note that I am running VC 2.5.0u1 Build 84767 and ESX 3.5.0u1 Build 82663.

I am like you and was lucky with VC and did not have a problem.  I was able to generate the CSRs using openssl then create the new certs from the MS CA web interface (https://<cert_server>/certsrv) and everything worked great.  Granted I had previously ignored certificate errors from the FQDN for the VC server which I installed the new cert on, but if I bring up that VC host's webpage over SSL/HTTPS, it does reflect the new cert. (Side question: can you reset the list of "ignored" hosts in the VC client?  Would this happen on uninstall/reinstall?)

My problem was with replacing the certs for the ESX hosts; I used the exact same method (with exception of .pfx generation) for generating keys & CSRs, cut/pasted the requests into the browser, moved the cert to the /etc/vmware/ssl directory, checked the perms ('644' for rui.crt, '600' for rui.key) and rebooted.  When the host came back up, it showed as unreachable in the client.  I then disconnected the host in VC, and tried to reconnect.  But instead of getting your error, I kept getting the error "Failed to install the VirtualCenter Agent service."  Looking at other threads, I tried all sorts of things to get the thing to connect such as restarting the "mgmt-vmware" and "vmware-vpxa" services, manually removing/installing the vpxa package, half a dozen reboots including the VC host, etc.  I finally checked the /etc/vmware/vpx/vpxa.log file and at the bottom I find the error:

                   http://... 'App' ... error  Re: SSL on Virtual Center and single Hosts  Failed: unrecognized file format: /etc/vmware/ssl/rui.crt

I then switched the cert and key files back the originals, restarted the services again, tried to connect again in VC and it worked first time.  Just to make sure nothing happened during the copy process, I thought I should double-check the validity of the files.  I used openssl ("openssl verify -CAfile MyCA.crt rui.key") to verify the key and it came back "OK."

So, it leaves me with a couple of glaring questions:

1. If I used the exact same process for VC as I am for the ESX hosts, why aren't they working?

2. If openssl can verify the certificate, what is the problem?

3. What am I doing wrong?

Hopefully we can get some more attention to this thread.

FYI

I managed to replace the certs on my VC and the hosts, then everything was running ok untill i needed to deploy from a template and the encryption configuration file for the sysprep stage did not work and i had to undo all my work.

This works: http://communities.vmware.com/thread/153794?tstart=0

Seb

Side question: can you reset the list of "ignored" hosts in the VC client?  Would this happen on uninstall/reinstall.

These sites are stored in the registry: HKEY_CURRENT_USER\Software\VMware\Virtual Infrastructure Client\Preferences\UI\SSLIgnore

The easiest way to find them is to open regedit and search for "SSLIgnore".

Delete the registry entries to undo the "ignore".

> Hi When i run the command to create the rui.csr file (openssl req -new -key rui.key > rui.csr) i get the following error

> " Unable to Load config info

> Unable to find 'distinguished_name' in config problems making Certificate Request

It sounds like you haven't set up the openssl.cnf config file correctly.

You could try passing the DN in the command line, using "-subj" something like this:

$ openssl req -new -nodes -md5 -key rui.key -subj '/C=US/ST=NY/L=Somewhere/organizationName=MyOrg/OU=MyDept/CN=fqdn.of.my.host' -days 3600 -out rui.csr