Windows 2008 SSL

[phyler]

I have an interesting issue. I have a wildcard SSL cert that was purchased from Network Solutions. It is *.domain.com. If I bind SSL to a website inside IIS 7 on a Windows 2008 box running on VMWare ESX 3.5 64607 the machine breaks. The VMWare tools no longer start up and the network looks disconnected yet if I go Edit the settings it shows the network is connected. I can't ping the machine at all. If I roll back to the snapshot I took right before I setup the SSL, everything works great.

I have this same setup working on both physical boxes and some MS VM's, it is just VMWare that has this issue. Anyone else ever see this?

Thanks!

Adam

[kjb007]

If you remove the cert, does the problem go away? The ESX host really does not look at specific applications running on the vm, per se, so I couldn't imagine it would have an issue with you loading a certificate on an IIS 7 server running on Windows 2008. I'll have to check this out myself to see if it makes a difference. If you look in your eventviewer, do you see any other errors?

There are other users experiencing issues with their network card appearing to get disconnected, but it has been due to other issues.

Are you running 32 or 64-bit 2008? Are you using the flexible enhanced driver, or the e1000?

-KjB

[phyler]

Once you add the cert and reboot, network connectivity is lost and the server will no longer start IIS so you can't remove the binding.

There are no errors in the event viewer.

I am running x64 2008 Standard Edition. I am using the enhanced driver.

The strange part is that the VMWare Tools fail to start as do several other services. I can do anything else I want to the box but as soon as I bind the cert to a website and reboot, everything goes nutty and I have to rollback to a snapshot. Everything works fine until a reboot too which is weird (i.e. the SSL cert works when you hit https://servername).

Any thoughts are appreciated.

[kjb007]

Remove the NIC and re-add it. I seem to remember having to use the regular NIC for 64 bit windows, and not the enhanced. When the vm comes back up, re-install the vmware tools.

-KjB

[phyler]

So, I was using the e1000, I switched it to vmxnet just to test. The machine does the same thing with either network adapter installed. I'm stumped at this point due to the fact that the machine works fine until I add the SSL cert.

I have switched it back to the e1000 in the mean time and will keep battling the issue.

Adam

[kjb007]

I'd like to see the log, if I could after you bind the cert and the server fails to come up. Can you post it here?

-KjB

[phyler]

Here is the log:

Task Completed : haTask-800-vim.VirtualMachine.powerOn-134492

Ticket issued for mks connections to user: vpxuser

Failed to validate VM IP address:

Hw info file: /etc/vmware/hostd/hwInfo.xml

Config target info loaded

Failed to validate VM IP address:

DISKLIB-VMFS : "/vmfs/volumes/685809e7-005a8d84/cpb-bdr-ms/cpb-bdr-ms-000005-delta.vmdk" : open successful (17) size = 644245094 40, hd = 0. Type 8

DISKLIB-VMFS : "/vmfs/volumes/685809e7-005a8d84/cpb-bdr-ms/cpb-bdr-ms-000005-delta.vmdk" : closed.

DISKLIB-VMFS : "/vmfs/volumes/685809e7-005a8d84/cpb-bdr-ms/cpb-bdr-ms-000005-delta.vmdk" : open successful (19) size = 644245094 40, hd = 0. Type 8

DISKLIB-VMFS : "/vmfs/volumes/685809e7-005a8d84/cpb-bdr-ms/cpb-bdr-ms-000004-delta.vmdk" : open successful (23) size = 644245094 40, hd = 0. Type 8

DISKLIB-VMFS : "/vmfs/volumes/685809e7-005a8d84/cpb-bdr-ms/cpb-bdr-ms-000002-delta.vmdk" : open successful (23) size = 644245094 40, hd = 0. Type 8

DISKLIB-VMFS : "/vmfs/volumes/685809e7-005a8d84/cpb-bdr-ms/cpb-bdr-ms-000003-delta.vmdk" : open successful (23) size = 644245094 40, hd = 0. Type 8

DISKLIB-VMFS : "/vmfs/volumes/685809e7-005a8d84/cpb-bdr-ms/cpb-bdr-ms-flat.vmdk" : open successful (23) size = 64424509440, hd = 0. Type 3

DISKLIB-VMFS : "/vmfs/volumes/685809e7-005a8d84/cpb-bdr-ms/cpb-bdr-ms-000005-delta.vmdk" : closed.

DISKLIB-VMFS : "/vmfs/volumes/685809e7-005a8d84/cpb-bdr-ms/cpb-bdr-ms-000004-delta.vmdk" : closed.

DISKLIB-VMFS : "/vmfs/volumes/685809e7-005a8d84/cpb-bdr-ms/cpb-bdr-ms-000002-delta.vmdk" : closed.

DISKLIB-VMFS : "/vmfs/volumes/685809e7-005a8d84/cpb-bdr-ms/cpb-bdr-ms-000003-delta.vmdk" : closed.

DISKLIB-VMFS : "/vmfs/volumes/685809e7-005a8d84/cpb-bdr-ms/cpb-bdr-ms-flat.vmdk" : closed.

Failed to validate VM IP address: unknown

Hw info file: /etc/vmware/hostd/hwInfo.xml

Config target info loaded

Failed to validate VM IP address: unknown

This is all I get, you can see on the fourth to last line what happened as soon as I bind the SSL cert. The last three lines are the reboot after I bind the SSL cert.

Thanks,

Adam

[jwahlen]

Did anyone figure this out. I have the same thing on 2 different ESX servers with 2 different Virtual servers. Anytime I have IIS 7 and a wildcard SSL it will run fine until I reboot than VM Tools stops working and Network fails. I have to remove nic from 2008 and reboot and then resetup the network settings.

[chadjoubert]

The issue is the Networks Solution certificate not the wildcard. Because Microsoft does not have the intermediates Certs on the server you need to install them UTNAddTrustServer_CA.crt, NetworkSolutions_CA.crt The root certificate AddTrustExternalCARoot.crt.

Start -> mmc -> File -> add/remove snapin -> Certificates then select Computer Account, local computer.

Right click on Trusted Root Certificates and Import the other certs. Allow the Wizard to choose the location.

[htoudiee]

The above solution is correct. In case anyone needs the UTNAddTrustServerCA Intermediate Cert, you can download it here.

*You need to import this cert info the Intermediate store.

[jcrowland]

I have encountered this exact issue and initially thought this was related to ESX and network drivers due to the extreme flakiness of the problem even though it didn't make logical sense. Same bit, stuck on "applying computer settings", not able to do much useful with the network on or in safe mode, but could ping. Most services just won't start (Teminal Services, IIS, etc...)

I run various 2008 IIS servers and unfortunately reproduced this exact issue on multiple servers, 32-bit, 64-bit, different SP's. I have so many SSL certificates from various vendors that hunting down the offender was difficult because there is no logging whatsoever in IIS7 or Windows 2008 to indicate what the problem is. It boggled my mind that one missing Intermediary cert could cause such systemic havoc without any warning... I felt like I was working with NT 3.51.

This Microsoft KB article decribes this problem without focusing on the SSL side of it. Sure enough, upon making the registry changes outlined, everything works upon reboot... seems to involved the SCM database and references SSL keys:

http://support.microsoft.com/default.aspx/kb/2004121

Be aware, the version of the MSFT KB posted now has obvious typos for the registry entry to change... misspelling Services and leaving out System.

MSFT's KB authors meant to say:

1. Open Registry Editor

2. Navigate to HKLM\System\CurrentControlSet\Services\HTTP and create the following Multi-string value: DependOnService

3. Double click the new DependOnService value that you created

4. Enter CRYPTSVC in the Value Data field and click OK

5. After you have made this change, you will need to reboot the server.

If I remove the DependOnService=CRYPTSVC, the server images break again upon reboot, if I add it, it works. If you read the KB article it references SSL keys, doesn't sound like MSFT has a 100% handle on it yet, but this worked for me.

Hope this helps someone else out there, I've been wrangling with this issue since Thankgsiving.

--John