VMware Horizon Community
TonyLyne
Contributor
Contributor
Jump to solution

Horizon 7.5 Connection Server SSL Update breaking deployment

Hi team,

This is a frustrating problem, surely someone has run into this before.

Have a lab deployment of HV 7.5 and in general it was working well. Until I updated the default self signed SSL certs to an external one (Wildcard Rapid SSL SHA 256), also same thing with an internal Windows CA issued cert.

I follow the tasks in this article to the letter Import a Signed Server Certificate into a Windows Certificate Store and from the outset it looks to be a simple update.

However, when I restart the Connection Server Service on the Connection Server we lose connectivity to it, we get the Page cannot be displayed error in a browser when trying to browse the admin pages. Also the Agents all loose connection with the brokers - example of a static RDSH workload below (Debug log from the agent)

2018-12-05T09:19:36.186+13:00 DEBUG (1B28-0450) <Thread-4> [JmsManager] Unable to connect to JMS server win2k12-cb1.home.lab com.vmware.vdi.logger.Logger.debug(Logger.java:44)

javax.jms.IllegalStateException: Session is closed

at com.swiftmq.jms.v750.SessionImpl.verifyState(Unknown Source)

at com.swiftmq.jms.v750.SessionImpl.createTemporaryTopic(Unknown Source)

at com.vmware.vdi.messagesecurity.swiftmq.BrokerUpdateUtility.updateOverJms(SourceFile:139)

at com.vmware.vdi.agent.messageserver.AgentJmsConfig.pairOverJms(SourceFile:376)

at com.vmware.vdi.agent.messageserver.JmsManager.connect(SourceFile:288)

at com.vmware.vdi.agent.messageserver.Main.Start(SourceFile:1265)

2018-12-05T09:19:36.186+13:00 WARN  (1B28-0450) <Thread-4> [JmsManager] Unable to connect to any listed host. The agent will continue to retry: [win2k12-cb1.home.lab]

2018-12-05T09:19:36.186+13:00 DEBUG (1B28-0450) <Thread-4> [JmsManager] Will not report initial error

2018-12-05T09:19:36.186+13:00 DEBUG (1B28-0450) <Thread-4> [JmsManager] Will attempt JMS Thumbprint exchange

2018-12-05T09:19:51.072+13:00 DEBUG (1B28-0450) <Thread-4> [JmsManager] Using connection broker win2k12-cb1.home.lab

2018-12-05T09:19:51.072+13:00 DEBUG (1B28-1930) <MessageFrameWorkDispatch> [MessageFrameWork] KeyVault service got operation=getEndEntityCertificates, ok=1, msecs=0

2018-12-05T09:19:51.102+13:00 DEBUG (1B28-1930) <MessageFrameWorkDispatch> [MessageFrameWork] KeyVault service got operation=getEndEntityCertificates, ok=1, msecs=15

2018-12-05T09:19:51.335+13:00 DEBUG (1B28-0450) <Thread-4> [JmsManager] username for swiftmq connection is: agent/f5ff457a-a849-4e35-b436-f2e141dd7539

2018-12-05T09:19:51.335+13:00 INFO  (1B28-0450) <Thread-4> [AgentJmsConfig] Attempting to securely pair agent for JMS communication

2018-12-05T09:19:51.335+13:00 DEBUG (1B28-0450) <Thread-4> [AgentMessageSecurityHandler] Configuring message security (ENHANCED).

2018-12-05T09:19:51.397+13:00 DEBUG (1B28-0450) <Thread-4> [BrokerUpdateUtility] Published CHANGEKEY request

2018-12-05T09:19:53.324+13:00 DEBUG (1868-1108) <TimerService> [ws_perfmon] System CPU use: 6%

2018-12-05T09:19:53.324+13:00 DEBUG (1868-1108) <TimerService> [ws_perfmon] Commit memory use: 8%

2018-12-05T09:19:53.386+13:00 DEBUG (1868-10EC) <MessageFrameWorkDispatch> [ws_winauth] CheckDomainMembershipStatus for domain home.lab

2018-12-05T09:19:53.510+13:00 DEBUG (1868-10EC) <MessageFrameWorkDispatch> [ws_winauth] CheckDomainMembershipStatus bind successful

2018-12-05T09:19:53.510+13:00 DEBUG (1868-10EC) <MessageFrameWorkDispatch> [wsnm_desktop] domainChecker: Cacheing domain membership status DOMAIN_CHECK_ERR_OK

2018-12-05T09:19:53.510+13:00 DEBUG (1868-10EC) <MessageFrameWorkDispatch> [wsnm_desktop] domainChecker: Setting timer for 3600 seconds

2018-12-05T09:19:57.118+13:00 DEBUG (1868-1354) <PSGC> [wsnm_psgc] Sending get-info request (mid=620)

2018-12-05T09:19:57.118+13:00 DEBUG (1868-1264) <PSGC:L> [wsnm_psgc] Good response received for get-info request (mid=620) in 0 milliseconds

2018-12-05T09:20:01.382+13:00 DEBUG (1B28-0450) <Thread-4> [BrokerUpdateUtility] Timeout waiting for success response

2018-12-05T09:20:18.712+13:00 DEBUG (1868-1354) <PSGC> [wsnm_psgc] Sending get-counters request (mid=621)

2018-12-05T09:20:18.712+13:00 DEBUG (1868-1264) <PSGC:L> [wsnm_psgc] Good response received for get-counters request (mid=621) in 0 milliseconds

2018-12-05T09:20:18.712+13:00 INFO  (1868-1264) <PSGC:L> [wsnm_psgc] Stats: count=0, hwm=0

2018-12-05T09:20:21.411+13:00 DEBUG (1B28-0450) <Thread-4> [BrokerUpdateUtility] Published CHANGEKEY request

2018-12-05T09:20:31.427+13:00 DEBUG (1B28-0450) <Thread-4> [BrokerUpdateUtility] Timeout waiting for success response

2018-12-05T09:20:51.732+13:00 DEBUG (1B28-0450) <Thread-4> [BrokerUpdateUtility] Published CHANGEKEY request

2018-12-05T09:20:51.903+13:00 DEBUG (1B28-0450) <Thread-4> [msgid] Validating message with ID: '9adf4353-9cfb-417e-bfb6-2023bd27000d'.

2018-12-05T09:20:51.903+13:00 WARN  (1B28-0450) <Thread-4> [JMSMessageSecurity] Message could not be validated: Message not signed

2018-12-05T09:20:51.903+13:00 WARN  (1B28-0450) <Thread-4> [JMSMessageSecurity] Identity validation failed: UNKNOWN

2018-12-05T09:20:51.903+13:00 DEBUG (1B28-0450) <Thread-4> [JMSMessageSecurity] Identity validation failure trace com.vmware.vdi.logger.Logger.debug(Logger.java:44)

java.lang.Exception: Identity validation failed: UNKNOWN is not known identity for: null

at com.vmware.vdi.messagesecurity.JMSMessageSecurity.a(SourceFile:572)

at com.vmware.vdi.messagesecurity.JMSMessageSecurity.validateAndCheckWithHandler(SourceFile:450)

at com.vmware.vdi.messagesecurity.swiftmq.BrokerUpdateUtility.a(SourceFile:236)

at com.vmware.vdi.messagesecurity.swiftmq.BrokerUpdateUtility.updateOverJms(SourceFile:162)

at com.vmware.vdi.agent.messageserver.AgentJmsConfig.pairOverJms(SourceFile:376)

at com.vmware.vdi.agent.messageserver.JmsManager.connect(SourceFile:288)

at com.vmware.vdi.agent.messageserver.Main.Start(SourceFile:1265)

2018-12-05T09:20:51.903+13:00 WARN  (1B28-0450) <Thread-4> [BrokerUpdateUtility] Dropping response as not validated

If I replace the SSL cert with the default self signed cert I can get back into the HV Console and can see the agent has still not able to connect (a reboot doesn't fix this, also a reinstall of the Agent on this test workload doesnt fix this!)

I've tried to drop the JMS security mode down to mixed or lower as outlined here Using the vdmutil Utility to Configure the JMS Message Security Mode thinking the Agent has picked up a corrupted or bad cert from the connection broker when the update was done, and then by forcing it to negotiate at a lower level I can fix it, but unfortunately I cant seem to drop the security mode down from enhanced in 7.x of view.

Below is the console view once I get the original self signed cert re-instated, note the agent is no longer reachable.

pastedImage_3.png

Anyone have any ideas on this, before I do a complete rebuild of the deployment?

What am I missing when it comes to updating SSL certs on the connection broker?

Reply
0 Kudos
1 Solution

Accepted Solutions
techguy129
Expert
Expert
Jump to solution

I don't see the private key attached to that certificate. It should look something like this. Notice the key icon

pastedImage_1.png

View solution in original post

Reply
0 Kudos
7 Replies
techguy129
Expert
Expert
Jump to solution

Did you update the friendly name of the certificate you imported?

Modify the Certificate Friendly Name

Reply
0 Kudos
TonyLyne
Contributor
Contributor
Jump to solution

Yes updated it to vdm and renamed the old one to vdm_old or something similar.

Reply
0 Kudos
techguy129
Expert
Expert
Jump to solution

If that isn't the issue and you verified the private key is in the Windows certificate store, you may need to uninstall and install the connection server.

People with the same error as you:

Horizon View Connection Server Errors |Virtualcloudz

View Admin Console not starting

Reply
0 Kudos
sjesse
Leadership
Leadership
Jump to solution

The certificate imported, did you include any intermediate ca's in a combined certificate, also is the root CA of those trusted by windows as well?

Reply
0 Kudos
TonyLyne
Contributor
Contributor
Jump to solution

Thanks folks for your help with this, much appreciated.

OK so this is getting frustrating, shouldn't be this hard!

So I ended up having to rebuild the deployment as it looks like the configuration database was corrupted when importing the cert and the keys for the VDA's were invalid. Being its a small lab initially it was quicker just to rebuild than try and find what the issue is. Rebuilding the environment was quick and I'm back up with the self signed certs.

If I however import a valid SSL cert into the connection broker and then set the friendly name to vdm as per instructions it makes the web console go off line again, with TLS errors in the web browser.

I can guarantee the SSL cert is valid, as its the same SSL cert and chain we use for our lab network load balancers and gateways, its a wildcard cert, see below.

pastedImage_2.png

pastedImage_3.png

pastedImage_4.png

Ive also tried an internal Lab Windows CA with no luck, exactly the same issue.

The documentation on VMWares web sites for SSL requirements is rather vague, so would be good to know exactly what the SSL cert requirements should be if its very particular about SSL types. I assume wildcard SSL certs are ok?

Any ideas?

Reply
0 Kudos
techguy129
Expert
Expert
Jump to solution

I don't see the private key attached to that certificate. It should look something like this. Notice the key icon

pastedImage_1.png

Reply
0 Kudos
TonyLyne
Contributor
Contributor
Jump to solution

You're right!

looks like the key didn't import with the cert, even though its there in the certificate file.

This is the strange bit. I couldn't get the key to import with the cert on the Windows 2012 R2 Connection broker, but could fine on a Windows Server 2016 server, so ended up importing it on the 2016 server, then re-exporting it with the key into a file from the 2016 server, and then importing it again on to the 2012 R2 broker server and it was away.

Must have been something weird going on with the original cert file I was using that messes with Windows Server 2012 R2 certificate manager, but I'm away and working now.

Nice work man....thanks for your help.

Reply
0 Kudos