7 Replies Latest reply on Dec 7, 2018 10:44 AM by TonyLyne

    Horizon 7.5 Connection Server SSL Update breaking deployment

    TonyLyne Lurker

      Hi team,

       

      This is a frustrating problem, surely someone has run into this before.

       

      Have a lab deployment of HV 7.5 and in general it was working well. Until I updated the default self signed SSL certs to an external one (Wildcard Rapid SSL SHA 256), also same thing with an internal Windows CA issued cert.

       

      I follow the tasks in this article to the letter Import a Signed Server Certificate into a Windows Certificate Store and from the outset it looks to be a simple update.

      However, when I restart the Connection Server Service on the Connection Server we lose connectivity to it, we get the Page cannot be displayed error in a browser when trying to browse the admin pages. Also the Agents all loose connection with the brokers - example of a static RDSH workload below (Debug log from the agent)

       

      2018-12-05T09:19:36.186+13:00 DEBUG (1B28-0450) <Thread-4> [JmsManager] Unable to connect to JMS server win2k12-cb1.home.lab com.vmware.vdi.logger.Logger.debug(Logger.java:44)

      javax.jms.IllegalStateException: Session is closed

      at com.swiftmq.jms.v750.SessionImpl.verifyState(Unknown Source)

      at com.swiftmq.jms.v750.SessionImpl.createTemporaryTopic(Unknown Source)

      at com.vmware.vdi.messagesecurity.swiftmq.BrokerUpdateUtility.updateOverJms(SourceFile:139)

      at com.vmware.vdi.agent.messageserver.AgentJmsConfig.pairOverJms(SourceFile:376)

      at com.vmware.vdi.agent.messageserver.JmsManager.connect(SourceFile:288)

      at com.vmware.vdi.agent.messageserver.Main.Start(SourceFile:1265)

      2018-12-05T09:19:36.186+13:00 WARN  (1B28-0450) <Thread-4> [JmsManager] Unable to connect to any listed host. The agent will continue to retry: [win2k12-cb1.home.lab]

      2018-12-05T09:19:36.186+13:00 DEBUG (1B28-0450) <Thread-4> [JmsManager] Will not report initial error

      2018-12-05T09:19:36.186+13:00 DEBUG (1B28-0450) <Thread-4> [JmsManager] Will attempt JMS Thumbprint exchange

      2018-12-05T09:19:51.072+13:00 DEBUG (1B28-0450) <Thread-4> [JmsManager] Using connection broker win2k12-cb1.home.lab

      2018-12-05T09:19:51.072+13:00 DEBUG (1B28-1930) <MessageFrameWorkDispatch> [MessageFrameWork] KeyVault service got operation=getEndEntityCertificates, ok=1, msecs=0

      2018-12-05T09:19:51.102+13:00 DEBUG (1B28-1930) <MessageFrameWorkDispatch> [MessageFrameWork] KeyVault service got operation=getEndEntityCertificates, ok=1, msecs=15

      2018-12-05T09:19:51.335+13:00 DEBUG (1B28-0450) <Thread-4> [JmsManager] username for swiftmq connection is: agent/f5ff457a-a849-4e35-b436-f2e141dd7539

      2018-12-05T09:19:51.335+13:00 INFO  (1B28-0450) <Thread-4> [AgentJmsConfig] Attempting to securely pair agent for JMS communication

      2018-12-05T09:19:51.335+13:00 DEBUG (1B28-0450) <Thread-4> [AgentMessageSecurityHandler] Configuring message security (ENHANCED).

      2018-12-05T09:19:51.397+13:00 DEBUG (1B28-0450) <Thread-4> [BrokerUpdateUtility] Published CHANGEKEY request

      2018-12-05T09:19:53.324+13:00 DEBUG (1868-1108) <TimerService> [ws_perfmon] System CPU use: 6%

      2018-12-05T09:19:53.324+13:00 DEBUG (1868-1108) <TimerService> [ws_perfmon] Commit memory use: 8%

      2018-12-05T09:19:53.386+13:00 DEBUG (1868-10EC) <MessageFrameWorkDispatch> [ws_winauth] CheckDomainMembershipStatus for domain home.lab

      2018-12-05T09:19:53.510+13:00 DEBUG (1868-10EC) <MessageFrameWorkDispatch> [ws_winauth] CheckDomainMembershipStatus bind successful

      2018-12-05T09:19:53.510+13:00 DEBUG (1868-10EC) <MessageFrameWorkDispatch> [wsnm_desktop] domainChecker: Cacheing domain membership status DOMAIN_CHECK_ERR_OK

      2018-12-05T09:19:53.510+13:00 DEBUG (1868-10EC) <MessageFrameWorkDispatch> [wsnm_desktop] domainChecker: Setting timer for 3600 seconds

      2018-12-05T09:19:57.118+13:00 DEBUG (1868-1354) <PSGC> [wsnm_psgc] Sending get-info request (mid=620)

      2018-12-05T09:19:57.118+13:00 DEBUG (1868-1264) <PSGC:L> [wsnm_psgc] Good response received for get-info request (mid=620) in 0 milliseconds

      2018-12-05T09:20:01.382+13:00 DEBUG (1B28-0450) <Thread-4> [BrokerUpdateUtility] Timeout waiting for success response

      2018-12-05T09:20:18.712+13:00 DEBUG (1868-1354) <PSGC> [wsnm_psgc] Sending get-counters request (mid=621)

      2018-12-05T09:20:18.712+13:00 DEBUG (1868-1264) <PSGC:L> [wsnm_psgc] Good response received for get-counters request (mid=621) in 0 milliseconds

      2018-12-05T09:20:18.712+13:00 INFO  (1868-1264) <PSGC:L> [wsnm_psgc] Stats: count=0, hwm=0

      2018-12-05T09:20:21.411+13:00 DEBUG (1B28-0450) <Thread-4> [BrokerUpdateUtility] Published CHANGEKEY request

      2018-12-05T09:20:31.427+13:00 DEBUG (1B28-0450) <Thread-4> [BrokerUpdateUtility] Timeout waiting for success response

      2018-12-05T09:20:51.732+13:00 DEBUG (1B28-0450) <Thread-4> [BrokerUpdateUtility] Published CHANGEKEY request

      2018-12-05T09:20:51.903+13:00 DEBUG (1B28-0450) <Thread-4> [msgid] Validating message with ID: '9adf4353-9cfb-417e-bfb6-2023bd27000d'.

      2018-12-05T09:20:51.903+13:00 WARN  (1B28-0450) <Thread-4> [JMSMessageSecurity] Message could not be validated: Message not signed

      2018-12-05T09:20:51.903+13:00 WARN  (1B28-0450) <Thread-4> [JMSMessageSecurity] Identity validation failed: UNKNOWN

      2018-12-05T09:20:51.903+13:00 DEBUG (1B28-0450) <Thread-4> [JMSMessageSecurity] Identity validation failure trace com.vmware.vdi.logger.Logger.debug(Logger.java:44)

      java.lang.Exception: Identity validation failed: UNKNOWN is not known identity for: null

      at com.vmware.vdi.messagesecurity.JMSMessageSecurity.a(SourceFile:572)

      at com.vmware.vdi.messagesecurity.JMSMessageSecurity.validateAndCheckWithHandler(SourceFile:450)

      at com.vmware.vdi.messagesecurity.swiftmq.BrokerUpdateUtility.a(SourceFile:236)

      at com.vmware.vdi.messagesecurity.swiftmq.BrokerUpdateUtility.updateOverJms(SourceFile:162)

      at com.vmware.vdi.agent.messageserver.AgentJmsConfig.pairOverJms(SourceFile:376)

      at com.vmware.vdi.agent.messageserver.JmsManager.connect(SourceFile:288)

      at com.vmware.vdi.agent.messageserver.Main.Start(SourceFile:1265)

      2018-12-05T09:20:51.903+13:00 WARN  (1B28-0450) <Thread-4> [BrokerUpdateUtility] Dropping response as not validated

       

      If I replace the SSL cert with the default self signed cert I can get back into the HV Console and can see the agent has still not able to connect (a reboot doesn't fix this, also a reinstall of the Agent on this test workload doesnt fix this!)

       

      I've tried to drop the JMS security mode down to mixed or lower as outlined here Using the vdmutil Utility to Configure the JMS Message Security Mode thinking the Agent has picked up a corrupted or bad cert from the connection broker when the update was done, and then by forcing it to negotiate at a lower level I can fix it, but unfortunately I cant seem to drop the security mode down from enhanced in 7.x of view.

       

      Below is the console view once I get the original self signed cert re-instated, note the agent is no longer reachable.

       

      Anyone have any ideas on this, before I do a complete rebuild of the deployment?

      What am I missing when it comes to updating SSL certs on the connection broker?