Horizon Connection Server update to 2206 blew up all our RDS Linuxes

[Perttu]

Hi all, 

After few Connection Server updates all our RDS Linuxes (manually registered unmanaged agents) went into unreachable state. From the /var/log/vmware/viewagent-debug.log of affected host I can find a following entry corresponding with the broker update timestamp. 

2022-08-30T10:22:56.301Z DEBUG <SwiftMQ-SessionPool-2> [msgid] Validating message with ID: 'null'.
2022-08-30T10:22:56.303Z DEBUG <pool-7-thread-4> [StandaloneAgent] vdi-rds-host1 received: Reconfigure[id:-6ea19e90:182e98bf8c9:-7845|serverdn:null|pooldn:null|asyncsessionseconds:null|msmode:null|disconnecttimeout:null|emptysessionlogoff:null|emptysessiontimeout:null|prelaunchsessiontimeout:null|maxsessions:null|brokerPublicKey:null|brokers:null]
2022-08-30T10:22:56.303Z INFO <pool-7-thread-4> [StandaloneAgent] Reconfiguring agent: vdi-rds-host1 with new Pool dn: null and brokers: null 

Sounds bit suspicious and problematic to me, probably a race condition on brokers' side. I hope this didn't hit your environments. I suppose I need to re-register these agents with agent re-installation.

[Perttu]

I also see that on an affected machine file /etc/vmware/viewagent-machine.cfg doesn't include broker data anymore, and it was updated during the CS update.

$ stat /etc/vmware/viewagent-machine.cfg
File: /etc/vmware/viewagent-machine.cfg
Size: 920 Blocks: 8 IO Block: 4096 regular file
Device: fd01h/64769d Inode: 1838769 Links: 1
Access: (0600/-rw-------) Uid: ( 0/ root) Gid: ( 0/ root)
Access: 2022-08-31 10:23:37.371999636 +0300
Modify: 2022-08-30 10:22:56.305406337 +0300
Change: 2022-08-30 10:22:56.305406337 +0300

$ cat /etc/vmware/viewagent-machine.cfg
{"asyncSessionSeconds":150,"name":"vdi-rds-host1","dn":"cn\u003d54a93ef8-c11e-4067-9472-b60bafdc1410,ou\u003dServers,dc\u003dvdi,dc\u003dvmware,dc\u003dint","messageSecurityMode":"ON","agentIdentity":"agent/54a93ef8-c11e-4067-9472-b60bafdc1410","agentPrivateKey":"MI..","agentPublicKey":"MI..","poolDn":"","useSVI":"0","useSysprep":"0"}

Next I try to manually add the broker data into that json, if it would trigger reconnection and then systemctl restart viewagent.

[Perttu]

And then I manually edited that file to include two missing attributes "brokerPublicKey" and "brokers" and restarted viewagent. Now I'm seeing in viewagent-debug.log lines such as

2022-08-31T12:09:59.458Z DEBUG <pool-3-thread-1> [msgid] Validating message with ID: 'null'.
2022-08-31T12:09:59.458Z WARN <pool-3-thread-1> [JMSMessageSecurity] Message could not be validated: Message not signed
2022-08-31T12:09:59.458Z WARN <pool-3-thread-1> [JMSMessageSecurity] Identity validation failed: UNKNOWN
2022-08-31T12:09:59.458Z DEBUG <pool-3-thread-1> [JMSMessageSecurity] Identity validation failure trace
java.lang.Exception: Identity validation failed: UNKNOWN is not known identity for: null
at com.vmware.vdi.messagesecurity.JMSMessageSecurity.a(SourceFile:582) [messagesecurity.jar:?]
omitted stacktrace
2022-08-31T12:09:59.459Z WARN <pool-3-thread-1> [BrokerUpdateUtility] Dropping response as not validated 

apparenty something is still missing.

[Perttu]

And the last missing part was change "messageSecurityMode":"ON" -> "messageSecurityMode":"ENHANCED" in the very same file /etc/vmware/viewagent-machine.cfg. Then connection was restored and the /etc/vmware/viewagent-machine.cfg automatically renewed itself (also some other lost data such as poolDn were restored).

Let's hope this helps somebody some day.