topic Re: Console Proxy not working behind Edge Load Balancer in VMware vCloud Director Discussions

Console Proxy not working behind Edge Load Balancer

Gabrie1 — Tue, 12 Mar 2013 08:45:18 GMT

I have a strange issue. I configured a load balancer using the vShield Edge. Behind the load balancer I have two vCloud Cells. The Web interface works fine for the users, but when trying to connect a VMRC to remotely view a screen it shows "connecting" and then the session is disconnected.

Then I shutdown the first cell and the users can make a VMRC connection again. When I enable the first cell again, the web interface continues to work, but the console proxy doesn't. Then I shutdown the second cell and now the console proxy works again.

Any tips?

Gabrie

Re: Console Proxy not working behind Edge Load Balancer

IamTHEvilONE — Fri, 15 Mar 2013 17:31:40 GMT

you probably have unique SSL Certificates on each cell for the console proxy. if you luck out and balance to the cell the created the session token, the certificate will match. However, when you balance to the node that did not make the token, there will be a thumbprint error and the session is not trusted.

Double check to make sure that the SSL Cert on cell 1 and 2 is the EXACT same for the consoleproxy entry.

Re: Console Proxy not working behind Edge Load Balancer

Gabrie1 — Sun, 17 Mar 2013 09:53:41 GMT

I copied my /etc/certificates.ks from the first cell to the second cell. Just to be sure I ran the vCloud configure script again on both cells, but I still get the disconnect.

Is there a way to check which certificate is in use and if they're really equal ?

Gabrie

Re: Console Proxy not working behind Edge Load Balancer

IamTHEvilONE — Sun, 17 Mar 2013 13:09:21 GMT

I don't know. if you ran the configure script and it didn't ask for the certificates.ks file, then it hasn't done anything to update the system.

somehwere in the $VCLOUD_HOME path, there are two certificates files .... i think certificate.ks and proxycertificate.ks. move those into /tmp then re-run the configure script. that'll force the script to re-create the individual certificates files.

you might be able to open SSL the console proxy addresses, but I haven't done that before.

Re: Console Proxy not working behind Edge Load Balancer

Gabrie1 — Mon, 18 Mar 2013 22:21:22 GMT

I found two files "certificates" and "proxycertificates" in /opt/vmware/vcloud-director/etc and deleted them both. Then I ran the configure script again and it asked for the certificates file. So I refered to /etc/certificates.ks and entered the password. I did this on both cells and then restarted the vmware-vcd service.

Unfortunately no change at all.

Think I'm going to call VMware Support because I'm out of options.

Re: Console Proxy not working behind Edge Load Balancer

cdickerson75 — Sat, 23 Mar 2013 04:40:56 GMT

Have a similar setup and was having the same problem. Two things to check. Go to c:\users\<login name>\appdata\local\temp\vmware-<login name> and open the latest vmware-vmrc-xxxx.log file. Near the end it should tell you why it's failing. For me it was a SSL thumbprint mismatch, which goes back to the different SSL certs on the cells. Also, make sure in vCloud director Administration-Public Address you have the correct VCD public console address specified. To fix my thumbprint mismatch, I just created a certificates.ks file on one cell and then copied it to the other cell and reran the configure script. Good luck getting VMware to help. I opened a case 6 days ago for this issue and have yet to get any help!

-Craig

Re: Console Proxy not working behind Edge Load Balancer

orthohin — Sat, 23 Mar 2013 07:04:20 GMT

Could you please have a look this kb. This may help you.

http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1026293

Regards,
Milton

Re: Console Proxy not working behind Edge Load Balancer

Gabrie1 — Sat, 23 Mar 2013 13:16:15 GMT

Thank you for the tips. Solved it, though it was different issue

Tunred out to be a timesync issue. See: http://www.chriscolotti.us/vmware/gotcha-ntp-can-affect-load-balanced-vcloud-vmrc/

This is actually already the end of my response, but to be complete I will include below what I've found when searching for the error. Also was affraid it was in the SSL certificates, but the logs below show no issue.

I double checked and the Public Addresses I used in vCloud administration -> public addresses page are:

VCD Public URL: https://vcloud-poc.xxxxxxxxx.lan/cloud

VCD Public console proxy address: vcloud-poc-console.xxxxxxxxx.lan (in the logs I can see this address is used and seems to work)

VCD public REST API base URL: https://vcloud-poc.xxxxxxxxxx.lan

I enabled both cells again and the connection failed as expected. In the vmware-vmrc log I found the following:

==================== FAILED CONNECTION ==========================

2013-03-23T13:06:04.235+01:00| vmrc| I120: Enabling HTTPS tunnellingvmClientCore::RunEmbeddedVMRC: parent PID: 4480, instance ID: "vmrc-np-t-{7A272BAB-D577-4974-950A-364D6EEDAE47}", modes: 2, messageMode: 2, features: 3

2013-03-23T13:06:04.235+01:00| vmrc| I120: vmClientCore::RunEmbeddedVMRC: eventName = "VMRC_EVENT_vmrc-np-t-{7A272BAB-D577-4974-950A-364D6EEDAE47}", monikerName = "VMRC_MONIKER_vmrc-np-t-{7A272BAB-D577-4974-950A-364D6EEDAE47}", monikerEvent = 494

2013-03-23T13:06:04.235+01:00| vmrc| W110: IMonikerImpl::vmClientCore::MonikerImpl<class vmClientCore::InvokeMgrDisp>::GetTimeOfLastChange invoked

2013-03-23T13:06:04.235+01:00| vmrc| I120: HOSTINFO 3097229574 @ 2669748Hz -> 0 @ 1000000000Hz

2013-03-23T13:06:04.235+01:00| vmrc| I120: HOSTINFO ((x * 3142097306) >> 23) + -1160120570722

2013-03-23T13:06:04.495+01:00| vmrc| I120: Setting proxy environment variable: "VMWARE_HTTPSPROXY="

2013-03-23T13:06:04.495+01:00| vmrc| I120: cui::vmrc::VMCnx::Connect: Connect to MOID "vm-934" on "vcloud-poc-console.xxxxxxx.lan"

2013-03-23T13:06:04.495+01:00| vmrc| I120: Resolving IP address for hostname vcloud-poc-console.xxxxxxx.lan

2013-03-23T13:06:04.495+01:00| vmrc| I120: Resolved to 172.17.1.94

2013-03-23T13:06:04.555+01:00| vthread-3| I120: VTHREAD initialize thread 3 "vthread-3" host id 2224

2013-03-23T13:06:04.615+01:00| vmrc| W110: SSL_IsVerifyEnabled: failed to open the product registry key. Falling back to default behavior: verification on. LastError = 0

2013-03-23T13:06:04.645+01:00| vmrc| I120: CertificateCheck::CheckCertEmbedded: allowSSLErrors: true

2013-03-23T13:06:04.645+01:00| vmrc| I120: CertificateCheck::CheckCertEmbedded: Found the following errors for vcloud-poc-console.xxxxxxx.lan's SSL certificate: {

2013-03-23T13:06:04.645+01:00| vmrc| I120: - 134217859

2013-03-23T13:06:04.645+01:00| vmrc| I120: - The certificate is based on an untrusted root.

2013-03-23T13:06:04.645+01:00| vmrc| I120: - A certificate in the host's chain is based on an untrusted root.

2013-03-23T13:06:04.645+01:00| vmrc| I120: - The host name used for the connection does not match the subject name on the host certificate.

2013-03-23T13:06:04.645+01:00| vmrc| I120: - The host's certificate is self-signed.

2013-03-23T13:06:04.645+01:00| vmrc| I120: }

2013-03-23T13:06:04.645+01:00| vmrc| I120: cui::CertificateCheck::CheckCertEmbedded - thumbprint for "vcloud-poc-console.xxxxxxx.lan" OK

2013-03-23T13:06:15.539+01:00| vmrc| I120: cui::vmrc::VMCnx::OnConnectAborted: Connect failed for MOID "vm-934" on "vcloud-poc-console.xxxxxxx.lan"

2013-03-23T13:06:15.539+01:00| vmrc| I120: cui::vmrc::VMCnxMgr::EmitConnectionStateSignal: Emitting "disconnected" signal (requested) for MOID "vm-934" on "vcloud-poc-console.xxxxxxx.lan" - reason 'An internal error occurred'

When I shutdown either one of the vCells, I get a succesful connection. See log:

============================== SUCCESS ============================

*********************** SUCCES ************************

2013-03-23T13:18:59.099+01:00| vmrc| I120: Enabling HTTPS tunnellingvmClientCore::RunEmbeddedVMRC: parent PID: 6464, instance ID: "vmrc-np-t-{650939BD-5C39-4E0A-BA30-976C4E3B3192}", modes: 4, messageMode: 2, features: 3

2013-03-23T13:18:59.101+01:00| vmrc| I120: vmClientCore::RunEmbeddedVMRC: eventName = "VMRC_EVENT_vmrc-np-t-{650939BD-5C39-4E0A-BA30-976C4E3B3192}", monikerName = "VMRC_MONIKER_vmrc-np-t-{650939BD-5C39-4E0A-BA30-976C4E3B3192}", monikerEvent = 524

2013-03-23T13:18:59.101+01:00| vmrc| W110: IMonikerImpl::vmClientCore::MonikerImpl<class vmClientCore::InvokeMgrDisp>::GetTimeOfLastChange invoked

2013-03-23T13:18:59.102+01:00| vmrc| I120: USBGW: Write arbitrator op:8 len:8

2013-03-23T13:18:59.270+01:00| vmrc| I120: Setting proxy environment variable: "VMWARE_HTTPSPROXY="

2013-03-23T13:18:59.270+01:00| vmrc| I120: cui::vmrc::VMCnx::Connect: Connect to MOID "vm-985" on "vcloud-poc-console.xxxxxxxxx.lan"

2013-03-23T13:18:59.270+01:00| vmrc| I120: Resolving IP address for hostname vcloud-poc-console.xxxxxxxxx.lan

2013-03-23T13:18:59.271+01:00| vmrc| I120: Resolved to 172.17.1.94

2013-03-23T13:18:59.301+01:00| vthread-5| I120: VTHREAD initialize thread 5 "vthread-5" host id 1696

2013-03-23T13:18:59.378+01:00| vmrc| W110: SSL_IsVerifyEnabled: failed to open the product registry key. Falling back to default behavior: verification on. LastError = 0

2013-03-23T13:18:59.389+01:00| vmrc| I120: CertificateCheck::CheckCertEmbedded: allowSSLErrors: true

2013-03-23T13:18:59.389+01:00| vmrc| I120: CertificateCheck::CheckCertEmbedded: Found the following errors for vcloud-poc-console.xxxxxxxxx.lan's SSL certificate: {

2013-03-23T13:18:59.389+01:00| vmrc| I120: - 134217859

2013-03-23T13:18:59.389+01:00| vmrc| I120: - The certificate is based on an untrusted root.

2013-03-23T13:18:59.389+01:00| vmrc| I120: - A certificate in the host's chain is based on an untrusted root.

2013-03-23T13:18:59.389+01:00| vmrc| I120: - The host name used for the connection does not match the subject name on the host certificate.

2013-03-23T13:18:59.389+01:00| vmrc| I120: - The host's certificate is self-signed.

2013-03-23T13:18:59.389+01:00| vmrc| I120: }

2013-03-23T13:18:59.389+01:00| vmrc| I120: cui::CertificateCheck::CheckCertEmbedded - thumbprint for "vcloud-poc-console.xxxxxxxxx.lan" OK

2013-03-23T13:19:00.851+01:00| vmrc| I120: cui::vmrc::VMCnx::OpenVM: Cnx connected for MOID "vm-985" on "vcloud-poc-console.xxxxxxxxx.lan"

2013-03-23T13:19:00.851+01:00| vmrc| I120: VMMgr::OpenVM (cnx=vcloud-poc-console.xxxxxxxxx.lan, moid=vm-985)

2013-03-23T13:19:01.590+01:00| vmrc| I120: cui::HostMgr::OpenHost - opening host with MOID "host-32"

2013-03-23T13:19:01.592+01:00| vmrc| I120: cui::HostMgr::OpenHost - opening host with MOID "host-32"

2013-03-23T13:19:01.592+01:00| vmrc| I120: cui::MsgMgrVIM::Manage: Suppressing VM questions

2013-03-23T13:19:01.594+01:00| vmrc| I120: VMStatus: Unknown guest os type: windows7Server64

2013-03-23T13:19:01.607+01:00| vmrc| I120: VMMgr: Finished opening VM at /vm/#_3F151E0_vm-985/ from vcloud-poc-console.xxxxxxxxx.lan:vm-985

2013-03-23T13:19:01.607+01:00| vmrc| I120: cui::vmrc::VMCnx::OnOpenVMDone: VM opened for MOID "vm-985" on "vcloud-poc-console.xxxxxxxxx.lan"

2013-03-23T13:19:01.607+01:00| vmrc| I120: cui::vmrc::VMCnxMgr::EmitConnectionStateSignal: Emitting "connected" signal (requested) for MOID "vm-985" on "vcloud-poc-console.xxxxxxxxx.lan" - reason 'Connection successful.'

2013-03-23T13:19:01.607+01:00| vmrc| I120: cui::vmrc::VMRC::OnActiveVMCnxChanged: Active VMCnx: MOID "vm-985" on "vcloud-poc-console.xxxxxxxxx.lan"

2013-03-23T13:19:01.618+01:00| vmrc| I120: SnapshotTree: Emitting refresh ([VMFS-042-R5-Tier1-CX3-40-01-PROD] APV001 (25a6245d-19ce-447c-84dd-9e49ff25c744)/APV001 (25a6245d-19ce-447c-84dd-9e49ff25c744).vmx)

2013-03-23T13:19:28.029+01:00| vmrc| I120: cui::vmrc::VMCnx::Disconnect: Disconnect from MOID "vm-985" on "vcloud-poc-console.xxxxxxxxx.lan" while "connected"

2013-03-23T13:19:28.029+01:00| vmrc| I120: cui::vmrc::VMCnx::DoDisconnect: Disconnecting from MOID "vm-985" on "vcloud-poc-console.xxxxxxxxx.lan"

2013-03-23T13:19:28.029+01:00| vmrc| I120: VMMgr::CloseVM: closing VM at /vm/#_3F151E0_vm-985/

2013-03-23T13:19:28.029+01:00| vmrc| I120: VMMgr::OnVMDestroyed: cleaning up after destroyed VM at /vm/#_3F151E0_vm-985/

2013-03-23T13:19:28.037+01:00| vmrc| I120: cui::vmrc::VMCnxMgr::EmitConnectionStateSignal: Emitting "disconnected" signal (requested) for MOID "vm-985" on "vcloud-poc-console.xxxxxxxxx.lan" - reason 'Disconnection successful.'

2013-03-23T13:19:28.037+01:00| vmrc| I120: cui::vmrc::VMRC::OnActiveVMCnxChanged: Active VMCnx: (none)

2013-03-23T13:19:28.087+01:00| vmrc| I120: Clean exit