Error Connecting to VM remote console ... after upgrade to VC 2.0.1 b 40644

[tenor0]

I have an XP SP2 - VC server that was working fine until I patched it to b 40644. I can connect to the remote console of any VM from a Vi Client running on the VC server;but if I try it from a Vi Client from any other host, I get the ERROR CONNECTING. NO CONNECTION COULD BE MADE BECAUSE THE TARGET MACHINE ACTIVELY REFUSED IT.

I can open the remote console to any VM if I use the Vi Client to connect directly to the ESX servers from any HOST, but using the VC server fails with the ERROR CONNECTING message but for the HOST running the VC server.

There is no firewall; I have re-initialized the DB; and I even reinstalled the VC server over the weekend. But as soon as I apply the patch b 40644 for the VC server..... you guessed it right... I get the ERROR CONNECTING.

I DO NOT see the same issue with a W2K3 VC server patched to b40644.

Any insight on how to resolve this issue will be greatly appreciated.

[esiebert7625]

Check your VC Server logs, this is most commonly a database or port issue.

To view the log files directly open Windows Explorer and go to the C:\Windows\Temp\VPX directory on the Virtual Center server

[tenor0]

These are the errors I see for each ESX server in my farm...

\[2007-05-14 11:36:27.674 'BaseLibs' 3700 warning] SSLVerifyCertAgainstSystemStore: The remote host certificate has these problems:

\* The host name used for the connection does not match the subject name on the host certificate

\* A certificate in the host's chain is based on an untrusted root.

\[2007-05-14 11:36:27.674 'BaseLibs' 3700 warning] SSLVerifyIsEnabled: failed to read registry value. Assuming verification is disabled. LastError = 0

\[2007-05-14 11:36:27.674 'BaseLibs' 3700 warning] SSLVerifyCertAgainstSystemStore: Certificate verification is disabled, so connection will proceed despite the error

\[2007-05-14 11:36:27.674 'BaseLibs' 3700 warning] SSLVerifyCertAgainstSystemStore: Subject mismatch: etlesx10 vs etl247-esx02.cqa.com[/i]

I do see the clients sychronising....

\[2007-05-14 11:36:34.496 'App' 3640 info] \[VpxLRO] -- BEGIN task-internal-9 -- host-64 -- VpxdInvtHostSyncHostLRO.Synchronize

\[2007-05-14 11:36:34.496 'App' 3640 info] \[VpxdHostSync] Synchronizing host: etl247-esx01.cqa.com (192.168.15.194)

\[2007-05-14 11:36:35.873 'App' 3640 info] \[VpxdHostSync] Retrieved host update to 32

\[2007-05-14 11:36:35.873 'App' 3640 info] \[VpxdHostSync] Completed host synchronization

\[2007-05-14 11:36:35.873 'App' 3640 info] \[VpxdInvtHost] Updating vpxa for etl247-esx01.cqa.com (spec. genno 28)[/i]

I do not see any error messages when I use a Vi Client from a different host ....

\[2007-05-14 12:09:53.325 'App' 3992 info] \[VpxLRO] -- BEGIN task-internal-82 -- group-d1 -- vim.ManagedEntity.GetEffectiveRole

\[2007-05-14 12:09:53.325 'App' 3992 info] \[VpxLRO] -- FINISH task-internal-82 -- group-d1 -- vim.ManagedEntity.GetEffectiveRole

\[2007-05-14 12:09:54.171 'App' 3992 info] \[VpxLRO] -- BEGIN task-internal-83 -- vm-76 -- vim.ManagedEntity.GetEffectiveRole

\[2007-05-14 12:09:54.171 'App' 3992 info] \[VpxLRO] -- FINISH task-internal-83 -- vm-76 -- vim.ManagedEntity.GetEffectiveRole

\[2007-05-14 12:09:59.430 'App' 3992 info] \[VpxLRO] -- BEGIN task-internal-84 -- vm-76 -- vim.VirtualMachine.GetConfig

\[2007-05-14 12:09:59.446 'App' 3992 info] \[VpxLRO] -- FINISH task-internal-84 -- vm-76 -- vim.VirtualMachine.GetConfig

\[2007-05-14 12:09:59.587 'App' 3992 info] \[VpxLRO] -- BEGIN task-internal-85 -- vm-76 -- vim.VirtualMachine.acquireMksTicket

\[2007-05-14 12:09:59.822 'App' 3992 info] \[VpxLRO] -- FINISH task-internal-85 -- vm-76 -- vim.VirtualMachine.acquireMksTicket

\[2007-05-14 12:10:03.939 'App' 3992 info] \[VpxLRO] -- BEGIN task-internal-86 -- vm-76 -- vim.VirtualMachine.acquireMksTicket

\[2007-05-14 12:10:04.189 'App' 3992 info] \[VpxLRO] -- FINISH task-internal-86 -- vm-76 -- vim.VirtualMachine.acquireMksTicket[/i]

Is there a specific ERROR message that I should look for that pertains to the remote console.

[tenor0]

Here is the output for netstat -a , ETL247-ESXVC01 is my VC server and it seems that it can connect to the ESX servers(esx64-1, esx64-2,etl247-esx01, etl247-esx02) and the VC server on port 902

TCP ETL247-ESXVC01:1903 esx64-2.cqa.com:902 ESTABLISHED

TCP ETL247-ESXVC01:1905 esx64-1.cqa.com:902 ESTABLISHED

TCP ETL247-ESXVC01:1913 etl247-esx01.cqa.com:902 ESTABLISHED

TCP ETL247-ESXVC01:1915 etl247-esx02.cqa.com:902 ESTABLISHED

TCP ETL247-ESXVC01:1925 ETL247-ESXVC01:http ESTABLISHED

TCP ETL247-ESXVC01:1931 ETL247-ESXVC01:902 ESTABLISHED

TCP ETL247-ESXVC01:1992 ETL247-ESXVC01:902 ESTABLISHED

But from anyother host, here CHARLIEVICTOR, running a Vi Client, it seems like a connection is only established to the VC server (192.168.15.179 - ETL247_ESXVC01).

TCP CharlieVictor:4911 192.168.15.179:http ESTABLISHED

TCP CharlieVictor:4912 192.168.15.179:902 ESTABLISHED

Do I need to be able to connect to the ESX server on port 902 to open a remote console to a VM running on a specific ESX server?

[esiebert7625]

Try verifying that your ESX servers that are managed by that VC server were upgraded to the new VC agent by typing “vpxa- v”

o VC 2.0.1 build number is 32042

o VC 2.0.1 build number is 32042

o VC 2.0.1 Patch 1 build number is 33643

o VC 2.0.1 Patch 2 build number is 40644

You also might try doing a “service vmware-vpxa restart” followed by “service mgmt-vmware restart” on the ESX servers.

[esiebert7625]

902 and 903 (depends if you use ESX and/or VC)

page 188 - http://www.vmware.com/pdf/vi3_server_config.pdf

902 Authentication traffic for the ESX Server host and virtual

machine configuration.

Use Port 902 for the following:

! VI Client access to the VirtualCenter Server.

! VirtualCenter Server access to ESX Server hosts.

! Direct VI Client access to ESX Server hosts.

! ESX Server host access to other ESX Server hosts for

migration and provisioning.

Incoming TCP,

outgoing UDP

903 Remote console traffic generated by user access to virtual

machines on a specific ESX Server host.

Use Port 903 for the following:

! VI Client access to virtual machine consoles.

! VI Web Access Client access to virtual machine consoles.

[tenor0]

\[root@etl247-esx01 root]# vpxa -v

VMware VirtualCenter Agent Daemon 2.0.1 build-40644[/i]

The restart of vpx services on the ESX did not help.

How would I check to see if PORT 903 can accept the Remote Console connetion?

Here is what I do not understand about the port, I can remote console when I direclty connect to the ESX server from the host CHARLIEVICTOR using Vi Client. I use netstat to verfiy the port 903 is indeed open. But when I connect to the VC server and try to remote console it does not work. So, essentially the PORT is OK....

[esiebert7625]

Can you load the VI client on the VC server and connect to the ESX and use remote console. If you can then that port should be open for VC Server also, if you can't then something on the VC server is blocking it.

If that does work then I think your certificate errors may be the problem. Are the ESX hosts you are trying to manage with this server being managed by another server by chance? A ESX host can only be managed by one VirtualCenter server at a time. You cannot use multiple VC server to manage the same ESX servers.

[tenor0]

I can connect to the remote console using the Vi Client on the VC server, which is what makes it bizarre. I do not have a firewall setup on my VC server or on the HOST running the Vi Client.

It is only from other hosts that are running Vi Client that cannot connect to the remote console through the VC server. But, I can connect to a remote console form any of the hosts if I directly connect to the ESX server through the Vi Client

I checked. I only have one VC managing my ESX servers. Is there a way to reset the certificates?

[esiebert7625]

This guide may help...

Replacing VirtualCenter Server Certificates - http://www.vmware.com/pdf/vi_vcserver_certificates.pdf

[tenor0]

Thank you. Will try it.

[tenor0]

I installed VC 2.0.1 b32042 just to check, and now everything works fine. So, there is not network or certificate issue.

What could have changed while installing b33643 or b40644 that could cause such an issue?

[esiebert7625]

Each version install of VC is a full installation of the program. Maybe they changed the certificate behavior in the later versions.

Based on this error:

"The host name used for the connection does not match the subject name on the host certificate"

it could have something to do with the DNS name of the VC server or ESX hosts, ie. common name versus fqdn. Certificates are usually tied to specific domain names, when you connect to VC you might try using the FQDN instead of the common name or vice versa. Same when adding the ESX host to VC. Also make sure your CN and FQDN both resolve ok in DNS.

[tenor0]

I have 4 ESX servers. Only two have the problem because their HOSTNAME was changed. So, I am not sure if is a certificated issue.

How do you completely uninstall VC server from a XP host i.e. is there any registry keys I would need to get rid off?

Also, what must I clear on the ESX server side if I am going to use a new VC server to manage them?

Do you think, changing the HOSTNAME on my VC Server and completely re-installing the VC server would help?

Thank you for all you help so far.

[tenor0]

To clarify "I have 4 ESX servers. Only two have the problem because their HOSTNAME was changed. So, I am not sure if is a certificated issue."[/i]

I only have the certificate error problem on 2 of the ESX servers. But I see the Remote console connect errors on all 4 ESX servers.

[esiebert7625]

To completely wipe VC, first remove the ESX hosts from VC which should disassociate them with that VC server. Then shut down VC, go to a CMD prompt and run "vpxd.exe –b" to delete the database. Un-install VC, then delete any leftover files from C:\Program Files\VMware, and delete any leftover registry entries from HKLM\Software\VMware

Then re-install VC and add the ESX hosts back in.

[zemotard]

Is it a potential risk for ESX host to update VC server ?

[esiebert7625]

Shouldn't harm ESX at all, the VC install is independent of ESX except for the agent install to the ESX servers