VMware Cloud Community
ikt
Enthusiast
Enthusiast

Successful VCSA 6.5 upgrade but then the webclient suddenly fails..

Hi,

I successfully upgraded my vsphere (VCSA + 3 hosts) from 6U2 to 6.5 yesterday.  It worked flawlessly and I could access everything with the web client (even the Update Manager stuff).

When I log in with the web client this morning, it accepts the login credentials and all the empty webclient frames show up, but when it is supposed to populate the frames with vsphere objects, it displays the following message in the top frame:

Could not connect to one or more vCenter Server systems:

https://vcenter.vmware.lan:443/sdk

As mentioned, it worked flawlessly last night.  Before leaving, the only thing I did was starting a Veeam B&R replication of all VMs (the new VCSA 6.5 VM was not included).  The replication had failed after processing about 50% of the VMs, but I have never had any problems with Veeam replication jobs before this.

Please note that my web browsers always yells about accessing my vcenter server as an unsecure site because of certificate issues (where do I find a how-to to register a valid certificate to a local site like mine?).  But as before I just created a security exception in my web browser and as usual it worked flawlessly yesterday with my new VCSA 6.5.

Thanks a lot for help on being able to log in to my VCSA 6.5 again 🙂

best regards Tor

15 Replies
ikt
Enthusiast
Enthusiast

Anyone....?

Or do I post this in the wrong forum..?

rgds Tor

Reply
0 Kudos
anksos
Enthusiast
Enthusiast

Hello ikt

Veeam Backup and Replication are not supported still with 6.5 version. As I have seen will be supported from 9.5 Update 1 version.

Thank you,

anksos

If you found it useful or you resolved the issue with my response just mark it as answered. Personal blog: https://anksos.wordpress.com
Reply
0 Kudos
ikt
Enthusiast
Enthusiast

I know, but I don't think that veeam has anything to do with this problem.

I have restarted the vcenter 6.5 server appliance and tried again from the webclient.

Here is a  screenshot from vcenter webclient after having been authenticated:

pastedImage_0.png

How do I troubleshoot this situation?  Anybody else having experienced this situation?

Thanks for help on how to proceed.

best regards


Tor

Reply
0 Kudos
Evan5
Enthusiast
Enthusiast

Hi - have you tried to connect with a different browser, just as a troubleshooting step?

Reply
0 Kudos
ikt
Enthusiast
Enthusiast

Hi,

I have always used Firefox, latest version.  Tried with IE 11 and got exactly the same result. 

To recall, I upgraded to vcsa 6.5? (latest version) upgraded hosts and vmware tools.  Could connect to vcenter server appliance with web client without any problem the whole day.  But this error manifested itself during login the next day.

Note that I have always had to save security exceptions with browsers due to the lack of trusted CA certificate for my vcenter server.  Could this be a certificate issue?

I also read that vcenter server could become unavailable if its ip address was changed.  I noticed that it was changed  from a temporary one to my old vcsa's address during the vcsa upgrade.  But I assume that ip change was done in the 'appropriate' manner.

What do I need to do now, in order to track down the root of my problem

regards

Tor

Reply
0 Kudos
Evan5
Enthusiast
Enthusiast

It's possible the problemis linked to the certificates.

Have a look at the following KB and hopefully this will help:

https://kb.vmware.com/kb/2050273

Reply
0 Kudos
ikt
Enthusiast
Enthusiast

Thanks for the tip.  I followed the instructions in the article and this is my psc_services file.  However there is no duplicate Name records here as there was in the article.  Does the data below seems ok, or does it indicate a problem?

root@vcenter [ ~ ]# cat /tmp/psc_services.txt

        Name: AboutInfo.vpx.name

        Description: AboutInfo.vpx.name

        Service Product: com.vmware.cis

        Service Type: vcenterserver

        Service ID: 6d4d5cd9-6684-4818-b281-6073cc2456cb

        Site ID: sso

        Node ID: 991c192c-24de-4598-b43b-b06a2864322b

        Owner ID: vpxd-50627b75-2f9b-4ec7-bc0f-11578987fc81@vsphere.lan

        Version: 6.5

        Endpoints:

                Type: com.vmware.cdc.provider

                Protocol: vmomi

                URL: https://vcenter.vmware.lan:443/sdk

                SSL trust: MIIDmjCCAoKgAwIBAgIJAMuh7mvjNCffM   ****  TRUNCATED  ****   VCoZIyoC+wKTDTDtXMISkNvI0WpnBGCSMI2mCTPqETOmyiFD9op3SkVccTT4RqVf9/Uzjgi5Cal0OE1XOviNSOQI=

                Endpoint Attributes:

                        cis.common.ep.localurl: http://localhost:8085/sdk

                Type: com.vmware.cis.common.resourcebundle

                Protocol: http

                URL: https://vcenter.vmware.lan:443/catalog/catalog.zip

                SSL trust: MIIDmjCCAoKgAwIBAgIJAMuh7mvjNCff  ****  TRUNCATED  ****  oZIyoC+wKTDTDtXMISkNvI0WpnBGCSMI2mCTPqETOmyiFD9op3SkVccTT4RqVf9/Uzjgi5Cal0OE1XOviNSOQI=

                Endpoint Attributes:

                        com.vmware.cis.common.resourcebundle.basename: cis.vc.action:cis.vc.alarm:cis.vc.auth:cis.vc.cluster:cis.vc.default:cis.vc.enum:cis.vc.evc:cis.vc.event:cis.vc.eventaux:cis.vc.fault:cis.vc.gos:cis.vc.host:cis.vc.locmsg:cis.vc.option:cis.vc.perf:cis.vc.question:cis.vc.stask:cis.vc.task:cis.vc.vm:cis.vc.profile:cis.vc.hostdiag.enum:cis.vc.hostdiag.eventaux:cis.vc.hostdiag.event:cis.vc.hostdiag.locmsg:cis.vc.VirtualCenter.enum:cis.vc.VirtualCenter.eventaux:cis.vc.VirtualCenter.event:cis.vc.VirtualCenter.extension:cis.vc.VirtualCenter.locmsg

                Type: com.vmware.vim.extension

                Protocol: vmomi

                URL: https://vcenter.vmware.lan:443/sdkTunnel

                SSL trust: MIIDmjCCAoKgAwIBAgIJAM  ****  TRUNCATED  ****  wKTDTDtXMISkNvI0WpnBGCSMI2mCTPqETOmyiFD9op3SkVccTT4RqVf9/Uzjgi5Cal0OE1XOviNSOQI=

                Endpoint Attributes:

                        cis.common.ep.localurl: http://localhost:8085/sdkTunnel

                Type: com.vmware.vim

                Protocol: vmomi

                URL: https://vcenter.vmware.lan:443/sdk

                SSL trust: MIIDmjCCAoKgAwIBAgIJAMuh7mvjNCffMA0GCSqGSIb3DQEBCwUAMIGLM  ****  TRUNCATED  ****  MISkNvI0WpnBGCSMI2mCTPqETOmyiFD9op3SkVccTT4RqVf9/Uzjgi5Cal0OE1XOviNSOQI=

                Endpoint Attributes:

                        cis.common.ep.localurl: http://localhost:8085/sdk

                Type: com.vmware.cis.data.provider

                Protocol: vmomi

                URL: https://vcenter.vmware.lan:443/sdk

                SSL trust: MIIDmjCCAoKgA  ****  TRUNCATED  ****  Sn+3ZVCoZIyoC+wKTDTDtXMISkNvI0WpnBGCSMI2mCTPqETOmyiFD9op3SkVccTT4RqVf9/Uzjgi5Cal0OE1XOviNSOQI=

                Endpoint Attributes:

                        cis.common.ep.localurl: http://localhost:8085/sdk

        Attributes:

                com.vmware.cis.cm.GroupInternalId: com.vmware.vim.vcenter

                com.vmware.vim.vcenter.instanceName: vcenter.vmware.lan

                com.vmware.cis.cm.ControlScript: service-control-default-vmon

                com.vmware.cis.cm.HostId: 50627b75-2f9b-4ec7-bc0f-11578987fc81

Reply
0 Kudos
Evan5
Enthusiast
Enthusiast

Hmmm. Tricky.

You mentioned that your IP Address changed during the upgrade. Is it still different to what it used to be? Have you tried putting it back to the original IP?

Have you checked DNS is resolving correctly?

Can you successfully login to the ESXi(s) host directly?

When you do, it will tell you the host is currently connected to vCenter at the following IP address and/or DNS name, check that is what it should be.

None of this is probably solving your problem but hopefully it will get you a step closer.

Reply
0 Kudos
ikt
Enthusiast
Enthusiast

Thanks again for following this up.

IP address, I just mentioned that I formerly had a VCSA 6U2 running and that I upgraded it according to the whitepapers to 6.5.  I assumed that my 'old' VCSA retained its IP address until a certain point in the upgrade process and that the 'new' VCSA then would be assigned to the old VCSA'a address.  I have not manually changed the VCSA's IP address.  Please also note that it worked flawlessly for a whole day, and this problem manifested itself the next morning.

I can successfully connect directly to any of my 3 ESXi hosts using both webclient and C# client.  FWIW I can also SSH into them and the VCSA and web access VCSA and PSC consoles.  It is only VCSA's webclient console that is failing to display any content...

The C# client displays the 'host is connected to vcenter server' message with the correct VCSA IP address.

I have an active Essentials Plus license.  Can I request support from vmware and let them access my environment via TemaViewer to track down the issue here, or will that cost me a fortune..?

regards Tor

Reply
0 Kudos
Evan5
Enthusiast
Enthusiast

The fact that it was working and then stopped is indeed strange. That's why I initially thought it may be a browser issue.

I think your best bet is to either open a ticket with VMware or failing that you could always install a new version of vCenter and just move your hosts into that. You will lose historical logs etc but you will be up and running sooner rather than later. If you use Distributed Switches you may have to recreate them too. A quick google search will gives you the pros and cons of this move but its not uncommon.

As for you licensing question, you generally have to have an active Support Agreement to get free support. If you log into your My VMware portal on the website you should be able to see your contract status.

Reply
0 Kudos
Evan5
Enthusiast
Enthusiast

One more thing worth checking is that all your Hosts and vCenter have their time synced. If they are way out of sync it tends to cause many problems.

This may account for why it worked and then didn't the next day. Bit of a long shot but worth a look.

Reply
0 Kudos
jbc974
Contributor
Contributor

Hi,

I'm facing the exact same issue.
Migration from Vcenter 6 (Windows) to VCA 6.5 with migration tool goes well.

Problems start from the moment I wan't to make any action through vsphere web client, like putting an esx in maintenance mode. Then I got the exact same issue as you (the esx do not get the maintenance mode command actually).

I've tryed migrating 3 times now and the problem occured everytime. I had no other option that remaining in vcenter 6.

I also suspected timesync but my vca and my hosts got the exact same time.

Philippe

Reply
0 Kudos
curriertech
Enthusiast
Enthusiast

Maybe not relevant but this was my similar experience going from 6.0 on Windows to 6.5 VCSA.  Database errors crashing vpxd, being the culprit.

After the webclient stops working, check to see if vpxd has stopped.  You can start it or reboot the appliance to get it going again, but it will fail again.  If vpxd is stopped, you can probably find errors like this in your logs:

2017-04-09T09:25:09.933Z error vpxd[7F1FE9E58700] [Originator@6876 sub=Default opID=HB-host-97553@32887-67f22e81] An unrecoverable problem has occurred, stopping the VMware VirtualCenter service. Error: Error[VdbODBCError] (-1) "ODBC error: (23505) - ERROR: duplicate key value violates unique constraint "vpxi_psa_path";

--> Error while executing the query" is returned when executing SQL statement "INSERT INTO VPX_PSA_PATH (HOST_ID, PATH_ID, KEY_VAL, LUN_NUMBER, SCSI_LUN_ID, HBA_ID, TARGET_ID, LUN_LINK_KEY, ADAPTER_LINK_KEY, TARGET_LINK_KEY) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?)"

Back when 6.5 first came out I upgraded and had this issue.  Support said it was a known issue and recommended I go back to 6.0, which I did.  Recently I upgraded again to 6.5b, and had the same result. 

-Josh.
Reply
0 Kudos
HNIS
Contributor
Contributor

We are having the exact same issue, though, the funny thing is we have 2 vcenter servers (one is DR.)  The DR one has no problems, but the main vcenter server is getting the error:

Could not connect to one or more vCenter Server systems:

https://vcenter.vmware.lan:443/sdk

I'm also getting a message is some areas about not having permissions to view this object or this object does not exist.  I have tried other logins besides my own with the same issue.

Restarting the VMware Identity Management Service worked for me for a moment, but then the error returned.  I have read elsewhere that users have reinstalled vcenter server, which I may end up trying very soon...

Please let me know if anyone has any other ideas...

Thanks!

Reply
0 Kudos
lmaxfield
Contributor
Contributor

Have also posted this solution on a similar-sounding issue involving vpxd service crashes -- VCSA 6.5 Web GUI Error

TLDR; If you have any v5.x Host Profiles, delete them.

We recently upgraded our v5.5 Windows vCenter with embedded SSO to a v6.5 vCSA with an external PSC. Everything seemed stable until we updated our ESXi hosts to v6.5, at which point interacting with the hosts in the Flash client (at least this seemed to be the most reliable trigger) would cause the vpxd service to crash.

None of the fixes in this thread worked for us, but a VMware tech stopped by today and we informed her that we were having this problem. She immediately told us to delete our v5.x Host Profiles, as they had recently seen this problem at another site. We had one v5.x Host Profile, and deleted it. After some testing, deleting the Host Profile appears to have resolved our vpxd service crashes.