VCenter Server 6.5 - Could not connect to one or more vCenter Server Systems

[ryedaleit]

Our Windows 2016 Management Server had unexpected shut down yesterday. Ever since, when we open the vSphere web Client we get "could not connect to one or more vCenter Server Systems https://<myserver>:443/sdk

I've trawled through the KB's and Forums for example this https://kb.vmware.com/s/article/2050273 but nothing on how to fix it on VCenter Server 6.5

Should we simply un-install and re-install. I don't see a repair option.

[daphnissov]

Then take this opportunity to ditch the Windows vCenter and deploy the vCSA. You'll be better off in the long run.

[LokeshHK]

Try with steps mentioned for 6.x , it should be common for both 6.0.x and 6.5.x versions.

Regards

Lokesh

[ryedaleit]

I hear a lot of advice to do that. But how? What do I need to create a vCSA? We only have a limited budget (if any).

[daphnissov]

Budget makes no difference because it doesn't cost anything extra. Actually, you'll be saving money by freeing up a Windows Server license. If you have basic ESXi hosts with no vDS, you can deploy a net new vCSA and join the hosts to it and shutdown that Windows vCenter. It's more complicated than that because this is essentially considered a "migration" upgrade even though you're moving horizontally from 6.5 => 6.5. See my blog for more caveats.

[ryedaleit]

Tried 6.X

As instructed I ran;

"%VMWARE_PYTHON_BIN%" "%VMWARE_CIS_HOME%\VMware Identity Services\lstool\scripts\lstool.py" list --url http://localhost:7080/lookupservice/sdk --type vcenterserver > c:\psc_services.txt

However there was no duplicate Service ID's in the output file, only one Service ID

[LokeshHK]

Then issue could be different., check the below thread where similar issue is resolved in VCSA and same steps could be applicable to Windows VC as well.

SOLVED: Could not connect to one or more vCenter Server systems:

To begin with check VPXD.log for similar errors.

Regards

Lokesh

[ryedaleit]

I'm new to Virtualisation, and we only have 4 Guest VM's, but it seems counterintuitive to me to put a VCenter Server Appliance on the same ESXi host that is also running the guest VM's? That's why I decided to use our existing 'Management Server' for VCenter Server...?

[daphnissov]

but it seems counterintuitive to me to put a VCenter Server Appliance on the same ESXi host that is also running the guest VM's?

It's not. This is extremely, extremely common.

That's why I decided to use our existing 'Management Server' for VCenter Server...?

Putting vCenter Server on a server which performs other functions is also a no-no. You're putting too many eggs in a basket and consequently dragging down performance of vCenter. Although you may be able to fix the problem, it won't correct upon your architecture which is what really needs to be changed here.

[sjesse]

We've been using the appliance for almost a year and we haven't had any problems. The problem with having vcenter on a shared windows server is that your virtual environment can be effected by those other services. For a small environment you only need 2 vcpu and 10gb of resources I think, if the esxi host this is on starts running into resource problems, you probably need another host anyway. A general practice if you have enough hosts is to make a managment cluster and a computer cluster, your general use vms would be on another host, and vcenter and any other management tools would be on the management cluster.

[dbnett]

I'll mirror what a few others have also said about not running vCenter on a shared Windows server. I know it doesn't resolve your issue, but a couple things I'll add are that running it on a Windows server is that the vCenter has to fight with the Windows OS for resources and you also have to contend with Windows updates, etc potentially breaking your vCenter installation. I've also been using the appliance for about a year now and it's come a long way. In your case I'd suggest using the Embedded Platform Service Controller deployment. The whole thing takes me about 45-60mins to deploy and configure in my test lab.

[ryedaleit]

I take on board all comment re:VCenter Appliance. However I'd like to get the Server in a working state before I attempt migrating to the Appliance.

I've seen in the postgresql log the following, which correlates with the time of our unexpected shut down

2018-10-09 11:14:54.445 BST 5bbc7f9e.1b20 0   LOG:  database system was interrupted; last known up at 2018-10-09 11:09:43 BST

2018-10-09 11:21:29.288 BST 5bbc7f86.e5c 0   LOG:  received fast shutdown request

2018-10-09 11:27:21.231 BST 5bbc7f9e.1b20 0   LOG:  database system was not properly shut down; automatic recovery in progress

2018-10-09 11:27:21.545 BST 5bbc7f9e.1b20 0   LOG:  redo starts at 5/515B6460

2018-10-09 13:02:55.439 BST 5bbc98ef.34c 0   LOG:  database system was interrupted while in recovery at 2018-10-09 11:27:21 BST

2018-10-09 13:02:55.439 BST 5bbc98ef.34c 0   HINT:  This probably means that some data is corrupted and you will have to use the last backup for recovery.

2018-10-09 13:09:38.655 BST 5bbc98e0.101c 0   LOG:  received fast shutdown request

2018-10-09 13:16:15.399 BST 5bbc98ef.34c 0   LOG:  database system was not properly shut down; automatic recovery in progress

2018-10-09 13:16:15.565 BST 5bbc98ef.34c 0   LOG:  redo starts at 5/515B6460

2018-10-09 14:13:33.563 BST 5bbca97d.186c 0   LOG:  database system was interrupted while in recovery at 2018-10-09 13:16:15 BST

2018-10-09 14:13:33.563 BST 5bbca97d.186c 0   HINT:  This probably means that some data is corrupted and you will have to use the last backup for recovery.

2018-10-09 14:20:30.418 BST 5bbca97a.2d68 0   LOG:  received fast shutdown request

2018-10-09 14:31:29.004 BST 5bbca97d.186c 0   LOG:  database system was not properly shut down; automatic recovery in progress

2018-10-09 14:31:29.112 BST 5bbca97d.186c 0   LOG:  redo starts at 5/515B6460

Could this be a clue or red herring?

[ryedaleit]

… or should I just go for it and migrate in it's current state? I have the VCSA downloaded and ready to go.....

[daphnissov]

It's up to you. You don't need a functional source vCenter 6.5 on Windows to deploy the vCSA 6.5.

[dbnett]

Coincidence or not...I'm currently working with VMware on an open case related to replacing the Machine SSL cert on an embedded PSC vCenter and after it fails and rolls back the cert changes I know get the same issue "https://<myserver>:443/sdk" after logging into the vSphere client. Do you know if there's been any changes to SSL certificates on your system?

Whether or not you want to take the time to troubleshoot this is obviously your call. My advice would be to deploy the vCenter appliance and then join your hosts to that instance and you're back up and running probably faster than than the time it'll take you to fix the current problem.

[ryedaleit]

Do you know if there's been any changes to SSL certificates on your system

No changes, it never had a certificate in the first place.

I'm defo' going to give the Appliance a go.

[sk84 · Accepted solution]

I know this error when the vpxd service is not running. Have you checked if the vmware-vpxd service is running? See: https://kb.vmware.com/s/article/2109881

If it's not running, you could try starting it and check the vpxd.log to see if you can find a cause for the error. See: https://kb.vmware.com/s/article/2110014

[ryedaleit]

I can see a few Services which return "does not seem to be registered with vMon" - is that normal?

c:\Program Files\VMware\vCenter Server\bin>SERVICE-CONTROL --start --all

Perform start operation. vmon_profile=ALL, svc_names=None, include_coreossvcs=True, include_leafossvcs=True

2018-10-12T08:24:55.576Z   Service vmware-cis-config does not seem to be registered with vMon. If this is unexpected please make sure your service config is a valid json. Also check vmon logs for warnings.

2018-10-12T08:24:55.577Z   State for service vmware-cis-config: RUNNING

Successfully started service vmware-cis-config

2018-10-12T08:24:55.667Z   Service VMWareAfdService does not seem to be registered with vMon. If this is unexpected please make sure your service config is a valid json. Also check vmon logs for warnings.

2018-10-12T08:24:55.668Z   State for service VMWareAfdService: RUNNING

Successfully started service VMWareAfdService

2018-10-12T08:24:55.793Z   Service VMWareDirectoryService does not seem to be registered with vMon. If this is unexpected please make sure your service config is a valid json. Also check vmon logs for warnings.

2018-10-12T08:24:55.796Z   State for service VMWareDirectoryService: RUNNING

Successfully started service VMWareDirectoryService

2018-10-12T08:24:55.885Z   Service VMWareCertificateService does not seem to be registered with vMon. If this is unexpected please make sure your service config is a valid json. Also check vmon logs for warnings.

2018-10-12T08:24:55.889Z   State for service VMWareCertificateService: RUNNING

Successfully started service VMWareCertificateService

2018-10-12T08:24:55.986Z   Service VMwareIdentityMgmtService does not seem to be registered with vMon. If this is unexpected please make sure your service config is a valid json. Also check vmon logs for warnings.

2018-10-12T08:24:55.989Z   State for service VMwareIdentityMgmtService: RUNNING

Successfully started service VMwareIdentityMgmtService

2018-10-12T08:24:56.068Z   Service VMwareSTS does not seem to be registered with vMon. If this is unexpected please make sure your service config is a valid json. Also check vmon logs for warnings.

2018-10-12T08:24:56.071Z   State for service VMwareSTS: RUNNING

Successfully started service VMwareSTS

2018-10-12T08:24:56.143Z   Service VMwareDNSService does not seem to be registered with vMon. If this is unexpected please make sure your service config is a valid json. Also check vmon logs for warnings.

2018-10-12T08:24:56.146Z   State for service VMwareDNSService: RUNNING

Successfully started service VMwareDNSService

2018-10-12T08:24:56.226Z   Service vmware-psc-client does not seem to be registered with vMon. If this is unexpected please make sure your service config is a valid json. Also check vmon logs for warnings.

2018-10-12T08:24:56.230Z   State for service vmware-psc-client: RUNNING

Successfully started service vmware-psc-client

[ryedaleit]

**UPDATE**

As per previous post, it obviously hadn't finished doing what it was doing, and eventually got "Successfully Started vmon services. Profile ALL" and now my Vcenter Server is back up and running!

Thanks for all the help. I'm now going to try and install the Appliance (a nice little Friday job) :smileyconfused:

[abd12]

ja genau

ich habe dieses Command getippt und lauft wieder Danke

c:\Program Files\VMware\vCenter Server\bin>SERVICE-CONTROL --start --all

As per previous post, it obviously hadn't finished doing what it was doing, and eventually got "Successfully Started vmon services