VSphere Client can't connect to vCenter Server Appliance

[MarkHaney]

This is a really odd problem.  I inherited this vmware ESXi5.1 cluster when I was hired to manage the network of a small BioTech company.  Up until now the vCenter Server appliance has been fine.  Then last Friday morning I simply could not connect to the the appliance using the vSphere Client. I get 'An unknown connection error occurred.  (The request failed because of a connection error. (Unable to connect to the remote server))

This is what I /can/ do:

Connect to the host and view the console of the appliance.  I can also SSH into the appliance as well as get to the WebGUI of the appliance.

This is what I've done to diagnose the problem:

Restarted the appliance (multiple times). 

Tried to follow the vmware kb articles for diagnosing the problem, but I gotta say, the terminology used is beyond confusing.  I'm still trying to figure out if I have VIMA or VMA.  Regardless, all the documentation centers on the host and I can see the host, can get to it by SSH and telnet.  I've also restarted the management network on the host.

I've not made any changes to the appliance or the host.  I was planning on installing vCLI on the appliance and needed to upload the SLES 11 SP2 SDKS to it so I could install the pre-requisite devel rpms.  I encountered a bit of an issue getting the ISOs uploaded via WinSCP, but thought that might have been the fact that my laptop suspended after being idle for so long (I started the copy late and just left it overnight since the SDK ISOs are 9+GB total.) I've got plenty of drive space on the appliance but deleted the ISOs anyway. 

What else can I try to get this fixed? Without the vSphere client I can't manage my cluster effectively.

[mrlesmithjr]

@MarkHaney It sounds like your vCenter service is not running on the appliance from the description here. Can you verify by logging into the WebGUI of the vcsa? https://vcsa:5480

Also did IP addresses or anything change? May need to regenerate SSL certs on the appliance.

[MarkHaney]

Well, it didn't occur to me until you mentioned regenerating the SSL certs.  The only change I've made was to add that host to DNS.  It's never been in DNS (and that was NOT my doing) and I added it since we've always attached to it via the IP address.  The problem is that I could still access the webgui without any problems.  I've pulled the DNS entry and I'm rebooting the appliance now to see if that fixes it.  If it does, I'll be the biggest moron on earth. 

For the record, how to I regenerate the certs if this works and I want to add it to DNS again?

I'm not a complete noob when it comes to vmware, but I'm still working on fixing all the problems on this cluster I inherited and it's tough getting up to speed on the latest vmware stuff and struggling to figure out the correct configuration on a system that was left totally devoid of documentation.  (FWIW, I was out of IT for 4 years and am just now getting back into it.

[admin]

Reason

This seems that the issue might be with the Vcenter connectivity with database.The vCenter Server Appliance has lost its ODBC configuration and is unable to start the embedded SQL database.   

Resolution

To resolve this issue, reconfigure the ODBC connection.

To reconfigure the ODBC connection:

1. Connect to the vCenter Server Appliance via SSH as the root user. The default password is  vmware.
   
2. Run this command to stop the vpxd service:
   
   service vmware-vpxd stop
   
3. Run this command at ssh Login prompt of ESX host.
   
   /usr/sbin/vpxd_odbcconfig
   
   You see output similar to:
   
   VMware vCenter Server Appliance ODBC Configuration
   
   Choose a database to use with VMware vCenter Server
   
   1) oracle
   2) embedded
   
4. Enter 2.
   
   You see output similar to:
   
   Configuring ODBC for embedded
   Operation was successful.
   Database already populated with version VirtualCenter Database 5.0
   Reinitialize database? (WARNING: WILL DELETE ALL RECORDS) [y(es)|N(O)]: OK, preserving the database. (select No to preserve the database)
   Restart vCenter Server now to activate the database changes [y/N]:y
   Successfully restarted vCenter Server

[MarkHaney]

No, deleting the DNS entry and restarting the appliance didn't fix it.  I can still get to the GUI at http://<host>:5480 just fine.  That wasn't even a problem before I deleted the DNS entry.

One other thing I did before the problem cropped up was try to upgrade the appliance:

Vendor:	VMware Inc.
Appliance Name:	VMware vCenter Server Appliance
Appliance Version:	5.1.0.10100 Build 1123965 ( Details... )
Available Updates
Appliance Version:	5.1.0.10300 Build 1364079 ( Details... )

Last Check:	Friday, December 6, 2013 10:00:04 PM GMT-05:00
Last Install:	"Failed to install updates(Error while running installation tests) on Thursday, December 5, 2013 10:05:33 AM GMT-05:00

This has been a problem for a while, and I've tried installing the update before and it's never caused problems even though it's failed.I don't see how that could be the cause, but you never know.

[mrlesmithjr]

You can check out this link on regenerating the SSL certs if you want to try that.

http://everythingshouldbevirtual.com/deploying-vmware-vcenter-server-appliance-5-1

[MarkHaney]

I sure would love to do that, however the host that the vCenterServer appliance is sitting on does not have the 'vpxd_odbcconfig' command. But, the server appliance has that command.  Is that what you meant?  See, this is my problem with the documentation and terminology.  To me 'ESX host or host system' means the hardware server that my VMs run on.  I have written plenty of documentation and the Vmware docs are just horrid.  The terminology is beyond confusing even for someone as experienced (in IT anyway) as I am.  I'm simply amazed any problems get figured out using the VMware KB.

[mrlesmithjr]

Yes run that on the vCenter Appliance. @albertolo may be onto your issue here.

[MarkHaney]

VMware vCenter Server Appliance ODBC Configuration

Choose a database to use with VMware vCenter Server

1) oracle

2) embedded

#? 2

Configuring ODBC for embedded

Error: Incompatible DB schema version.

Database  configuration has failed.

Would you like to try again or modify the settings manually?

[A]gain/[m]anually :

Hmm, This is interesting.  What do I now?

[mrlesmithjr]

So interestingly enough I ran into a similar issue last night with my VCSA 5.5 giving similar issues. My issue ended up to be that I needed to reauthenticate my VCSA to Active Directory. Are you using AD authentication on your appliance?

[MarkHaney]

Well, yes and no.  I've tried to get AD authentication working and have set it up in the WebUI, but it's never allowed me to authenticate using the vSphere Client to the VCSA using my AD credentials.  I've determined that vCLI isn't on the VCSA and based on what I read on the KB I need that to actually join the VCSA to an AD domain.  Granted I could be sorely mistaken by that, but I do know that the KB instructs you to use some command (which escapes me ATM) from the vi-admin user with the --add-domain switch.  The problem with that is that there is no 'vi-admin' user on the VCSA. 

Disclaimer:  This setup is not my doing.  This cluster was setup by a third party IT shop a year before I was hired.  I've managed to correct some significant deficiencies in the setup of the cluster, but by no means have I corrected (or even found) them all.  I would have expected the vi-admin user to be in the VCSA by default, but it's not in this case.

Does that help at all?

[MarkHaney]

Now, this is interesting, it /does/ appear as if I have my VCSA sitting with a mix of 5.1.0a and 5.1.0c rpms installed.  Here's what I'm seeing on the system:

This is the 5.1.0c update ISO mounted on the VCSA:

vcenter1-srv:/mnt/cdrom/update/package-pool # ls VMware-vPostgres*

VMware-vPostgres-1.0.1.1211-541401.x86_64.rpm

VMware-vPostgres-jdbc-5.1.0-820920.x86_64.rpm

VMware-vPostgres-client-1.0.1.1211-541401.x86_64.rpm

VMware-vPostgres-server-extensions-1.0.1.1211-541401.x86_64.rpm

And THIS is what RPM sees installed on the server:

vcenter1-srv:~ # rpm -qa | grep VMware-vPostgres

VMware-vPostgres-client-1.0.1.1211-541401

VMware-vPostgres-jdbc-5.1.0-820920

VMware-vPostgres-1.0.1.1211-541401

VMware-vPostgres-server-extensions-1.0.1.1211-541401

So, it looks like I have a system sitting between 5.1.0a and 5.1.0c:

Appliance Name:	VMware vCenter Server Appliance
Appliance Version:	5.1.0.10100 Build 1123965 ( Details... )
VMware vCenter Server Appliance Update to version 5.1.0 Update 1a
Available Updates
Appliance Version:	5.1.0.10300 Build 1364079 ( Details... )
VMware vCenter Server Appliance Update to version 5.1.0 Update 1c

Question is, how do I fix it?

[Anthony_D]

Have exact same problem only from vCenter Appliance version 5.0.0.3324 Build 472350 update to 5.0.0.8512 Build 1302764 (Error while running post install scripts).

Attempted ODBC configuration for Embedded with same error cannot connect to DB.

Did you end up resolving the issue?

I'm starting to think it will be easier removing vm's from inventory installing new version of vCenter Applicance and reconnecting hosts to be managed by new server. I didn't have any special setup in vCenter so it doesn't hold any special configuration information that needs to be restored.

Let me know how you resolved your problem much appreciated.

[MarkHaney]

Actually, yes I did.  But the solution I found may or may not be helpful to you since my problem was with 5.1, not 5.0.  This is the link I found that helped me fix mine: http://www.vaspects.com/2013/05/07/workaround-for-vcenter-appliance-update-delay/

It gives you a method of updating the appliance from the command line, but it's not for the faint of heart.  Even then, I'm not convinced the appliance is completely normalized since I'm having an issue with VMs appearing on datastores even though the VMs are no longer ON that datastore. (See: iSCSI volumes and Virtual Machines).  I can't say for certain that my current problem is related to the failed update problem I posted about here, but it does seem an odd coincidence. 

The Appliance is actually stable and has given me no trouble at all since the forced update, all version numbers match, etc, and it's been rock solid with all the moves and migrations and such that I've done in the last 2 weeks or so. 

I hope the link above helps.