darkdragon001
Enthusiast
Enthusiast

vCenter 6 fails to restart after IP change

Jump to solution

I changed the vCenter 6 IP from within web client.

Since I had problems connecting to the new IP, I restarted the server via vSphere Client directly on the host using VMware Tools.

After the restart, I can't connect to vCenter any more. Trying to connect via https shows me the following error:

503 Service Unavailable (Failed to connect to endpoint: [N7Vmacore4Http20NamedPipeServiceSpecE:0x7f4158068bf0] _serverNamespace = / _isRedirect = false _pipeName =/var/run/vmware/vpxd-webserver-pipe)

Then I directly used the console, enabled ssh and bash and logged in there.

I tried to check the current IP settings via the script used in version 5.5 which still exists (/opt/vmware/share/vami/vami_config_net) but fails with multiple errors.

Then, I tried to reset the certificates via certificate manager (/usr/lib/vmware-vmca/bin/certificate-manager), but I could not login via SSO (seems that SSO service could also not be started).

Then, I had a look in the log files, and noticed a problem with vmware-vpxd:

VC SSL certificate does not exist, it will be generated by vpxd

This did not help either. I tried to manually start the service, with the same result.

How can I fix my vCenter?

1 Solution

Accepted Solutions
Talon256
Contributor
Contributor

I got this error after changing the vCenter's IP address

  1. I used #6 https://rlevchenko.com/2016/03/24/vcenter-503-service-unavailable/ to enable the shell.
  2. ssh into the vcenter server
  3. Command> shell
  4. vi /etc/hosts changed the IP there
  5. service-control --stop --all
  6. service-control --start --all

That got me fixed

View solution in original post

30 Replies
NutanixCanuck
Contributor
Contributor

Same thing happened to me today

0 Kudos
NutanixCanuck
Contributor
Contributor

I was able to start vmware-vxpd from the CLI but still no luck

emon:                                                                                                                                               unused

Checking for service rpcbind                                                                                                                                                       running

                                                                                                                                                                                   unknown

Checking for service sendmail:                                                                                                                                                     unused

/dev/ttyS0 at 0x03f8 (irq = 4) is a 16550A

/dev/ttyS1 at 0x02f8 (irq = 3) is a 16550A                                                                                                                                         running

/usr/sbin/FOO not installed

/bin/snmpd is not running

                                                                                                                                                                                   unknown

                                                                                                                                                                                   unknown

Checking for service sshd                                                                                                                                                          running

Checking for stunnel (SSL tunnel):                                                                                                                                                 unused

Checking for service syslog:                                                                                                                                                       running

Checking for service uuidd                                                                                                                                                         unused

running (standalone: 5311)

running (standalone: 5367)

Warning: vmci status: unimplemented

running (standalone: 5417)

VMware CIS License Service is running: PID:10693, Wrapper:STARTED, Java:STARTED

VMware Component Manager is running: PID:10134, Wrapper:STARTED, Java:STARTED

VMware ESX Agent Manager is not running.

VMware Inventory Service is not running.

VMware Message Bus Config Service is not running.

VMware ESXi dump collector is not running

VMware Performance Charts Service is not running.

Checking for /usr/bin/rbd-watchdog-linux:                                                                                                                                          unused

VMware HTTP Reverse Proxy is running.

VMware Service Control Agent is running: PID:10937, Wrapper:STARTED, Java:STARTED

VMware vSphere Profile-Driven Storage Service is running: PID:18707, Wrapper:STARTED, Java:STARTED

ensure environment variables are set

Checking for VMware STS IDM Server ...                                                                                                                                             running

ensure environment variables are set

Checking for service vmware-stsd                                                                                                                                                   running

syslog is running, PID: 11999

VMware Common Logging Service is running: PID:11339, Wrapper:STARTED, Java:STARTED

vmtoolsd is running

VGAuth daemon .

vAPI Endpoint is running: PID:11563, Wrapper:STARTED, Java:STARTED

VMware Content Library Service is running: PID:19194, Wrapper:STARTED, Java:STARTED

Last login: Wed May 13 12:29:51 UTC 2015 on pts/0

pg_ctl: server is running (PID: 12178)

/opt/vmware/vpostgres/9.3/bin/postgres "-D" "/storage/db/vpostgres"

VMware vCenter workflow manager is not running.

vmware-vpxd is running

VMware vService Manager is running: PID:27602, Wrapper:STARTED, Java:STARTED

/usr/java/jre-vmware/bin/vmware-vws is running.

Warning: vsock status: unimplemented

VMware vSphere Web Client is running: PID:27919, Wrapper:STARTED, Java:STARTED

Checking for service xinetd:                                                                                                                                                       unused

Checking for ypbind:                                     

0 Kudos
hullt
Enthusiast
Enthusiast

Did you happen to try option 4?  Regenerate a new VMCA Root Cert?

0 Kudos
rcstevensonaz
Contributor
Contributor

As a reference to others, Tim's comment refers to Option 4 when you run /usr/lib/vmware-vmca/bin/certificate-manager

But, it is asking for a valid SSO password to perform certificate operations.  Can anyone explain what that is (it is not the root password).

0 Kudos
npadmani
Virtuoso
Virtuoso

valid SSO administrator password is the password associated with account administrator@<sso domain>.

Note: Default <sso dmain> is vsphere.local, but since you are dealing with vcsa6, at the time of deployment that name can be customized

Narendra Padmani VCIX6-DCV | VCIX7-CMA | VCI | TOGAF 9 Certified
rcstevensonaz
Contributor
Contributor

Thanks for clarifying.  In my case, the root and the administrator@vsphere.local are the same password.

Is there any issue with special characters (e.g., "(", ")", and "^")?

0 Kudos
darkdragon001
Enthusiast
Enthusiast

I only noticed issues with passwords more than 20 characters length.

0 Kudos
tomsjo
Contributor
Contributor

I too have the same problem.

Been trying to figure this one out for a few days. Even if i regenerate the certs in the psc (i use external psc) it will not play ball, still hangs on starting vpxd, then reboots.

These are the last lines in my vpxd.log

2015-05-28T14:55:30.746Z error vpxd[7F37346B97A0] [Originator@6876 sub=componentManagerUtil] AcquireToken for solution (vpxd-e0ba3305-3310-4e62-9e36-0339ae5c0291@vsphere.local): SsoException: Unexpected SOAP fault: ns0:MessageExpired; request failed.

2015-05-28T14:55:30.746Z error vpxd[7F37346B97A0] [Originator@6876 sub=licenseServiceDiscovery] Failed to discover license service endpoint: N9SsoClient12SsoExceptionE(Unexpected SOAP fault: ns0:MessageExpired; request failed.)

2015-05-28T14:55:30.752Z warning vpxd[7F37346B97A0] [Originator@6876 sub=VpxProfiler] ServerApp::Init [TotalTime] took 6690 ms

2015-05-28T14:55:30.753Z error vpxd[7F37346B97A0] [Originator@6876 sub=Main] [VpxdMain] Failed to initialize N9SsoClient12SsoExceptionE(Unexpected SOAP fault: ns0:MessageExpired; request failed.)

2015-05-28T14:55:30.753Z error vpxd[7F37346B97A0] [Originator@6876 sub=Default] Failed to intialize VMware VirtualCenter. Shutting down...

Any ideas?

0 Kudos
admin
Immortal
Immortal

darkdragon001‌, NutanixCanuck‌, tomsjo

I ran into this issue as well, but I was able to find the answer in the documentation. vCenter 6.0 doesn't support changing the 'FQDN' setting. When you first setup the VM\Appliance it suggests to use a FQDN, but a static IP address is an okay alternative. Long story short, if you use a FQDN, then you can change the IP or use DHCP, but you cannot change the FQDN. If you used an IP address, then you cannot change the IP address nor change the FQDN setting. (If you use DHCP, then you need to make certain that the assignment is static.)

This shortcoming is best documented in the 'Installing vCenter Server on a Windows Virtual Machine or Physical Server' section of the documentation.

http://pubs.vmware.com/vsphere-60/index.jsp#com.vmware.vsphere.install.doc/GUID-A9CDCCC4-A9E9-44B0-A...

Important

The name that you type is encoded in the SSL certificate of the system. The components communicate with each other by using this name. The system name must be either a static IP address or a fully qualified domain name (FQDN). Make sure that the system name does not change. You cannot change the system name after the installation completes.

And, there are subtle warnings in the 'Deploying the vCenter Server Appliance' section.

http://pubs.vmware.com/vsphere-60/index.jsp#com.vmware.vsphere.install.doc/GUID-86FD9381-88B0-40F1-B...

http://pubs.vmware.com/vsphere-60/index.jsp#com.vmware.vsphere.install.doc/GUID-E278E0F2-8983-4246-9...

On the Network Settings page, set up the network settings.

The IP address or the FQDN of the appliance is used as a system name. It is recommended to use an FQDN. However, if you want to use an IP address, use static IP address allocation for the appliance, because IP addresses allocated by DHCP might change.

...

If you use an IP address as a system name, you cannot change the IP address and update the DNS settings after deployment.

The issue with the script (/usr/lib/vmware-vmca/bin/certificate-manager) is not working is because the FQDN\system name has changed, so it cannot find the SSO server.

I just redeployed the VC, but I would imagine that you may be able to recover by going back to the original FQDN, or IP. Maybe reverting a snapshot would work too.

darkdragon001
Enthusiast
Enthusiast

Thanks welchg for the valuable hints to the documentation.

Maybe my problems is a combination of multiple issues: I used a hostname as FQDN, submitted my own 'external' certificates and then changed the IP address.

0 Kudos
gray13
Contributor
Contributor

Hi,

I just met the exactly same issue...

my solution is simple...

i restart the AD server.

actually i shutdown the AD to check whether i can use the standalone VCSA6.. but the applicance is running fine but when i login to the webclient hit the same error.

So u can try to reconfigure your AD and try to re-add your VCSA6 item in the AD...

btw.. anyone tried RHEL7 LDAP to replace windows AD? can give some advice?

this AD eat a lot of memory and not that useful if i dont install mssql... but can not shut down at all...

hope it helps ..

0 Kudos
malabelle
Enthusiast
Enthusiast

Hi, I have this problem and when I try to Option 7 or 4 (revert or regenerate) it asks for the password...

But it does not recognize the password.. Is there a default one?

Any hint?

vExpert '16, VCAP-DCA, VCAP-DCD
0 Kudos
malabelle
Enthusiast
Enthusiast

2015-09-10T18:42:07.590Z info vpxd[7F767C1D07A0] [Originator@6876 sub=[SSO][SsoCertificateManagerImpl]] [CreateServiceContent] Try to connect to SSO VMOMI endpoint.

2015-09-10T18:42:07.591Z warning vpxd[7F766BBA8700] [Originator@6876 sub=Default] Failed to resolve address; <resolver p:0x00007f76646d9130, 'vcenterserver-fqdn:443'>, e: asio.netdb:1(Host not found (authoritative))

2015-09-10T18:42:07.591Z error vpxd[7F766BBA8700] [Originator@6876 sub=HttpConnectionPool-000001] [ConnectComplete] Connect failed to <cs p:00007f76646dbb10, TCP:vcenterserver-fqdn:443>; cnx: (null), error: N7Vmacore15SystemExceptionE(Host not found (authoritative))

2015-09-10T18:42:07.591Z warning vpxd[7F767C1D07A0] [Originator@6876 sub=[SSO][SsoCertificateManagerImpl]] [RetryOnConnectionFailure] SystemException while trying to connect to SSO Admin server: N7Vmacore15SystemExceptionE(Host not found (authoritative))

2015-09-10T18:42:07.591Z info vpxd[7F767C1D07A0] [Originator@6876 sub=[SSO][SsoCertificateManagerImpl]] [CreateServiceContent] Will attempt to connect again in 10 seconds.

2015-09-10T18:42:17.591Z info vpxd[7F767C1D07A0] [Originator@6876 sub=[SSO][SsoCertificateManagerImpl]] [CreateServiceContent] Try to connect to SSO VMOMI endpoint.

2015-09-10T18:42:17.592Z warning vpxd[7F766BB27700] [Originator@6876 sub=Default] Failed to resolve address; <resolver p:0x00007f76646d9130, 'vcenterserver-fqdn:443'>, e: asio.netdb:1(Host not found (authoritative))

2015-09-10T18:42:17.592Z error vpxd[7F766BB27700] [Originator@6876 sub=HttpConnectionPool-000001] [ConnectComplete] Connect failed to <cs p:00007f76646dbb10, TCP:vcenterserver-fqdn:443>; cnx: (null), error: N7Vmacore15SystemExceptionE(Host not found (authoritative))

2015-09-10T18:42:17.592Z warning vpxd[7F767C1D07A0] [Originator@6876 sub=[SSO][SsoCertificateManagerImpl]] [RetryOnConnectionFailure] SystemException while trying to connect to SSO Admin server: N7Vmacore15SystemExceptionE(Host not found (authoritative))

2015-09-10T18:42:17.592Z error vpxd[7F767C1D07A0] [Originator@6876 sub=[SSO][SsoCertificateManagerImpl]] [RetryOnConnectionFailure] Max attempts (10) reached. Giving up ...

2015-09-10T18:42:17.592Z error vpxd[7F767C1D07A0] [Originator@6876 sub=[SSO][SsoFactory_CreateFacade]] Unable to create SSO facade: N7Vmacore15SystemExceptionE(Host not found (authoritative)).

2015-09-10T18:42:17.592Z warning vpxd[7F767C1D07A0] [Originator@6876 sub=VpxProfiler] Init [Vpx::Common::Sso::SsoFactory_CreateFacade(sslContext, ssoFacadeConstPtr)] took 90015 ms

2015-09-10T18:42:17.592Z error vpxd[7F767C1D07A0] [Originator@6876 sub=Main] [Init] Init failed: Vpx::Common::Sso::SsoFactory_CreateFacade(sslContext, ssoFacadeConstPtr)

--> Backtrace:

-->

--> [backtrace begin] product: VMware VirtualCenter, version: 6.0.0, build: build-2776510, tag: vpxd

--> backtrace[00] libvmacore.so[0x003C00C4]: Vmacore::System::Stacktrace::CaptureWork(unsigned int)

--> backtrace[01] libvmacore.so[0x001EDBD3]: Vmacore::System::SystemFactoryImpl::CreateQuickBacktrace(Vmacore::Ref<Vmacore::System::Backtrace>&)

--> backtrace[02] libvmacore.so[0x001988DD]: Vmacore::Throwable::Throwable(std::string const&)

--> backtrace[03] vpxd[0x00BD075E]: Vmomi::Fault::SystemError::Exception::Exception(std::string const&)

vExpert '16, VCAP-DCA, VCAP-DCD
0 Kudos
malabelle
Enthusiast
Enthusiast

i tried

service-control --start --all

Stdout = Starting VMware Inventory Service...

Waiting for VMware Inventory Service...................................................................................................

WARNING: VMware Inventory Service may have failed to start.

Stderr =

2015-09-10T19:33:24.061Z   {

    "resolution": null,

    "detail": [

        {

            "args": [

                "Command: ['/sbin/service', u'vmware-invsvc', 'start']\nStderr: "

            ],

            "id": "install.ciscommon.command.errinvoke",

            "localized": "An error occurred while invoking external command : 'Command: ['/sbin/service', u'vmware-invsvc', 'start']\nStderr: '",

            "translatable": "An error occurred while invoking external command : '%(0)s'"

        }

    ],

    "componentKey": null,

    "problemId": null

}

ERROR:root:Unable to start service vmware-invsvc, Exception: {

    "resolution": null,

    "detail": [

        {

            "args": [

                "vmware-invsvc"

            ],

            "id": "install.ciscommon.service.failstart",

            "localized": "An error occurred while starting service 'vmware-invsvc'",

            "translatable": "An error occurred while starting service '%(0)s'"

        }

    ],

    "componentKey": null,

    "problemId": null

}

Unable to start service vmware-invsvc, Exception: {

    "resolution": null,

    "detail": [

        {

            "args": [

                "vmware-invsvc"

            ],

            "id": "install.ciscommon.service.failstart",

            "localized": "An error occurred while starting service 'vmware-invsvc'",

            "translatable": "An error occurred while starting service '%(0)s'"

        }

    ],

    "componentKey": null,

    "problemId": null

}

vExpert '16, VCAP-DCA, VCAP-DCD
0 Kudos
kegman21
Contributor
Contributor

There doesn't seem to be many sources at this point with the 'failed to connect to endpoint' error and while my issue was not caused by changing an IP I received the same error when a common time could not be found for multiple components (AD, PSC, VC) within my lab. 

I recently moved the domain's PDC emulator to a new server that turned out to have  inaccurate time which eventually spread throughout the domain... once I corrected the time, rebooting the VCSA VC corrected my 'failed to connect to endpoint' error (VCSA PSC was on a separate VM which started successfully without common time). 

Definitely not suggesting it will solve everyone's problem, but wanted to share as it could be worth validating that time is common for the AD domain, PSC, and VC when getting the below error.

0 Kudos
PerAtOdenSoluti
Contributor
Contributor

Hi,

I have the exact same issue as you had. Vxpd is startable from cli but inventory is not...

Did you solve your issue and if so, how?

Crossing my fingers I found someone with answers...

/Per

0 Kudos
mohdhanifk
Enthusiast
Enthusiast

Hi,

Even we faced similar issue and had to log case with support team.

0 Kudos
welnet
Contributor
Contributor

I ran into this issue today when changing the ip-address of the vcenter appliance. It seems the old ip address is not changed in the hosts file on the appliance. to correct this login to appliance management on port 5480 and enable ssh login & bash access. login with ssh and drop into the shell, edit the /etc/hosts file with vi and change the ip-address. After a reboot of the VM and restarting all services in the shell everything was working again.

beth22
Enthusiast
Enthusiast

Please check if Platform Service Controller services are started up and try to start vCenter Server services again?

And which vCenter Server deployment has been deployed? Embedded or External PSC?

0 Kudos