1 2 3 Previous Next 30 Replies Latest reply on Jun 13, 2017 7:44 AM by ShDog

    vCenter 6 fails to restart after IP change

    darkdragon001 Enthusiast

      I changed the vCenter 6 IP from within web client.

      Since I had problems connecting to the new IP, I restarted the server via vSphere Client directly on the host using VMware Tools.

      After the restart, I can't connect to vCenter any more. Trying to connect via https shows me the following error:

      503 Service Unavailable (Failed to connect to endpoint: [N7Vmacore4Http20NamedPipeServiceSpecE:0x7f4158068bf0] _serverNamespace = / _isRedirect = false _pipeName =/var/run/vmware/vpxd-webserver-pipe)

      Then I directly used the console, enabled ssh and bash and logged in there.

      I tried to check the current IP settings via the script used in version 5.5 which still exists (/opt/vmware/share/vami/vami_config_net) but fails with multiple errors.

      Then, I tried to reset the certificates via certificate manager (/usr/lib/vmware-vmca/bin/certificate-manager), but I could not login via SSO (seems that SSO service could also not be started).

      Then, I had a look in the log files, and noticed a problem with vmware-vpxd:

      VC SSL certificate does not exist, it will be generated by vpxd

      This did not help either. I tried to manually start the service, with the same result.

       

      How can I fix my vCenter?

        • 1. Re: vCenter 6 fails to restart after IP change
          NutanixCanuck Lurker

          Same thing happened to me today

          • 2. Re: vCenter 6 fails to restart after IP change
            NutanixCanuck Lurker

            I was able to start vmware-vxpd from the CLI but still no luck

             

            emon:                                                                                                                                               unused

            Checking for service rpcbind                                                                                                                                                       running

                                                                                                                                                                                               unknown

            Checking for service sendmail:                                                                                                                                                     unused

            /dev/ttyS0 at 0x03f8 (irq = 4) is a 16550A

            /dev/ttyS1 at 0x02f8 (irq = 3) is a 16550A                                                                                                                                         running

            /usr/sbin/FOO not installed

            /bin/snmpd is not running

                                                                                                                                                                                               unknown

                                                                                                                                                                                               unknown

            Checking for service sshd                                                                                                                                                          running

            Checking for stunnel (SSL tunnel):                                                                                                                                                 unused

            Checking for service syslog:                                                                                                                                                       running

            Checking for service uuidd                                                                                                                                                         unused

             

             

            running (standalone: 5311)

            running (standalone: 5367)

            Warning: vmci status: unimplemented

            running (standalone: 5417)

            VMware CIS License Service is running: PID:10693, Wrapper:STARTED, Java:STARTED

            VMware Component Manager is running: PID:10134, Wrapper:STARTED, Java:STARTED

            VMware ESX Agent Manager is not running.

            VMware Inventory Service is not running.

            VMware Message Bus Config Service is not running.

            VMware ESXi dump collector is not running

            VMware Performance Charts Service is not running.

            Checking for /usr/bin/rbd-watchdog-linux:                                                                                                                                          unused

            VMware HTTP Reverse Proxy is running.

            VMware Service Control Agent is running: PID:10937, Wrapper:STARTED, Java:STARTED

            VMware vSphere Profile-Driven Storage Service is running: PID:18707, Wrapper:STARTED, Java:STARTED

            ensure environment variables are set

            Checking for VMware STS IDM Server ...                                                                                                                                             running

            ensure environment variables are set

            Checking for service vmware-stsd                                                                                                                                                   running

            syslog is running, PID: 11999

            VMware Common Logging Service is running: PID:11339, Wrapper:STARTED, Java:STARTED

            vmtoolsd is running

            VGAuth daemon .

            vAPI Endpoint is running: PID:11563, Wrapper:STARTED, Java:STARTED

            VMware Content Library Service is running: PID:19194, Wrapper:STARTED, Java:STARTED

            Last login: Wed May 13 12:29:51 UTC 2015 on pts/0

            pg_ctl: server is running (PID: 12178)

            /opt/vmware/vpostgres/9.3/bin/postgres "-D" "/storage/db/vpostgres"

            VMware vCenter workflow manager is not running.

            vmware-vpxd is running

            VMware vService Manager is running: PID:27602, Wrapper:STARTED, Java:STARTED

            /usr/java/jre-vmware/bin/vmware-vws is running.

            Warning: vsock status: unimplemented

            VMware vSphere Web Client is running: PID:27919, Wrapper:STARTED, Java:STARTED

            Checking for service xinetd:                                                                                                                                                       unused

            Checking for ypbind:                                     

            • 3. Re: vCenter 6 fails to restart after IP change
              hullt Enthusiast

              Did you happen to try option 4?  Regenerate a new VMCA Root Cert?

              • 4. Re: vCenter 6 fails to restart after IP change
                rcstevensonaz Lurker

                As a reference to others, Tim's comment refers to Option 4 when you run /usr/lib/vmware-vmca/bin/certificate-manager

                 

                But, it is asking for a valid SSO password to perform certificate operations.  Can anyone explain what that is (it is not the root password).

                • 5. Re: vCenter 6 fails to restart after IP change
                  npadmani Master
                  VMware EmployeesvExpert

                  valid SSO administrator password is the password associated with account administrator@<sso domain>.

                  Note: Default <sso dmain> is vsphere.local, but since you are dealing with vcsa6, at the time of deployment that name can be customized

                  • 6. Re: vCenter 6 fails to restart after IP change
                    rcstevensonaz Lurker

                    Thanks for clarifying.  In my case, the root and the administrator@vsphere.local are the same password.

                     

                    Is there any issue with special characters (e.g., "(", ")", and "^")?

                    • 7. Re: vCenter 6 fails to restart after IP change
                      darkdragon001 Enthusiast

                      I only noticed issues with passwords more than 20 characters length.

                      • 8. Re: vCenter 6 fails to restart after IP change
                        tomsjo Lurker

                        I too have the same problem.

                         

                        Been trying to figure this one out for a few days. Even if i regenerate the certs in the psc (i use external psc) it will not play ball, still hangs on starting vpxd, then reboots.

                        These are the last lines in my vpxd.log

                         

                        2015-05-28T14:55:30.746Z error vpxd[7F37346B97A0] [Originator@6876 sub=componentManagerUtil] AcquireToken for solution (vpxd-e0ba3305-3310-4e62-9e36-0339ae5c0291@vsphere.local): SsoException: Unexpected SOAP fault: ns0:MessageExpired; request failed.

                        2015-05-28T14:55:30.746Z error vpxd[7F37346B97A0] [Originator@6876 sub=licenseServiceDiscovery] Failed to discover license service endpoint: N9SsoClient12SsoExceptionE(Unexpected SOAP fault: ns0:MessageExpired; request failed.)

                        2015-05-28T14:55:30.752Z warning vpxd[7F37346B97A0] [Originator@6876 sub=VpxProfiler] ServerApp::Init [TotalTime] took 6690 ms

                        2015-05-28T14:55:30.753Z error vpxd[7F37346B97A0] [Originator@6876 sub=Main] [VpxdMain] Failed to initialize N9SsoClient12SsoExceptionE(Unexpected SOAP fault: ns0:MessageExpired; request failed.)

                        2015-05-28T14:55:30.753Z error vpxd[7F37346B97A0] [Originator@6876 sub=Default] Failed to intialize VMware VirtualCenter. Shutting down...

                         

                        Any ideas?

                        • 9. Re: vCenter 6 fails to restart after IP change
                          Novice

                          darkdragon001, NutanixCanuck, tomsjo

                           

                          I ran into this issue as well, but I was able to find the answer in the documentation. vCenter 6.0 doesn't support changing the 'FQDN' setting. When you first setup the VM\Appliance it suggests to use a FQDN, but a static IP address is an okay alternative. Long story short, if you use a FQDN, then you can change the IP or use DHCP, but you cannot change the FQDN. If you used an IP address, then you cannot change the IP address nor change the FQDN setting. (If you use DHCP, then you need to make certain that the assignment is static.)

                           

                          This shortcoming is best documented in the 'Installing vCenter Server on a Windows Virtual Machine or Physical Server' section of the documentation.

                           

                          http://pubs.vmware.com/vsphere-60/index.jsp#com.vmware.vsphere.install.doc/GUID-A9CDCCC4-A9E9-44B0-ABED-4ECB83C432A0.html

                          Important

                          The name that you type is encoded in the SSL certificate of the system. The components communicate with each other by using this name. The system name must be either a static IP address or a fully qualified domain name (FQDN). Make sure that the system name does not change. You cannot change the system name after the installation completes.

                           

                          And, there are subtle warnings in the 'Deploying the vCenter Server Appliance' section.

                           

                          http://pubs.vmware.com/vsphere-60/index.jsp#com.vmware.vsphere.install.doc/GUID-86FD9381-88B0-40F1-B516-6CC782ED9A14.html

                          http://pubs.vmware.com/vsphere-60/index.jsp#com.vmware.vsphere.install.doc/GUID-E278E0F2-8983-4246-9903-50C4022F1273.html

                          On the Network Settings page, set up the network settings.

                          The IP address or the FQDN of the appliance is used as a system name. It is recommended to use an FQDN. However, if you want to use an IP address, use static IP address allocation for the appliance, because IP addresses allocated by DHCP might change.

                          ...

                          If you use an IP address as a system name, you cannot change the IP address and update the DNS settings after deployment.

                           

                           

                          The issue with the script (/usr/lib/vmware-vmca/bin/certificate-manager) is not working is because the FQDN\system name has changed, so it cannot find the SSO server.

                           

                          I just redeployed the VC, but I would imagine that you may be able to recover by going back to the original FQDN, or IP. Maybe reverting a snapshot would work too.

                          1 person found this helpful
                          • 10. Re: vCenter 6 fails to restart after IP change
                            darkdragon001 Enthusiast

                            Thanks welchg for the valuable hints to the documentation.

                             

                            Maybe my problems is a combination of multiple issues: I used a hostname as FQDN, submitted my own 'external' certificates and then changed the IP address.

                            • 11. Re: vCenter 6 fails to restart after IP change
                              gray13 Novice

                              Hi,

                              I just met the exactly same issue...

                              my solution is simple...

                              i restart the AD server.

                              actually i shutdown the AD to check whether i can use the standalone VCSA6.. but the applicance is running fine but when i login to the webclient hit the same error.

                              So u can try to reconfigure your AD and try to re-add your VCSA6 item in the AD...

                               

                              btw.. anyone tried RHEL7 LDAP to replace windows AD? can give some advice?

                              this AD eat a lot of memory and not that useful if i dont install mssql... but can not shut down at all...

                               

                              hope it helps ..

                              • 12. Re: vCenter 6 fails to restart after IP change
                                malabelle Enthusiast

                                Hi, I have this problem and when I try to Option 7 or 4 (revert or regenerate) it asks for the password...

                                 

                                But it does not recognize the password.. Is there a default one?

                                Any hint?

                                • 13. Re: vCenter 6 fails to restart after IP change
                                  malabelle Enthusiast

                                  2015-09-10T18:42:07.590Z info vpxd[7F767C1D07A0] [Originator@6876 sub=[SSO][SsoCertificateManagerImpl]] [CreateServiceContent] Try to connect to SSO VMOMI endpoint.

                                  2015-09-10T18:42:07.591Z warning vpxd[7F766BBA8700] [Originator@6876 sub=Default] Failed to resolve address; <resolver p:0x00007f76646d9130, 'vcenterserver-fqdn:443'>, e: asio.netdb:1(Host not found (authoritative))

                                  2015-09-10T18:42:07.591Z error vpxd[7F766BBA8700] [Originator@6876 sub=HttpConnectionPool-000001] [ConnectComplete] Connect failed to <cs p:00007f76646dbb10, TCP:vcenterserver-fqdn:443>; cnx: (null), error: N7Vmacore15SystemExceptionE(Host not found (authoritative))

                                  2015-09-10T18:42:07.591Z warning vpxd[7F767C1D07A0] [Originator@6876 sub=[SSO][SsoCertificateManagerImpl]] [RetryOnConnectionFailure] SystemException while trying to connect to SSO Admin server: N7Vmacore15SystemExceptionE(Host not found (authoritative))

                                  2015-09-10T18:42:07.591Z info vpxd[7F767C1D07A0] [Originator@6876 sub=[SSO][SsoCertificateManagerImpl]] [CreateServiceContent] Will attempt to connect again in 10 seconds.

                                  2015-09-10T18:42:17.591Z info vpxd[7F767C1D07A0] [Originator@6876 sub=[SSO][SsoCertificateManagerImpl]] [CreateServiceContent] Try to connect to SSO VMOMI endpoint.

                                  2015-09-10T18:42:17.592Z warning vpxd[7F766BB27700] [Originator@6876 sub=Default] Failed to resolve address; <resolver p:0x00007f76646d9130, 'vcenterserver-fqdn:443'>, e: asio.netdb:1(Host not found (authoritative))

                                  2015-09-10T18:42:17.592Z error vpxd[7F766BB27700] [Originator@6876 sub=HttpConnectionPool-000001] [ConnectComplete] Connect failed to <cs p:00007f76646dbb10, TCP:vcenterserver-fqdn:443>; cnx: (null), error: N7Vmacore15SystemExceptionE(Host not found (authoritative))

                                  2015-09-10T18:42:17.592Z warning vpxd[7F767C1D07A0] [Originator@6876 sub=[SSO][SsoCertificateManagerImpl]] [RetryOnConnectionFailure] SystemException while trying to connect to SSO Admin server: N7Vmacore15SystemExceptionE(Host not found (authoritative))

                                  2015-09-10T18:42:17.592Z error vpxd[7F767C1D07A0] [Originator@6876 sub=[SSO][SsoCertificateManagerImpl]] [RetryOnConnectionFailure] Max attempts (10) reached. Giving up ...

                                  2015-09-10T18:42:17.592Z error vpxd[7F767C1D07A0] [Originator@6876 sub=[SSO][SsoFactory_CreateFacade]] Unable to create SSO facade: N7Vmacore15SystemExceptionE(Host not found (authoritative)).

                                  2015-09-10T18:42:17.592Z warning vpxd[7F767C1D07A0] [Originator@6876 sub=VpxProfiler] Init [Vpx::Common::Sso::SsoFactory_CreateFacade(sslContext, ssoFacadeConstPtr)] took 90015 ms

                                  2015-09-10T18:42:17.592Z error vpxd[7F767C1D07A0] [Originator@6876 sub=Main] [Init] Init failed: Vpx::Common::Sso::SsoFactory_CreateFacade(sslContext, ssoFacadeConstPtr)

                                  --> Backtrace:

                                  -->

                                  --> [backtrace begin] product: VMware VirtualCenter, version: 6.0.0, build: build-2776510, tag: vpxd

                                  --> backtrace[00] libvmacore.so[0x003C00C4]: Vmacore::System::Stacktrace::CaptureWork(unsigned int)

                                  --> backtrace[01] libvmacore.so[0x001EDBD3]: Vmacore::System::SystemFactoryImpl::CreateQuickBacktrace(Vmacore::Ref<Vmacore::System::Backtrace>&)

                                  --> backtrace[02] libvmacore.so[0x001988DD]: Vmacore::Throwable::Throwable(std::string const&)

                                  --> backtrace[03] vpxd[0x00BD075E]: Vmomi::Fault::SystemError::Exception::Exception(std::string const&)

                                  • 14. Re: vCenter 6 fails to restart after IP change
                                    malabelle Enthusiast

                                    i tried

                                    service-control --start --all

                                     

                                     

                                     

                                    Stdout = Starting VMware Inventory Service...

                                    Waiting for VMware Inventory Service...................................................................................................

                                    WARNING: VMware Inventory Service may have failed to start.

                                     

                                     

                                    Stderr =

                                    2015-09-10T19:33:24.061Z   {

                                        "resolution": null,

                                        "detail": [

                                            {

                                                "args": [

                                                    "Command: ['/sbin/service', u'vmware-invsvc', 'start']\nStderr: "

                                                ],

                                                "id": "install.ciscommon.command.errinvoke",

                                                "localized": "An error occurred while invoking external command : 'Command: ['/sbin/service', u'vmware-invsvc', 'start']\nStderr: '",

                                                "translatable": "An error occurred while invoking external command : '%(0)s'"

                                            }

                                        ],

                                        "componentKey": null,

                                        "problemId": null

                                    }

                                    ERROR:root:Unable to start service vmware-invsvc, Exception: {

                                        "resolution": null,

                                        "detail": [

                                            {

                                                "args": [

                                                    "vmware-invsvc"

                                                ],

                                                "id": "install.ciscommon.service.failstart",

                                                "localized": "An error occurred while starting service 'vmware-invsvc'",

                                                "translatable": "An error occurred while starting service '%(0)s'"

                                            }

                                        ],

                                        "componentKey": null,

                                        "problemId": null

                                    }

                                    Unable to start service vmware-invsvc, Exception: {

                                        "resolution": null,

                                        "detail": [

                                            {

                                                "args": [

                                                    "vmware-invsvc"

                                                ],

                                                "id": "install.ciscommon.service.failstart",

                                                "localized": "An error occurred while starting service 'vmware-invsvc'",

                                                "translatable": "An error occurred while starting service '%(0)s'"

                                            }

                                        ],

                                        "componentKey": null,

                                        "problemId": null

                                    }

                                    1 2 3 Previous Next