1 2 Previous Next 19 Replies Latest reply on Dec 7, 2018 12:33 AM by Smerchla

    Cannot Start vCenter Server Appliance - "Waiting for vpxd to initialize: ....failed"

    edmandsj Novice

      I am stumped. At first, I was able to at least login to vsphere and do operations. The client would close like every 15 minutes so it got really annoying. Now I am stuck with no launch to vsphere. I must log into each host individually. ESXi hosts without a vsphere is so limited. Please advise.

       

       

      Here is the /var/log/messages

      vsphere-fmrif:~ # tail -f /var/log/messages
      May 31 09:50:32  vsphere-fmrif watchdog-vpxd: [4773] Begin '/usr/sbin/vpxd', min-uptime =  30, max-quick-failures = 5, max-total-failures = 1000000, bg_pid_file =  ''
      May 31 09:50:32 vsphere-fmrif watchdog-vpxd: Executing '/usr/sbin/vpxd'
      May  31 09:50:34 vsphere-fmrif kernel: [   59.780777] warning: process  `vpxd-worker' used the deprecated sysctl system call with 1.1.
      May 31 09:50:40 vsphere-fmrif sshd[4944]: Accepted password for root from 128.231.212.117 port 51160 ssh2
      May 31 09:50:44 vsphere-fmrif watchdog-vpxd: '/usr/sbin/vpxd' exited after 12 seconds (quick failure 1) 134
      May 31 09:50:44 vsphere-fmrif watchdog-vpxd: Executing '/usr/sbin/vpxd'
      May 31 09:50:45 vsphere-fmrif kernel: [   70.438974] warning: process `vpxd' used the deprecated sysctl system call with 1.1.
      May 31 09:50:51 vsphere-fmrif watchdog-vpxd: '/usr/sbin/vpxd' exited after 7 seconds (quick failure 2) 134
      May 31 09:50:51 vsphere-fmrif watchdog-vpxd: Executing '/usr/sbin/vpxd'
      May 31 09:50:52 vsphere-fmrif kernel: [   77.370397] warning: process `vpxd' used the deprecated sysctl system call with 1.1.
      May 31 09:50:58 vsphere-fmrif watchdog-vpxd: '/usr/sbin/vpxd' exited after 7 seconds (quick failure 3) 134
      May 31 09:50:58 vsphere-fmrif watchdog-vpxd: Executing '/usr/sbin/vpxd'
      May 31 09:50:58 vsphere-fmrif kernel: [   83.923261] warning: process `vpxd' used the deprecated sysctl system call with 1.1.
      May 31 09:51:05 vsphere-fmrif watchdog-vpxd: '/usr/sbin/vpxd' exited after 7 seconds (quick failure 4) 134
      May 31 09:51:05 vsphere-fmrif watchdog-vpxd: Executing '/usr/sbin/vpxd'
      May  31 09:51:06 vsphere-fmrif kernel: [   91.720959] warning: process  `vpxd-worker' used the deprecated sysctl system call with 1.1.
      May 31 09:51:12 vsphere-fmrif auditd[3012]: Audit daemon rotating log files
      May 31 09:51:13 vsphere-fmrif watchdog-vpxd: '/usr/sbin/vpxd' exited after 8 seconds (quick failure 5) 134
      May 31 09:51:13 vsphere-fmrif watchdog-vpxd: Executing '/usr/sbin/vpxd'
      May 31 09:51:19 vsphere-fmrif watchdog-vpxd: '/usr/sbin/vpxd' exited after 6 seconds (quick failure 6) 134
      May 31 09:51:19 vsphere-fmrif watchdog-vpxd: End '/usr/sbin/vpxd', failure limit reached

       

      /var/log/warn

       

      May 31 02:40:23 vsphere-fmrif watchdog-vpxd: '/usr/sbin/vpxd' exited after 62 seconds 1
      May 31 02:41:26 vsphere-fmrif watchdog-vpxd: '/usr/sbin/vpxd' exited after 63 seconds 1
      May 31 02:42:28 vsphere-fmrif watchdog-vpxd: '/usr/sbin/vpxd' exited after 62 seconds 1
      May 31 02:43:31 vsphere-fmrif watchdog-vpxd: '/usr/sbin/vpxd' exited after 63 seconds 1
      May 31 02:44:33 vsphere-fmrif watchdog-vpxd: '/usr/sbin/vpxd' exited after 62 seconds 1
      May 31 02:45:36 vsphere-fmrif watchdog-vpxd: '/usr/sbin/vpxd' exited after 63 seconds 1
      May 31 02:46:38 vsphere-fmrif watchdog-vpxd: '/usr/sbin/vpxd' exited after 62 seconds 1
      May 31 02:47:41 vsphere-fmrif watchdog-vpxd: '/usr/sbin/vpxd' exited after 63 seconds 1
      May 31 02:48:44 vsphere-fmrif watchdog-vpxd: '/usr/sbin/vpxd' exited after 63 seconds 1
      May 31 02:49:46 vsphere-fmrif watchdog-vpxd: '/usr/sbin/vpxd' exited after 62 seconds 1
      May 31 02:50:49 vsphere-fmrif watchdog-vpxd: '/usr/sbin/vpxd' exited after 63 seconds 1
      May 31 02:51:51 vsphere-fmrif watchdog-vpxd: '/usr/sbin/vpxd' exited after 62 seconds 1
      May 31 02:52:54 vsphere-fmrif watchdog-vpxd: '/usr/sbin/vpxd' exited after 63 seconds 1
      May 31 02:53:56 vsphere-fmrif watchdog-vpxd: '/usr/sbin/vpxd' exited after 62 seconds 1
      May 31 02:54:59 vsphere-fmrif watchdog-vpxd: '/usr/sbin/vpxd' exited after 63 seconds 1
      May 31 02:56:01 vsphere-fmrif watchdog-vpxd: '/usr/sbin/vpxd' exited after 62 seconds 1
      May 31 02:57:04 vsphere-fmrif watchdog-vpxd: '/usr/sbin/vpxd' exited after 63 seconds 1
      May 31 02:58:06 vsphere-fmrif watchdog-vpxd: '/usr/sbin/vpxd' exited after 62 seconds 1
      May 31 02:59:09 vsphere-fmrif watchdog-vpxd: '/usr/sbin/vpxd' exited after 63 seconds 1
      May 31 03:00:12 vsphere-fmrif watchdog-vpxd: '/usr/sbin/vpxd' exited after 63 seconds 1
      May 31 03:01:14 vsphere-fmrif watchdog-vpxd: '/usr/sbin/vpxd' exited after 62 seconds 1
      May 31 03:01:52 vsphere-fmrif portmap[25177]: connect from 128.231.212.168 to callit(390109): request from unauthorized host
      May 31 03:01:52 vsphere-fmrif portmap[25178]: connect from 128.231.212.177 to callit(390109): request from unauthorized host
      May 31 03:01:53 vsphere-fmrif portmap[25179]: connect from 128.231.212.121 to callit(390109): request from unauthorized host
      May 31 03:02:17 vsphere-fmrif watchdog-vpxd: '/usr/sbin/vpxd' exited after 63 seconds 1
      May 31 03:03:07 vsphere-fmrif portmap[25275]: connect from 128.231.212.168 to callit(390109): request from unauthorized host
      May 31 03:03:19 vsphere-fmrif watchdog-vpxd: '/usr/sbin/vpxd' exited after 62 seconds 1
      May 31 03:04:04 vsphere-fmrif portmap[25358]: connect from 128.231.212.177 to callit(390109): request from unauthorized host
      May 31 03:04:19 vsphere-fmrif portmap[25368]: connect from 128.231.212.121 to callit(390109): request from unauthorized host
      May 31 03:04:22 vsphere-fmrif watchdog-vpxd: '/usr/sbin/vpxd' exited after 62 seconds 1
      May 31 03:05:25 vsphere-fmrif watchdog-vpxd: '/usr/sbin/vpxd' exited after 63 seconds 1
      May 31 03:06:01 vsphere-fmrif portmap[25531]: connect from 128.231.212.168 to callit(390109): request from unauthorized host
      May 31 03:06:27 vsphere-fmrif watchdog-vpxd: '/usr/sbin/vpxd' exited after 62 seconds 1
      May 31 03:06:31 vsphere-fmrif portmap[25603]: connect from 128.231.212.121 to callit(390109): request from unauthorized host
      May 31 03:07:00 vsphere-fmrif portmap[25618]: connect from 128.231.212.177 to callit(390109): request from unauthorized host
      May 31 03:07:30 vsphere-fmrif watchdog-vpxd: '/usr/sbin/vpxd' exited after 63 seconds 1
      May 31 03:08:32 vsphere-fmrif watchdog-vpxd: '/usr/sbin/vpxd' exited after 62 seconds 1
      May 31 03:09:35 vsphere-fmrif watchdog-vpxd: '/usr/sbin/vpxd' exited after 63 seconds 1
      May 31 03:10:38 vsphere-fmrif watchdog-vpxd: '/usr/sbin/vpxd' exited after 63 seconds 1
      May 31 03:11:40 vsphere-fmrif watchdog-vpxd: '/usr/sbin/vpxd' exited after 62 seconds 1
      May 31 03:12:20 vsphere-fmrif portmap[26072]: connect from 128.231.212.177 to callit(390109): request from unauthorized host
      May 31 03:12:43 vsphere-fmrif watchdog-vpxd: '/usr/sbin/vpxd' exited after 63 seconds 1
      May 31 03:12:54 vsphere-fmrif portmap[26144]: connect from 128.231.212.168 to callit(390109): request from unauthorized host
      May 31 03:13:45 vsphere-fmrif watchdog-vpxd: '/usr/sbin/vpxd' exited after 62 seconds 1
      May 31 03:13:48 vsphere-fmrif portmap[26227]: connect from 128.231.212.177 to callit(390109): request from unauthorized host
      May 31 03:14:17 vsphere-fmrif portmap[26242]: connect from 128.231.212.121 to callit(390109): request from unauthorized host
      May 31 03:14:48 vsphere-fmrif watchdog-vpxd: '/usr/sbin/vpxd' exited after 63 seconds 1
      May 31 03:15:09 vsphere-fmrif portmap[26348]: connect from 128.231.212.177 to callit(390109): request from unauthorized host
      May 31 03:15:17 vsphere-fmrif portmap[26354]: connect from 128.231.212.177 to callit(390109): request from unauthorized host
      May 31 03:15:51 vsphere-fmrif watchdog-vpxd: '/usr/sbin/vpxd' exited after 63 seconds 1
      May 31 03:16:03 vsphere-fmrif portmap[26435]: connect from 128.231.212.168 to callit(390109): request from unauthorized host
      May 31 03:16:53 vsphere-fmrif watchdog-vpxd: '/usr/sbin/vpxd' exited after 62 seconds 1
      May 31 03:16:53 vsphere-fmrif portmap[26472]: connect from 128.231.212.177 to callit(390109): request from unauthorized host
      May 31 03:16:59 vsphere-fmrif portmap[26519]: connect from 128.231.212.121 to callit(390109): request from unauthorized host
      May 31 03:17:56 vsphere-fmrif watchdog-vpxd: '/usr/sbin/vpxd' exited after 63 seconds 1
      May 31 03:18:30 vsphere-fmrif portmap[26621]: connect from 128.231.212.121 to callit(390109): request from unauthorized host
      May 31 03:18:58 vsphere-fmrif watchdog-vpxd: '/usr/sbin/vpxd' exited after 62 seconds 1
      May 31 03:20:01 vsphere-fmrif watchdog-vpxd: '/usr/sbin/vpxd' exited after 63 seconds 1
      May 31 03:21:03 vsphere-fmrif watchdog-vpxd: '/usr/sbin/vpxd' exited after 62 seconds 1
      May 31 03:21:37 vsphere-fmrif portmap[26897]: connect from 128.231.212.168 to callit(390109): request from unauthorized host
      May 31 03:22:06 vsphere-fmrif watchdog-vpxd: '/usr/sbin/vpxd' exited after 63 seconds 1
      May 31 03:23:09 vsphere-fmrif watchdog-vpxd: '/usr/sbin/vpxd' exited after 63 seconds 1
      May 31 03:24:11 vsphere-fmrif watchdog-vpxd: '/usr/sbin/vpxd' exited after 62 seconds 1
      May 31 03:24:22 vsphere-fmrif portmap[27146]: connect from 128.231.212.168 to callit(390109): request from unauthorized host
      May 31 03:25:14 vsphere-fmrif watchdog-vpxd: '/usr/sbin/vpxd' exited after 63 seconds 1
      May 31 03:26:16 vsphere-fmrif watchdog-vpxd: '/usr/sbin/vpxd' exited after 62 seconds 1
      May 31 03:26:40 vsphere-fmrif portmap[27330]: connect from 128.231.212.168 to callit(390109): request from unauthorized host
      May 31 03:27:19 vsphere-fmrif watchdog-vpxd: '/usr/sbin/vpxd' exited after 63 seconds 1
      May 31 03:28:21 vsphere-fmrif watchdog-vpxd: '/usr/sbin/vpxd' exited after 62 seconds 1
      May 31 03:29:24 vsphere-fmrif watchdog-vpxd: '/usr/sbin/vpxd' exited after 63 seconds 1
      May 31 03:30:27 vsphere-fmrif watchdog-vpxd: '/usr/sbin/vpxd' exited after 63 seconds 1
      May 31 03:31:29 vsphere-fmrif watchdog-vpxd: '/usr/sbin/vpxd' exited after 62 seconds 1
      May 31 03:32:32 vsphere-fmrif watchdog-vpxd: '/usr/sbin/vpxd' exited after 63 seconds 1
      May 31 03:33:35 vsphere-fmrif watchdog-vpxd: '/usr/sbin/vpxd' exited after 63 seconds 1
      May 31 03:34:37 vsphere-fmrif watchdog-vpxd: '/usr/sbin/vpxd' exited after 62 seconds 1
      May 31 03:35:40 vsphere-fmrif watchdog-vpxd: '/usr/sbin/vpxd' exited after 63 seconds 1

       

      /var/log/vmware/vpx/vpxd.log

       

       

       

      From earlier:

      vsphere-fmrif:~ # tail -f -n 100 /var/log/vmware/vpx/vpxd.log

      Section for VMware VirtualCenter, pid=25488, version=5.0.0, build=build-455964, option=Release

       

      ------ In-memory logs start --------

      2012-05-31T09:11:29.623-04:00 [7FFFF3B09700 info 'Hooks'] Hooks Initialized

       

      ------ In-memory logs end   --------

      2012-05-31T09:11:29.633-04:00 [7FFFF3B09700 info 'Default'] Initialized channel manager

      2012-05-31T09:11:29.641-04:00 [7FFFF3B09700 info 'Default'] Current working directory: /storage/log/vmware/vpx

      2012-05-31T09:11:29.642-04:00 [7FFFF3B09700 info 'Default'] Log path: /var/log/vmware/vpx

      2012-05-31T09:11:29.642-04:00 [7FFFF3B09700 info 'Default'] Initializing SSL

      2012-05-31T09:11:29.645-04:00  [7FFFF3B09700 info 'Default'] Vmacore::InitSSL: doVersionCheck = true,  handshakeTimeoutUs = 120000000

      2012-05-31T09:11:29.646-04:00 [7FFFF3B09700 info 'Default'] Changed working directory to /var/log/vmware/vpx

      2012-05-31T09:11:29.666-04:00 [7FFFF3B09700 info 'Default'] Starting VMware VirtualCenter 5.0.0 build-455964

      2012-05-31T09:11:29.666-04:00 [7FFFF3B09700 info 'Default'] Log directory: /var/log/vmware/vpx.

      2012-05-31T09:11:29.667-04:00 [7FFFF3B09700 info 'Default'] Account name: root

      2012-05-31T09:11:29.667-04:00 [7FFFF3B09700 info 'Default'] [Init:835] Calling: InfoDeclSchema(gDB)

      2012-05-31T09:11:29.667-04:00  [7FFFF3B09700 info 'Default'] [Init:837] Calling:  VpxCallbackDesc::Init(MakeFunctor(this,  &ServerApp::RequestShutdown))

      2012-05-31T09:11:29.667-04:00 [7FFFF3B09700 info 'Default'] [Init:838] Calling: VpxCryptInit()

      2012-05-31T09:11:29.667-04:00 [7FFFF3B09700 info 'Default'] [Init:839] Calling: VpxLRO::Init()

      2012-05-31T09:11:29.667-04:00 [7FFFF3B09700 info 'Default'] [Init:840] Calling: VpxLroList::Init(ltud)

      2012-05-31T09:11:29.667-04:00 [7FFFF3B09700 info 'Default'] [VpxLRO] 640 max LROs

      2012-05-31T09:11:29.667-04:00 [7FFFF3B09700 info 'Default'] [VpxLRO] 12 reserved internal LROs

      2012-05-31T09:11:29.667-04:00 [7FFFF3B09700 info 'Default'] [VpxLRO] 12 reserved blocker LROs

      2012-05-31T09:11:29.667-04:00 [7FFFF3B09700 info 'Default'] [VpxLRO] 12 reserved short LROs

      2012-05-31T09:11:29.667-04:00 [7FFFF3B09700 info 'Default'] [VpxLRO] 8 reserved long LROs

      2012-05-31T09:11:29.667-04:00 [7FFFF3B09700 info 'Default'] [VpxLRO] 600-second completed task lifetime

      2012-05-31T09:11:29.667-04:00 [7FFFF3B09700 info 'Default'] [VpxLRO] 60-second minimum completed task lifetime

      2012-05-31T09:11:29.667-04:00 [7FFFF3B09700 info 'Default'] [VpxLRO] 200 maximum completed tasks

      2012-05-31T09:11:29.667-04:00 [7FFFF3B09700 info 'Default'] [VpxLRO] 600-second unregistered task lifetime

      2012-05-31T09:11:29.667-04:00 [7FFFF3B09700 info 'Default'] [Init:841] Calling: VpxdCharacterizeThreadpool(ltud)

      2012-05-31T09:11:29.667-04:00  [7FFFF3B09700 info 'ThreadPool'] [SetConfig] Thread info: IoMin: 2,  IoMax: 1990, TaskMin: 59, TaskMax: 1990, Max Thread: 3980, Keepalive:  61, thread kill: 600, max fds: -1

      2012-05-31T09:11:29.667-04:00 [7FFFF3B09700 info 'Default'] [Init:842] Calling: VpxdCertificate_Load(gDB, CERTIFICATE_VMDBPATH )

      2012-05-31T09:11:29.668-04:00  [7FFFF3B09700 info 'Default'] [Init:843] Calling:  VpxdVdb::Init(VpxdVdb::GetVcVdbInstId(), false, false, NULL)

      2012-05-31T09:11:29.668-04:00 [7FFFF3B09700 info 'Default'] Registry Item DB 5 value is ''

      2012-05-31T09:11:32.078-04:00  [7FFFF3B09700 info 'Default'] [VdbSchemaLoader::LookupSchemaFromDB]  Loaded schema and index information from DB

      2012-05-31T09:11:32.099-04:00 [7FFFF3B09700 info 'Default'] [Init:844] Calling: VpxdDbSchemaHash::CheckSchema()

      2012-05-31T09:11:32.100-04:00  [7FFFF3B09700 info 'Default'] [VpxdDbSchemaHash::CheckSchema]: schema  file: /usr/lib/vmware-vpx/VCDB_db2.sql

      2012-05-31T09:11:32.104-04:00 [7FFFF3B09700 info 'Default'] [Init:845] Calling: VpxdDbMonitoring::Init()

      2012-05-31T09:11:32.120-04:00 [7FFFF3B09700 info 'Default'] [Init:848] Calling: VpxdLdap::Init()

      2012-05-31T09:11:32.121-04:00 [7FFFF3B09700 info 'Default'] [LdapBackup] Making sure LDAP instance VMwareVCMSDS is running

      2012-05-31T09:11:32.121-04:00 [7FFFF3B09700 info 'Default'] [LdapBackup] Checking service ldap for running

      2012-05-31T09:11:32.179-04:00 [7FFFF3B09700 info 'Default'] [Init:849] Calling: VpxResultFilter::Init()

      2012-05-31T09:11:32.179-04:00  [7FFFF3B09700 info 'Default'] Looking for vpx result filter at  /usr/lib/vmware-vpx/vpxResultFilter.xml

      2012-05-31T09:11:32.184-04:00 [7FFFF3B09700 info 'Default'] [Init:853] Calling: VpxdLock::Init(gDB)

      2012-05-31T09:11:32.184-04:00 [7FFFF3B09700 info 'Default'] [Init:854] Calling: VpxdMoLock::Init()

      2012-05-31T09:11:32.184-04:00 [7FFFF3B09700 info 'Default'] [Init:855] Calling: VpxdStateLock::Init()

      2012-05-31T09:11:32.184-04:00 [7FFFF3B09700 info 'Default'] [Init:856] Calling: VpxdMoDataTable::Init()

      2012-05-31T09:11:32.184-04:00  [7FFFF3B09700 info 'Default'] [Init:858] Calling:  VpxdHeartbeat::Init(MakeFunctor(this, &ServerApp::RequestShutdown))

      2012-05-31T09:11:32.184-04:00 [7FFFF3B09700 info 'Default'] [Init:859] Calling: VpxdHostCnx::Init(gDB)

      2012-05-31T09:11:32.189-04:00 [7FFFEAA06710 info 'Default' opID=SWI-75fe915a] Pre-loading symbols...

      2012-05-31T09:11:32.190-04:00 [7FFFEAA06710 info 'Default' opID=SWI-75fe915a] Done pre-loading symbols.

      2012-05-31T09:11:32.196-04:00 [7FFFF3B09700 info 'Default'] [Init:861] Calling: VpxdVmomi::Init()

      2012-05-31T09:11:32.198-04:00  [7FFFF3B09700 info 'Default'] Looking for vpx result filter at  /usr/lib/vmware-vpx/vpxResultFilter.xml

      2012-05-31T09:11:32.200-04:00 [7FFFF3B09700 info 'Default'] [Init:862] Calling: QueryServiceProviderHelper::Init()

      2012-05-31T09:11:32.201-04:00 [7FFFF3B09700 info 'QueryServiceProvider'] Initializing QueryServiceProviderHelperImpl

      2012-05-31T09:11:32.240-04:00 [7FFFF3B09700 info 'QueryServiceProvider'] Starting with generation number: 5230290

      2012-05-31T09:11:32.240-04:00 [7FFFF3B09700 info 'Default'] [Init:863] Calling: ServiceDirectory_Init()

      2012-05-31T09:11:32.240-04:00  [7FFFF3B09700 info 'Default'] [ServiceDirectory] Local instance UUID:  DB25B17D-3703-4779-BAFB-845B64AC33AA

      2012-05-31T09:11:32.245-04:00 [7FFFF3B09700 warning 'VpxProfiler'] ServerApp::Init [TotalTime] took 2578 ms

      2012-05-31T09:11:32.246-04:00 [7FFFF3B09700 error 'Default'] [VpxdMain] Failed to initialize: Name or service not known

      2012-05-31T09:11:32.246-04:00 [7FFFF3B09700 error 'Default'] Failed to intialize VMware VirtualCenter. Shutting down...

      2012-05-31T09:11:32.246-04:00 [7FFFF3B09700 info 'Default'] Wrote uptime information

      2012-05-31T09:12:32.701-04:00 [7FFFF3B09700 info 'Default'] Forcing shutdown of VMware VirtualCenter now

       

       

       

      and the process itself

      vsphere-fmrif:~ # service vmware-vpxd status
      vmware-vpxd is stopped
      tomcat is stopped
      vsphere-fmrif:~ # service vmware-vpxd start
      Waiting for embedded DB2 database to startup: .success
      Cleaning session lock table: success
      Verifying EULA acceptance: success
      Shutting down ldap-server..done
      Starting ldap-server..done
      Starting vmware-vpxd: success
      Waiting for vpxd to initialize: ....failed
        • 1. Re: Cannot Start vCenter Server Appliance - "Waiting for vpxd to initialize: ....failed"
          Troy Clavell Guru
          vExpertUser Moderators

          sometimes this happens if you are using the built in DB and you configure your timezone.  the appliance will use UTC and if it is changed the local DB will not know how to access records in the future.  Did you change the timezone?  If so, you may try importing the appliance again and this time don't change the timezone.

          • 2. Re: Cannot Start vCenter Server Appliance - "Waiting for vpxd to initialize: ....failed"
            edmandsj Novice

            Well the appliance has worked for the past 4-6 months. I'm not sure what would have changed.

             

            What are my options? Just thinking out loud below

             

            Fix Time zone settings or time settings on exisiting vCenter Appliance

            Sync time somehow?

            "Reset DB Contents" --> How disruptive is this? What will I lose?

            Deploy new vCenter Appliance and Import old embedded database

             

            If all else fails and I need to use VDR to recover my vcenter server (to get the database), what will I need to do to put the database back in place?

            • 3. Re: Cannot Start vCenter Server Appliance - "Waiting for vpxd to initialize: ....failed"
              edmandsj Novice

              oh, may have found the culprit

               

              vsphere-fmrif:~ # df -h
              Filesystem            Size  Used Avail Use% Mounted on
              /dev/sda3             9.8G  4.2G  5.1G  46% /
              devtmpfs              2.0G  104K  2.0G   1% /dev
              tmpfs                 2.0G     0  2.0G   0% /dev/shm
              /dev/sda1             130M   18M  105M  15% /boot
              /dev/sdb1              20G   20G     0 100% /storage/core
              /dev/sdb2              20G  892M   18G   5% /storage/log
              /dev/sdb3              20G  2.7G   17G  14% /storage/db

               

               

              http://vmwire.com/category/vcsa-2/

              • 4. Re: Cannot Start vCenter Server Appliance - "Waiting for vpxd to initialize: ....failed"
                edmandsj Novice

                So I cleared out the /storage/core mount point. It quickly fills up again, though. It's already at 25% again. Either way I still cannot login. It is most likely the time thing. If I change it back to the original time(not sure what it was at this point) what will happen?

                 

                If you think rebuilding a new VCSA VM is the best possible way, just say so.

                 

                 

                side note: backup database from /storage/db ?

                • 5. Re: Cannot Start vCenter Server Appliance - "Waiting for vpxd to initialize: ....failed"
                  Troy Clavell Guru
                  User ModeratorsvExpert

                  a fresh start is always an option, but you'll have to add all your ESX Hosts back in, create any new Clusters, Resource Pools, etc.  My initial response was thinking you were just bringing the appliance on-line for the first time.

                   

                  Since this environment has been online for quite sometime, it may be worth your while to contact VMware Support to see if they can help in fixing the existing issue.

                  • 6. Re: Cannot Start vCenter Server Appliance - "Waiting for vpxd to initialize: ....failed"
                    roddam Novice

                    Hi,

                     

                       Did you manage to resolve this without a fresh install? A month ago my friend came along and suggested that I should change the time to UTC, at that time vCenter and vpxd were working fine. I changed it. Now I am doing negative tests in my environment, and one of the tests was to restart the vpxd service and it is failing with the same error as you saw.

                       Even rebooting the vCenter vApp does not help. I get the same error failied to initialize vpxd.

                       Let me know how you resolved this issue. I can initialize the contents of the DB, but what I dont want to lose is the customized alarms and some defualt alarms that I modified.

                     

                    Thanks!

                    • 7. Re: Cannot Start vCenter Server Appliance - "Waiting for vpxd to initialize: ....failed"
                      edmandsj Novice

                      Fresh install is the only remedy from what I could tell. Take it as an opportunity to check your settings and so on. Sorry man.

                       

                      --

                       

                      Justin Edmands

                      • 8. Re: Cannot Start vCenter Server Appliance - "Waiting for vpxd to initialize: ....failed"
                        roddam Novice

                        Thanks!

                         

                        But this is going to be such a pain. I guess I will try a few things before I did that.

                        Yup, I will try to see the positive side of it! will be hard though

                        • 9. Re: Cannot Start vCenter Server Appliance - "Waiting for vpxd to initialize: ....failed"
                          lubela Lurker

                          fresh install only option.  vmware support takes 3 days to even come up with anything.. even for a sev1.

                          • 10. Re: Cannot Start vCenter Server Appliance - "Waiting for vpxd to initialize: ....failed"
                            aSmoot Novice

                            My lab instance, a vSphere 5.1 vCenter Server Appliance with 2 clusters, 8 hosts (all vSphere 5.1), 80 VMs, exhibits these exact same symptoms.  It is using the built in database, no external.    I can say for certain that the VCSA time zone has not been touched since initial install, so that is not the issue.    My issues started a couple of weeks ago or more.  The  vSphere Client would just crash every 15 minutes or so, but then I could log back in so I just did that and worked on other more pressing matters, the lab is important, but it would have to wait.  It progessed to much worse this week and vSphere client or Web Client not being able to stay logged for more than a couple of seconds.    I haven't had time to deal with this until now, so here we go.   

                             

                            You know you are in trouble when on the VCSA management page, vCenter Server > Summary > Storage > Coredumps = 100%!   The core dumps or "coredumps" are stored on the VCSA in /storage/core with files named core.vpxd-worker.xxxx where 'x' is a number.  I emptied that via SSH with the rm command, but of course it just started filling up again with the same files.    Time to break out the toolbox.

                             

                             

                            After 20 minutes of research and an easy fix, I made a change that has let me login and perform operations in the vSphere Client.  So far, so good, no more core dump files 2 hours later.  It may be best to perform a basic, surface-level health check (or use LucD's)  before rebuilding a VCSA from scratch.  That can be a lot of work and will have consequences for VDS, Clusters, policies, etc.  Did you document everything?    

                             

                            These are the steps and actions I took to get the lab back online:

                             

                            From the VCSA management page, stop and started vCenter and Inventory services, but no effect.

                             

                            From the VCSA management page, reboot the appliance, but no effect.

                             

                            Stopped vCenter Server and Inventory Services to keep them off during troubleshooting

                             

                            Logged into each host with vSphere Client first to check all the basics, DNS name, time, verify that IPV6 is disabled (I had problems with that enabled before even though it wasn't in use), etc.  

                             

                            Then (still in vSphere client) check the host logs Home > Administration > System Logs for odd patterns or excessive errors in any of the logs (there are several, click the drop-down).    The first 4 hosts I check all looked normal, but one was having what looked like an excessive amount of errors related to snapshot delta disks from snapshots.   The problem rang a bell, there was a VM I noticed I had trouble with recently that has a bunch of snapshots and delta disks from some failed backups.   I didn't have time to look into it a month or so when I first saw that (it is lab and we never have the time do we?).

                             

                            The VM was already shut down, so I removed it from the host inventory, then SSH'd to the host and ran /sbin/services.sh restart to restart the management agents --but you can also use the DCUI to accomplish same.   

                             

                            Started vCenter Server and Inventory Services and voila, I could log in and manage my lab again.

                             

                            Side notes:

                            I got this message on all of the VMs that were running on the host.   The VMs never went down, but HA was attempted:

                            vSphere HA virtual machine failover failed

                             

                            I have to work on the snapshot problem which I think was caused during backups by the VMware Data Recovery Appliance, but I don't know the cause.

                            A number of different issues could be causing this behavior with your vCenter services, but if you can isolate it down to a particular host or even VM as was my case, you can save yourself a lot of trouble.  Always have a plan or you may need your resume.   Good luck!
                            Cheers!

                             

                            Message was edited by: aSmoot

                            • 11. Re: Cannot Start vCenter Server Appliance - "Waiting for vpxd to initialize: ....failed"
                              doubleH Expert

                              Thank you for your message. I am currently out of the office and will return Apr 16, 2013. I will respond upon my return.

                               

                              For support related issues please contact the IT Service Desk servicedesk@camhydro.com or x2700.

                               

                              Thank you

                              Heath

                              • 12. Re: Cannot Start vCenter Server Appliance - "Waiting for vpxd to initialize: ....failed"
                                Lynn123 Lurker

                                Hi there,

                                I have exact same problem on the linux- based vCenter server appliance 5.5 installed with OVA on the ESXi host 5.5. 

                                I used:

                                Name: VMware-vCenter-Server-Appliance-5.5.0.20200-2183109_OVF10.ova

                                Release Date: 2014-10-09

                                Build Number: 2183109

                                 

                                All services can start except the vCenter.  It always says :

                                Starting vmware-vpxd: success

                                "Waiting for vpxd to initialized: .failed'.

                                 

                                I have checked the time on ESXi host and the vCenter and they matched since both the ESXi host and vCenter sync to same ntp server.

                                My DataBase is embedded and SSO is also embedded.  "Test Settings" is successful.

                                 

                                 

                                in /var/log/messages:

                                2014-10-31T21:16:19+00:00 localhost vami-sfcbd: pam_succeed_if(vami-sfcb:auth): requirement "uid eq 0" was met by user "root"

                                2014-10-31T21:16:23+00:00 localhost lsassd[3997]: 0x7f57668fa700:Failed to run provider specific request (request code = 12, provider = 'lsa-activedirectory-provider') -> error = 2692, symbol = NERR_SetupNotJoined, client pid = 1815

                                2014-10-31T21:16:23+00:00 localhost lwsmd: Stopping service: eventlog

                                2014-10-31T21:16:24+00:00 localhost lwsmd: Starting service: eventlog

                                2014-10-31T21:16:24+00:00 localhost lsassd[3997]: 0x7f57668fa700:Failed to run provider specific request (request code = 12, provider = 'lsa-activedirectory-provider') -> error = 2692, symbol = NERR_SetupNotJoined, client pid = 1967

                                2014-10-31T21:16:50+00:00 localhost lsassd[3997]: 0x7f57668fa700:Failed to run provider specific request (request code = 12, provider = 'lsa-activedirectory-provider') -> error = 2692, symbol = NERR_SetupNotJoined, client pid = 2305

                                2014-10-31T21:16:53+00:00 localhost lsassd[3997]: 0x7f57668fa700:Failed to run provider specific request (request code = 12, provider = 'lsa-activedirectory-provider') -> error = 2692, symbol = NERR_SetupNotJoined, client pid = 2609

                                2014-10-31T21:16:53+00:00 localhost lwsmd: Stopping service: eventlog

                                2014-10-31T21:17:07+00:00 localhost watchdog-vpxd: [3080] Begin '/usr/sbin/vpxd', min-uptime = 30, max-quick-failures = 5, max-total-failures = 1000000, bg_pid_file = ''

                                2014-10-31T21:17:07+00:00 localhost watchdog-vpxd: Executing '/usr/sbin/vpxd'

                                2014-10-31T21:17:07+00:00 localhost watchdog-vpxd: '/usr/sbin/vpxd' exited after 0 seconds (quick failure 1) 134

                                2014-10-31T21:17:07+00:00 localhost watchdog-vpxd: Executing '/usr/sbin/vpxd'

                                2014-10-31T21:17:07+00:00 localhost watchdog-vpxd: '/usr/sbin/vpxd' exited after 0 seconds (quick failure 2) 134

                                2014-10-31T21:17:07+00:00 localhost watchdog-vpxd: Executing '/usr/sbin/vpxd'

                                2014-10-31T21:17:08+00:00 localhost watchdog-vpxd: '/usr/sbin/vpxd' exited after 1 seconds (quick failure 3) 134

                                2014-10-31T21:17:08+00:00 localhost watchdog-vpxd: Executing '/usr/sbin/vpxd'

                                2014-10-31T21:17:08+00:00 localhost watchdog-vpxd: '/usr/sbin/vpxd' exited after 0 seconds (quick failure 4) 134

                                2014-10-31T21:17:08+00:00 localhost watchdog-vpxd: Executing '/usr/sbin/vpxd'

                                2014-10-31T21:17:08+00:00 localhost watchdog-vpxd: '/usr/sbin/vpxd' exited after 0 seconds (quick failure 5) 134

                                2014-10-31T21:17:08+00:00 localhost watchdog-vpxd: Executing '/usr/sbin/vpxd'

                                2014-10-31T21:17:09+00:00 localhost watchdog-vpxd: '/usr/sbin/vpxd' exited after 1 seconds (quick failure 6) 134

                                2014-10-31T21:17:09+00:00 localhost watchdog-vpxd: End '/usr/sbin/vpxd', failure limit reached

                                2014-10-31T21:19:19+00:00 localhost syslog-ng[1289]: Log statistics; dropped='pipe(/dev/xconsole)=0', dropped='pipe(/dev/tty10)=0', processed='center(queued)=1704', processed='center(received)=1169', processed='destination(messages)=469', processed='destination(mailinfo)=2', processed='destination(mailwarn)=4', processed='destination(authlog)=38', processed='destination(localmessages)=10', processed='destination(cron)=25', processed='destination(mailerr)=0', processed='destination(ldapmessages)=694', processed='destination(netmgm)=0', processed='destination(warn)=156', processed='destination(console)=150', processed='destination(mail)=6', processed='destination(null)=0', processed='destination(xconsole)=150', processed='destination(firewall)=0', processed='destination(acpid)=0', processed='source(src)=1169'

                                2014-10-31T21:20:01+00:00 localhost /usr/sbin/cron[3432]: (root) CMD ([ -x /usr/lib64/sa/sa1 ] && exec /usr/lib64/sa/sa1 -S ALL 1 1)

                                2014-10-31T21:20:01+00:00 localhost /usr/sbin/cron[3435]: (root) CMD ( test -x /usr/sbin/vpxd_periodic && /usr/sbin/vpxd_periodic >/dev/null 2>&1)

                                2014-10-31T21:30:01+00:00 localhost /usr/sbin/cron[3733]: (root) CMD ( test -x /usr/sbin/vpxd_periodic && /usr/sbin/vpxd_periodic >/dev/null 2>&1)

                                2014-10-31T21:30:01+00:00 localhost /usr/sbin/cron[3735]: (root) CMD ([ -x /usr/lib64/sa/sa1 ] && exec /usr/lib64/sa/sa1 -S ALL 1 1)

                                 

                                I followed instruction on VMware KB: vpxd process on the vCenter Server Appliance 5.x fails with the error: Signal 11 receivedhttp://kb.vmware.com/selfservice/microsites/search.do?cmd=displayKC&docType=kc&externalId=2012167&sliceId=2&docTypeID=DT_KB_1_1&dialogID=413762600&stateId=0%200%20426598848

                                and modified /etc/vmware-vpx/vpx.cfg file as sugggested by adding the ThreadStackSizedKb line.  See below. 

                                <threadPool>

                                       <TaskMax>90</TaskMax>
                                       <ThreadStackSizeKb>512</ThreadStackSizeKb>
                                       <threadNamePrefix>vpxd</threadNamePrefix>

                                </threadPool>

                                 

                                However, it doesn't work.  

                                Anyone knows about this problem please help.

                                • 13. Re: Cannot Start vCenter Server Appliance - "Waiting for vpxd to initialize: ....failed"
                                  bhanekom Lurker

                                  I had an issue with the vpxd failing to initialize (vpxd service started succesfully), and found that there was a message in the var/log/vmware/vpx/vpxd.log that it was unable to open the file var/log/vmware/journal/1414758110.0.lck - no such file exists. I looked in the location and a directory of that name had been created. Once I removed this directory and restarted the vpxd service I got back my vCenter server and was able to log in using both the vsphere client and web client. I have not seen anyone mention this error in their logs posted - but this resolved the service failing to initialise issue on my setup. I am on ESXi 5.5 with a Single vCenter and 3 hosts

                                  • 14. Re: Cannot Start vCenter Server Appliance - "Waiting for vpxd to initialize: ....failed"
                                    brokentwig Lurker

                                    I had the same issue and it turned out the /storage/db drive was out of space.  I followed WMware's article: VMware KB: Increasing the disk space in vCenter Server Appliance 5.x and I'm good.

                                    1 2 Previous Next