Cannot Start vCenter Server Appliance - "Waiting f...

edmandsj · ‎05-31-2012

I am stumped. At first, I was able to at least login to vsphere and do operations. The client would close like every 15 minutes so it got really annoying. Now I am stuck with no launch to vsphere. I must log into each host individually. ESXi hosts without a vsphere is so limited. Please advise.

Here is the /var/log/messages

vsphere-fmrif:~ # tail -f /var/log/messages
May 31 09:50:32 vsphere-fmrif watchdog-vpxd: [4773] Begin '/usr/sbin/vpxd', min-uptime = 30, max-quick-failures = 5, max-total-failures = 1000000, bg_pid_file = ''
May 31 09:50:32 vsphere-fmrif watchdog-vpxd: Executing '/usr/sbin/vpxd'
May 31 09:50:34 vsphere-fmrif kernel: [   59.780777] warning: process `vpxd-worker' used the deprecated sysctl system call with 1.1.
May 31 09:50:40 vsphere-fmrif sshd[4944]: Accepted password for root from 128.231.212.117 port 51160 ssh2
May 31 09:50:44 vsphere-fmrif watchdog-vpxd: '/usr/sbin/vpxd' exited after 12 seconds (quick failure 1) 134
May 31 09:50:44 vsphere-fmrif watchdog-vpxd: Executing '/usr/sbin/vpxd'
May 31 09:50:45 vsphere-fmrif kernel: [   70.438974] warning: process `vpxd' used the deprecated sysctl system call with 1.1.
May 31 09:50:51 vsphere-fmrif watchdog-vpxd: '/usr/sbin/vpxd' exited after 7 seconds (quick failure 2) 134
May 31 09:50:51 vsphere-fmrif watchdog-vpxd: Executing '/usr/sbin/vpxd'
May 31 09:50:52 vsphere-fmrif kernel: [   77.370397] warning: process `vpxd' used the deprecated sysctl system call with 1.1.
May 31 09:50:58 vsphere-fmrif watchdog-vpxd: '/usr/sbin/vpxd' exited after 7 seconds (quick failure 3) 134
May 31 09:50:58 vsphere-fmrif watchdog-vpxd: Executing '/usr/sbin/vpxd'
May 31 09:50:58 vsphere-fmrif kernel: [   83.923261] warning: process `vpxd' used the deprecated sysctl system call with 1.1.
May 31 09:51:05 vsphere-fmrif watchdog-vpxd: '/usr/sbin/vpxd' exited after 7 seconds (quick failure 4) 134
May 31 09:51:05 vsphere-fmrif watchdog-vpxd: Executing '/usr/sbin/vpxd'
May 31 09:51:06 vsphere-fmrif kernel: [   91.720959] warning: process `vpxd-worker' used the deprecated sysctl system call with 1.1.
May 31 09:51:12 vsphere-fmrif auditd[3012]: Audit daemon rotating log files
May 31 09:51:13 vsphere-fmrif watchdog-vpxd: '/usr/sbin/vpxd' exited after 8 seconds (quick failure 5) 134
May 31 09:51:13 vsphere-fmrif watchdog-vpxd: Executing '/usr/sbin/vpxd'
May 31 09:51:19 vsphere-fmrif watchdog-vpxd: '/usr/sbin/vpxd' exited after 6 seconds (quick failure 6) 134
May 31 09:51:19 vsphere-fmrif watchdog-vpxd: End '/usr/sbin/vpxd', failure limit reached

/var/log/warn

May 31 02:40:23 vsphere-fmrif watchdog-vpxd: '/usr/sbin/vpxd' exited after 62 seconds 1
May 31 02:41:26 vsphere-fmrif watchdog-vpxd: '/usr/sbin/vpxd' exited after 63 seconds 1
May 31 02:42:28 vsphere-fmrif watchdog-vpxd: '/usr/sbin/vpxd' exited after 62 seconds 1
May 31 02:43:31 vsphere-fmrif watchdog-vpxd: '/usr/sbin/vpxd' exited after 63 seconds 1
May 31 02:44:33 vsphere-fmrif watchdog-vpxd: '/usr/sbin/vpxd' exited after 62 seconds 1
May 31 02:45:36 vsphere-fmrif watchdog-vpxd: '/usr/sbin/vpxd' exited after 63 seconds 1
May 31 02:46:38 vsphere-fmrif watchdog-vpxd: '/usr/sbin/vpxd' exited after 62 seconds 1
May 31 02:47:41 vsphere-fmrif watchdog-vpxd: '/usr/sbin/vpxd' exited after 63 seconds 1
May 31 02:48:44 vsphere-fmrif watchdog-vpxd: '/usr/sbin/vpxd' exited after 63 seconds 1
May 31 02:49:46 vsphere-fmrif watchdog-vpxd: '/usr/sbin/vpxd' exited after 62 seconds 1
May 31 02:50:49 vsphere-fmrif watchdog-vpxd: '/usr/sbin/vpxd' exited after 63 seconds 1
May 31 02:51:51 vsphere-fmrif watchdog-vpxd: '/usr/sbin/vpxd' exited after 62 seconds 1
May 31 02:52:54 vsphere-fmrif watchdog-vpxd: '/usr/sbin/vpxd' exited after 63 seconds 1
May 31 02:53:56 vsphere-fmrif watchdog-vpxd: '/usr/sbin/vpxd' exited after 62 seconds 1
May 31 02:54:59 vsphere-fmrif watchdog-vpxd: '/usr/sbin/vpxd' exited after 63 seconds 1
May 31 02:56:01 vsphere-fmrif watchdog-vpxd: '/usr/sbin/vpxd' exited after 62 seconds 1
May 31 02:57:04 vsphere-fmrif watchdog-vpxd: '/usr/sbin/vpxd' exited after 63 seconds 1
May 31 02:58:06 vsphere-fmrif watchdog-vpxd: '/usr/sbin/vpxd' exited after 62 seconds 1
May 31 02:59:09 vsphere-fmrif watchdog-vpxd: '/usr/sbin/vpxd' exited after 63 seconds 1
May 31 03:00:12 vsphere-fmrif watchdog-vpxd: '/usr/sbin/vpxd' exited after 63 seconds 1
May 31 03:01:14 vsphere-fmrif watchdog-vpxd: '/usr/sbin/vpxd' exited after 62 seconds 1
May 31 03:01:52 vsphere-fmrif portmap[25177]: connect from 128.231.212.168 to callit(390109): request from unauthorized host
May 31 03:01:52 vsphere-fmrif portmap[25178]: connect from 128.231.212.177 to callit(390109): request from unauthorized host
May 31 03:01:53 vsphere-fmrif portmap[25179]: connect from 128.231.212.121 to callit(390109): request from unauthorized host
May 31 03:02:17 vsphere-fmrif watchdog-vpxd: '/usr/sbin/vpxd' exited after 63 seconds 1
May 31 03:03:07 vsphere-fmrif portmap[25275]: connect from 128.231.212.168 to callit(390109): request from unauthorized host
May 31 03:03:19 vsphere-fmrif watchdog-vpxd: '/usr/sbin/vpxd' exited after 62 seconds 1
May 31 03:04:04 vsphere-fmrif portmap[25358]: connect from 128.231.212.177 to callit(390109): request from unauthorized host
May 31 03:04:19 vsphere-fmrif portmap[25368]: connect from 128.231.212.121 to callit(390109): request from unauthorized host
May 31 03:04:22 vsphere-fmrif watchdog-vpxd: '/usr/sbin/vpxd' exited after 62 seconds 1
May 31 03:05:25 vsphere-fmrif watchdog-vpxd: '/usr/sbin/vpxd' exited after 63 seconds 1
May 31 03:06:01 vsphere-fmrif portmap[25531]: connect from 128.231.212.168 to callit(390109): request from unauthorized host
May 31 03:06:27 vsphere-fmrif watchdog-vpxd: '/usr/sbin/vpxd' exited after 62 seconds 1
May 31 03:06:31 vsphere-fmrif portmap[25603]: connect from 128.231.212.121 to callit(390109): request from unauthorized host
May 31 03:07:00 vsphere-fmrif portmap[25618]: connect from 128.231.212.177 to callit(390109): request from unauthorized host
May 31 03:07:30 vsphere-fmrif watchdog-vpxd: '/usr/sbin/vpxd' exited after 63 seconds 1
May 31 03:08:32 vsphere-fmrif watchdog-vpxd: '/usr/sbin/vpxd' exited after 62 seconds 1
May 31 03:09:35 vsphere-fmrif watchdog-vpxd: '/usr/sbin/vpxd' exited after 63 seconds 1
May 31 03:10:38 vsphere-fmrif watchdog-vpxd: '/usr/sbin/vpxd' exited after 63 seconds 1
May 31 03:11:40 vsphere-fmrif watchdog-vpxd: '/usr/sbin/vpxd' exited after 62 seconds 1
May 31 03:12:20 vsphere-fmrif portmap[26072]: connect from 128.231.212.177 to callit(390109): request from unauthorized host
May 31 03:12:43 vsphere-fmrif watchdog-vpxd: '/usr/sbin/vpxd' exited after 63 seconds 1
May 31 03:12:54 vsphere-fmrif portmap[26144]: connect from 128.231.212.168 to callit(390109): request from unauthorized host
May 31 03:13:45 vsphere-fmrif watchdog-vpxd: '/usr/sbin/vpxd' exited after 62 seconds 1
May 31 03:13:48 vsphere-fmrif portmap[26227]: connect from 128.231.212.177 to callit(390109): request from unauthorized host
May 31 03:14:17 vsphere-fmrif portmap[26242]: connect from 128.231.212.121 to callit(390109): request from unauthorized host
May 31 03:14:48 vsphere-fmrif watchdog-vpxd: '/usr/sbin/vpxd' exited after 63 seconds 1
May 31 03:15:09 vsphere-fmrif portmap[26348]: connect from 128.231.212.177 to callit(390109): request from unauthorized host
May 31 03:15:17 vsphere-fmrif portmap[26354]: connect from 128.231.212.177 to callit(390109): request from unauthorized host
May 31 03:15:51 vsphere-fmrif watchdog-vpxd: '/usr/sbin/vpxd' exited after 63 seconds 1
May 31 03:16:03 vsphere-fmrif portmap[26435]: connect from 128.231.212.168 to callit(390109): request from unauthorized host
May 31 03:16:53 vsphere-fmrif watchdog-vpxd: '/usr/sbin/vpxd' exited after 62 seconds 1
May 31 03:16:53 vsphere-fmrif portmap[26472]: connect from 128.231.212.177 to callit(390109): request from unauthorized host
May 31 03:16:59 vsphere-fmrif portmap[26519]: connect from 128.231.212.121 to callit(390109): request from unauthorized host
May 31 03:17:56 vsphere-fmrif watchdog-vpxd: '/usr/sbin/vpxd' exited after 63 seconds 1
May 31 03:18:30 vsphere-fmrif portmap[26621]: connect from 128.231.212.121 to callit(390109): request from unauthorized host
May 31 03:18:58 vsphere-fmrif watchdog-vpxd: '/usr/sbin/vpxd' exited after 62 seconds 1
May 31 03:20:01 vsphere-fmrif watchdog-vpxd: '/usr/sbin/vpxd' exited after 63 seconds 1
May 31 03:21:03 vsphere-fmrif watchdog-vpxd: '/usr/sbin/vpxd' exited after 62 seconds 1
May 31 03:21:37 vsphere-fmrif portmap[26897]: connect from 128.231.212.168 to callit(390109): request from unauthorized host
May 31 03:22:06 vsphere-fmrif watchdog-vpxd: '/usr/sbin/vpxd' exited after 63 seconds 1
May 31 03:23:09 vsphere-fmrif watchdog-vpxd: '/usr/sbin/vpxd' exited after 63 seconds 1
May 31 03:24:11 vsphere-fmrif watchdog-vpxd: '/usr/sbin/vpxd' exited after 62 seconds 1
May 31 03:24:22 vsphere-fmrif portmap[27146]: connect from 128.231.212.168 to callit(390109): request from unauthorized host
May 31 03:25:14 vsphere-fmrif watchdog-vpxd: '/usr/sbin/vpxd' exited after 63 seconds 1
May 31 03:26:16 vsphere-fmrif watchdog-vpxd: '/usr/sbin/vpxd' exited after 62 seconds 1
May 31 03:26:40 vsphere-fmrif portmap[27330]: connect from 128.231.212.168 to callit(390109): request from unauthorized host
May 31 03:27:19 vsphere-fmrif watchdog-vpxd: '/usr/sbin/vpxd' exited after 63 seconds 1
May 31 03:28:21 vsphere-fmrif watchdog-vpxd: '/usr/sbin/vpxd' exited after 62 seconds 1
May 31 03:29:24 vsphere-fmrif watchdog-vpxd: '/usr/sbin/vpxd' exited after 63 seconds 1
May 31 03:30:27 vsphere-fmrif watchdog-vpxd: '/usr/sbin/vpxd' exited after 63 seconds 1
May 31 03:31:29 vsphere-fmrif watchdog-vpxd: '/usr/sbin/vpxd' exited after 62 seconds 1
May 31 03:32:32 vsphere-fmrif watchdog-vpxd: '/usr/sbin/vpxd' exited after 63 seconds 1
May 31 03:33:35 vsphere-fmrif watchdog-vpxd: '/usr/sbin/vpxd' exited after 63 seconds 1
May 31 03:34:37 vsphere-fmrif watchdog-vpxd: '/usr/sbin/vpxd' exited after 62 seconds 1
May 31 03:35:40 vsphere-fmrif watchdog-vpxd: '/usr/sbin/vpxd' exited after 63 seconds 1

/var/log/vmware/vpx/vpxd.log

From earlier:

vsphere-fmrif:~ # tail -f -n 100 /var/log/vmware/vpx/vpxd.log

Section for VMware VirtualCenter, pid=25488, version=5.0.0, build=build-455964, option=Release

------ In-memory logs start --------

2012-05-31T09:11:29.623-04:00 [7FFFF3B09700 info 'Hooks'] Hooks Initialized

------ In-memory logs end --------

2012-05-31T09:11:29.633-04:00 [7FFFF3B09700 info 'Default'] Initialized channel manager

2012-05-31T09:11:29.641-04:00 [7FFFF3B09700 info 'Default'] Current working directory: /storage/log/vmware/vpx

2012-05-31T09:11:29.642-04:00 [7FFFF3B09700 info 'Default'] Log path: /var/log/vmware/vpx

2012-05-31T09:11:29.642-04:00 [7FFFF3B09700 info 'Default'] Initializing SSL

2012-05-31T09:11:29.645-04:00 [7FFFF3B09700 info 'Default'] Vmacore::InitSSL: doVersionCheck = true, handshakeTimeoutUs = 120000000

2012-05-31T09:11:29.646-04:00 [7FFFF3B09700 info 'Default'] Changed working directory to /var/log/vmware/vpx

2012-05-31T09:11:29.666-04:00 [7FFFF3B09700 info 'Default'] Starting VMware VirtualCenter 5.0.0 build-455964

2012-05-31T09:11:29.666-04:00 [7FFFF3B09700 info 'Default'] Log directory: /var/log/vmware/vpx.

2012-05-31T09:11:29.667-04:00 [7FFFF3B09700 info 'Default'] Account name: root

2012-05-31T09:11:29.667-04:00 [7FFFF3B09700 info 'Default'] [Init:835] Calling: InfoDeclSchema(gDB)

2012-05-31T09:11:29.667-04:00 [7FFFF3B09700 info 'Default'] [Init:837] Calling: VpxCallbackDesc::Init(MakeFunctor(this, &ServerApp::RequestShutdown))

2012-05-31T09:11:29.667-04:00 [7FFFF3B09700 info 'Default'] [Init:838] Calling: VpxCryptInit()

2012-05-31T09:11:29.667-04:00 [7FFFF3B09700 info 'Default'] [Init:839] Calling: VpxLRO::Init()

2012-05-31T09:11:29.667-04:00 [7FFFF3B09700 info 'Default'] [Init:840] Calling: VpxLroList::Init(ltud)

2012-05-31T09:11:29.667-04:00 [7FFFF3B09700 info 'Default'] [VpxLRO] 640 max LROs

2012-05-31T09:11:29.667-04:00 [7FFFF3B09700 info 'Default'] [VpxLRO] 12 reserved internal LROs

2012-05-31T09:11:29.667-04:00 [7FFFF3B09700 info 'Default'] [VpxLRO] 12 reserved blocker LROs

2012-05-31T09:11:29.667-04:00 [7FFFF3B09700 info 'Default'] [VpxLRO] 12 reserved short LROs

2012-05-31T09:11:29.667-04:00 [7FFFF3B09700 info 'Default'] [VpxLRO] 8 reserved long LROs

2012-05-31T09:11:29.667-04:00 [7FFFF3B09700 info 'Default'] [VpxLRO] 600-second completed task lifetime

2012-05-31T09:11:29.667-04:00 [7FFFF3B09700 info 'Default'] [VpxLRO] 60-second minimum completed task lifetime

2012-05-31T09:11:29.667-04:00 [7FFFF3B09700 info 'Default'] [VpxLRO] 200 maximum completed tasks

2012-05-31T09:11:29.667-04:00 [7FFFF3B09700 info 'Default'] [VpxLRO] 600-second unregistered task lifetime

2012-05-31T09:11:29.667-04:00 [7FFFF3B09700 info 'Default'] [Init:841] Calling: VpxdCharacterizeThreadpool(ltud)

2012-05-31T09:11:29.667-04:00 [7FFFF3B09700 info 'ThreadPool'] [SetConfig] Thread info: IoMin: 2, IoMax: 1990, TaskMin: 59, TaskMax: 1990, Max Thread: 3980, Keepalive: 61, thread kill: 600, max fds: -1

2012-05-31T09:11:29.667-04:00 [7FFFF3B09700 info 'Default'] [Init:842] Calling: VpxdCertificate_Load(gDB, CERTIFICATE_VMDBPATH )

2012-05-31T09:11:29.668-04:00 [7FFFF3B09700 info 'Default'] [Init:843] Calling: VpxdVdb::Init(VpxdVdb::GetVcVdbInstId(), false, false, NULL)

2012-05-31T09:11:29.668-04:00 [7FFFF3B09700 info 'Default'] Registry Item DB 5 value is ''

2012-05-31T09:11:32.078-04:00 [7FFFF3B09700 info 'Default'] [VdbSchemaLoader::LookupSchemaFromDB] Loaded schema and index information from DB

2012-05-31T09:11:32.099-04:00 [7FFFF3B09700 info 'Default'] [Init:844] Calling: VpxdDbSchemaHash::CheckSchema()

2012-05-31T09:11:32.100-04:00 [7FFFF3B09700 info 'Default'] [VpxdDbSchemaHash::CheckSchema]: schema file: /usr/lib/vmware-vpx/VCDB_db2.sql

2012-05-31T09:11:32.104-04:00 [7FFFF3B09700 info 'Default'] [Init:845] Calling: VpxdDbMonitoring::Init()

2012-05-31T09:11:32.120-04:00 [7FFFF3B09700 info 'Default'] [Init:848] Calling: VpxdLdap::Init()

2012-05-31T09:11:32.121-04:00 [7FFFF3B09700 info 'Default'] [LdapBackup] Making sure LDAP instance VMwareVCMSDS is running

2012-05-31T09:11:32.121-04:00 [7FFFF3B09700 info 'Default'] [LdapBackup] Checking service ldap for running

2012-05-31T09:11:32.179-04:00 [7FFFF3B09700 info 'Default'] [Init:849] Calling: VpxResultFilter::Init()

2012-05-31T09:11:32.179-04:00 [7FFFF3B09700 info 'Default'] Looking for vpx result filter at /usr/lib/vmware-vpx/vpxResultFilter.xml

2012-05-31T09:11:32.184-04:00 [7FFFF3B09700 info 'Default'] [Init:853] Calling: VpxdLock::Init(gDB)

2012-05-31T09:11:32.184-04:00 [7FFFF3B09700 info 'Default'] [Init:854] Calling: VpxdMoLock::Init()

2012-05-31T09:11:32.184-04:00 [7FFFF3B09700 info 'Default'] [Init:855] Calling: VpxdStateLock::Init()

2012-05-31T09:11:32.184-04:00 [7FFFF3B09700 info 'Default'] [Init:856] Calling: VpxdMoDataTable::Init()

2012-05-31T09:11:32.184-04:00 [7FFFF3B09700 info 'Default'] [Init:858] Calling: VpxdHeartbeat::Init(MakeFunctor(this, &ServerApp::RequestShutdown))

2012-05-31T09:11:32.184-04:00 [7FFFF3B09700 info 'Default'] [Init:859] Calling: VpxdHostCnx::Init(gDB)

2012-05-31T09:11:32.189-04:00 [7FFFEAA06710 info 'Default' opID=SWI-75fe915a] Pre-loading symbols...

2012-05-31T09:11:32.190-04:00 [7FFFEAA06710 info 'Default' opID=SWI-75fe915a] Done pre-loading symbols.

2012-05-31T09:11:32.196-04:00 [7FFFF3B09700 info 'Default'] [Init:861] Calling: VpxdVmomi::Init()

2012-05-31T09:11:32.198-04:00 [7FFFF3B09700 info 'Default'] Looking for vpx result filter at /usr/lib/vmware-vpx/vpxResultFilter.xml

2012-05-31T09:11:32.200-04:00 [7FFFF3B09700 info 'Default'] [Init:862] Calling: QueryServiceProviderHelper::Init()

2012-05-31T09:11:32.201-04:00 [7FFFF3B09700 info 'QueryServiceProvider'] Initializing QueryServiceProviderHelperImpl

2012-05-31T09:11:32.240-04:00 [7FFFF3B09700 info 'QueryServiceProvider'] Starting with generation number: 5230290

2012-05-31T09:11:32.240-04:00 [7FFFF3B09700 info 'Default'] [Init:863] Calling: ServiceDirectory_Init()

2012-05-31T09:11:32.240-04:00 [7FFFF3B09700 info 'Default'] [ServiceDirectory] Local instance UUID: DB25B17D-3703-4779-BAFB-845B64AC33AA

2012-05-31T09:11:32.245-04:00 [7FFFF3B09700 warning 'VpxProfiler'] ServerApp::Init [TotalTime] took 2578 ms

2012-05-31T09:11:32.246-04:00 [7FFFF3B09700 error 'Default'] [VpxdMain] Failed to initialize: Name or service not known

2012-05-31T09:11:32.246-04:00 [7FFFF3B09700 error 'Default'] Failed to intialize VMware VirtualCenter. Shutting down...

2012-05-31T09:11:32.246-04:00 [7FFFF3B09700 info 'Default'] Wrote uptime information

2012-05-31T09:12:32.701-04:00 [7FFFF3B09700 info 'Default'] Forcing shutdown of VMware VirtualCenter now

and the process itself

vsphere-fmrif:~ # service vmware-vpxd status
vmware-vpxd is stopped
tomcat is stopped
vsphere-fmrif:~ # service vmware-vpxd start
Waiting for embedded DB2 database to startup: .success
Cleaning session lock table: success
Verifying EULA acceptance: success
Shutting down ldap-server..done
Starting ldap-server..done
Starting vmware-vpxd: success
Waiting for vpxd to initialize: ....failed

Troy_Clavell · ‎05-31-2012

sometimes this happens if you are using the built in DB and you configure your timezone. the appliance will use UTC and if it is changed the local DB will not know how to access records in the future. Did you change the timezone? If so, you may try importing the appliance again and this time don't change the timezone.

edmandsj · ‎05-31-2012

Well the appliance has worked for the past 4-6 months. I'm not sure what would have changed.

What are my options? Just thinking out loud below

Fix Time zone settings or time settings on exisiting vCenter Appliance

Sync time somehow?

"Reset DB Contents" --> How disruptive is this? What will I lose?

Deploy new vCenter Appliance and Import old embedded database

If all else fails and I need to use VDR to recover my vcenter server (to get the database), what will I need to do to put the database back in place?

edmandsj · ‎05-31-2012

oh, may have found the culprit

vsphere-fmrif:~ # df -h
Filesystem            Size Used Avail Use% Mounted on
/dev/sda3             9.8G 4.2G 5.1G 46% /
devtmpfs              2.0G 104K 2.0G   1% /dev
tmpfs                 2.0G     0 2.0G   0% /dev/shm
/dev/sda1             130M   18M 105M 15% /boot
/dev/sdb1              20G   20G     0 100% /storage/core
/dev/sdb2              20G 892M   18G   5% /storage/log
/dev/sdb3              20G 2.7G   17G 14% /storage/db

http://vmwire.com/category/vcsa-2/

edmandsj · ‎05-31-2012

So I cleared out the /storage/core mount point. It quickly fills up again, though. It's already at 25% again. Either way I still cannot login. It is most likely the time thing. If I change it back to the original time(not sure what it was at this point) what will happen?

If you think rebuilding a new VCSA VM is the best possible way, just say so.

side note: backup database from /storage/db ?

Troy_Clavell · ‎05-31-2012

a fresh start is always an option, but you'll have to add all your ESX Hosts back in, create any new Clusters, Resource Pools, etc. My initial response was thinking you were just bringing the appliance on-line for the first time.

Since this environment has been online for quite sometime, it may be worth your while to contact VMware Support to see if they can help in fixing the existing issue.

roddam · ‎11-12-2012

Hi,

Did you manage to resolve this without a fresh install? A month ago my friend came along and suggested that I should change the time to UTC, at that time vCenter and vpxd were working fine. I changed it. Now I am doing negative tests in my environment, and one of the tests was to restart the vpxd service and it is failing with the same error as you saw.

Even rebooting the vCenter vApp does not help. I get the same error failied to initialize vpxd.

Let me know how you resolved this issue. I can initialize the contents of the DB, but what I dont want to lose is the customized alarms and some defualt alarms that I modified.

Thanks!

edmandsj · ‎11-12-2012

Fresh install is the only remedy from what I could tell. Take it as an opportunity to check your settings and so on. Sorry man.

--

Justin Edmands

roddam · ‎11-12-2012

Thanks!

But this is going to be such a pain. I guess I will try a few things before I did that.

Yup, I will try to see the positive side of it! will be hard though

lubela · ‎03-04-2013

fresh install only option. vmware support takes 3 days to even come up with anything.. even for a sev1.

aSmoot · ‎04-15-2013

My lab instance, a vSphere 5.1 vCenter Server Appliance with 2 clusters, 8 hosts (all vSphere 5.1), 80 VMs, exhibits these exact same symptoms. It is using the built in database, no external. I can say for certain that the VCSA time zone has not been touched since initial install, so that is not the issue. My issues started a couple of weeks ago or more. The vSphere Client would just crash every 15 minutes or so, but then I could log back in so I just did that and worked on other more pressing matters, the lab is important, but it would have to wait. It progessed to much worse this week and vSphere client or Web Client not being able to stay logged for more than a couple of seconds. I haven't had time to deal with this until now, so here we go.

You know you are in trouble when on the VCSA management page, vCenter Server > Summary > Storage > Coredumps = 100%! The core dumps or "coredumps" are stored on the VCSA in /storage/core with files named core.vpxd-worker.xxxx where 'x' is a number. I emptied that via SSH with the rm command, but of course it just started filling up again with the same files. Time to break out the toolbox.

After 20 minutes of research and an easy fix, I made a change that has let me login and perform operations in the vSphere Client. So far, so good, no more core dump files 2 hours later. It may be best to perform a basic, surface-level health check (or use LucD's) before rebuilding a VCSA from scratch. That can be a lot of work and will have consequences for VDS, Clusters, policies, etc. Did you document everything?

These are the steps and actions I took to get the lab back online:

From the VCSA management page, stop and started vCenter and Inventory services, but no effect.

From the VCSA management page, reboot the appliance, but no effect.

Stopped vCenter Server and Inventory Services to keep them off during troubleshooting

Logged into each host with vSphere Client first to check all the basics, DNS name, time, verify that IPV6 is disabled (I had problems with that enabled before even though it wasn't in use), etc.

Then (still in vSphere client) check the host logs Home > Administration > System Logs for odd patterns or excessive errors in any of the logs (there are several, click the drop-down). The first 4 hosts I check all looked normal, but one was having what looked like an excessive amount of errors related to snapshot delta disks from snapshots. The problem rang a bell, there was a VM I noticed I had trouble with recently that has a bunch of snapshots and delta disks from some failed backups. I didn't have time to look into it a month or so when I first saw that (it is lab and we never have the time do we?).

The VM was already shut down, so I removed it from the host inventory, then SSH'd to the host and ran /sbin/services.sh restart to restart the management agents --but you can also use the DCUI to accomplish same.

Started vCenter Server and Inventory Services and voila, I could log in and manage my lab again.

Side notes:

I got this message on all of the VMs that were running on the host. The VMs never went down, but HA was attempted:

vSphere HA virtual machine failover failed

I have to work on the snapshot problem which I think was caused during backups by the VMware Data Recovery Appliance, but I don't know the cause.

A number of different issues could be causing this behavior with your vCenter services, but if you can isolate it down to a particular host or even VM as was my case, you can save yourself a lot of trouble. Always have a plan or you may need your resume. Good luck!

Cheers!

Message was edited by: aSmoot

doubleH · ‎04-15-2013

Thank you for your message. I am currently out of the office and will return Apr 16, 2013. I will respond upon my return.

For support related issues please contact the IT Service Desk servicedesk@camhydro.com or x2700.

Thank you

Heath

If you found this or any other post helpful please consider the use of the Helpfull/Correct buttons to award points

Lynn123 · ‎10-31-2014

Hi there,

I have exact same problem on the linux- based vCenter server appliance 5.5 installed with OVA on the ESXi host 5.5.

I used:

Name: VMware-vCenter-Server-Appliance-5.5.0.20200-2183109_OVF10.ova

Release Date: 2014-10-09

Build Number: 2183109

All services can start except the vCenter. It always says :

Starting vmware-vpxd: success

"Waiting for vpxd to initialized: .failed'.

I have checked the time on ESXi host and the vCenter and they matched since both the ESXi host and vCenter sync to same ntp server.

My DataBase is embedded and SSO is also embedded. "Test Settings" is successful.

in /var/log/messages:

2014-10-31T21:16:19+00:00 localhost vami-sfcbd: pam_succeed_if(vami-sfcb:auth): requirement "uid eq 0" was met by user "root"

2014-10-31T21:16:23+00:00 localhost lsassd[3997]: 0x7f57668fa700:Failed to run provider specific request (request code = 12, provider = 'lsa-activedirectory-provider') -> error = 2692, symbol = NERR_SetupNotJoined, client pid = 1815

2014-10-31T21:16:23+00:00 localhost lwsmd: Stopping service: eventlog

2014-10-31T21:16:24+00:00 localhost lwsmd: Starting service: eventlog

2014-10-31T21:16:24+00:00 localhost lsassd[3997]: 0x7f57668fa700:Failed to run provider specific request (request code = 12, provider = 'lsa-activedirectory-provider') -> error = 2692, symbol = NERR_SetupNotJoined, client pid = 1967

2014-10-31T21:16:50+00:00 localhost lsassd[3997]: 0x7f57668fa700:Failed to run provider specific request (request code = 12, provider = 'lsa-activedirectory-provider') -> error = 2692, symbol = NERR_SetupNotJoined, client pid = 2305

2014-10-31T21:16:53+00:00 localhost lsassd[3997]: 0x7f57668fa700:Failed to run provider specific request (request code = 12, provider = 'lsa-activedirectory-provider') -> error = 2692, symbol = NERR_SetupNotJoined, client pid = 2609

2014-10-31T21:16:53+00:00 localhost lwsmd: Stopping service: eventlog

2014-10-31T21:17:07+00:00 localhost watchdog-vpxd: [3080] Begin '/usr/sbin/vpxd', min-uptime = 30, max-quick-failures = 5, max-total-failures = 1000000, bg_pid_file = ''

2014-10-31T21:17:07+00:00 localhost watchdog-vpxd: Executing '/usr/sbin/vpxd'

2014-10-31T21:17:07+00:00 localhost watchdog-vpxd: '/usr/sbin/vpxd' exited after 0 seconds (quick failure 1) 134

2014-10-31T21:17:07+00:00 localhost watchdog-vpxd: Executing '/usr/sbin/vpxd'

2014-10-31T21:17:07+00:00 localhost watchdog-vpxd: '/usr/sbin/vpxd' exited after 0 seconds (quick failure 2) 134

2014-10-31T21:17:07+00:00 localhost watchdog-vpxd: Executing '/usr/sbin/vpxd'

2014-10-31T21:17:08+00:00 localhost watchdog-vpxd: '/usr/sbin/vpxd' exited after 1 seconds (quick failure 3) 134

2014-10-31T21:17:08+00:00 localhost watchdog-vpxd: Executing '/usr/sbin/vpxd'

2014-10-31T21:17:08+00:00 localhost watchdog-vpxd: '/usr/sbin/vpxd' exited after 0 seconds (quick failure 4) 134

2014-10-31T21:17:08+00:00 localhost watchdog-vpxd: Executing '/usr/sbin/vpxd'

2014-10-31T21:17:08+00:00 localhost watchdog-vpxd: '/usr/sbin/vpxd' exited after 0 seconds (quick failure 5) 134

2014-10-31T21:17:08+00:00 localhost watchdog-vpxd: Executing '/usr/sbin/vpxd'

2014-10-31T21:17:09+00:00 localhost watchdog-vpxd: '/usr/sbin/vpxd' exited after 1 seconds (quick failure 6) 134

2014-10-31T21:17:09+00:00 localhost watchdog-vpxd: End '/usr/sbin/vpxd', failure limit reached

2014-10-31T21:19:19+00:00 localhost syslog-ng[1289]: Log statistics; dropped='pipe(/dev/xconsole)=0', dropped='pipe(/dev/tty10)=0', processed='center(queued)=1704', processed='center(received)=1169', processed='destination(messages)=469', processed='destination(mailinfo)=2', processed='destination(mailwarn)=4', processed='destination(authlog)=38', processed='destination(localmessages)=10', processed='destination(cron)=25', processed='destination(mailerr)=0', processed='destination(ldapmessages)=694', processed='destination(netmgm)=0', processed='destination(warn)=156', processed='destination(console)=150', processed='destination(mail)=6', processed='destination(null)=0', processed='destination(xconsole)=150', processed='destination(firewall)=0', processed='destination(acpid)=0', processed='source(src)=1169'

2014-10-31T21:20:01+00:00 localhost /usr/sbin/cron[3432]: (root) CMD ([ -x /usr/lib64/sa/sa1 ] && exec /usr/lib64/sa/sa1 -S ALL 1 1)

2014-10-31T21:20:01+00:00 localhost /usr/sbin/cron[3435]: (root) CMD ( test -x /usr/sbin/vpxd_periodic && /usr/sbin/vpxd_periodic >/dev/null 2>&1)

2014-10-31T21:30:01+00:00 localhost /usr/sbin/cron[3733]: (root) CMD ( test -x /usr/sbin/vpxd_periodic && /usr/sbin/vpxd_periodic >/dev/null 2>&1)

2014-10-31T21:30:01+00:00 localhost /usr/sbin/cron[3735]: (root) CMD ([ -x /usr/lib64/sa/sa1 ] && exec /usr/lib64/sa/sa1 -S ALL 1 1)

I followed instruction on VMware KB: vpxd process on the vCenter Server Appliance 5.x fails with the error: Signal 11 received

and modified /etc/vmware-vpx/vpx.cfg file as sugggested by adding the ThreadStackSizedKb line. See below.

However, it doesn't work.

Anyone knows about this problem please help.

bhanekom · ‎11-20-2014

I had an issue with the vpxd failing to initialize (vpxd service started succesfully), and found that there was a message in the var/log/vmware/vpx/vpxd.log that it was unable to open the file var/log/vmware/journal/1414758110.0.lck - no such file exists. I looked in the location and a directory of that name had been created. Once I removed this directory and restarted the vpxd service I got back my vCenter server and was able to log in using both the vsphere client and web client. I have not seen anyone mention this error in their logs posted - but this resolved the service failing to initialise issue on my setup. I am on ESXi 5.5 with a Single vCenter and 3 hosts

brokentwig · ‎09-22-2015

I had the same issue and it turned out the /storage/db drive was out of space. I followed WMware's article: VMware KB: Increasing the disk space in vCenter Server Appliance 5.x and I'm good.

Shneider82 · ‎02-01-2016

you must increase space of hard drive from the vsphere console

emorillo · ‎05-02-2017

My datastore ran out of space, the vcenter froze, I cleared space on the datastore, resumed the vcsa but it was unresponsive.

Restarted vcenter and got the "Waiting for vpxd to initialize failed".

What solved it was running these instructions I found here:

From: http://www.colabug.com/thread-1093735-1-1.html

To get over this, log on to the VCVA using SSH

Run the command “ /usr/sbin/vpxd_odbcconfig ”

You see output similar to:

VMware vCenter Server Appliance ODBC Configuration

Choose a database to use with VMware vCenter Server

1) oracle

2) embedded

Enter 2 (should the DB be the embedded PostgresSQL

You see output similar to:

Configuring ODBC for embedded

Operation was successful.

Database already populated with version VirtualCenter Database 5.5

Reinitialize database? (WARNING: WILL DELETE ALL RECORDS) [y(es)|N(O)]: OK, preserving the database. (select No to preserve the database)

Restart vCenter Server now to activate the database changes [y/N]:y

Successfully restarted vCenter Server

This should start the vpxd service

Here's my output:

Starting vmware-vpxd: success

Waiting for vpxd to initialize: ............................................................failed

vcsa:~ # /usr/sbin/vpxd_odbcconfig

WARNING - Use of this command is not supported unless you are doing this

with the consultation of VMware Technical Support or a VMware published

technical article.

VMware vCenter Server Appliance ODBC Configuration

Choose a database to use with VMware vCenter Server

1) oracle

2) embedded

#? 2

Configuring ODBC for embedded

Error: VPXD must be stopped to perform this operation.

Database configuration has failed.

Would you like to try again or modify the settings manually?

[A]gain/[m]anually : m

vcsa:~ # service vmware-vpxd stop

Stopping tomcat: success

Stopping vmware-vpxd: success

Shutting down ldap-server..done

vcsa:~ # /usr/sbin/vpxd_odbcconfig

WARNING - Use of this command is not supported unless you are doing this

with the consultation of VMware Technical Support or a VMware published

technical article.

VMware vCenter Server Appliance ODBC Configuration

Choose a database to use with VMware vCenter Server

1) oracle

2) embedded

#? 2

Configuring ODBC for embedded

Operation was successful.

Database already populated with version VirtualCenter Database 5.5

Reinitialize database? (WARNING: WILL DELETE ALL RECORDS) [y(es)|N(O)]: n

OK, preserving the database.

Restart vCenter Server now to activate the database changes [y/N]:y

Successfully restarted vCenter Server

vcsa:~ #

Success!!

rinku02bansal · ‎06-05-2017

We have setup VMware appliance in our setup but after changing some configuration in inventory.cfg , Vmware-vpxd services will not starting even after reverting the changes. Attaching logs below. kindly help us ASAP.

Waiting for the embedded database to start up: success

Verifying EULA acceptance: success

Executing pre-startup scripts...

Updating the vCenter endpoint in the Lookup Service.

Intializing registration provider...

Getting SSL certificates for https://192.168.20.100:7444/lookupservice/sdk

Service with name 'vpxd-localhost.localdom-543843fd-d8da-4c7f-9bd7-8acd95edeb60' and ID 'local:9c27a9ee-0ca0-42e7-8767-7704f847d240' was updated.

Return code is: Success

Shutting down ldap-server..done

Starting ldap-server..done

Starting vmware-vpxd: success

Waiting for vpxd to initialize: .failed <-- this is going to be fail every time.

localhost:~ #

On System log , We will get :-

localhost:~ # cat /var/log/messages

2017-06-05T10:13:18+00:00 localhost watchdog-vpxd: PID file /var/run/vmware/watchdog-vpxd.PID does not exist

2017-06-05T10:13:18+00:00 localhost watchdog-vpxd: Unable to terminate watchdog: No running watchdog process for vpxd

2017-06-05T10:13:19+00:00 localhost watchdog-vpxd: PID file /var/run/vmware/watchdog-vpxd.PID does not exist

2017-06-05T10:13:19+00:00 localhost watchdog-vpxd: Unable to terminate watchdog: No running watchdog process for vpxd

2017-06-05T10:13:27+00:00 localhost watchdog-vpxd: [12418] Begin '/usr/sbin/vpxd', min-uptime = 30, max-quick-failures = 5, max-total-failures = 1000000, bg_pid_file = ''

2017-06-05T10:13:27+00:00 localhost watchdog-vpxd: Executing '/usr/sbin/vpxd'

2017-06-05T10:13:27+00:00 localhost watchdog-vpxd: '/usr/sbin/vpxd' exited after 0 seconds (quick failure 1) 1

2017-06-05T10:13:27+00:00 localhost watchdog-vpxd: Executing '/usr/sbin/vpxd'

2017-06-05T10:13:28+00:00 localhost watchdog-vpxd: '/usr/sbin/vpxd' exited after 1 seconds (quick failure 2) 1

2017-06-05T10:13:28+00:00 localhost watchdog-vpxd: Executing '/usr/sbin/vpxd'

2017-06-05T10:13:28+00:00 localhost watchdog-vpxd: '/usr/sbin/vpxd' exited after 0 seconds (quick failure 3) 1

2017-06-05T10:13:28+00:00 localhost watchdog-vpxd: Executing '/usr/sbin/vpxd'

2017-06-05T10:13:28+00:00 localhost watchdog-vpxd: '/usr/sbin/vpxd' exited after 0 seconds (quick failure 4) 1

2017-06-05T10:13:28+00:00 localhost watchdog-vpxd: Executing '/usr/sbin/vpxd'

2017-06-05T10:13:29+00:00 localhost watchdog-vpxd: '/usr/sbin/vpxd' exited after 1 seconds (quick failure 5) 1

2017-06-05T10:13:29+00:00 localhost watchdog-vpxd: Executing '/usr/sbin/vpxd'

2017-06-05T10:13:29+00:00 localhost watchdog-vpxd: '/usr/sbin/vpxd' exited after 0 seconds (quick failure 6) 1

2017-06-05T10:13:29+00:00 localhost watchdog-vpxd: End '/usr/sbin/vpxd', failure limit reached

2017-06-05T10:14:13+00:00 localhost watchdog-vpxd: PID file /var/run/vmware/watchdog-vpxd.PID does not exist

2017-06-05T10:14:13+00:00 localhost watchdog-vpxd: Unable to terminate watchdog: No running watchdog process for vpxd

2017-06-05T10:14:14+00:00 localhost watchdog-vpxd: PID file /var/run/vmware/watchdog-vpxd.PID does not exist

2017-06-05T10:14:14+00:00 localhost watchdog-vpxd: Unable to terminate watchdog: No running watchdog process for vpxd

2017-06-05T10:15:56+00:00 localhost watchdog-vpxd: [4936] Begin '/usr/sbin/vpxd', min-uptime = 30, max-quick-failures = 5, max-total-failures = 1000000, bg_pid_file = ''

2017-06-05T10:15:56+00:00 localhost watchdog-vpxd: Executing '/usr/sbin/vpxd'

2017-06-05T10:15:56+00:00 localhost kernel: [ 82.191937] warning: process `vpxd' used the deprecated sysctl system call with 1.1.

2017-06-05T10:15:56+00:00 localhost watchdog-vpxd: '/usr/sbin/vpxd' exited after 0 seconds (quick failure 1) 1

2017-06-05T10:15:56+00:00 localhost watchdog-vpxd: Executing '/usr/sbin/vpxd'

2017-06-05T10:15:57+00:00 localhost watchdog-vpxd: '/usr/sbin/vpxd' exited after 1 seconds (quick failure 2) 1

2017-06-05T10:15:57+00:00 localhost watchdog-vpxd: Executing '/usr/sbin/vpxd'

2017-06-05T10:15:57+00:00 localhost watchdog-vpxd: '/usr/sbin/vpxd' exited after 0 seconds (quick failure 3) 1

2017-06-05T10:15:57+00:00 localhost watchdog-vpxd: Executing '/usr/sbin/vpxd'

2017-06-05T10:15:57+00:00 localhost watchdog-vpxd: '/usr/sbin/vpxd' exited after 0 seconds (quick failure 4) 1

2017-06-05T10:15:57+00:00 localhost watchdog-vpxd: Executing '/usr/sbin/vpxd'

2017-06-05T10:15:58+00:00 localhost watchdog-vpxd: '/usr/sbin/vpxd' exited after 1 seconds (quick failure 5) 1

2017-06-05T10:15:58+00:00 localhost watchdog-vpxd: Executing '/usr/sbin/vpxd'

2017-06-05T10:15:58+00:00 localhost watchdog-vpxd: '/usr/sbin/vpxd' exited after 0 seconds (quick failure 6) 1

2017-06-05T10:15:58+00:00 localhost watchdog-vpxd: End '/usr/sbin/vpxd', failure limit reached

2017-06-05T10:20:01+00:00 localhost /usr/sbin/cron[5640]: (root) CMD ( test -x /usr/sbin/vpxd_periodic && /usr/sbin/vpxd_periodic >/dev/null 2>&1)

On vpxd.log, I also get some error.

localhost:/var/log/vmware/vpx # tailf -n 100 vpxd-171.log

Section for VMware VirtualCenter, pid=27456, version=5.5.0, build=2001466, option=Release

------ In-memory logs start --------

mem> 2017-06-06T04:14:20.657Z [7F9131494740 verbose 'vpxdvpxdSignal'] Signal handler for signal 1 is installed

mem> 2017-06-06T04:14:20.657Z [7F9131494740 verbose 'vpxdvpxdSignal'] Signal handler for signal 2 is installed

mem> 2017-06-06T04:14:20.657Z [7F9131494740 verbose 'vpxdvpxdSignal'] Signal handler for signal 15 is installed

mem> 2017-06-06T04:14:20.657Z [7F9131494740 verbose 'vpxdvpxdSignal'] Signal handler for signal 10 is installed

mem> 2017-06-06T04:14:20.657Z [7F9131494740 info 'Hooks'] Hooks Initialized

------ In-memory logs end --------

2017-06-06T04:14:20.660Z [7F9131494740 info 'Default'] Logging uses fast path: true

2017-06-06T04:14:20.660Z [7F9131494740 info 'Default'] Handling bora/lib logs with VmaCore facilities

2017-06-06T04:14:20.660Z [7F9131494740 info 'Default'] Initialized channel manager

2017-06-06T04:14:20.663Z [7F9131494740 info 'Default'] Current working directory: /storage/log/vmware/vpx

2017-06-06T04:14:20.663Z [7F9131494740 info 'ThreadPool'] Thread pool on asio: Min Io, Max Io, Min Task, Max Task, Max Concurency: 2, 200, 2, 90, 2147483647

2017-06-06T04:14:20.663Z [7F9131494740 info 'ThreadPool'] Thread enlisted

2017-06-06T04:14:20.663Z [7F9131494740 info 'Default'] Log path: /var/log/vmware/vpx

2017-06-06T04:14:20.663Z [7F9131494740 info 'Default'] Initializing SSL

2017-06-06T04:14:20.663Z [7F91278A4700 info 'ThreadPool'] Thread enlisted

2017-06-06T04:14:20.663Z [7F91279A6700 info 'ThreadPool'] Thread enlisted

2017-06-06T04:14:20.663Z [7F9127823700 info 'ThreadPool'] Thread enlisted

2017-06-06T04:14:20.663Z [7F9127925700 info 'ThreadPool'] Thread enlisted

2017-06-06T04:14:20.664Z [7F9131494740 info 'Default'] Vmacore::InitSSL: handshakeTimeoutUs = 120000000

2017-06-06T04:14:20.665Z [7F91277A2700 info 'ThreadPool'] Thread enlisted

2017-06-06T04:14:20.665Z [7F9131494740 info 'vpxdvpxdDaemon'] Changed working directory to /var/log/vmware/vpx

2017-06-06T04:14:20.666Z [7F9131494740 error 'vpxdvpxdDaemon'] Unhandled exception:

--> not well-formed (invalid token)

2017-06-06T04:14:20.667Z [7F9131494740 error 'vpxdvpxdDaemon'] Backtrace:

--> backtrace[00] rip 00007f912b3f9014 Vmacore::System::Stacktrace::CaptureWork(unsigned int)

--> backtrace[01] rip 00007f912b2e3132 Vmacore::System::SystemFactoryImpl::CreateQuickBacktrace(Vmacore::Ref<Vmacore::System::Backtrace>&)

--> backtrace[02] rip 00007f912b2448a5 Vmacore::Throwable::Throwable(std::string const&)

--> backtrace[03] rip 00007f912b28b566 /usr/lib/vmware-vpx/libvmacore.so(+0x285566) [0x7f912b28b566]

--> backtrace[04] rip 00007f912b23e85d /usr/lib/vmware-vpx/libvmacore.so(+0x23885d) [0x7f912b23e85d]

--> backtrace[05] rip 00007f9132abfd07 /usr/lib/vmware-vpx/vpxd(+0x14aad07) [0x7f9132abfd07]

--> backtrace[06] rip 00007f9132a3f0c5 /usr/lib/vmware-vpx/vpxd(+0x142a0c5) [0x7f9132a3f0c5]

--> backtrace[07] rip 00007f91324c3ca3 /usr/lib/vmware-vpx/vpxd(+0xeaeca3) [0x7f91324c3ca3]

--> backtrace[08] rip 00007f91324a6c2e /usr/lib/vmware-vpx/vpxd(+0xe91c2e) [0x7f91324a6c2e]

--> backtrace[09] rip 00007f9129369c16 /lib64/libc.so.6(__libc_start_main+0xe6) [0x7f9129369c16]

--> backtrace[10] rip 00007f91324a5e89 /usr/lib/vmware-vpx/vpxd(+0xe90e89) [0x7f91324a5e89]

I have gone through the above solutions for storage space full but i checked on our setup its not there.

localhost:/var/log/vmware/vpx # df -h

Filesystem Size Used Avail Use% Mounted on

/dev/sda3 9.8G 3.7G 5.6G 40% /

udev 4.0G 104K 4.0G 1% /dev

tmpfs 4.0G 0 4.0G 0% /dev/shm

/dev/sda1 128M 21M 101M 18% /boot

/dev/sdb1 20G 173M 19G 1% /storage/core

/dev/sdb2 20G 305M 19G 2% /storage/log

/dev/sdb3 60G 1.7G 55G 4% /storage/db

localhost:/var/log/vmware/vpx # df -ih

Filesystem Inodes IUsed IFree IUse% Mounted on

/dev/sda3 632K 60K 573K 10% /

udev 0 0 0 - /dev

tmpfs 1001K 2 1001K 1% /dev/shm

/dev/sda1 34K 40 34K 1% /boot

/dev/sdb1 1.3M 12 1.3M 1% /storage/core

/dev/sdb2 1.3M 293 1.3M 1% /storage/log

/dev/sdb3 3.8M 3.7K 3.8M 1% /storage/db

localhost:/var/log/vmware/vpx #

citylights · ‎02-24-2018

This absolutely worked for me. Thank you!

In my case, I do not believe I had ran out of space first (df -h was reporting fine).

Smerchla · ‎12-07-2018

Hello I have same problem

after restarting I cannot connect to vcenter. This is a my log file. Please Help solve this problem

Section for VMware VirtualCenter, pid=4819, version=5.5.0, build=2001466, option=Release

------ In-memory logs start --------

mem> 2018-12-07T08:20:43.272Z [7FA96E132740 verbose 'vpxdvpxdSignal'] Signal handler for signal 1 is installed

mem> 2018-12-07T08:20:43.272Z [7FA96E132740 verbose 'vpxdvpxdSignal'] Signal handler for signal 2 is installed

mem> 2018-12-07T08:20:43.272Z [7FA96E132740 verbose 'vpxdvpxdSignal'] Signal handler for signal 15 is installed

mem> 2018-12-07T08:20:43.272Z [7FA96E132740 verbose 'vpxdvpxdSignal'] Signal handler for signal 10 is installed

mem> 2018-12-07T08:20:43.272Z [7FA96E132740 info 'Hooks'] Hooks Initialized

------ In-memory logs end --------

2018-12-07T08:20:43.281Z [7FA96E132740 info 'Default'] Logging uses fast path: true

2018-12-07T08:20:43.281Z [7FA96E132740 info 'Default'] Handling bora/lib logs with VmaCore facilities

2018-12-07T08:20:43.281Z [7FA96E132740 info 'Default'] Initialized channel manager

2018-12-07T08:20:43.288Z [7FA96E132740 info 'Default'] Current working directory: /storage/log/vmware/vpx

2018-12-07T08:20:43.288Z [7FA964685700 info 'ThreadPool'] Thread enlisted

2018-12-07T08:20:43.288Z [7FA964502700 info 'ThreadPool'] Thread enlisted

2018-12-07T08:20:43.288Z [7FA96E132740 info 'ThreadPool'] Thread pool on asio: Min Io, Max Io, Min Task, Max Task, Max Concurency: 2, 200, 2, 90, 2147483647

2018-12-07T08:20:43.288Z [7FA96E132740 info 'ThreadPool'] Thread enlisted

2018-12-07T08:20:43.288Z [7FA964604700 info 'ThreadPool'] Thread enlisted

2018-12-07T08:20:43.288Z [7FA964583700 info 'ThreadPool'] Thread enlisted

2018-12-07T08:20:43.288Z [7FA96E132740 info 'Default'] Log path: /var/log/vmware/vpx

2018-12-07T08:20:43.288Z [7FA96E132740 info 'Default'] Initializing SSL

2018-12-07T08:20:43.291Z [7FA96E132740 info 'Default'] Vmacore::InitSSL: handshakeTimeoutUs = 120000000

2018-12-07T08:20:43.292Z [7FA964481700 info 'ThreadPool'] Thread enlisted

2018-12-07T08:20:43.293Z [7FA96E132740 info 'vpxdvpxdDaemon'] Changed working directory to /var/log/vmware/vpx

2018-12-07T08:20:43.306Z [7FA96E132740 info 'Default'] Starting VMware VirtualCenter 5.5.0 build-2001466

2018-12-07T08:20:43.306Z [7FA96E132740 info 'Default'] Log directory: /var/log/vmware/vpx.

2018-12-07T08:20:43.307Z [7FA96E132740 info 'vpxdvpxdMain'] Account name: root

2018-12-07T08:20:43.307Z [7FA96E132740 info 'commonvpxLro'] [VpxLRO] 640 max LROs

2018-12-07T08:20:43.307Z [7FA96E132740 info 'commonvpxLro'] [VpxLRO] 12 reserved internal LROs

2018-12-07T08:20:43.307Z [7FA96E132740 info 'commonvpxLro'] [VpxLRO] 12 reserved blocker LROs

2018-12-07T08:20:43.307Z [7FA96E132740 info 'commonvpxLro'] [VpxLRO] 12 reserved short LROs

2018-12-07T08:20:43.307Z [7FA96E132740 info 'commonvpxLro'] [VpxLRO] 8 reserved long LROs

2018-12-07T08:20:43.307Z [7FA96E132740 info 'commonvpxLro'] [VpxLRO] 600-second completed task lifetime

2018-12-07T08:20:43.307Z [7FA96E132740 info 'commonvpxLro'] [VpxLRO] 60-second minimum completed task lifetime

2018-12-07T08:20:43.307Z [7FA96E132740 info 'commonvpxLro'] [VpxLRO] 200 maximum completed tasks

2018-12-07T08:20:43.307Z [7FA96E132740 info 'commonvpxLro'] [VpxLRO] 600-second unregistered task lifetime

2018-12-07T08:20:43.308Z [7FA96E132740 info 'utilvpxdVdb'] Registry Item DB 5 value is ''

2018-12-07T08:20:43.308Z [7FA96E132740 info 'utilvpxdVdb'] [VpxdVdb::SetDBType] Logging in to DSN: VMware VirtualCenter with username vc

2018-12-07T08:20:43.477Z [7FA96E132740 info 'Default'] [VdbSchemaLoader::LookupSchemaFromDB] Loaded schema and index information from DB

2018-12-07T08:20:43.494Z [7FA96E132740 info 'utilvpxdDbSchemaHash'] [VpxdDbSchemaHash::GetSchemaFilename]: sqlDir: /usr/lib/vmware-vpx/sql/

2018-12-07T08:20:43.494Z [7FA96E132740 info 'utilvpxdDbSchemaHash'] [VpxdDbSchemaHash::CheckSchema]: schema file: /usr/lib/vmware-vpx/sql/VCDB_PostgreSQL.sql

2018-12-07T08:20:43.508Z [7FA96E132740 info 'linuxvpxLdap_linux'] [LdapBackup] Making sure LDAP instance VMwareVCMSDS is running

2018-12-07T08:20:43.508Z [7FA96E132740 info 'linuxvpxLdap_linux'] [LdapBackup] Checking service ldap for running

2018-12-07T08:20:43.533Z [7FA96E132740 info 'Default'] Looking for vpx result filter at /usr/lib/vmware-vpx/vpxResultFilter.xml

2018-12-07T08:20:43.538Z [7FA964604700 info 'vpxdvpxdMain' opID=SWI-6058ed8] Pre-loading symbols...

2018-12-07T08:20:43.540Z [7FA964604700 info 'vpxdvpxdMain' opID=SWI-6058ed8] Done pre-loading symbols.

2018-12-07T08:20:43.541Z [7FA96E132740 info 'MoLock'] LRO tracing: enabled: false

2018-12-07T08:20:43.541Z [7FA96E132740 info 'MoLock'] LRO tracing: LRO name:

2018-12-07T08:20:43.556Z [7FA96E132740 info 'Default'] Looking for vpx result filter at /usr/lib/vmware-vpx/vpxResultFilter.xml

2018-12-07T08:20:43.561Z [7FA96E132740 info 'QueryServiceProvider'] Initializing QueryServiceProviderHelperImpl

2018-12-07T08:20:43.605Z [7FA96E132740 info 'QueryServiceProvider'] Starting with generation number: 7085462

2018-12-07T08:20:43.605Z [7FA96E132740 info 'vpxservicesMoServiceDirectory'] [ServiceDirectory] Local instance UUID: 0FCF384F-1EEA-4A00-9424-673FC0F7F363

2018-12-07T08:20:43.608Z [7FA96E132740 error 'linuxvpxLdap_linux'] [LDAP Client] Failed to modify LDAP entry cn=VIMAPI,cn=0FCF384F-1EEA-4A00-9424-673FC0F7F363,ou=Instances,dc=virtualcenter,dc=vmware,dc=int: 0x1 (Operations error)

2018-12-07T08:20:43.608Z [7FA96E132740 error 'linuxvpxLdap_linux'] [LDAP Client] LdapApi::modify_s: key (vmw-vc-SSLThumbprint)

2018-12-07T08:20:43.608Z [7FA96E132740 error 'vpxservicesMoServiceDirectory'] [ServiceDirectory::UpdateServiceInfo] Error while modifying service-endpoint SSL thumbprint cn=VIMAPI,cn=0FCF384F-1EEA-4A00-9424-673FC0F7F363,ou=Instances,dc=virtualcenter,dc=vmware,dc=int

2018-12-07T08:20:43.608Z [7FA96E132740 error 'vpxdvpxdMain'] [Init] Init failed: Ldap error

--> Backtrace:

--> backtrace[00] rip 00007fa968097014 Vmacore::System::Stacktrace::CaptureWork(unsigned int)

--> backtrace[01] rip 00007fa967f81132 Vmacore::System::SystemFactoryImpl::CreateQuickBacktrace(Vmacore::Ref<Vmacore::System::Backtrace>&)

--> backtrace[02] rip 00007fa967ee28a5 Vmacore::Throwable::Throwable(std::string const&)

--> backtrace[03] rip 00007fa96f16331f Vmomi::Fault::SystemError::Exception::Exception(std::string const&)

--> backtrace[04] rip 00007fa97060c60b /usr/lib/vmware-vpx/vpxd(+0x235960b) [0x7fa97060c60b]

--> backtrace[05] rip 00007fa97060f0fd /usr/lib/vmware-vpx/vpxd(+0x235c0fd) [0x7fa97060f0fd]

--> backtrace[06] rip 00007fa97060f645 /usr/lib/vmware-vpx/vpxd(+0x235c645) [0x7fa97060f645]

--> backtrace[07] rip 00007fa96f14daaf /usr/lib/vmware-vpx/vpxd(+0xe9aaaf) [0x7fa96f14daaf]

--> backtrace[08] rip 00007fa96f161d43 /usr/lib/vmware-vpx/vpxd(+0xeaed43) [0x7fa96f161d43]

--> backtrace[09] rip 00007fa96f144c2e /usr/lib/vmware-vpx/vpxd(+0xe91c2e) [0x7fa96f144c2e]

--> backtrace[10] rip 00007fa966007c16 /lib64/libc.so.6(__libc_start_main+0xe6) [0x7fa966007c16]

--> backtrace[11] rip 00007fa96f143e89 /usr/lib/vmware-vpx/vpxd(+0xe90e89) [0x7fa96f143e89]

-->

2018-12-07T08:20:43.612Z [7FA96E132740 error 'Default'] Failed to intialize VMware VirtualCenter. Shutting down...

2018-12-07T08:20:43.612Z [7FA96E132740 info 'vpxdvpxdSupportManager'] Wrote uptime information

All

Cannot Start vCenter Server Appliance - "Waiting for vpxd to initialize: ....failed"