I am stumped. At first, I was able to at least login to vsphere and do operations. The client would close like every 15 minutes so it got really annoying. Now I am stuck with no launch to vsphere. I must log into each host individually. ESXi hosts without a vsphere is so limited. Please advise.
Here is the /var/log/messages
vsphere-fmrif:~ # tail -f /var/log/messages
May 31 09:50:32 vsphere-fmrif watchdog-vpxd: [4773] Begin '/usr/sbin/vpxd', min-uptime = 30, max-quick-failures = 5, max-total-failures = 1000000, bg_pid_file = ''
May 31 09:50:32 vsphere-fmrif watchdog-vpxd: Executing '/usr/sbin/vpxd'
May 31 09:50:34 vsphere-fmrif kernel: [ 59.780777] warning: process `vpxd-worker' used the deprecated sysctl system call with 1.1.
May 31 09:50:40 vsphere-fmrif sshd[4944]: Accepted password for root from 128.231.212.117 port 51160 ssh2
May 31 09:50:44 vsphere-fmrif watchdog-vpxd: '/usr/sbin/vpxd' exited after 12 seconds (quick failure 1) 134
May 31 09:50:44 vsphere-fmrif watchdog-vpxd: Executing '/usr/sbin/vpxd'
May 31 09:50:45 vsphere-fmrif kernel: [ 70.438974] warning: process `vpxd' used the deprecated sysctl system call with 1.1.
May 31 09:50:51 vsphere-fmrif watchdog-vpxd: '/usr/sbin/vpxd' exited after 7 seconds (quick failure 2) 134
May 31 09:50:51 vsphere-fmrif watchdog-vpxd: Executing '/usr/sbin/vpxd'
May 31 09:50:52 vsphere-fmrif kernel: [ 77.370397] warning: process `vpxd' used the deprecated sysctl system call with 1.1.
May 31 09:50:58 vsphere-fmrif watchdog-vpxd: '/usr/sbin/vpxd' exited after 7 seconds (quick failure 3) 134
May 31 09:50:58 vsphere-fmrif watchdog-vpxd: Executing '/usr/sbin/vpxd'
May 31 09:50:58 vsphere-fmrif kernel: [ 83.923261] warning: process `vpxd' used the deprecated sysctl system call with 1.1.
May 31 09:51:05 vsphere-fmrif watchdog-vpxd: '/usr/sbin/vpxd' exited after 7 seconds (quick failure 4) 134
May 31 09:51:05 vsphere-fmrif watchdog-vpxd: Executing '/usr/sbin/vpxd'
May 31 09:51:06 vsphere-fmrif kernel: [ 91.720959] warning: process `vpxd-worker' used the deprecated sysctl system call with 1.1.
May 31 09:51:12 vsphere-fmrif auditd[3012]: Audit daemon rotating log files
May 31 09:51:13 vsphere-fmrif watchdog-vpxd: '/usr/sbin/vpxd' exited after 8 seconds (quick failure 5) 134
May 31 09:51:13 vsphere-fmrif watchdog-vpxd: Executing '/usr/sbin/vpxd'
May 31 09:51:19 vsphere-fmrif watchdog-vpxd: '/usr/sbin/vpxd' exited after 6 seconds (quick failure 6) 134
May 31 09:51:19 vsphere-fmrif watchdog-vpxd: End '/usr/sbin/vpxd', failure limit reached
/var/log/warn
May 31 02:40:23 vsphere-fmrif watchdog-vpxd: '/usr/sbin/vpxd' exited after 62 seconds 1
May 31 02:41:26 vsphere-fmrif watchdog-vpxd: '/usr/sbin/vpxd' exited after 63 seconds 1
May 31 02:42:28 vsphere-fmrif watchdog-vpxd: '/usr/sbin/vpxd' exited after 62 seconds 1
May 31 02:43:31 vsphere-fmrif watchdog-vpxd: '/usr/sbin/vpxd' exited after 63 seconds 1
May 31 02:44:33 vsphere-fmrif watchdog-vpxd: '/usr/sbin/vpxd' exited after 62 seconds 1
May 31 02:45:36 vsphere-fmrif watchdog-vpxd: '/usr/sbin/vpxd' exited after 63 seconds 1
May 31 02:46:38 vsphere-fmrif watchdog-vpxd: '/usr/sbin/vpxd' exited after 62 seconds 1
May 31 02:47:41 vsphere-fmrif watchdog-vpxd: '/usr/sbin/vpxd' exited after 63 seconds 1
May 31 02:48:44 vsphere-fmrif watchdog-vpxd: '/usr/sbin/vpxd' exited after 63 seconds 1
May 31 02:49:46 vsphere-fmrif watchdog-vpxd: '/usr/sbin/vpxd' exited after 62 seconds 1
May 31 02:50:49 vsphere-fmrif watchdog-vpxd: '/usr/sbin/vpxd' exited after 63 seconds 1
May 31 02:51:51 vsphere-fmrif watchdog-vpxd: '/usr/sbin/vpxd' exited after 62 seconds 1
May 31 02:52:54 vsphere-fmrif watchdog-vpxd: '/usr/sbin/vpxd' exited after 63 seconds 1
May 31 02:53:56 vsphere-fmrif watchdog-vpxd: '/usr/sbin/vpxd' exited after 62 seconds 1
May 31 02:54:59 vsphere-fmrif watchdog-vpxd: '/usr/sbin/vpxd' exited after 63 seconds 1
May 31 02:56:01 vsphere-fmrif watchdog-vpxd: '/usr/sbin/vpxd' exited after 62 seconds 1
May 31 02:57:04 vsphere-fmrif watchdog-vpxd: '/usr/sbin/vpxd' exited after 63 seconds 1
May 31 02:58:06 vsphere-fmrif watchdog-vpxd: '/usr/sbin/vpxd' exited after 62 seconds 1
May 31 02:59:09 vsphere-fmrif watchdog-vpxd: '/usr/sbin/vpxd' exited after 63 seconds 1
May 31 03:00:12 vsphere-fmrif watchdog-vpxd: '/usr/sbin/vpxd' exited after 63 seconds 1
May 31 03:01:14 vsphere-fmrif watchdog-vpxd: '/usr/sbin/vpxd' exited after 62 seconds 1
May 31 03:01:52 vsphere-fmrif portmap[25177]: connect from 128.231.212.168 to callit(390109): request from unauthorized host
May 31 03:01:52 vsphere-fmrif portmap[25178]: connect from 128.231.212.177 to callit(390109): request from unauthorized host
May 31 03:01:53 vsphere-fmrif portmap[25179]: connect from 128.231.212.121 to callit(390109): request from unauthorized host
May 31 03:02:17 vsphere-fmrif watchdog-vpxd: '/usr/sbin/vpxd' exited after 63 seconds 1
May 31 03:03:07 vsphere-fmrif portmap[25275]: connect from 128.231.212.168 to callit(390109): request from unauthorized host
May 31 03:03:19 vsphere-fmrif watchdog-vpxd: '/usr/sbin/vpxd' exited after 62 seconds 1
May 31 03:04:04 vsphere-fmrif portmap[25358]: connect from 128.231.212.177 to callit(390109): request from unauthorized host
May 31 03:04:19 vsphere-fmrif portmap[25368]: connect from 128.231.212.121 to callit(390109): request from unauthorized host
May 31 03:04:22 vsphere-fmrif watchdog-vpxd: '/usr/sbin/vpxd' exited after 62 seconds 1
May 31 03:05:25 vsphere-fmrif watchdog-vpxd: '/usr/sbin/vpxd' exited after 63 seconds 1
May 31 03:06:01 vsphere-fmrif portmap[25531]: connect from 128.231.212.168 to callit(390109): request from unauthorized host
May 31 03:06:27 vsphere-fmrif watchdog-vpxd: '/usr/sbin/vpxd' exited after 62 seconds 1
May 31 03:06:31 vsphere-fmrif portmap[25603]: connect from 128.231.212.121 to callit(390109): request from unauthorized host
May 31 03:07:00 vsphere-fmrif portmap[25618]: connect from 128.231.212.177 to callit(390109): request from unauthorized host
May 31 03:07:30 vsphere-fmrif watchdog-vpxd: '/usr/sbin/vpxd' exited after 63 seconds 1
May 31 03:08:32 vsphere-fmrif watchdog-vpxd: '/usr/sbin/vpxd' exited after 62 seconds 1
May 31 03:09:35 vsphere-fmrif watchdog-vpxd: '/usr/sbin/vpxd' exited after 63 seconds 1
May 31 03:10:38 vsphere-fmrif watchdog-vpxd: '/usr/sbin/vpxd' exited after 63 seconds 1
May 31 03:11:40 vsphere-fmrif watchdog-vpxd: '/usr/sbin/vpxd' exited after 62 seconds 1
May 31 03:12:20 vsphere-fmrif portmap[26072]: connect from 128.231.212.177 to callit(390109): request from unauthorized host
May 31 03:12:43 vsphere-fmrif watchdog-vpxd: '/usr/sbin/vpxd' exited after 63 seconds 1
May 31 03:12:54 vsphere-fmrif portmap[26144]: connect from 128.231.212.168 to callit(390109): request from unauthorized host
May 31 03:13:45 vsphere-fmrif watchdog-vpxd: '/usr/sbin/vpxd' exited after 62 seconds 1
May 31 03:13:48 vsphere-fmrif portmap[26227]: connect from 128.231.212.177 to callit(390109): request from unauthorized host
May 31 03:14:17 vsphere-fmrif portmap[26242]: connect from 128.231.212.121 to callit(390109): request from unauthorized host
May 31 03:14:48 vsphere-fmrif watchdog-vpxd: '/usr/sbin/vpxd' exited after 63 seconds 1
May 31 03:15:09 vsphere-fmrif portmap[26348]: connect from 128.231.212.177 to callit(390109): request from unauthorized host
May 31 03:15:17 vsphere-fmrif portmap[26354]: connect from 128.231.212.177 to callit(390109): request from unauthorized host
May 31 03:15:51 vsphere-fmrif watchdog-vpxd: '/usr/sbin/vpxd' exited after 63 seconds 1
May 31 03:16:03 vsphere-fmrif portmap[26435]: connect from 128.231.212.168 to callit(390109): request from unauthorized host
May 31 03:16:53 vsphere-fmrif watchdog-vpxd: '/usr/sbin/vpxd' exited after 62 seconds 1
May 31 03:16:53 vsphere-fmrif portmap[26472]: connect from 128.231.212.177 to callit(390109): request from unauthorized host
May 31 03:16:59 vsphere-fmrif portmap[26519]: connect from 128.231.212.121 to callit(390109): request from unauthorized host
May 31 03:17:56 vsphere-fmrif watchdog-vpxd: '/usr/sbin/vpxd' exited after 63 seconds 1
May 31 03:18:30 vsphere-fmrif portmap[26621]: connect from 128.231.212.121 to callit(390109): request from unauthorized host
May 31 03:18:58 vsphere-fmrif watchdog-vpxd: '/usr/sbin/vpxd' exited after 62 seconds 1
May 31 03:20:01 vsphere-fmrif watchdog-vpxd: '/usr/sbin/vpxd' exited after 63 seconds 1
May 31 03:21:03 vsphere-fmrif watchdog-vpxd: '/usr/sbin/vpxd' exited after 62 seconds 1
May 31 03:21:37 vsphere-fmrif portmap[26897]: connect from 128.231.212.168 to callit(390109): request from unauthorized host
May 31 03:22:06 vsphere-fmrif watchdog-vpxd: '/usr/sbin/vpxd' exited after 63 seconds 1
May 31 03:23:09 vsphere-fmrif watchdog-vpxd: '/usr/sbin/vpxd' exited after 63 seconds 1
May 31 03:24:11 vsphere-fmrif watchdog-vpxd: '/usr/sbin/vpxd' exited after 62 seconds 1
May 31 03:24:22 vsphere-fmrif portmap[27146]: connect from 128.231.212.168 to callit(390109): request from unauthorized host
May 31 03:25:14 vsphere-fmrif watchdog-vpxd: '/usr/sbin/vpxd' exited after 63 seconds 1
May 31 03:26:16 vsphere-fmrif watchdog-vpxd: '/usr/sbin/vpxd' exited after 62 seconds 1
May 31 03:26:40 vsphere-fmrif portmap[27330]: connect from 128.231.212.168 to callit(390109): request from unauthorized host
May 31 03:27:19 vsphere-fmrif watchdog-vpxd: '/usr/sbin/vpxd' exited after 63 seconds 1
May 31 03:28:21 vsphere-fmrif watchdog-vpxd: '/usr/sbin/vpxd' exited after 62 seconds 1
May 31 03:29:24 vsphere-fmrif watchdog-vpxd: '/usr/sbin/vpxd' exited after 63 seconds 1
May 31 03:30:27 vsphere-fmrif watchdog-vpxd: '/usr/sbin/vpxd' exited after 63 seconds 1
May 31 03:31:29 vsphere-fmrif watchdog-vpxd: '/usr/sbin/vpxd' exited after 62 seconds 1
May 31 03:32:32 vsphere-fmrif watchdog-vpxd: '/usr/sbin/vpxd' exited after 63 seconds 1
May 31 03:33:35 vsphere-fmrif watchdog-vpxd: '/usr/sbin/vpxd' exited after 63 seconds 1
May 31 03:34:37 vsphere-fmrif watchdog-vpxd: '/usr/sbin/vpxd' exited after 62 seconds 1
May 31 03:35:40 vsphere-fmrif watchdog-vpxd: '/usr/sbin/vpxd' exited after 63 seconds 1
/var/log/vmware/vpx/vpxd.log
From earlier:
vsphere-fmrif:~ # tail -f -n 100 /var/log/vmware/vpx/vpxd.log
Section for VMware VirtualCenter, pid=25488, version=5.0.0, build=build-455964, option=Release
------ In-memory logs start --------
2012-05-31T09:11:29.623-04:00 [7FFFF3B09700 info 'Hooks'] Hooks Initialized
------ In-memory logs end --------
2012-05-31T09:11:29.633-04:00 [7FFFF3B09700 info 'Default'] Initialized channel manager
2012-05-31T09:11:29.641-04:00 [7FFFF3B09700 info 'Default'] Current working directory: /storage/log/vmware/vpx
2012-05-31T09:11:29.642-04:00 [7FFFF3B09700 info 'Default'] Log path: /var/log/vmware/vpx
2012-05-31T09:11:29.642-04:00 [7FFFF3B09700 info 'Default'] Initializing SSL
2012-05-31T09:11:29.645-04:00 [7FFFF3B09700 info 'Default'] Vmacore::InitSSL: doVersionCheck = true, handshakeTimeoutUs = 120000000
2012-05-31T09:11:29.646-04:00 [7FFFF3B09700 info 'Default'] Changed working directory to /var/log/vmware/vpx
2012-05-31T09:11:29.666-04:00 [7FFFF3B09700 info 'Default'] Starting VMware VirtualCenter 5.0.0 build-455964
2012-05-31T09:11:29.666-04:00 [7FFFF3B09700 info 'Default'] Log directory: /var/log/vmware/vpx.
2012-05-31T09:11:29.667-04:00 [7FFFF3B09700 info 'Default'] Account name: root
2012-05-31T09:11:29.667-04:00 [7FFFF3B09700 info 'Default'] [Init:835] Calling: InfoDeclSchema(gDB)
2012-05-31T09:11:29.667-04:00 [7FFFF3B09700 info 'Default'] [Init:837] Calling: VpxCallbackDesc::Init(MakeFunctor(this, &ServerApp::RequestShutdown))
2012-05-31T09:11:29.667-04:00 [7FFFF3B09700 info 'Default'] [Init:838] Calling: VpxCryptInit()
2012-05-31T09:11:29.667-04:00 [7FFFF3B09700 info 'Default'] [Init:839] Calling: VpxLRO::Init()
2012-05-31T09:11:29.667-04:00 [7FFFF3B09700 info 'Default'] [Init:840] Calling: VpxLroList::Init(ltud)
2012-05-31T09:11:29.667-04:00 [7FFFF3B09700 info 'Default'] [VpxLRO] 640 max LROs
2012-05-31T09:11:29.667-04:00 [7FFFF3B09700 info 'Default'] [VpxLRO] 12 reserved internal LROs
2012-05-31T09:11:29.667-04:00 [7FFFF3B09700 info 'Default'] [VpxLRO] 12 reserved blocker LROs
2012-05-31T09:11:29.667-04:00 [7FFFF3B09700 info 'Default'] [VpxLRO] 12 reserved short LROs
2012-05-31T09:11:29.667-04:00 [7FFFF3B09700 info 'Default'] [VpxLRO] 8 reserved long LROs
2012-05-31T09:11:29.667-04:00 [7FFFF3B09700 info 'Default'] [VpxLRO] 600-second completed task lifetime
2012-05-31T09:11:29.667-04:00 [7FFFF3B09700 info 'Default'] [VpxLRO] 60-second minimum completed task lifetime
2012-05-31T09:11:29.667-04:00 [7FFFF3B09700 info 'Default'] [VpxLRO] 200 maximum completed tasks
2012-05-31T09:11:29.667-04:00 [7FFFF3B09700 info 'Default'] [VpxLRO] 600-second unregistered task lifetime
2012-05-31T09:11:29.667-04:00 [7FFFF3B09700 info 'Default'] [Init:841] Calling: VpxdCharacterizeThreadpool(ltud)
2012-05-31T09:11:29.667-04:00 [7FFFF3B09700 info 'ThreadPool'] [SetConfig] Thread info: IoMin: 2, IoMax: 1990, TaskMin: 59, TaskMax: 1990, Max Thread: 3980, Keepalive: 61, thread kill: 600, max fds: -1
2012-05-31T09:11:29.667-04:00 [7FFFF3B09700 info 'Default'] [Init:842] Calling: VpxdCertificate_Load(gDB, CERTIFICATE_VMDBPATH )
2012-05-31T09:11:29.668-04:00 [7FFFF3B09700 info 'Default'] [Init:843] Calling: VpxdVdb::Init(VpxdVdb::GetVcVdbInstId(), false, false, NULL)
2012-05-31T09:11:29.668-04:00 [7FFFF3B09700 info 'Default'] Registry Item DB 5 value is ''
2012-05-31T09:11:32.078-04:00 [7FFFF3B09700 info 'Default'] [VdbSchemaLoader::LookupSchemaFromDB] Loaded schema and index information from DB
2012-05-31T09:11:32.099-04:00 [7FFFF3B09700 info 'Default'] [Init:844] Calling: VpxdDbSchemaHash::CheckSchema()
2012-05-31T09:11:32.100-04:00 [7FFFF3B09700 info 'Default'] [VpxdDbSchemaHash::CheckSchema]: schema file: /usr/lib/vmware-vpx/VCDB_db2.sql
2012-05-31T09:11:32.104-04:00 [7FFFF3B09700 info 'Default'] [Init:845] Calling: VpxdDbMonitoring::Init()
2012-05-31T09:11:32.120-04:00 [7FFFF3B09700 info 'Default'] [Init:848] Calling: VpxdLdap::Init()
2012-05-31T09:11:32.121-04:00 [7FFFF3B09700 info 'Default'] [LdapBackup] Making sure LDAP instance VMwareVCMSDS is running
2012-05-31T09:11:32.121-04:00 [7FFFF3B09700 info 'Default'] [LdapBackup] Checking service ldap for running
2012-05-31T09:11:32.179-04:00 [7FFFF3B09700 info 'Default'] [Init:849] Calling: VpxResultFilter::Init()
2012-05-31T09:11:32.179-04:00 [7FFFF3B09700 info 'Default'] Looking for vpx result filter at /usr/lib/vmware-vpx/vpxResultFilter.xml
2012-05-31T09:11:32.184-04:00 [7FFFF3B09700 info 'Default'] [Init:853] Calling: VpxdLock::Init(gDB)
2012-05-31T09:11:32.184-04:00 [7FFFF3B09700 info 'Default'] [Init:854] Calling: VpxdMoLock::Init()
2012-05-31T09:11:32.184-04:00 [7FFFF3B09700 info 'Default'] [Init:855] Calling: VpxdStateLock::Init()
2012-05-31T09:11:32.184-04:00 [7FFFF3B09700 info 'Default'] [Init:856] Calling: VpxdMoDataTable::Init()
2012-05-31T09:11:32.184-04:00 [7FFFF3B09700 info 'Default'] [Init:858] Calling: VpxdHeartbeat::Init(MakeFunctor(this, &ServerApp::RequestShutdown))
2012-05-31T09:11:32.184-04:00 [7FFFF3B09700 info 'Default'] [Init:859] Calling: VpxdHostCnx::Init(gDB)
2012-05-31T09:11:32.189-04:00 [7FFFEAA06710 info 'Default' opID=SWI-75fe915a] Pre-loading symbols...
2012-05-31T09:11:32.190-04:00 [7FFFEAA06710 info 'Default' opID=SWI-75fe915a] Done pre-loading symbols.
2012-05-31T09:11:32.196-04:00 [7FFFF3B09700 info 'Default'] [Init:861] Calling: VpxdVmomi::Init()
2012-05-31T09:11:32.198-04:00 [7FFFF3B09700 info 'Default'] Looking for vpx result filter at /usr/lib/vmware-vpx/vpxResultFilter.xml
2012-05-31T09:11:32.200-04:00 [7FFFF3B09700 info 'Default'] [Init:862] Calling: QueryServiceProviderHelper::Init()
2012-05-31T09:11:32.201-04:00 [7FFFF3B09700 info 'QueryServiceProvider'] Initializing QueryServiceProviderHelperImpl
2012-05-31T09:11:32.240-04:00 [7FFFF3B09700 info 'QueryServiceProvider'] Starting with generation number: 5230290
2012-05-31T09:11:32.240-04:00 [7FFFF3B09700 info 'Default'] [Init:863] Calling: ServiceDirectory_Init()
2012-05-31T09:11:32.240-04:00 [7FFFF3B09700 info 'Default'] [ServiceDirectory] Local instance UUID: DB25B17D-3703-4779-BAFB-845B64AC33AA
2012-05-31T09:11:32.245-04:00 [7FFFF3B09700 warning 'VpxProfiler'] ServerApp::Init [TotalTime] took 2578 ms
2012-05-31T09:11:32.246-04:00 [7FFFF3B09700 error 'Default'] [VpxdMain] Failed to initialize: Name or service not known
2012-05-31T09:11:32.246-04:00 [7FFFF3B09700 error 'Default'] Failed to intialize VMware VirtualCenter. Shutting down...
2012-05-31T09:11:32.246-04:00 [7FFFF3B09700 info 'Default'] Wrote uptime information
2012-05-31T09:12:32.701-04:00 [7FFFF3B09700 info 'Default'] Forcing shutdown of VMware VirtualCenter now
and the process itself
vsphere-fmrif:~ # service vmware-vpxd status
vmware-vpxd is stopped
tomcat is stopped
vsphere-fmrif:~ # service vmware-vpxd start
Waiting for embedded DB2 database to startup: .success
Cleaning session lock table: success
Verifying EULA acceptance: success
Shutting down ldap-server..done
Starting ldap-server..done
Starting vmware-vpxd: success
Waiting for vpxd to initialize: ....failed
sometimes this happens if you are using the built in DB and you configure your timezone. the appliance will use UTC and if it is changed the local DB will not know how to access records in the future. Did you change the timezone? If so, you may try importing the appliance again and this time don't change the timezone.
Well the appliance has worked for the past 4-6 months. I'm not sure what would have changed.
What are my options? Just thinking out loud below
Fix Time zone settings or time settings on exisiting vCenter Appliance
Sync time somehow?
"Reset DB Contents" --> How disruptive is this? What will I lose?
Deploy new vCenter Appliance and Import old embedded database
If all else fails and I need to use VDR to recover my vcenter server (to get the database), what will I need to do to put the database back in place?
oh, may have found the culprit
vsphere-fmrif:~ # df -h
Filesystem Size Used Avail Use% Mounted on
/dev/sda3 9.8G 4.2G 5.1G 46% /
devtmpfs 2.0G 104K 2.0G 1% /dev
tmpfs 2.0G 0 2.0G 0% /dev/shm
/dev/sda1 130M 18M 105M 15% /boot
/dev/sdb1 20G 20G 0 100% /storage/core
/dev/sdb2 20G 892M 18G 5% /storage/log
/dev/sdb3 20G 2.7G 17G 14% /storage/db
So I cleared out the /storage/core mount point. It quickly fills up again, though. It's already at 25% again. Either way I still cannot login. It is most likely the time thing. If I change it back to the original time(not sure what it was at this point) what will happen?
If you think rebuilding a new VCSA VM is the best possible way, just say so.
side note: backup database from /storage/db ?
a fresh start is always an option, but you'll have to add all your ESX Hosts back in, create any new Clusters, Resource Pools, etc. My initial response was thinking you were just bringing the appliance on-line for the first time.
Since this environment has been online for quite sometime, it may be worth your while to contact VMware Support to see if they can help in fixing the existing issue.
Hi,
Did you manage to resolve this without a fresh install? A month ago my friend came along and suggested that I should change the time to UTC, at that time vCenter and vpxd were working fine. I changed it. Now I am doing negative tests in my environment, and one of the tests was to restart the vpxd service and it is failing with the same error as you saw.
Even rebooting the vCenter vApp does not help. I get the same error failied to initialize vpxd.
Let me know how you resolved this issue. I can initialize the contents of the DB, but what I dont want to lose is the customized alarms and some defualt alarms that I modified.
Thanks!
Fresh install is the only remedy from what I could tell. Take it as an opportunity to check your settings and so on. Sorry man.
--
Justin Edmands
Thanks!
But this is going to be such a pain. I guess I will try a few things before I did that.
Yup, I will try to see the positive side of it! will be hard though
fresh install only option. vmware support takes 3 days to even come up with anything.. even for a sev1.
My lab instance, a vSphere 5.1 vCenter Server Appliance with 2 clusters, 8 hosts (all vSphere 5.1), 80 VMs, exhibits these exact same symptoms. It is using the built in database, no external. I can say for certain that the VCSA time zone has not been touched since initial install, so that is not the issue. My issues started a couple of weeks ago or more. The vSphere Client would just crash every 15 minutes or so, but then I could log back in so I just did that and worked on other more pressing matters, the lab is important, but it would have to wait. It progessed to much worse this week and vSphere client or Web Client not being able to stay logged for more than a couple of seconds. I haven't had time to deal with this until now, so here we go.
You know you are in trouble when on the VCSA management page, vCenter Server > Summary > Storage > Coredumps = 100%! The core dumps or "coredumps" are stored on the VCSA in /storage/core with files named core.vpxd-worker.xxxx where 'x' is a number. I emptied that via SSH with the rm command, but of course it just started filling up again with the same files. Time to break out the toolbox.
After 20 minutes of research and an easy fix, I made a change that has let me login and perform operations in the vSphere Client. So far, so good, no more core dump files 2 hours later. It may be best to perform a basic, surface-level health check (or use LucD's) before rebuilding a VCSA from scratch. That can be a lot of work and will have consequences for VDS, Clusters, policies, etc. Did you document everything?
These are the steps and actions I took to get the lab back online:
From the VCSA management page, stop and started vCenter and Inventory services, but no effect.
From the VCSA management page, reboot the appliance, but no effect.
Stopped vCenter Server and Inventory Services to keep them off during troubleshooting
Logged into each host with vSphere Client first to check all the basics, DNS name, time, verify that IPV6 is disabled (I had problems with that enabled before even though it wasn't in use), etc.
Then (still in vSphere client) check the host logs Home > Administration > System Logs for odd patterns or excessive errors in any of the logs (there are several, click the drop-down). The first 4 hosts I check all looked normal, but one was having what looked like an excessive amount of errors related to snapshot delta disks from snapshots. The problem rang a bell, there was a VM I noticed I had trouble with recently that has a bunch of snapshots and delta disks from some failed backups. I didn't have time to look into it a month or so when I first saw that (it is lab and we never have the time do we?).
The VM was already shut down, so I removed it from the host inventory, then SSH'd to the host and ran /sbin/services.sh restart to restart the management agents --but you can also use the DCUI to accomplish same.
Started vCenter Server and Inventory Services and voila, I could log in and manage my lab again.
Side notes:
I got this message on all of the VMs that were running on the host. The VMs never went down, but HA was attempted:
vSphere HA virtual machine failover failed
I have to work on the snapshot problem which I think was caused during backups by the VMware Data Recovery Appliance, but I don't know the cause.
Message was edited by: aSmoot
Thank you for your message. I am currently out of the office and will return Apr 16, 2013. I will respond upon my return.
For support related issues please contact the IT Service Desk servicedesk@camhydro.com or x2700.
Thank you
Heath
Hi there,
I have exact same problem on the linux- based vCenter server appliance 5.5 installed with OVA on the ESXi host 5.5.
I used:
Name: VMware-vCenter-Server-Appliance-5.5.0.20200-2183109_OVF10.ova
Release Date: 2014-10-09
Build Number: 2183109
All services can start except the vCenter. It always says :
Starting vmware-vpxd: success
"Waiting for vpxd to initialized: .failed'.
I have checked the time on ESXi host and the vCenter and they matched since both the ESXi host and vCenter sync to same ntp server.
My DataBase is embedded and SSO is also embedded. "Test Settings" is successful.
in /var/log/messages:
2014-10-31T21:16:19+00:00 localhost vami-sfcbd: pam_succeed_if(vami-sfcb:auth): requirement "uid eq 0" was met by user "root"
2014-10-31T21:16:23+00:00 localhost lsassd[3997]: 0x7f57668fa700:Failed to run provider specific request (request code = 12, provider = 'lsa-activedirectory-provider') -> error = 2692, symbol = NERR_SetupNotJoined, client pid = 1815
2014-10-31T21:16:23+00:00 localhost lwsmd: Stopping service: eventlog
2014-10-31T21:16:24+00:00 localhost lwsmd: Starting service: eventlog
2014-10-31T21:16:24+00:00 localhost lsassd[3997]: 0x7f57668fa700:Failed to run provider specific request (request code = 12, provider = 'lsa-activedirectory-provider') -> error = 2692, symbol = NERR_SetupNotJoined, client pid = 1967
2014-10-31T21:16:50+00:00 localhost lsassd[3997]: 0x7f57668fa700:Failed to run provider specific request (request code = 12, provider = 'lsa-activedirectory-provider') -> error = 2692, symbol = NERR_SetupNotJoined, client pid = 2305
2014-10-31T21:16:53+00:00 localhost lsassd[3997]: 0x7f57668fa700:Failed to run provider specific request (request code = 12, provider = 'lsa-activedirectory-provider') -> error = 2692, symbol = NERR_SetupNotJoined, client pid = 2609
2014-10-31T21:16:53+00:00 localhost lwsmd: Stopping service: eventlog
2014-10-31T21:17:07+00:00 localhost watchdog-vpxd: [3080] Begin '/usr/sbin/vpxd', min-uptime = 30, max-quick-failures = 5, max-total-failures = 1000000, bg_pid_file = ''
2014-10-31T21:17:07+00:00 localhost watchdog-vpxd: Executing '/usr/sbin/vpxd'
2014-10-31T21:17:07+00:00 localhost watchdog-vpxd: '/usr/sbin/vpxd' exited after 0 seconds (quick failure 1) 134
2014-10-31T21:17:07+00:00 localhost watchdog-vpxd: Executing '/usr/sbin/vpxd'
2014-10-31T21:17:07+00:00 localhost watchdog-vpxd: '/usr/sbin/vpxd' exited after 0 seconds (quick failure 2) 134
2014-10-31T21:17:07+00:00 localhost watchdog-vpxd: Executing '/usr/sbin/vpxd'
2014-10-31T21:17:08+00:00 localhost watchdog-vpxd: '/usr/sbin/vpxd' exited after 1 seconds (quick failure 3) 134
2014-10-31T21:17:08+00:00 localhost watchdog-vpxd: Executing '/usr/sbin/vpxd'
2014-10-31T21:17:08+00:00 localhost watchdog-vpxd: '/usr/sbin/vpxd' exited after 0 seconds (quick failure 4) 134
2014-10-31T21:17:08+00:00 localhost watchdog-vpxd: Executing '/usr/sbin/vpxd'
2014-10-31T21:17:08+00:00 localhost watchdog-vpxd: '/usr/sbin/vpxd' exited after 0 seconds (quick failure 5) 134
2014-10-31T21:17:08+00:00 localhost watchdog-vpxd: Executing '/usr/sbin/vpxd'
2014-10-31T21:17:09+00:00 localhost watchdog-vpxd: '/usr/sbin/vpxd' exited after 1 seconds (quick failure 6) 134
2014-10-31T21:17:09+00:00 localhost watchdog-vpxd: End '/usr/sbin/vpxd', failure limit reached
2014-10-31T21:19:19+00:00 localhost syslog-ng[1289]: Log statistics; dropped='pipe(/dev/xconsole)=0', dropped='pipe(/dev/tty10)=0', processed='center(queued)=1704', processed='center(received)=1169', processed='destination(messages)=469', processed='destination(mailinfo)=2', processed='destination(mailwarn)=4', processed='destination(authlog)=38', processed='destination(localmessages)=10', processed='destination(cron)=25', processed='destination(mailerr)=0', processed='destination(ldapmessages)=694', processed='destination(netmgm)=0', processed='destination(warn)=156', processed='destination(console)=150', processed='destination(mail)=6', processed='destination(null)=0', processed='destination(xconsole)=150', processed='destination(firewall)=0', processed='destination(acpid)=0', processed='source(src)=1169'
2014-10-31T21:20:01+00:00 localhost /usr/sbin/cron[3432]: (root) CMD ([ -x /usr/lib64/sa/sa1 ] && exec /usr/lib64/sa/sa1 -S ALL 1 1)
2014-10-31T21:20:01+00:00 localhost /usr/sbin/cron[3435]: (root) CMD ( test -x /usr/sbin/vpxd_periodic && /usr/sbin/vpxd_periodic >/dev/null 2>&1)
2014-10-31T21:30:01+00:00 localhost /usr/sbin/cron[3733]: (root) CMD ( test -x /usr/sbin/vpxd_periodic && /usr/sbin/vpxd_periodic >/dev/null 2>&1)
2014-10-31T21:30:01+00:00 localhost /usr/sbin/cron[3735]: (root) CMD ([ -x /usr/lib64/sa/sa1 ] && exec /usr/lib64/sa/sa1 -S ALL 1 1)
I followed instruction on VMware KB: vpxd process on the vCenter Server Appliance 5.x fails with the error: Signal 11 received
and modified /etc/vmware-vpx/vpx.cfg file as sugggested by adding the ThreadStackSizedKb line. See below.
<threadPool>
<TaskMax>90</TaskMax>
<ThreadStackSizeKb>512</ThreadStackSizeKb>
<threadNamePrefix>vpxd</threadNamePrefix>
</threadPool>
However, it doesn't work.
Anyone knows about this problem please help.
I had an issue with the vpxd failing to initialize (vpxd service started succesfully), and found that there was a message in the var/log/vmware/vpx/vpxd.log that it was unable to open the file var/log/vmware/journal/1414758110.0.lck - no such file exists. I looked in the location and a directory of that name had been created. Once I removed this directory and restarted the vpxd service I got back my vCenter server and was able to log in using both the vsphere client and web client. I have not seen anyone mention this error in their logs posted - but this resolved the service failing to initialise issue on my setup. I am on ESXi 5.5 with a Single vCenter and 3 hosts
I had the same issue and it turned out the /storage/db drive was out of space. I followed WMware's article: VMware KB: Increasing the disk space in vCenter Server Appliance 5.x and I'm good.
you must increase space of hard drive from the vsphere console
My datastore ran out of space, the vcenter froze, I cleared space on the datastore, resumed the vcsa but it was unresponsive.
Restarted vcenter and got the "Waiting for vpxd to initialize failed".
What solved it was running these instructions I found here:
From: http://www.colabug.com/thread-1093735-1-1.html
To get over this, log on to the VCVA using SSH
Run the command “ /usr/sbin/vpxd_odbcconfig ”
You see output similar to:
VMware vCenter Server Appliance ODBC Configuration
Choose a database to use with VMware vCenter Server
1) oracle
2) embedded
Enter 2 (should the DB be the embedded PostgresSQL
You see output similar to:
Configuring ODBC for embedded
Operation was successful.
Database already populated with version VirtualCenter Database 5.5
Reinitialize database? (WARNING: WILL DELETE ALL RECORDS) [y(es)|N(O)]: OK, preserving the database. (select No to preserve the database)
Restart vCenter Server now to activate the database changes [y/N]:y
Successfully restarted vCenter Server
This should start the vpxd service
Here's my output:
Starting vmware-vpxd: success
Waiting for vpxd to initialize: ............................................................failed
vcsa:~ # /usr/sbin/vpxd_odbcconfig
WARNING - Use of this command is not supported unless you are doing this
with the consultation of VMware Technical Support or a VMware published
technical article.
VMware vCenter Server Appliance ODBC Configuration
Choose a database to use with VMware vCenter Server
1) oracle
2) embedded
#? 2
Configuring ODBC for embedded
Error: VPXD must be stopped to perform this operation.
Database configuration has failed.
Would you like to try again or modify the settings manually?
[A]gain/[m]anually : m
vcsa:~ # service vmware-vpxd stop
Stopping tomcat: success
Stopping vmware-vpxd: success
Shutting down ldap-server..done
vcsa:~ # /usr/sbin/vpxd_odbcconfig
WARNING - Use of this command is not supported unless you are doing this
with the consultation of VMware Technical Support or a VMware published
technical article.
VMware vCenter Server Appliance ODBC Configuration
Choose a database to use with VMware vCenter Server
1) oracle
2) embedded
#? 2
Configuring ODBC for embedded
Operation was successful.
Database already populated with version VirtualCenter Database 5.5
Reinitialize database? (WARNING: WILL DELETE ALL RECORDS) [y(es)|N(O)]: n
OK, preserving the database.
Restart vCenter Server now to activate the database changes [y/N]:y
Successfully restarted vCenter Server
vcsa:~ #
Success!!
We have setup VMware appliance in our setup but after changing some configuration in inventory.cfg , Vmware-vpxd services will not starting even after reverting the changes. Attaching logs below. kindly help us ASAP.
Waiting for the embedded database to start up: success
Verifying EULA acceptance: success
Executing pre-startup scripts...
Updating the vCenter endpoint in the Lookup Service.
Intializing registration provider...
Getting SSL certificates for https://192.168.20.100:7444/lookupservice/sdk
Service with name 'vpxd-localhost.localdom-543843fd-d8da-4c7f-9bd7-8acd95edeb60' and ID 'local:9c27a9ee-0ca0-42e7-8767-7704f847d240' was updated.
Return code is: Success
Shutting down ldap-server..done
Starting ldap-server..done
Starting vmware-vpxd: success
Waiting for vpxd to initialize: .failed <-- this is going to be fail every time.
localhost:~ #
On System log , We will get :-
localhost:~ # cat /var/log/messages
2017-06-05T10:13:18+00:00 localhost watchdog-vpxd: PID file /var/run/vmware/watchdog-vpxd.PID does not exist
2017-06-05T10:13:18+00:00 localhost watchdog-vpxd: Unable to terminate watchdog: No running watchdog process for vpxd
2017-06-05T10:13:19+00:00 localhost watchdog-vpxd: PID file /var/run/vmware/watchdog-vpxd.PID does not exist
2017-06-05T10:13:19+00:00 localhost watchdog-vpxd: Unable to terminate watchdog: No running watchdog process for vpxd
2017-06-05T10:13:27+00:00 localhost watchdog-vpxd: [12418] Begin '/usr/sbin/vpxd', min-uptime = 30, max-quick-failures = 5, max-total-failures = 1000000, bg_pid_file = ''
2017-06-05T10:13:27+00:00 localhost watchdog-vpxd: Executing '/usr/sbin/vpxd'
2017-06-05T10:13:27+00:00 localhost watchdog-vpxd: '/usr/sbin/vpxd' exited after 0 seconds (quick failure 1) 1
2017-06-05T10:13:27+00:00 localhost watchdog-vpxd: Executing '/usr/sbin/vpxd'
2017-06-05T10:13:28+00:00 localhost watchdog-vpxd: '/usr/sbin/vpxd' exited after 1 seconds (quick failure 2) 1
2017-06-05T10:13:28+00:00 localhost watchdog-vpxd: Executing '/usr/sbin/vpxd'
2017-06-05T10:13:28+00:00 localhost watchdog-vpxd: '/usr/sbin/vpxd' exited after 0 seconds (quick failure 3) 1
2017-06-05T10:13:28+00:00 localhost watchdog-vpxd: Executing '/usr/sbin/vpxd'
2017-06-05T10:13:28+00:00 localhost watchdog-vpxd: '/usr/sbin/vpxd' exited after 0 seconds (quick failure 4) 1
2017-06-05T10:13:28+00:00 localhost watchdog-vpxd: Executing '/usr/sbin/vpxd'
2017-06-05T10:13:29+00:00 localhost watchdog-vpxd: '/usr/sbin/vpxd' exited after 1 seconds (quick failure 5) 1
2017-06-05T10:13:29+00:00 localhost watchdog-vpxd: Executing '/usr/sbin/vpxd'
2017-06-05T10:13:29+00:00 localhost watchdog-vpxd: '/usr/sbin/vpxd' exited after 0 seconds (quick failure 6) 1
2017-06-05T10:13:29+00:00 localhost watchdog-vpxd: End '/usr/sbin/vpxd', failure limit reached
2017-06-05T10:14:13+00:00 localhost watchdog-vpxd: PID file /var/run/vmware/watchdog-vpxd.PID does not exist
2017-06-05T10:14:13+00:00 localhost watchdog-vpxd: Unable to terminate watchdog: No running watchdog process for vpxd
2017-06-05T10:14:14+00:00 localhost watchdog-vpxd: PID file /var/run/vmware/watchdog-vpxd.PID does not exist
2017-06-05T10:14:14+00:00 localhost watchdog-vpxd: Unable to terminate watchdog: No running watchdog process for vpxd
2017-06-05T10:15:56+00:00 localhost watchdog-vpxd: [4936] Begin '/usr/sbin/vpxd', min-uptime = 30, max-quick-failures = 5, max-total-failures = 1000000, bg_pid_file = ''
2017-06-05T10:15:56+00:00 localhost watchdog-vpxd: Executing '/usr/sbin/vpxd'
2017-06-05T10:15:56+00:00 localhost kernel: [ 82.191937] warning: process `vpxd' used the deprecated sysctl system call with 1.1.
2017-06-05T10:15:56+00:00 localhost watchdog-vpxd: '/usr/sbin/vpxd' exited after 0 seconds (quick failure 1) 1
2017-06-05T10:15:56+00:00 localhost watchdog-vpxd: Executing '/usr/sbin/vpxd'
2017-06-05T10:15:57+00:00 localhost watchdog-vpxd: '/usr/sbin/vpxd' exited after 1 seconds (quick failure 2) 1
2017-06-05T10:15:57+00:00 localhost watchdog-vpxd: Executing '/usr/sbin/vpxd'
2017-06-05T10:15:57+00:00 localhost watchdog-vpxd: '/usr/sbin/vpxd' exited after 0 seconds (quick failure 3) 1
2017-06-05T10:15:57+00:00 localhost watchdog-vpxd: Executing '/usr/sbin/vpxd'
2017-06-05T10:15:57+00:00 localhost watchdog-vpxd: '/usr/sbin/vpxd' exited after 0 seconds (quick failure 4) 1
2017-06-05T10:15:57+00:00 localhost watchdog-vpxd: Executing '/usr/sbin/vpxd'
2017-06-05T10:15:58+00:00 localhost watchdog-vpxd: '/usr/sbin/vpxd' exited after 1 seconds (quick failure 5) 1
2017-06-05T10:15:58+00:00 localhost watchdog-vpxd: Executing '/usr/sbin/vpxd'
2017-06-05T10:15:58+00:00 localhost watchdog-vpxd: '/usr/sbin/vpxd' exited after 0 seconds (quick failure 6) 1
2017-06-05T10:15:58+00:00 localhost watchdog-vpxd: End '/usr/sbin/vpxd', failure limit reached
2017-06-05T10:20:01+00:00 localhost /usr/sbin/cron[5640]: (root) CMD ( test -x /usr/sbin/vpxd_periodic && /usr/sbin/vpxd_periodic >/dev/null 2>&1)
On vpxd.log, I also get some error.
localhost:/var/log/vmware/vpx # tailf -n 100 vpxd-171.log
Section for VMware VirtualCenter, pid=27456, version=5.5.0, build=2001466, option=Release
------ In-memory logs start --------
mem> 2017-06-06T04:14:20.657Z [7F9131494740 verbose 'vpxdvpxdSignal'] Signal handler for signal 1 is installed
mem> 2017-06-06T04:14:20.657Z [7F9131494740 verbose 'vpxdvpxdSignal'] Signal handler for signal 2 is installed
mem> 2017-06-06T04:14:20.657Z [7F9131494740 verbose 'vpxdvpxdSignal'] Signal handler for signal 15 is installed
mem> 2017-06-06T04:14:20.657Z [7F9131494740 verbose 'vpxdvpxdSignal'] Signal handler for signal 10 is installed
mem> 2017-06-06T04:14:20.657Z [7F9131494740 info 'Hooks'] Hooks Initialized
------ In-memory logs end --------
2017-06-06T04:14:20.660Z [7F9131494740 info 'Default'] Logging uses fast path: true
2017-06-06T04:14:20.660Z [7F9131494740 info 'Default'] Handling bora/lib logs with VmaCore facilities
2017-06-06T04:14:20.660Z [7F9131494740 info 'Default'] Initialized channel manager
2017-06-06T04:14:20.663Z [7F9131494740 info 'Default'] Current working directory: /storage/log/vmware/vpx
2017-06-06T04:14:20.663Z [7F9131494740 info 'ThreadPool'] Thread pool on asio: Min Io, Max Io, Min Task, Max Task, Max Concurency: 2, 200, 2, 90, 2147483647
2017-06-06T04:14:20.663Z [7F9131494740 info 'ThreadPool'] Thread enlisted
2017-06-06T04:14:20.663Z [7F9131494740 info 'Default'] Log path: /var/log/vmware/vpx
2017-06-06T04:14:20.663Z [7F9131494740 info 'Default'] Initializing SSL
2017-06-06T04:14:20.663Z [7F91278A4700 info 'ThreadPool'] Thread enlisted
2017-06-06T04:14:20.663Z [7F91279A6700 info 'ThreadPool'] Thread enlisted
2017-06-06T04:14:20.663Z [7F9127823700 info 'ThreadPool'] Thread enlisted
2017-06-06T04:14:20.663Z [7F9127925700 info 'ThreadPool'] Thread enlisted
2017-06-06T04:14:20.664Z [7F9131494740 info 'Default'] Vmacore::InitSSL: handshakeTimeoutUs = 120000000
2017-06-06T04:14:20.665Z [7F91277A2700 info 'ThreadPool'] Thread enlisted
2017-06-06T04:14:20.665Z [7F9131494740 info 'vpxdvpxdDaemon'] Changed working directory to /var/log/vmware/vpx
2017-06-06T04:14:20.666Z [7F9131494740 error 'vpxdvpxdDaemon'] Unhandled exception:
--> not well-formed (invalid token)
2017-06-06T04:14:20.667Z [7F9131494740 error 'vpxdvpxdDaemon'] Backtrace:
--> backtrace[00] rip 00007f912b3f9014 Vmacore::System::Stacktrace::CaptureWork(unsigned int)
--> backtrace[01] rip 00007f912b2e3132 Vmacore::System::SystemFactoryImpl::CreateQuickBacktrace(Vmacore::Ref<Vmacore::System::Backtrace>&)
--> backtrace[02] rip 00007f912b2448a5 Vmacore::Throwable::Throwable(std::string const&)
--> backtrace[03] rip 00007f912b28b566 /usr/lib/vmware-vpx/libvmacore.so(+0x285566) [0x7f912b28b566]
--> backtrace[04] rip 00007f912b23e85d /usr/lib/vmware-vpx/libvmacore.so(+0x23885d) [0x7f912b23e85d]
--> backtrace[05] rip 00007f9132abfd07 /usr/lib/vmware-vpx/vpxd(+0x14aad07) [0x7f9132abfd07]
--> backtrace[06] rip 00007f9132a3f0c5 /usr/lib/vmware-vpx/vpxd(+0x142a0c5) [0x7f9132a3f0c5]
--> backtrace[07] rip 00007f91324c3ca3 /usr/lib/vmware-vpx/vpxd(+0xeaeca3) [0x7f91324c3ca3]
--> backtrace[08] rip 00007f91324a6c2e /usr/lib/vmware-vpx/vpxd(+0xe91c2e) [0x7f91324a6c2e]
--> backtrace[09] rip 00007f9129369c16 /lib64/libc.so.6(__libc_start_main+0xe6) [0x7f9129369c16]
--> backtrace[10] rip 00007f91324a5e89 /usr/lib/vmware-vpx/vpxd(+0xe90e89) [0x7f91324a5e89]
I have gone through the above solutions for storage space full but i checked on our setup its not there.
localhost:/var/log/vmware/vpx # df -h
Filesystem Size Used Avail Use% Mounted on
/dev/sda3 9.8G 3.7G 5.6G 40% /
udev 4.0G 104K 4.0G 1% /dev
tmpfs 4.0G 0 4.0G 0% /dev/shm
/dev/sda1 128M 21M 101M 18% /boot
/dev/sdb1 20G 173M 19G 1% /storage/core
/dev/sdb2 20G 305M 19G 2% /storage/log
/dev/sdb3 60G 1.7G 55G 4% /storage/db
localhost:/var/log/vmware/vpx # df -ih
Filesystem Inodes IUsed IFree IUse% Mounted on
/dev/sda3 632K 60K 573K 10% /
udev 0 0 0 - /dev
tmpfs 1001K 2 1001K 1% /dev/shm
/dev/sda1 34K 40 34K 1% /boot
/dev/sdb1 1.3M 12 1.3M 1% /storage/core
/dev/sdb2 1.3M 293 1.3M 1% /storage/log
/dev/sdb3 3.8M 3.7K 3.8M 1% /storage/db
localhost:/var/log/vmware/vpx #
This absolutely worked for me. Thank you!
In my case, I do not believe I had ran out of space first (df -h was reporting fine).
Hello I have same problem
after restarting I cannot connect to vcenter. This is a my log file. Please Help solve this problem
Section for VMware VirtualCenter, pid=4819, version=5.5.0, build=2001466, option=Release
------ In-memory logs start --------
mem> 2018-12-07T08:20:43.272Z [7FA96E132740 verbose 'vpxdvpxdSignal'] Signal handler for signal 1 is installed
mem> 2018-12-07T08:20:43.272Z [7FA96E132740 verbose 'vpxdvpxdSignal'] Signal handler for signal 2 is installed
mem> 2018-12-07T08:20:43.272Z [7FA96E132740 verbose 'vpxdvpxdSignal'] Signal handler for signal 15 is installed
mem> 2018-12-07T08:20:43.272Z [7FA96E132740 verbose 'vpxdvpxdSignal'] Signal handler for signal 10 is installed
mem> 2018-12-07T08:20:43.272Z [7FA96E132740 info 'Hooks'] Hooks Initialized
------ In-memory logs end --------
2018-12-07T08:20:43.281Z [7FA96E132740 info 'Default'] Logging uses fast path: true
2018-12-07T08:20:43.281Z [7FA96E132740 info 'Default'] Handling bora/lib logs with VmaCore facilities
2018-12-07T08:20:43.281Z [7FA96E132740 info 'Default'] Initialized channel manager
2018-12-07T08:20:43.288Z [7FA96E132740 info 'Default'] Current working directory: /storage/log/vmware/vpx
2018-12-07T08:20:43.288Z [7FA964685700 info 'ThreadPool'] Thread enlisted
2018-12-07T08:20:43.288Z [7FA964502700 info 'ThreadPool'] Thread enlisted
2018-12-07T08:20:43.288Z [7FA96E132740 info 'ThreadPool'] Thread pool on asio: Min Io, Max Io, Min Task, Max Task, Max Concurency: 2, 200, 2, 90, 2147483647
2018-12-07T08:20:43.288Z [7FA96E132740 info 'ThreadPool'] Thread enlisted
2018-12-07T08:20:43.288Z [7FA964604700 info 'ThreadPool'] Thread enlisted
2018-12-07T08:20:43.288Z [7FA964583700 info 'ThreadPool'] Thread enlisted
2018-12-07T08:20:43.288Z [7FA96E132740 info 'Default'] Log path: /var/log/vmware/vpx
2018-12-07T08:20:43.288Z [7FA96E132740 info 'Default'] Initializing SSL
2018-12-07T08:20:43.291Z [7FA96E132740 info 'Default'] Vmacore::InitSSL: handshakeTimeoutUs = 120000000
2018-12-07T08:20:43.292Z [7FA964481700 info 'ThreadPool'] Thread enlisted
2018-12-07T08:20:43.293Z [7FA96E132740 info 'vpxdvpxdDaemon'] Changed working directory to /var/log/vmware/vpx
2018-12-07T08:20:43.306Z [7FA96E132740 info 'Default'] Starting VMware VirtualCenter 5.5.0 build-2001466
2018-12-07T08:20:43.306Z [7FA96E132740 info 'Default'] Log directory: /var/log/vmware/vpx.
2018-12-07T08:20:43.307Z [7FA96E132740 info 'vpxdvpxdMain'] Account name: root
2018-12-07T08:20:43.307Z [7FA96E132740 info 'commonvpxLro'] [VpxLRO] 640 max LROs
2018-12-07T08:20:43.307Z [7FA96E132740 info 'commonvpxLro'] [VpxLRO] 12 reserved internal LROs
2018-12-07T08:20:43.307Z [7FA96E132740 info 'commonvpxLro'] [VpxLRO] 12 reserved blocker LROs
2018-12-07T08:20:43.307Z [7FA96E132740 info 'commonvpxLro'] [VpxLRO] 12 reserved short LROs
2018-12-07T08:20:43.307Z [7FA96E132740 info 'commonvpxLro'] [VpxLRO] 8 reserved long LROs
2018-12-07T08:20:43.307Z [7FA96E132740 info 'commonvpxLro'] [VpxLRO] 600-second completed task lifetime
2018-12-07T08:20:43.307Z [7FA96E132740 info 'commonvpxLro'] [VpxLRO] 60-second minimum completed task lifetime
2018-12-07T08:20:43.307Z [7FA96E132740 info 'commonvpxLro'] [VpxLRO] 200 maximum completed tasks
2018-12-07T08:20:43.307Z [7FA96E132740 info 'commonvpxLro'] [VpxLRO] 600-second unregistered task lifetime
2018-12-07T08:20:43.308Z [7FA96E132740 info 'utilvpxdVdb'] Registry Item DB 5 value is ''
2018-12-07T08:20:43.308Z [7FA96E132740 info 'utilvpxdVdb'] [VpxdVdb::SetDBType] Logging in to DSN: VMware VirtualCenter with username vc
2018-12-07T08:20:43.477Z [7FA96E132740 info 'Default'] [VdbSchemaLoader::LookupSchemaFromDB] Loaded schema and index information from DB
2018-12-07T08:20:43.494Z [7FA96E132740 info 'utilvpxdDbSchemaHash'] [VpxdDbSchemaHash::GetSchemaFilename]: sqlDir: /usr/lib/vmware-vpx/sql/
2018-12-07T08:20:43.494Z [7FA96E132740 info 'utilvpxdDbSchemaHash'] [VpxdDbSchemaHash::CheckSchema]: schema file: /usr/lib/vmware-vpx/sql/VCDB_PostgreSQL.sql
2018-12-07T08:20:43.508Z [7FA96E132740 info 'linuxvpxLdap_linux'] [LdapBackup] Making sure LDAP instance VMwareVCMSDS is running
2018-12-07T08:20:43.508Z [7FA96E132740 info 'linuxvpxLdap_linux'] [LdapBackup] Checking service ldap for running
2018-12-07T08:20:43.533Z [7FA96E132740 info 'Default'] Looking for vpx result filter at /usr/lib/vmware-vpx/vpxResultFilter.xml
2018-12-07T08:20:43.538Z [7FA964604700 info 'vpxdvpxdMain' opID=SWI-6058ed8] Pre-loading symbols...
2018-12-07T08:20:43.540Z [7FA964604700 info 'vpxdvpxdMain' opID=SWI-6058ed8] Done pre-loading symbols.
2018-12-07T08:20:43.541Z [7FA96E132740 info 'MoLock'] LRO tracing: enabled: false
2018-12-07T08:20:43.541Z [7FA96E132740 info 'MoLock'] LRO tracing: LRO name:
2018-12-07T08:20:43.556Z [7FA96E132740 info 'Default'] Looking for vpx result filter at /usr/lib/vmware-vpx/vpxResultFilter.xml
2018-12-07T08:20:43.561Z [7FA96E132740 info 'QueryServiceProvider'] Initializing QueryServiceProviderHelperImpl
2018-12-07T08:20:43.605Z [7FA96E132740 info 'QueryServiceProvider'] Starting with generation number: 7085462
2018-12-07T08:20:43.605Z [7FA96E132740 info 'vpxservicesMoServiceDirectory'] [ServiceDirectory] Local instance UUID: 0FCF384F-1EEA-4A00-9424-673FC0F7F363
2018-12-07T08:20:43.608Z [7FA96E132740 error 'linuxvpxLdap_linux'] [LDAP Client] Failed to modify LDAP entry cn=VIMAPI,cn=0FCF384F-1EEA-4A00-9424-673FC0F7F363,ou=Instances,dc=virtualcenter,dc=vmware,dc=int: 0x1 (Operations error)
2018-12-07T08:20:43.608Z [7FA96E132740 error 'linuxvpxLdap_linux'] [LDAP Client] LdapApi::modify_s: key (vmw-vc-SSLThumbprint)
2018-12-07T08:20:43.608Z [7FA96E132740 error 'vpxservicesMoServiceDirectory'] [ServiceDirectory::UpdateServiceInfo] Error while modifying service-endpoint SSL thumbprint cn=VIMAPI,cn=0FCF384F-1EEA-4A00-9424-673FC0F7F363,ou=Instances,dc=virtualcenter,dc=vmware,dc=int
2018-12-07T08:20:43.608Z [7FA96E132740 error 'vpxdvpxdMain'] [Init] Init failed: Ldap error
--> Backtrace:
--> backtrace[00] rip 00007fa968097014 Vmacore::System::Stacktrace::CaptureWork(unsigned int)
--> backtrace[01] rip 00007fa967f81132 Vmacore::System::SystemFactoryImpl::CreateQuickBacktrace(Vmacore::Ref<Vmacore::System::Backtrace>&)
--> backtrace[02] rip 00007fa967ee28a5 Vmacore::Throwable::Throwable(std::string const&)
--> backtrace[03] rip 00007fa96f16331f Vmomi::Fault::SystemError::Exception::Exception(std::string const&)
--> backtrace[04] rip 00007fa97060c60b /usr/lib/vmware-vpx/vpxd(+0x235960b) [0x7fa97060c60b]
--> backtrace[05] rip 00007fa97060f0fd /usr/lib/vmware-vpx/vpxd(+0x235c0fd) [0x7fa97060f0fd]
--> backtrace[06] rip 00007fa97060f645 /usr/lib/vmware-vpx/vpxd(+0x235c645) [0x7fa97060f645]
--> backtrace[07] rip 00007fa96f14daaf /usr/lib/vmware-vpx/vpxd(+0xe9aaaf) [0x7fa96f14daaf]
--> backtrace[08] rip 00007fa96f161d43 /usr/lib/vmware-vpx/vpxd(+0xeaed43) [0x7fa96f161d43]
--> backtrace[09] rip 00007fa96f144c2e /usr/lib/vmware-vpx/vpxd(+0xe91c2e) [0x7fa96f144c2e]
--> backtrace[10] rip 00007fa966007c16 /lib64/libc.so.6(__libc_start_main+0xe6) [0x7fa966007c16]
--> backtrace[11] rip 00007fa96f143e89 /usr/lib/vmware-vpx/vpxd(+0xe90e89) [0x7fa96f143e89]
-->
2018-12-07T08:20:43.612Z [7FA96E132740 error 'Default'] Failed to intialize VMware VirtualCenter. Shutting down...
2018-12-07T08:20:43.612Z [7FA96E132740 info 'vpxdvpxdSupportManager'] Wrote uptime information