We have setup VMware appliance in our setup but after changing some configuration in inventory.cfg , Vmware-vpxd services will not starting even after reverting the changes. Attaching logs below. kindly help us ASAP.
Waiting for the embedded database to start up: success
Verifying EULA acceptance: success
Executing pre-startup scripts...
Updating the vCenter endpoint in the Lookup Service.
Intializing registration provider...
Getting SSL certificates for https://192.168.20.100:7444/lookupservice/sdk
Service with name 'vpxd-localhost.localdom-543843fd-d8da-4c7f-9bd7-8acd95edeb60' and ID 'local:9c27a9ee-0ca0-42e7-8767-7704f847d240' was updated.
Return code is: Success
Shutting down ldap-server..done
Starting ldap-server..done
Starting vmware-vpxd: success
Waiting for vpxd to initialize: .failed <-- this is going to be fail every time.
localhost:~ #
On System log , We will get :-
localhost:~ # cat /var/log/messages
2017-06-05T10:13:18+00:00 localhost watchdog-vpxd: PID file /var/run/vmware/watchdog-vpxd.PID does not exist
2017-06-05T10:13:18+00:00 localhost watchdog-vpxd: Unable to terminate watchdog: No running watchdog process for vpxd
2017-06-05T10:13:19+00:00 localhost watchdog-vpxd: PID file /var/run/vmware/watchdog-vpxd.PID does not exist
2017-06-05T10:13:19+00:00 localhost watchdog-vpxd: Unable to terminate watchdog: No running watchdog process for vpxd
2017-06-05T10:13:27+00:00 localhost watchdog-vpxd: [12418] Begin '/usr/sbin/vpxd', min-uptime = 30, max-quick-failures = 5, max-total-failures = 1000000, bg_pid_file = ''
2017-06-05T10:13:27+00:00 localhost watchdog-vpxd: Executing '/usr/sbin/vpxd'
2017-06-05T10:13:27+00:00 localhost watchdog-vpxd: '/usr/sbin/vpxd' exited after 0 seconds (quick failure 1) 1
2017-06-05T10:13:27+00:00 localhost watchdog-vpxd: Executing '/usr/sbin/vpxd'
2017-06-05T10:13:28+00:00 localhost watchdog-vpxd: '/usr/sbin/vpxd' exited after 1 seconds (quick failure 2) 1
2017-06-05T10:13:28+00:00 localhost watchdog-vpxd: Executing '/usr/sbin/vpxd'
2017-06-05T10:13:28+00:00 localhost watchdog-vpxd: '/usr/sbin/vpxd' exited after 0 seconds (quick failure 3) 1
2017-06-05T10:13:28+00:00 localhost watchdog-vpxd: Executing '/usr/sbin/vpxd'
2017-06-05T10:13:28+00:00 localhost watchdog-vpxd: '/usr/sbin/vpxd' exited after 0 seconds (quick failure 4) 1
2017-06-05T10:13:28+00:00 localhost watchdog-vpxd: Executing '/usr/sbin/vpxd'
2017-06-05T10:13:29+00:00 localhost watchdog-vpxd: '/usr/sbin/vpxd' exited after 1 seconds (quick failure 5) 1
2017-06-05T10:13:29+00:00 localhost watchdog-vpxd: Executing '/usr/sbin/vpxd'
2017-06-05T10:13:29+00:00 localhost watchdog-vpxd: '/usr/sbin/vpxd' exited after 0 seconds (quick failure 6) 1
2017-06-05T10:13:29+00:00 localhost watchdog-vpxd: End '/usr/sbin/vpxd', failure limit reached
2017-06-05T10:14:13+00:00 localhost watchdog-vpxd: PID file /var/run/vmware/watchdog-vpxd.PID does not exist
2017-06-05T10:14:13+00:00 localhost watchdog-vpxd: Unable to terminate watchdog: No running watchdog process for vpxd
2017-06-05T10:14:14+00:00 localhost watchdog-vpxd: PID file /var/run/vmware/watchdog-vpxd.PID does not exist
2017-06-05T10:14:14+00:00 localhost watchdog-vpxd: Unable to terminate watchdog: No running watchdog process for vpxd
2017-06-05T10:15:56+00:00 localhost watchdog-vpxd: [4936] Begin '/usr/sbin/vpxd', min-uptime = 30, max-quick-failures = 5, max-total-failures = 1000000, bg_pid_file = ''
2017-06-05T10:15:56+00:00 localhost watchdog-vpxd: Executing '/usr/sbin/vpxd'
2017-06-05T10:15:56+00:00 localhost kernel: [ 82.191937] warning: process `vpxd' used the deprecated sysctl system call with 1.1.
2017-06-05T10:15:56+00:00 localhost watchdog-vpxd: '/usr/sbin/vpxd' exited after 0 seconds (quick failure 1) 1
2017-06-05T10:15:56+00:00 localhost watchdog-vpxd: Executing '/usr/sbin/vpxd'
2017-06-05T10:15:57+00:00 localhost watchdog-vpxd: '/usr/sbin/vpxd' exited after 1 seconds (quick failure 2) 1
2017-06-05T10:15:57+00:00 localhost watchdog-vpxd: Executing '/usr/sbin/vpxd'
2017-06-05T10:15:57+00:00 localhost watchdog-vpxd: '/usr/sbin/vpxd' exited after 0 seconds (quick failure 3) 1
2017-06-05T10:15:57+00:00 localhost watchdog-vpxd: Executing '/usr/sbin/vpxd'
2017-06-05T10:15:57+00:00 localhost watchdog-vpxd: '/usr/sbin/vpxd' exited after 0 seconds (quick failure 4) 1
2017-06-05T10:15:57+00:00 localhost watchdog-vpxd: Executing '/usr/sbin/vpxd'
2017-06-05T10:15:58+00:00 localhost watchdog-vpxd: '/usr/sbin/vpxd' exited after 1 seconds (quick failure 5) 1
2017-06-05T10:15:58+00:00 localhost watchdog-vpxd: Executing '/usr/sbin/vpxd'
2017-06-05T10:15:58+00:00 localhost watchdog-vpxd: '/usr/sbin/vpxd' exited after 0 seconds (quick failure 6) 1
2017-06-05T10:15:58+00:00 localhost watchdog-vpxd: End '/usr/sbin/vpxd', failure limit reached
2017-06-05T10:20:01+00:00 localhost /usr/sbin/cron[5640]: (root) CMD ( test -x /usr/sbin/vpxd_periodic && /usr/sbin/vpxd_periodic >/dev/null 2>&1)
On vpxd.log, I also get some error.
localhost:/var/log/vmware/vpx # tailf -n 100 vpxd-171.log
Section for VMware VirtualCenter, pid=27456, version=5.5.0, build=2001466, option=Release
------ In-memory logs start --------
mem> 2017-06-06T04:14:20.657Z [7F9131494740 verbose 'vpxdvpxdSignal'] Signal handler for signal 1 is installed
mem> 2017-06-06T04:14:20.657Z [7F9131494740 verbose 'vpxdvpxdSignal'] Signal handler for signal 2 is installed
mem> 2017-06-06T04:14:20.657Z [7F9131494740 verbose 'vpxdvpxdSignal'] Signal handler for signal 15 is installed
mem> 2017-06-06T04:14:20.657Z [7F9131494740 verbose 'vpxdvpxdSignal'] Signal handler for signal 10 is installed
mem> 2017-06-06T04:14:20.657Z [7F9131494740 info 'Hooks'] Hooks Initialized
------ In-memory logs end --------
2017-06-06T04:14:20.660Z [7F9131494740 info 'Default'] Logging uses fast path: true
2017-06-06T04:14:20.660Z [7F9131494740 info 'Default'] Handling bora/lib logs with VmaCore facilities
2017-06-06T04:14:20.660Z [7F9131494740 info 'Default'] Initialized channel manager
2017-06-06T04:14:20.663Z [7F9131494740 info 'Default'] Current working directory: /storage/log/vmware/vpx
2017-06-06T04:14:20.663Z [7F9131494740 info 'ThreadPool'] Thread pool on asio: Min Io, Max Io, Min Task, Max Task, Max Concurency: 2, 200, 2, 90, 2147483647
2017-06-06T04:14:20.663Z [7F9131494740 info 'ThreadPool'] Thread enlisted
2017-06-06T04:14:20.663Z [7F9131494740 info 'Default'] Log path: /var/log/vmware/vpx
2017-06-06T04:14:20.663Z [7F9131494740 info 'Default'] Initializing SSL
2017-06-06T04:14:20.663Z [7F91278A4700 info 'ThreadPool'] Thread enlisted
2017-06-06T04:14:20.663Z [7F91279A6700 info 'ThreadPool'] Thread enlisted
2017-06-06T04:14:20.663Z [7F9127823700 info 'ThreadPool'] Thread enlisted
2017-06-06T04:14:20.663Z [7F9127925700 info 'ThreadPool'] Thread enlisted
2017-06-06T04:14:20.664Z [7F9131494740 info 'Default'] Vmacore::InitSSL: handshakeTimeoutUs = 120000000
2017-06-06T04:14:20.665Z [7F91277A2700 info 'ThreadPool'] Thread enlisted
2017-06-06T04:14:20.665Z [7F9131494740 info 'vpxdvpxdDaemon'] Changed working directory to /var/log/vmware/vpx
2017-06-06T04:14:20.666Z [7F9131494740 error 'vpxdvpxdDaemon'] Unhandled exception:
--> not well-formed (invalid token)
2017-06-06T04:14:20.667Z [7F9131494740 error 'vpxdvpxdDaemon'] Backtrace:
--> backtrace[00] rip 00007f912b3f9014 Vmacore::System::Stacktrace::CaptureWork(unsigned int)
--> backtrace[01] rip 00007f912b2e3132 Vmacore::System::SystemFactoryImpl::CreateQuickBacktrace(Vmacore::Ref<Vmacore::System::Backtrace>&)
--> backtrace[02] rip 00007f912b2448a5 Vmacore::Throwable::Throwable(std::string const&)
--> backtrace[03] rip 00007f912b28b566 /usr/lib/vmware-vpx/libvmacore.so(+0x285566) [0x7f912b28b566]
--> backtrace[04] rip 00007f912b23e85d /usr/lib/vmware-vpx/libvmacore.so(+0x23885d) [0x7f912b23e85d]
--> backtrace[05] rip 00007f9132abfd07 /usr/lib/vmware-vpx/vpxd(+0x14aad07) [0x7f9132abfd07]
--> backtrace[06] rip 00007f9132a3f0c5 /usr/lib/vmware-vpx/vpxd(+0x142a0c5) [0x7f9132a3f0c5]
--> backtrace[07] rip 00007f91324c3ca3 /usr/lib/vmware-vpx/vpxd(+0xeaeca3) [0x7f91324c3ca3]
--> backtrace[08] rip 00007f91324a6c2e /usr/lib/vmware-vpx/vpxd(+0xe91c2e) [0x7f91324a6c2e]
--> backtrace[09] rip 00007f9129369c16 /lib64/libc.so.6(__libc_start_main+0xe6) [0x7f9129369c16]
--> backtrace[10] rip 00007f91324a5e89 /usr/lib/vmware-vpx/vpxd(+0xe90e89) [0x7f91324a5e89]
I have gone through the above solutions for storage space full but i checked on our setup its not there.
localhost:/var/log/vmware/vpx # df -h
Filesystem Size Used Avail Use% Mounted on
/dev/sda3 9.8G 3.7G 5.6G 40% /
udev 4.0G 104K 4.0G 1% /dev
tmpfs 4.0G 0 4.0G 0% /dev/shm
/dev/sda1 128M 21M 101M 18% /boot
/dev/sdb1 20G 173M 19G 1% /storage/core
/dev/sdb2 20G 305M 19G 2% /storage/log
/dev/sdb3 60G 1.7G 55G 4% /storage/db
localhost:/var/log/vmware/vpx # df -ih
Filesystem Inodes IUsed IFree IUse% Mounted on
/dev/sda3 632K 60K 573K 10% /
udev 0 0 0 - /dev
tmpfs 1001K 2 1001K 1% /dev/shm
/dev/sda1 34K 40 34K 1% /boot
/dev/sdb1 1.3M 12 1.3M 1% /storage/core
/dev/sdb2 1.3M 293 1.3M 1% /storage/log
/dev/sdb3 3.8M 3.7K 3.8M 1% /storage/db
localhost:/var/log/vmware/vpx #