I had to reboot this server for a maintenance. since then the services are not coming up. Is it because of /storage/archive being 100% full ?
root@XXXXXXX [ ~ ]# service-control --status
Running:
lwsmd vmafdd vmcad vmdird vmdnsd vmware-analytics vmware-certificatemanagement vmware-cis-license vmware-eam vmware-pod vmware-postgres-archiver vmware-rhttpproxy vmware-sca vmware-statsmonitor vmware-sts-idmd vmware-stsd vmware-vapi-endpoint vmware-vmon vmware-vpostgres vmware-vpxd-svcs vsphere-client vsphere-ui
Stopped:
applmgmt pschealth vmcam vmonapi vmware-cm vmware-content-library vmware-imagebuilder vmware-mbcs vmware-netdumper vmware-perfcharts vmware-rbd-watchdog vmware-sps vmware-topologysvc vmware-updatemgr vmware-vcha vmware-vpxd vmware-vsan-health vmware-vsm vsan-dps
2020-03-09T14:51:41.420Z Done running command
2020-03-09T14:51:41.421Z Successfully started service vmware-vmon
2020-03-09T14:51:41.421Z Running command: ['/usr/bin/systemctl', 'unset-environment', 'VMON_PROFILE']
2020-03-09T14:51:41.430Z Done running command
Successfully started service vmware-vmon
2020-03-09T14:51:41.431Z Running command: ['/usr/lib/vmware-vmon/vmon-cli', '--batchstart', 'ALL']
2020-03-09T14:58:49.248Z Done running command
Service-control failed. Error: Failed to start services in profile ALL. RC=1, stderr=Failed to start topologysvc, vmonapi, applmgmt, cm services. Error: Operation timed out
2020-05-15T05:13:14.486-04:00 | ERROR | state-manager1 | DefaultStateManager | Could not initialize endpoint runtime state.
com.vmware.vapi.endpoint.config.ConfigurationException: Failed to retrieve SSO settings.
at com.vmware.vapi.endpoint.cis.SsoSettingsBuilder.buildInitial(SsoSettingsBuilder.java:48)
at com.vmware.vapi.state.impl.DefaultStateManager.build(DefaultStateManager.java:353)
at com.vmware.vapi.state.impl.DefaultStateManager$1.doInitialConfig(DefaultStateManager.java:167)
at com.vmware.vapi.state.impl.DefaultStateManager$1.run(DefaultStateManager.java:150)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: com.vmware.vapi.endpoint.config.ConfigurationException: com.vmware.vim.vmomi.client.common.UnexpectedStatusCodeException: Unexpected status code: 503
at com.vmware.vapi.endpoint.cis.ls.LookupServiceClientWrapper.lookup(LookupServiceClientWrapper.java:292)
at com.vmware.vapi.endpoint.cis.ls.LookupServiceClientWrapper.lookupSso(LookupServiceClientWrapper.java:206)
at com.vmware.vapi.endpoint.cis.SsoSettingsBuilder.ssoSettings(SsoSettingsBuilder.java:107)
at com.vmware.vapi.endpoint.cis.SsoSettingsBuilder.buildInitial(SsoSettingsBuilder.java:41)
... 10 more
Caused by: com.vmware.vim.vmomi.client.common.UnexpectedStatusCodeException: Unexpected status code: 503
at com.vmware.vim.vmomi.client.common.Response$Status.getStatus(Response.java:58)
at com.vmware.vim.vmomi.client.http.impl.HttpExchangeBase.parseResponse(HttpExchangeBase.java:156)
at com.vmware.vim.vmomi.client.http.impl.HttpExchange.run(HttpExchange.java:53)
at com.vmware.vim.vmomi.client.http.impl.HttpProtocolBindingBase.executeRunnable(HttpProtocolBindingBase.java:226)
at com.vmware.vim.vmomi.client.http.impl.HttpProtocolBindingImpl.send(HttpProtocolBindingImpl.java:106)
at com.vmware.vim.vmomi.client.common.impl.MethodInvocationHandlerImpl$CallExecutor.sendCall(MethodInvocationHandlerImpl.java:629)
at com.vmware.vim.vmomi.client.common.impl.MethodInvocationHandlerImpl$CallExecutor.executeCall(MethodInvocationHandlerImpl.java:610)
at com.vmware.vim.vmomi.client.common.impl.MethodInvocationHandlerImpl.completeCall(MethodInvocationHandlerImpl.java:360)
at com.vmware.vim.vmomi.client.common.impl.MethodInvocationHandlerImpl.invokeOperation(MethodInvocationHandlerImpl.java:311)
at com.vmware.vim.vmomi.client.common.impl.MethodInvocationHandlerImpl.invoke(MethodInvocationHandlerImpl.java:184)
at com.sun.proxy.$Proxy38.list(Unknown Source)
at com.vmware.vapi.endpoint.cis.ls.LookupServiceClientWrapper.lookup(LookupServiceClientWrapper.java:288)
... 13 more
# df -h
Filesystem Size Used Avail Use% Mounted on
devtmpfs 9.8G 0 9.8G 0% /dev
tmpfs 9.8G 864K 9.8G 1% /dev/shm
tmpfs 9.8G 684K 9.8G 1% /run
tmpfs 9.8G 0 9.8G 0% /sys/fs/cgroup
/dev/sda3 11G 6.1G 4.0G 61% /
tmpfs 9.8G 912K 9.8G 1% /tmp
/dev/mapper/dblog_vg-dblog 15G 470M 14G 4% /storage/dblog
/dev/mapper/core_vg-core 50G 831M 46G 2% /storage/core
/dev/mapper/db_vg-db 9.8G 324M 8.9G 4% /storage/db
/dev/sda1 120M 34M 78M 31% /boot
/dev/mapper/netdump_vg-netdump 985M 1.3M 916M 1% /storage/netdump
/dev/mapper/log_vg-log 9.8G 6.6G 2.7G 71% /storage/log
/dev/mapper/autodeploy_vg-autodeploy 9.8G 23M 9.2G 1% /storage/autodeploy
/dev/mapper/imagebuilder_vg-imagebuilder 9.8G 23M 9.2G 1% /storage/imagebuilder
/dev/mapper/archive_vg-archive 50G 47G 20M 100% /storage/archive
/dev/mapper/updatemgr_vg-updatemgr 99G 64M 94G 1% /storage/updatemgr
/dev/mapper/seat_vg-seat 25G 1.1G 23G 5% /storage/seat
2020-05-26T08:13:43.671-04:00 warning rhttpproxy[79242] [Originator@6876 sub=Proxy Req 04617 Tunnel] Error connecting tunnel on TCP socket: (null): Connection refused: The remote service is not running, OR is overloaded, OR a firewall is rejecting connections.
2020-05-26T08:13:53.697-04:00 warning rhttpproxy[79240] [Originator@6876 sub=Default] Failed to connect socket; <io_obj p:0x00007f4ff80059e0, h:19, <TCP '127.0.0.1 : 46684'>, <TCP '127.0.0.1 : 8089'>>, e: 111(Connection refused)
2020-05-26T08:13:53.697-04:00 warning rhttpproxy[79240] [Originator@6876 sub=Proxy Req 04618 Tunnel] Error connecting tunnel on TCP socket: (null): Connection refused: The remote service is not running, OR is overloaded, OR a firewall is rejecting connections.
2020-05-26T08:14:03.729-04:00 warning rhttpproxy[79240] [Originator@6876 sub=Default] Failed to connect socket; <io_obj p:0x00007f50380166f0, h:21, <TCP '127.0.0.1 : 46706'>, <TCP '127.0.0.1 : 8089'>>, e: 111(Connection refused)
2020-05-26T08:14:03.729-04:00 warning rhttpproxy[79240] [Originator@6876 sub=Proxy Req 04620 Tunnel] Error connecting tunnel on TCP socket: (null): Connection refused: The remote service is not running, OR is overloaded, OR a firewall is rejecting connections.
2020-05-26T08:14:13.755-04:00 warning rhttpproxy[79234] [Originator@6876 sub=Default] Failed to connect socket; <io_obj p:0x00007f50500134d0, h:19, <TCP '127.0.0.1 : 46850'>, <TCP '127.0.0.1 : 8089'>>, e: 111(Connection refused)
2020-05-26T08:14:13.755-04:00 warning rhttpproxy[79234] [Originator@6876 sub=Proxy Req 04642 Tunnel] Error connecting tunnel on TCP socket: (null): Connection refused: The remote service is not running, OR is overloaded, OR a firewall is rejecting connections.
