Hello, for months now I have been unable to backup my vCenter Server Appliance using the "Backup" button found on the summary tab of the web GUI for the appliance. The backup use to work just fine before a somewhat recent update to the appliance.
After entering the required information at the "Enter backup details" screen I get "Validating input" for a moment and then I get: "Invalid vCenter Server Status: All required services are not up! Stopped services: 'statsmonitor'."
I attempted to start 'statsmonitor' manually by executing: "service-control --start statsmonitor", the command prompt hangs for quite a while but eventually produces:
Perform start operation. vmon_profile=None, svc_names=['statsmonitor'], include_coreossvcs=False, include_leafossvcs=False
2018-01-05T19:33:04.441Z Service statsmonitor state STOPPED
Error executing start on service statsmonitor. Details {
"resolution": null,
"detail": [
{
"args": [
"statsmonitor"
],
"id": "install.ciscommon.service.failstart",
"localized": "An error occurred while starting service 'statsmonitor'",
"translatable": "An error occurred while starting service '%(0)s'"
}
],
"componentKey": null,
"problemId": null
}
Service-control failed. Error {
"resolution": null,
"detail": [
{
"args": [
"statsmonitor"
],
"id": "install.ciscommon.service.failstart",
"localized": "An error occurred while starting service 'statsmonitor'",
"translatable": "An error occurred while starting service '%(0)s'"
}
],
"componentKey": null,
"problemId": null
}
Command>
I'm also experiencing the dreaded "Health Status > Overall Health: Unknown" issue.
I'm running:
Type: vCenter Server with an embedded Platform Services Controller
Product: VMware vCenter Server Appliance
Version: 6.5.0.13000
As of today, I get "Update Status: Latest updates already installed on vCSA, nothing to stage/install" when I click "Check Updates".
Please advise.
Cheers,
Kajetan.
In order to troubleshoot why statsmonitor isn't starting, you'll need to grab StatsMonitor.log which is a symlink to the current-numbered log file within /var/log/vmware/applmgmt.
Hi and thanks for replying. I looked for StatsMonitor.log in /var/log but it doesn't appear to be there. Is that the correct location to look in? I'm quite new to this sort of thing.
[root@esxi3:/var/log] ls
Xorg.log iofilter-init.log upitd.log
auth.log iofiltervpd.log usb.log
boot.gz ipmi vitd.log
clomd.log jumpstart-stdout.log vmauthd.log
cmmdsTimeMachine.log kickstart.log vmkdevmgr.log
cmmdsTimeMachineDump.log lacp.log vmkernel.log
configRP.log nfcd.log vmkeventd.log
ddecomd.log osfsd.log vmksummary.log
dhclient.log rabbitmqproxy.log vmkwarning.log
epd.log rhttpproxy.log vmware
esxcli-software.log sdrsinjector.log vobd.log
esxupdate.log shell.log vprobe.log
fdm.log smbios.bin vpxa.log
hbrca.log storagerm.log vsanmgmt.log
hostd-probe.log swapobjd.log vsansystem.log
hostd.log sysboot.log vsanvpd.log
hostdCgiServer.log syslog.log vvold.log
hostprofiletrace.log tallylog
[root@esxi3:/var/log]
You're logged into an ESXi host there and not the vCSA.
Got ya. OK, here's what I got from the appliance it's self:
root@vcenter [ /var/log ]# ls -l
total 75084
drwxr-xr-x 2 root root 4096 Jan 12 03:16 audit
-rw------- 1 root root 0 Oct 21 2016 auth.log
-rw------- 1 root utmp 0 Jan 1 03:10 btmp
-rw------- 1 root utmp 0 Dec 22 13:32 btmp-20180101
-rw-r--r-- 1 root root 769162 Dec 22 13:54 cloud-init.log
lrwxrwxrwx 1 root root 18 Feb 24 2017 cm -> /var/log/vmware/cm
-rw------- 1 root root 0 Oct 21 2016 cron
drwx------ 2 root root 4096 Jan 12 03:02 devicelist
-rw-r----- 1 dnsmasq dnsmasq 14628401 Jan 12 09:49 dnsmasq.log
-rw-r----- 1 dnsmasq root 3936501 Dec 24 03:50 dnsmasq.log-20171224
-rw-r----- 1 dnsmasq dnsmasq 19743305 Jan 1 03:09 dnsmasq.log-20180101
-rw-r----- 1 dnsmasq dnsmasq 16412825 Jan 7 03:23 dnsmasq.log-20180107
-rw-r--r-- 1 root root 7078 Feb 24 2017 fbStatusInt.json
drwxr-xr-x 2 root root 4096 Dec 7 00:51 firstboot
-rw-r--r-- 1 root root 0 Oct 21 2016 installer.log
-rw-r--r-- 1 root root 58 Feb 24 2017 invoked_procs
drwxr-sr-x 4 root systemd-journal 4096 Nov 29 19:53 journal
-rw-r----- 1 root tty 292 Jan 12 09:33 lastlog
-rw-r----- 1 root tty 48 Jan 5 13:55 lastlog.1.gz
-rw-r----- 1 root utmp 317 Dec 22 13:35 lastlog.2.gz
-rw-r----- 1 root tty 48 Dec 8 17:34 lastlog.3.gz
-rw-r----- 1 root tty 18671 Nov 24 14:11 lastlog.4.gz
-rw-r----- 1 root tty 48 Aug 11 15:42 lastlog.5.gz
-rw-r----- 1 root root 323 Feb 24 2017 lastlog.6.gz
-rw-r----- 1 root root 42880 Jan 12 09:48 messages
-rw-r----- 1 root root 16159 Jan 3 03:34 messages.10.gz
-rw-r----- 1 root root 16603 Jan 2 03:49 messages.11.gz
-rw-r----- 1 root root 16022 Jan 1 03:10 messages.12.gz
-rw-r----- 1 root root 16527 Dec 31 03:38 messages.13.gz
-rw-r----- 1 root root 16481 Dec 30 03:37 messages.14.gz
-rw-r----- 1 root root 16331 Dec 29 03:49 messages.15.gz
-rw-r----- 1 root root 16739 Jan 12 03:16 messages.1.gz
-rw-r----- 1 root root 15904 Jan 11 03:07 messages.2.gz
-rw-r----- 1 root root 16351 Jan 10 03:40 messages.3.gz
-rw-r----- 1 root root 16149 Jan 9 03:28 messages.4.gz
-rw-r----- 1 root root 16401 Jan 8 03:43 messages.5.gz
-rw-r----- 1 root root 16569 Jan 7 03:24 messages.6.gz
-rw-r----- 1 root root 23597 Jan 6 03:18 messages.7.gz
-rw-r----- 1 root root 16545 Jan 5 03:50 messages.8.gz
-rw-r----- 1 root root 16167 Jan 4 03:24 messages.9.gz
drwxr-xr-x 2 root root 4096 Dec 7 00:51 postthaw
drwxr-xr-x 2 root root 4096 Dec 7 00:51 prefreeze
-rw-r--r-- 1 root root 6713393 Aug 11 15:20 procstate
-rw-r--r-- 1 root root 1736148 Aug 5 03:40 procstate-20170805.gz
-rw-r--r-- 1 root root 1698375 Aug 6 03:10 procstate-20170806.gz
-rw-r--r-- 1 root root 3432763 Aug 7 03:10 procstate-20170807.gz
-rw-r--r-- 1 root root 1773281 Aug 8 03:40 procstate-20170808.gz
-rw-r--r-- 1 root root 1726535 Aug 9 03:30 procstate-20170809.gz
-rw-r--r-- 1 root root 1749844 Aug 10 03:40 procstate-20170810.gz
-rw-r--r-- 1 root root 1735955 Aug 11 03:40 procstate-20170811.gz
lrwxrwxrwx 1 root root 19 Feb 24 2017 remote -> /storage/log/remote
drwxr-xr-x 2 root root 4096 Dec 7 00:51 restore
drwx------ 2 root root 4096 Jan 12 03:02 rpmcheck
-rw-r--r-- 1 root root 2160 Dec 22 13:53 sa
drwx------ 2 root root 4096 Jan 12 03:05 sgidlist
-rw------- 1 root root 50895 Dec 22 13:32 stigreport.log
drwx------ 2 root root 4096 Jan 12 03:05 suidlist
-rw------- 1 root root 64 Jan 12 09:33 tallylog
-rw------- 1 root root 24 Aug 11 15:42 tallylog.10.gz
-rw------- 1 root root 24 Aug 4 17:24 tallylog.11.gz
-rw------- 1 root root 24 Jul 7 2017 tallylog.12.gz
-rw------- 1 root root 24 Jun 2 2017 tallylog.13.gz
-rw------- 1 root root 24 May 5 2017 tallylog.14.gz
-rw------- 1 root root 24 Apr 28 2017 tallylog.15.gz
-rw------- 1 root root 24 Jan 5 13:55 tallylog.1.gz
-rw------- 1 root root 24 Dec 22 16:22 tallylog.2.gz
-rw------- 1 root root 4130 Dec 8 17:34 tallylog.3.gz
-rw------- 1 root root 24 Dec 1 12:12 tallylog.4.gz
-rw------- 1 root root 4129 Nov 24 14:11 tallylog.5.gz
-rw------- 1 root root 24 Oct 17 17:52 tallylog.6.gz
-rw------- 1 root root 24 Oct 11 14:12 tallylog.7.gz
-rw------- 1 root root 24 Oct 6 16:46 tallylog.8.gz
-rw------- 1 root root 24 Sep 8 13:17 tallylog.9.gz
-rw-r--r-- 1 root root 22642 Feb 24 2017 tmp-cmd.out
lrwxrwxrwx 1 root root 19 Feb 24 2017 vmware -> /storage/log/vmware
-rw-r--r-- 1 root root 14243 Dec 22 13:54 vmware-vgauthsvc.log.0
-rw-r--r-- 1 root root 65388 Dec 22 13:55 vmware-vmsvc.log
-rw-rw-r-- 1 root utmp 4224 Jan 12 09:33 wtmp
drwxr-xr-x 2 root root 4096 Apr 13 2017 YaST2
Again, no sign of the specific log file.
Refer to my previous message:
In order to troubleshoot why statsmonitor isn't starting, you'll need to grab StatsMonitor.log which is a symlink to the current-numbered log file within /var/log/vmware/applmgmt.
Here we go:
2018-01-05T14:33:04.482-05:00 Section for VMware VirtualCenter, pid=17192, version=6.5.0, build=7312210, option=Release
2018-01-05T14:33:04.482-05:00 verbose StatsMonitor[7FC82B787780] [Originator@6876 sub=Default] Dumping early logs:
------ Early init logs start --------
2018-01-05T14:33:04.478-05:00 info -[7FC82B787780] [Originator@6876 sub=Default] Glibc malloc guards disabled.
2018-01-05T14:33:04.478-05:00 info -[7FC82B787780] [Originator@6876 sub=Default] Initialized SystemFactory
------ Early init logs end --------
2018-01-05T14:33:04.482-05:00 info StatsMonitor[7FC82B787780] [Originator@6876 sub=Default] Logging uses fast path: true
2018-01-05T14:33:04.482-05:00 info StatsMonitor[7FC82B787780] [Originator@6876 sub=Default] The bora/lib logs WILL be handled by VmaCore
2018-01-05T14:33:04.482-05:00 info StatsMonitor[7FC82B787780] [Originator@6876 sub=Default] Initialized channel manager
2018-01-05T14:33:04.482-05:00 info StatsMonitor[7FC82B787780] [Originator@6876 sub=Default] Current working directory: /storage/log/vmware/vmon
2018-01-05T14:33:04.482-05:00 info StatsMonitor[7FC82B787780] [Originator@6876 sub=ThreadPool] Catch work item exceptions disabled.
2018-01-05T14:33:04.482-05:00 info StatsMonitor[7FC82B787780] [Originator@6876 sub=FairScheduler] Priority level 4 is now active.
2018-01-05T14:33:04.482-05:00 info StatsMonitor[7FC82B787780] [Originator@6876 sub=WorkQueue.vmacoreDefaultIOCompletionQueue] Created: WorkQueue.vmacoreDefaultIOCompletionQueue, type = fair, priority = 4 , itemWeight = 1
2018-01-05T14:33:04.482-05:00 info StatsMonitor[7FC82B787780] [Originator@6876 sub=FairScheduler] Priority level 8 is now active.
2018-01-05T14:33:04.482-05:00 info StatsMonitor[7FC82B787780] [Originator@6876 sub=WorkQueue.vmacoreDefaultIOQueue] Created: WorkQueue.vmacoreDefaultIOQueue, type = fair, priority = 8 , itemWeight = 1
2018-01-05T14:33:04.482-05:00 info StatsMonitor[7FC82B787780] [Originator@6876 sub=FairScheduler] Priority level 16 is now active.
2018-01-05T14:33:04.482-05:00 info StatsMonitor[7FC82B787780] [Originator@6876 sub=WorkQueue.vmacoreDefaultLongTaskQueue] Created: WorkQueue.vmacoreDefaultLongTaskQueue, type = fair, priority = 16 , itemWeight = 1
2018-01-05T14:33:04.482-05:00 info StatsMonitor[7FC82B787780] [Originator@6876 sub=ThreadPool] Registered foreign worker - allocated: 1, idle: 0
2018-01-05T14:33:04.482-05:00 info StatsMonitor[7FC82B787780] [Originator@6876 sub=ThreadPool] Registered foreign worker - allocated: 2, idle: 0
2018-01-05T14:33:04.482-05:00 info StatsMonitor[7FC82B787780] [Originator@6876 sub=ThreadPool] Registered foreign worker - allocated: 3, idle: 0
2018-01-05T14:33:04.482-05:00 info StatsMonitor[7FC82B787780] [Originator@6876 sub=ThreadPool] Registered foreign worker - allocated: 4, idle: 0
2018-01-05T14:33:04.482-05:00 info StatsMonitor[7FC82B745700] [Originator@6876 sub=ThreadPool] Thread enlisted
2018-01-05T14:33:04.482-05:00 info StatsMonitor[7FC82B745700] [Originator@6876 sub=ThreadPool] Entering worker thread loop
2018-01-05T14:33:04.482-05:00 info StatsMonitor[7FC82B682700] [Originator@6876 sub=ThreadPool] Thread enlisted
2018-01-05T14:33:04.482-05:00 info StatsMonitor[7FC82B704700] [Originator@6876 sub=ThreadPool] Thread enlisted
2018-01-05T14:33:04.482-05:00 info StatsMonitor[7FC82B786700] [Originator@6876 sub=ThreadPool] Thread enlisted
2018-01-05T14:33:04.482-05:00 info StatsMonitor[7FC82B682700] [Originator@6876 sub=ThreadPool] Entering IO thread loop
2018-01-05T14:33:04.482-05:00 info StatsMonitor[7FC82B704700] [Originator@6876 sub=ThreadPool] Entering worker thread loop
2018-01-05T14:33:04.482-05:00 info StatsMonitor[7FC82B786700] [Originator@6876 sub=ThreadPool] Entering worker thread loop
2018-01-05T14:33:04.482-05:00 info StatsMonitor[7FC82B6C3700] [Originator@6876 sub=ThreadPool] Thread enlisted
2018-01-05T14:33:04.482-05:00 info StatsMonitor[7FC82B6C3700] [Originator@6876 sub=ThreadPool] Entering worker thread loop
2018-01-05T14:33:04.482-05:00 info StatsMonitor[7FC82B787780] [Originator@6876 sub=ThreadPool] Thread pool fair initial threads spawned. IO: 2, Min workers: 4, Max workers: 13, Reservation ratio: 9
2018-01-05T14:33:04.482-05:00 info StatsMonitor[7FC82B787780] [Originator@6876 sub=ThreadPool] Thread enlisted
2018-01-05T14:33:04.482-05:00 info StatsMonitor[7FC82B641700] [Originator@6876 sub=ThreadPool] Thread enlisted
2018-01-05T14:33:04.482-05:00 info StatsMonitor[7FC82B787780] [Originator@6876 sub=Default] Syscommand enabled: true
2018-01-05T14:33:04.482-05:00 info StatsMonitor[7FC82B641700] [Originator@6876 sub=ThreadPool] Entering IO thread loop
2018-01-05T14:33:04.482-05:00 info StatsMonitor[7FC82B787780] [Originator@6876 sub=Default] ReaperManager Initialized
2018-01-05T14:33:04.482-05:00 info StatsMonitor[7FC82B600700] [Originator@6876 sub=ThreadPool] Thread enlisted
2018-01-05T14:33:04.482-05:00 info StatsMonitor[7FC82B787780] [Originator@6876 sub=StatsMonitor] Setting up signal handlers
2018-01-05T14:33:04.483-05:00 info StatsMonitor[7FC82B600700] [Originator@6876 sub=ThreadPool] Entering fair thread loop
2018-01-05T14:33:04.483-05:00 error StatsMonitor[7FC82B787780] [Originator@6876 sub=StatsMonitor] Failed to register handler for signal: 0
2018-01-05T14:33:04.483-05:00 info StatsMonitor[7FC82B787780] [Originator@6876 sub=StatsMonitor] Initializing
2018-01-05T14:33:04.483-05:00 info StatsMonitor[7FC82B787780] [Originator@6876 sub=LinuxStatsProvider(94432600755760)] Registered 162 stats (sources:114, derivatives:48)
2018-01-05T14:33:04.483-05:00 info StatsMonitor[7FC82B787780] [Originator@6876 sub=StatsMonitor] Found 12 file systems and 7 dirs in config for monitoring.
2018-01-05T14:33:04.483-05:00 info StatsMonitor[7FC82B787780] [Originator@6876 sub=LinuxStorageStatsProvider(94432600923648)] Registered 3 stats
2018-01-05T14:33:04.483-05:00 info StatsMonitor[7FC82B787780] [Originator@6876 sub=StatsMonitor] SqliteStorageEngine using SQLite version: 3.17.0
2018-01-05T14:37:04.450-05:00 info StatsMonitor[7FC82B787780] [Originator@6876 sub=StatsMonitor] Received signal 15
2018-01-05T14:37:04.450-05:00 info StatsMonitor[7FC82B787780] [Originator@6876 sub=StatsMonitor] Shutting down
Hmm, it doesn't appear to have much information in it. You might want to open an SR (if you have support) to see what other logs to pull.
Yeah, I don't think our license includes support.