VMware Cloud Community
Kajetan
Contributor
Contributor

vCenter Server Appliance > Backup Appliance Fails

Hello, for months now I have been unable to backup my vCenter Server Appliance using the "Backup" button found on the summary tab of the web GUI for the appliance.  The backup use to work just fine before a somewhat recent update to the appliance.

After entering the required information at the "Enter backup details" screen I get "Validating input" for a moment and then I get: "Invalid vCenter Server Status: All required services are not up! Stopped services: 'statsmonitor'."

I attempted to start 'statsmonitor' manually by executing: "service-control --start statsmonitor", the command prompt hangs for quite a while but eventually produces:

Perform start operation. vmon_profile=None, svc_names=['statsmonitor'], include_coreossvcs=False, include_leafossvcs=False

2018-01-05T19:33:04.441Z   Service statsmonitor state STOPPED

Error executing start on service statsmonitor. Details {

    "resolution": null,

    "detail": [

        {

            "args": [

                "statsmonitor"

            ],

            "id": "install.ciscommon.service.failstart",

            "localized": "An error occurred while starting service 'statsmonitor'",

            "translatable": "An error occurred while starting service '%(0)s'"

        }

    ],

    "componentKey": null,

    "problemId": null

}

Service-control failed. Error {

    "resolution": null,

    "detail": [

        {

            "args": [

                "statsmonitor"

            ],

            "id": "install.ciscommon.service.failstart",

            "localized": "An error occurred while starting service 'statsmonitor'",

            "translatable": "An error occurred while starting service '%(0)s'"

        }

    ],

    "componentKey": null,

    "problemId": null

}

Command>

I'm also experiencing the dreaded "Health Status > Overall Health: Unknown" issue.

I'm running:

Type: vCenter Server with an embedded Platform Services Controller

Product: VMware vCenter Server Appliance

Version: 6.5.0.13000

As of today, I get "Update Status: Latest updates already installed on vCSA, nothing to stage/install" when I click "Check Updates".

Please advise.

Cheers,

Kajetan.

Reply
0 Kudos
8 Replies
daphnissov
Immortal
Immortal

In order to troubleshoot why statsmonitor isn't starting, you'll need to grab StatsMonitor.log which is a symlink to the current-numbered log file within /var/log/vmware/applmgmt.

Reply
0 Kudos
Kajetan
Contributor
Contributor

Hi and thanks for replying.  I looked for StatsMonitor.log in /var/log but it doesn't appear to be there.  Is that the correct location to look in?  I'm quite new to this sort of thing.

[root@esxi3:/var/log] ls

Xorg.log                  iofilter-init.log         upitd.log

auth.log                  iofiltervpd.log           usb.log

boot.gz                   ipmi                      vitd.log

clomd.log                 jumpstart-stdout.log      vmauthd.log

cmmdsTimeMachine.log      kickstart.log             vmkdevmgr.log

cmmdsTimeMachineDump.log  lacp.log                  vmkernel.log

configRP.log              nfcd.log                  vmkeventd.log

ddecomd.log               osfsd.log                 vmksummary.log

dhclient.log              rabbitmqproxy.log         vmkwarning.log

epd.log                   rhttpproxy.log            vmware

esxcli-software.log       sdrsinjector.log          vobd.log

esxupdate.log             shell.log                 vprobe.log

fdm.log                   smbios.bin                vpxa.log

hbrca.log                 storagerm.log             vsanmgmt.log

hostd-probe.log           swapobjd.log              vsansystem.log

hostd.log                 sysboot.log               vsanvpd.log

hostdCgiServer.log        syslog.log                vvold.log

hostprofiletrace.log      tallylog

[root@esxi3:/var/log]

Reply
0 Kudos
daphnissov
Immortal
Immortal

You're logged into an ESXi host there and not the vCSA.

Reply
0 Kudos
Kajetan
Contributor
Contributor

Got ya.  OK, here's what I got  from the appliance it's self:

root@vcenter [ /var/log ]# ls -l

total 75084

drwxr-xr-x 2 root    root                4096 Jan 12 03:16 audit

-rw------- 1 root    root                   0 Oct 21  2016 auth.log

-rw------- 1 root    utmp                   0 Jan  1 03:10 btmp

-rw------- 1 root    utmp                   0 Dec 22 13:32 btmp-20180101

-rw-r--r-- 1 root    root              769162 Dec 22 13:54 cloud-init.log

lrwxrwxrwx 1 root    root                  18 Feb 24  2017 cm -> /var/log/vmware/cm

-rw------- 1 root    root                   0 Oct 21  2016 cron

drwx------ 2 root    root                4096 Jan 12 03:02 devicelist

-rw-r----- 1 dnsmasq dnsmasq         14628401 Jan 12 09:49 dnsmasq.log

-rw-r----- 1 dnsmasq root             3936501 Dec 24 03:50 dnsmasq.log-20171224

-rw-r----- 1 dnsmasq dnsmasq         19743305 Jan  1 03:09 dnsmasq.log-20180101

-rw-r----- 1 dnsmasq dnsmasq         16412825 Jan  7 03:23 dnsmasq.log-20180107

-rw-r--r-- 1 root    root                7078 Feb 24  2017 fbStatusInt.json

drwxr-xr-x 2 root    root                4096 Dec  7 00:51 firstboot

-rw-r--r-- 1 root    root                   0 Oct 21  2016 installer.log

-rw-r--r-- 1 root    root                  58 Feb 24  2017 invoked_procs

drwxr-sr-x 4 root    systemd-journal     4096 Nov 29 19:53 journal

-rw-r----- 1 root    tty                  292 Jan 12 09:33 lastlog

-rw-r----- 1 root    tty                   48 Jan  5 13:55 lastlog.1.gz

-rw-r----- 1 root    utmp                 317 Dec 22 13:35 lastlog.2.gz

-rw-r----- 1 root    tty                   48 Dec  8 17:34 lastlog.3.gz

-rw-r----- 1 root    tty                18671 Nov 24 14:11 lastlog.4.gz

-rw-r----- 1 root    tty                   48 Aug 11 15:42 lastlog.5.gz

-rw-r----- 1 root    root                 323 Feb 24  2017 lastlog.6.gz

-rw-r----- 1 root    root               42880 Jan 12 09:48 messages

-rw-r----- 1 root    root               16159 Jan  3 03:34 messages.10.gz

-rw-r----- 1 root    root               16603 Jan  2 03:49 messages.11.gz

-rw-r----- 1 root    root               16022 Jan  1 03:10 messages.12.gz

-rw-r----- 1 root    root               16527 Dec 31 03:38 messages.13.gz

-rw-r----- 1 root    root               16481 Dec 30 03:37 messages.14.gz

-rw-r----- 1 root    root               16331 Dec 29 03:49 messages.15.gz

-rw-r----- 1 root    root               16739 Jan 12 03:16 messages.1.gz

-rw-r----- 1 root    root               15904 Jan 11 03:07 messages.2.gz

-rw-r----- 1 root    root               16351 Jan 10 03:40 messages.3.gz

-rw-r----- 1 root    root               16149 Jan  9 03:28 messages.4.gz

-rw-r----- 1 root    root               16401 Jan  8 03:43 messages.5.gz

-rw-r----- 1 root    root               16569 Jan  7 03:24 messages.6.gz

-rw-r----- 1 root    root               23597 Jan  6 03:18 messages.7.gz

-rw-r----- 1 root    root               16545 Jan  5 03:50 messages.8.gz

-rw-r----- 1 root    root               16167 Jan  4 03:24 messages.9.gz

drwxr-xr-x 2 root    root                4096 Dec  7 00:51 postthaw

drwxr-xr-x 2 root    root                4096 Dec  7 00:51 prefreeze

-rw-r--r-- 1 root    root             6713393 Aug 11 15:20 procstate

-rw-r--r-- 1 root    root             1736148 Aug  5 03:40 procstate-20170805.gz

-rw-r--r-- 1 root    root             1698375 Aug  6 03:10 procstate-20170806.gz

-rw-r--r-- 1 root    root             3432763 Aug  7 03:10 procstate-20170807.gz

-rw-r--r-- 1 root    root             1773281 Aug  8 03:40 procstate-20170808.gz

-rw-r--r-- 1 root    root             1726535 Aug  9 03:30 procstate-20170809.gz

-rw-r--r-- 1 root    root             1749844 Aug 10 03:40 procstate-20170810.gz

-rw-r--r-- 1 root    root             1735955 Aug 11 03:40 procstate-20170811.gz

lrwxrwxrwx 1 root    root                  19 Feb 24  2017 remote -> /storage/log/remote

drwxr-xr-x 2 root    root                4096 Dec  7 00:51 restore

drwx------ 2 root    root                4096 Jan 12 03:02 rpmcheck

-rw-r--r-- 1 root    root                2160 Dec 22 13:53 sa

drwx------ 2 root    root                4096 Jan 12 03:05 sgidlist

-rw------- 1 root    root               50895 Dec 22 13:32 stigreport.log

drwx------ 2 root    root                4096 Jan 12 03:05 suidlist

-rw------- 1 root    root                  64 Jan 12 09:33 tallylog

-rw------- 1 root    root                  24 Aug 11 15:42 tallylog.10.gz

-rw------- 1 root    root                  24 Aug  4 17:24 tallylog.11.gz

-rw------- 1 root    root                  24 Jul  7  2017 tallylog.12.gz

-rw------- 1 root    root                  24 Jun  2  2017 tallylog.13.gz

-rw------- 1 root    root                  24 May  5  2017 tallylog.14.gz

-rw------- 1 root    root                  24 Apr 28  2017 tallylog.15.gz

-rw------- 1 root    root                  24 Jan  5 13:55 tallylog.1.gz

-rw------- 1 root    root                  24 Dec 22 16:22 tallylog.2.gz

-rw------- 1 root    root                4130 Dec  8 17:34 tallylog.3.gz

-rw------- 1 root    root                  24 Dec  1 12:12 tallylog.4.gz

-rw------- 1 root    root                4129 Nov 24 14:11 tallylog.5.gz

-rw------- 1 root    root                  24 Oct 17 17:52 tallylog.6.gz

-rw------- 1 root    root                  24 Oct 11 14:12 tallylog.7.gz

-rw------- 1 root    root                  24 Oct  6 16:46 tallylog.8.gz

-rw------- 1 root    root                  24 Sep  8 13:17 tallylog.9.gz

-rw-r--r-- 1 root    root               22642 Feb 24  2017 tmp-cmd.out

lrwxrwxrwx 1 root    root                  19 Feb 24  2017 vmware -> /storage/log/vmware

-rw-r--r-- 1 root    root               14243 Dec 22 13:54 vmware-vgauthsvc.log.0

-rw-r--r-- 1 root    root               65388 Dec 22 13:55 vmware-vmsvc.log

-rw-rw-r-- 1 root    utmp                4224 Jan 12 09:33 wtmp

drwxr-xr-x 2 root    root                4096 Apr 13  2017 YaST2

Again, no sign of the specific log file.

Reply
0 Kudos
daphnissov
Immortal
Immortal

Refer to my previous message:

In order to troubleshoot why statsmonitor isn't starting, you'll need to grab StatsMonitor.log which is a symlink to the current-numbered log file within /var/log/vmware/applmgmt.

Reply
0 Kudos
Kajetan
Contributor
Contributor

Here we go:

2018-01-05T14:33:04.482-05:00 Section for VMware VirtualCenter, pid=17192, version=6.5.0, build=7312210, option=Release

2018-01-05T14:33:04.482-05:00 verbose StatsMonitor[7FC82B787780] [Originator@6876 sub=Default] Dumping early logs:

------ Early init logs start --------

2018-01-05T14:33:04.478-05:00 info -[7FC82B787780] [Originator@6876 sub=Default] Glibc malloc guards disabled.

2018-01-05T14:33:04.478-05:00 info -[7FC82B787780] [Originator@6876 sub=Default] Initialized SystemFactory

------ Early init logs end   --------

2018-01-05T14:33:04.482-05:00 info StatsMonitor[7FC82B787780] [Originator@6876 sub=Default] Logging uses fast path: true

2018-01-05T14:33:04.482-05:00 info StatsMonitor[7FC82B787780] [Originator@6876 sub=Default] The bora/lib logs WILL be handled by VmaCore

2018-01-05T14:33:04.482-05:00 info StatsMonitor[7FC82B787780] [Originator@6876 sub=Default] Initialized channel manager

2018-01-05T14:33:04.482-05:00 info StatsMonitor[7FC82B787780] [Originator@6876 sub=Default] Current working directory: /storage/log/vmware/vmon

2018-01-05T14:33:04.482-05:00 info StatsMonitor[7FC82B787780] [Originator@6876 sub=ThreadPool] Catch work item exceptions disabled.

2018-01-05T14:33:04.482-05:00 info StatsMonitor[7FC82B787780] [Originator@6876 sub=FairScheduler] Priority level 4 is now active.

2018-01-05T14:33:04.482-05:00 info StatsMonitor[7FC82B787780] [Originator@6876 sub=WorkQueue.vmacoreDefaultIOCompletionQueue] Created: WorkQueue.vmacoreDefaultIOCompletionQueue, type = fair, priority = 4 , itemWeight = 1

2018-01-05T14:33:04.482-05:00 info StatsMonitor[7FC82B787780] [Originator@6876 sub=FairScheduler] Priority level 8 is now active.

2018-01-05T14:33:04.482-05:00 info StatsMonitor[7FC82B787780] [Originator@6876 sub=WorkQueue.vmacoreDefaultIOQueue] Created: WorkQueue.vmacoreDefaultIOQueue, type = fair, priority = 8 , itemWeight = 1

2018-01-05T14:33:04.482-05:00 info StatsMonitor[7FC82B787780] [Originator@6876 sub=FairScheduler] Priority level 16 is now active.

2018-01-05T14:33:04.482-05:00 info StatsMonitor[7FC82B787780] [Originator@6876 sub=WorkQueue.vmacoreDefaultLongTaskQueue] Created: WorkQueue.vmacoreDefaultLongTaskQueue, type = fair, priority = 16 , itemWeight = 1

2018-01-05T14:33:04.482-05:00 info StatsMonitor[7FC82B787780] [Originator@6876 sub=ThreadPool] Registered foreign worker - allocated: 1, idle: 0

2018-01-05T14:33:04.482-05:00 info StatsMonitor[7FC82B787780] [Originator@6876 sub=ThreadPool] Registered foreign worker - allocated: 2, idle: 0

2018-01-05T14:33:04.482-05:00 info StatsMonitor[7FC82B787780] [Originator@6876 sub=ThreadPool] Registered foreign worker - allocated: 3, idle: 0

2018-01-05T14:33:04.482-05:00 info StatsMonitor[7FC82B787780] [Originator@6876 sub=ThreadPool] Registered foreign worker - allocated: 4, idle: 0

2018-01-05T14:33:04.482-05:00 info StatsMonitor[7FC82B745700] [Originator@6876 sub=ThreadPool] Thread enlisted

2018-01-05T14:33:04.482-05:00 info StatsMonitor[7FC82B745700] [Originator@6876 sub=ThreadPool] Entering worker thread loop

2018-01-05T14:33:04.482-05:00 info StatsMonitor[7FC82B682700] [Originator@6876 sub=ThreadPool] Thread enlisted

2018-01-05T14:33:04.482-05:00 info StatsMonitor[7FC82B704700] [Originator@6876 sub=ThreadPool] Thread enlisted

2018-01-05T14:33:04.482-05:00 info StatsMonitor[7FC82B786700] [Originator@6876 sub=ThreadPool] Thread enlisted

2018-01-05T14:33:04.482-05:00 info StatsMonitor[7FC82B682700] [Originator@6876 sub=ThreadPool] Entering IO thread loop

2018-01-05T14:33:04.482-05:00 info StatsMonitor[7FC82B704700] [Originator@6876 sub=ThreadPool] Entering worker thread loop

2018-01-05T14:33:04.482-05:00 info StatsMonitor[7FC82B786700] [Originator@6876 sub=ThreadPool] Entering worker thread loop

2018-01-05T14:33:04.482-05:00 info StatsMonitor[7FC82B6C3700] [Originator@6876 sub=ThreadPool] Thread enlisted

2018-01-05T14:33:04.482-05:00 info StatsMonitor[7FC82B6C3700] [Originator@6876 sub=ThreadPool] Entering worker thread loop

2018-01-05T14:33:04.482-05:00 info StatsMonitor[7FC82B787780] [Originator@6876 sub=ThreadPool] Thread pool fair initial threads spawned. IO: 2, Min workers: 4, Max workers: 13, Reservation ratio: 9

2018-01-05T14:33:04.482-05:00 info StatsMonitor[7FC82B787780] [Originator@6876 sub=ThreadPool] Thread enlisted

2018-01-05T14:33:04.482-05:00 info StatsMonitor[7FC82B641700] [Originator@6876 sub=ThreadPool] Thread enlisted

2018-01-05T14:33:04.482-05:00 info StatsMonitor[7FC82B787780] [Originator@6876 sub=Default] Syscommand enabled: true

2018-01-05T14:33:04.482-05:00 info StatsMonitor[7FC82B641700] [Originator@6876 sub=ThreadPool] Entering IO thread loop

2018-01-05T14:33:04.482-05:00 info StatsMonitor[7FC82B787780] [Originator@6876 sub=Default] ReaperManager Initialized

2018-01-05T14:33:04.482-05:00 info StatsMonitor[7FC82B600700] [Originator@6876 sub=ThreadPool] Thread enlisted

2018-01-05T14:33:04.482-05:00 info StatsMonitor[7FC82B787780] [Originator@6876 sub=StatsMonitor] Setting up signal handlers

2018-01-05T14:33:04.483-05:00 info StatsMonitor[7FC82B600700] [Originator@6876 sub=ThreadPool] Entering fair thread loop

2018-01-05T14:33:04.483-05:00 error StatsMonitor[7FC82B787780] [Originator@6876 sub=StatsMonitor] Failed to register handler for signal: 0

2018-01-05T14:33:04.483-05:00 info StatsMonitor[7FC82B787780] [Originator@6876 sub=StatsMonitor] Initializing

2018-01-05T14:33:04.483-05:00 info StatsMonitor[7FC82B787780] [Originator@6876 sub=LinuxStatsProvider(94432600755760)] Registered 162 stats (sources:114, derivatives:48)

2018-01-05T14:33:04.483-05:00 info StatsMonitor[7FC82B787780] [Originator@6876 sub=StatsMonitor] Found 12 file systems and 7 dirs in config for monitoring.

2018-01-05T14:33:04.483-05:00 info StatsMonitor[7FC82B787780] [Originator@6876 sub=LinuxStorageStatsProvider(94432600923648)] Registered 3 stats

2018-01-05T14:33:04.483-05:00 info StatsMonitor[7FC82B787780] [Originator@6876 sub=StatsMonitor] SqliteStorageEngine using SQLite version: 3.17.0

2018-01-05T14:37:04.450-05:00 info StatsMonitor[7FC82B787780] [Originator@6876 sub=StatsMonitor] Received signal 15

2018-01-05T14:37:04.450-05:00 info StatsMonitor[7FC82B787780] [Originator@6876 sub=StatsMonitor] Shutting down

Reply
0 Kudos
daphnissov
Immortal
Immortal

Hmm, it doesn't appear to have much information in it. You might want to open an SR (if you have support) to see what other logs to pull.

Reply
0 Kudos
Kajetan
Contributor
Contributor

Yeah, I don't think our license includes support.

Reply
0 Kudos