8 Replies Latest reply on Jan 12, 2018 2:14 PM by Kajetan

    vCenter Server Appliance > Backup Appliance Fails

    Kajetan Lurker

      Hello, for months now I have been unable to backup my vCenter Server Appliance using the "Backup" button found on the summary tab of the web GUI for the appliance.  The backup use to work just fine before a somewhat recent update to the appliance.

       

      After entering the required information at the "Enter backup details" screen I get "Validating input" for a moment and then I get: "Invalid vCenter Server Status: All required services are not up! Stopped services: 'statsmonitor'."

       

      I attempted to start 'statsmonitor' manually by executing: "service-control --start statsmonitor", the command prompt hangs for quite a while but eventually produces:

       

      Perform start operation. vmon_profile=None, svc_names=['statsmonitor'], include_coreossvcs=False, include_leafossvcs=False

      2018-01-05T19:33:04.441Z   Service statsmonitor state STOPPED

      Error executing start on service statsmonitor. Details {

          "resolution": null,

          "detail": [

              {

                  "args": [

                      "statsmonitor"

                  ],

                  "id": "install.ciscommon.service.failstart",

                  "localized": "An error occurred while starting service 'statsmonitor'",

                  "translatable": "An error occurred while starting service '%(0)s'"

              }

          ],

          "componentKey": null,

          "problemId": null

      }

      Service-control failed. Error {

          "resolution": null,

          "detail": [

              {

                  "args": [

                      "statsmonitor"

                  ],

                  "id": "install.ciscommon.service.failstart",

                  "localized": "An error occurred while starting service 'statsmonitor'",

                  "translatable": "An error occurred while starting service '%(0)s'"

              }

          ],

          "componentKey": null,

          "problemId": null

      }

      Command>

       

      I'm also experiencing the dreaded "Health Status > Overall Health: Unknown" issue.

       

      I'm running:

      Type: vCenter Server with an embedded Platform Services Controller

      Product: VMware vCenter Server Appliance

      Version: 6.5.0.13000

       

      As of today, I get "Update Status: Latest updates already installed on vCSA, nothing to stage/install" when I click "Check Updates".

       

      Please advise.

       

      Cheers,

      Kajetan.

        • 1. Re: vCenter Server Appliance > Backup Appliance Fails
          daphnissov Champion
          Community WarriorsvExpert

          In order to troubleshoot why statsmonitor isn't starting, you'll need to grab StatsMonitor.log which is a symlink to the current-numbered log file within /var/log/vmware/applmgmt.

          • 2. Re: vCenter Server Appliance > Backup Appliance Fails
            Kajetan Lurker

            Hi and thanks for replying.  I looked for StatsMonitor.log in /var/log but it doesn't appear to be there.  Is that the correct location to look in?  I'm quite new to this sort of thing.

             

            [root@esxi3:/var/log] ls

            Xorg.log                  iofilter-init.log         upitd.log

            auth.log                  iofiltervpd.log           usb.log

            boot.gz                   ipmi                      vitd.log

            clomd.log                 jumpstart-stdout.log      vmauthd.log

            cmmdsTimeMachine.log      kickstart.log             vmkdevmgr.log

            cmmdsTimeMachineDump.log  lacp.log                  vmkernel.log

            configRP.log              nfcd.log                  vmkeventd.log

            ddecomd.log               osfsd.log                 vmksummary.log

            dhclient.log              rabbitmqproxy.log         vmkwarning.log

            epd.log                   rhttpproxy.log            vmware

            esxcli-software.log       sdrsinjector.log          vobd.log

            esxupdate.log             shell.log                 vprobe.log

            fdm.log                   smbios.bin                vpxa.log

            hbrca.log                 storagerm.log             vsanmgmt.log

            hostd-probe.log           swapobjd.log              vsansystem.log

            hostd.log                 sysboot.log               vsanvpd.log

            hostdCgiServer.log        syslog.log                vvold.log

            hostprofiletrace.log      tallylog

            [root@esxi3:/var/log]

            • 3. Re: vCenter Server Appliance > Backup Appliance Fails
              daphnissov Champion
              vExpertCommunity Warriors

              You're logged into an ESXi host there and not the vCSA.

              • 4. Re: vCenter Server Appliance > Backup Appliance Fails
                Kajetan Lurker

                Got ya.  OK, here's what I got  from the appliance it's self:

                 

                root@vcenter [ /var/log ]# ls -l

                total 75084

                drwxr-xr-x 2 root    root                4096 Jan 12 03:16 audit

                -rw------- 1 root    root                   0 Oct 21  2016 auth.log

                -rw------- 1 root    utmp                   0 Jan  1 03:10 btmp

                -rw------- 1 root    utmp                   0 Dec 22 13:32 btmp-20180101

                -rw-r--r-- 1 root    root              769162 Dec 22 13:54 cloud-init.log

                lrwxrwxrwx 1 root    root                  18 Feb 24  2017 cm -> /var/log/vmware/cm

                -rw------- 1 root    root                   0 Oct 21  2016 cron

                drwx------ 2 root    root                4096 Jan 12 03:02 devicelist

                -rw-r----- 1 dnsmasq dnsmasq         14628401 Jan 12 09:49 dnsmasq.log

                -rw-r----- 1 dnsmasq root             3936501 Dec 24 03:50 dnsmasq.log-20171224

                -rw-r----- 1 dnsmasq dnsmasq         19743305 Jan  1 03:09 dnsmasq.log-20180101

                -rw-r----- 1 dnsmasq dnsmasq         16412825 Jan  7 03:23 dnsmasq.log-20180107

                -rw-r--r-- 1 root    root                7078 Feb 24  2017 fbStatusInt.json

                drwxr-xr-x 2 root    root                4096 Dec  7 00:51 firstboot

                -rw-r--r-- 1 root    root                   0 Oct 21  2016 installer.log

                -rw-r--r-- 1 root    root                  58 Feb 24  2017 invoked_procs

                drwxr-sr-x 4 root    systemd-journal     4096 Nov 29 19:53 journal

                -rw-r----- 1 root    tty                  292 Jan 12 09:33 lastlog

                -rw-r----- 1 root    tty                   48 Jan  5 13:55 lastlog.1.gz

                -rw-r----- 1 root    utmp                 317 Dec 22 13:35 lastlog.2.gz

                -rw-r----- 1 root    tty                   48 Dec  8 17:34 lastlog.3.gz

                -rw-r----- 1 root    tty                18671 Nov 24 14:11 lastlog.4.gz

                -rw-r----- 1 root    tty                   48 Aug 11 15:42 lastlog.5.gz

                -rw-r----- 1 root    root                 323 Feb 24  2017 lastlog.6.gz

                -rw-r----- 1 root    root               42880 Jan 12 09:48 messages

                -rw-r----- 1 root    root               16159 Jan  3 03:34 messages.10.gz

                -rw-r----- 1 root    root               16603 Jan  2 03:49 messages.11.gz

                -rw-r----- 1 root    root               16022 Jan  1 03:10 messages.12.gz

                -rw-r----- 1 root    root               16527 Dec 31 03:38 messages.13.gz

                -rw-r----- 1 root    root               16481 Dec 30 03:37 messages.14.gz

                -rw-r----- 1 root    root               16331 Dec 29 03:49 messages.15.gz

                -rw-r----- 1 root    root               16739 Jan 12 03:16 messages.1.gz

                -rw-r----- 1 root    root               15904 Jan 11 03:07 messages.2.gz

                -rw-r----- 1 root    root               16351 Jan 10 03:40 messages.3.gz

                -rw-r----- 1 root    root               16149 Jan  9 03:28 messages.4.gz

                -rw-r----- 1 root    root               16401 Jan  8 03:43 messages.5.gz

                -rw-r----- 1 root    root               16569 Jan  7 03:24 messages.6.gz

                -rw-r----- 1 root    root               23597 Jan  6 03:18 messages.7.gz

                -rw-r----- 1 root    root               16545 Jan  5 03:50 messages.8.gz

                -rw-r----- 1 root    root               16167 Jan  4 03:24 messages.9.gz

                drwxr-xr-x 2 root    root                4096 Dec  7 00:51 postthaw

                drwxr-xr-x 2 root    root                4096 Dec  7 00:51 prefreeze

                -rw-r--r-- 1 root    root             6713393 Aug 11 15:20 procstate

                -rw-r--r-- 1 root    root             1736148 Aug  5 03:40 procstate-20170805.gz

                -rw-r--r-- 1 root    root             1698375 Aug  6 03:10 procstate-20170806.gz

                -rw-r--r-- 1 root    root             3432763 Aug  7 03:10 procstate-20170807.gz

                -rw-r--r-- 1 root    root             1773281 Aug  8 03:40 procstate-20170808.gz

                -rw-r--r-- 1 root    root             1726535 Aug  9 03:30 procstate-20170809.gz

                -rw-r--r-- 1 root    root             1749844 Aug 10 03:40 procstate-20170810.gz

                -rw-r--r-- 1 root    root             1735955 Aug 11 03:40 procstate-20170811.gz

                lrwxrwxrwx 1 root    root                  19 Feb 24  2017 remote -> /storage/log/remote

                drwxr-xr-x 2 root    root                4096 Dec  7 00:51 restore

                drwx------ 2 root    root                4096 Jan 12 03:02 rpmcheck

                -rw-r--r-- 1 root    root                2160 Dec 22 13:53 sa

                drwx------ 2 root    root                4096 Jan 12 03:05 sgidlist

                -rw------- 1 root    root               50895 Dec 22 13:32 stigreport.log

                drwx------ 2 root    root                4096 Jan 12 03:05 suidlist

                -rw------- 1 root    root                  64 Jan 12 09:33 tallylog

                -rw------- 1 root    root                  24 Aug 11 15:42 tallylog.10.gz

                -rw------- 1 root    root                  24 Aug  4 17:24 tallylog.11.gz

                -rw------- 1 root    root                  24 Jul  7  2017 tallylog.12.gz

                -rw------- 1 root    root                  24 Jun  2  2017 tallylog.13.gz

                -rw------- 1 root    root                  24 May  5  2017 tallylog.14.gz

                -rw------- 1 root    root                  24 Apr 28  2017 tallylog.15.gz

                -rw------- 1 root    root                  24 Jan  5 13:55 tallylog.1.gz

                -rw------- 1 root    root                  24 Dec 22 16:22 tallylog.2.gz

                -rw------- 1 root    root                4130 Dec  8 17:34 tallylog.3.gz

                -rw------- 1 root    root                  24 Dec  1 12:12 tallylog.4.gz

                -rw------- 1 root    root                4129 Nov 24 14:11 tallylog.5.gz

                -rw------- 1 root    root                  24 Oct 17 17:52 tallylog.6.gz

                -rw------- 1 root    root                  24 Oct 11 14:12 tallylog.7.gz

                -rw------- 1 root    root                  24 Oct  6 16:46 tallylog.8.gz

                -rw------- 1 root    root                  24 Sep  8 13:17 tallylog.9.gz

                -rw-r--r-- 1 root    root               22642 Feb 24  2017 tmp-cmd.out

                lrwxrwxrwx 1 root    root                  19 Feb 24  2017 vmware -> /storage/log/vmware

                -rw-r--r-- 1 root    root               14243 Dec 22 13:54 vmware-vgauthsvc.log.0

                -rw-r--r-- 1 root    root               65388 Dec 22 13:55 vmware-vmsvc.log

                -rw-rw-r-- 1 root    utmp                4224 Jan 12 09:33 wtmp

                drwxr-xr-x 2 root    root                4096 Apr 13  2017 YaST2

                 

                 

                Again, no sign of the specific log file.

                • 5. Re: vCenter Server Appliance > Backup Appliance Fails
                  daphnissov Champion
                  vExpertCommunity Warriors

                  Refer to my previous message:

                   

                  In order to troubleshoot why statsmonitor isn't starting, you'll need to grab StatsMonitor.log which is a symlink to the current-numbered log file within /var/log/vmware/applmgmt.

                  • 6. Re: vCenter Server Appliance > Backup Appliance Fails
                    Kajetan Lurker

                    Here we go:

                     

                    2018-01-05T14:33:04.482-05:00 Section for VMware VirtualCenter, pid=17192, version=6.5.0, build=7312210, option=Release

                    2018-01-05T14:33:04.482-05:00 verbose StatsMonitor[7FC82B787780] [Originator@6876 sub=Default] Dumping early logs:

                     

                    ------ Early init logs start --------

                    2018-01-05T14:33:04.478-05:00 info -[7FC82B787780] [Originator@6876 sub=Default] Glibc malloc guards disabled.

                    2018-01-05T14:33:04.478-05:00 info -[7FC82B787780] [Originator@6876 sub=Default] Initialized SystemFactory

                     

                    ------ Early init logs end   --------

                    2018-01-05T14:33:04.482-05:00 info StatsMonitor[7FC82B787780] [Originator@6876 sub=Default] Logging uses fast path: true

                    2018-01-05T14:33:04.482-05:00 info StatsMonitor[7FC82B787780] [Originator@6876 sub=Default] The bora/lib logs WILL be handled by VmaCore

                    2018-01-05T14:33:04.482-05:00 info StatsMonitor[7FC82B787780] [Originator@6876 sub=Default] Initialized channel manager

                    2018-01-05T14:33:04.482-05:00 info StatsMonitor[7FC82B787780] [Originator@6876 sub=Default] Current working directory: /storage/log/vmware/vmon

                    2018-01-05T14:33:04.482-05:00 info StatsMonitor[7FC82B787780] [Originator@6876 sub=ThreadPool] Catch work item exceptions disabled.

                    2018-01-05T14:33:04.482-05:00 info StatsMonitor[7FC82B787780] [Originator@6876 sub=FairScheduler] Priority level 4 is now active.

                    2018-01-05T14:33:04.482-05:00 info StatsMonitor[7FC82B787780] [Originator@6876 sub=WorkQueue.vmacoreDefaultIOCompletionQueue] Created: WorkQueue.vmacoreDefaultIOCompletionQueue, type = fair, priority = 4 , itemWeight = 1

                    2018-01-05T14:33:04.482-05:00 info StatsMonitor[7FC82B787780] [Originator@6876 sub=FairScheduler] Priority level 8 is now active.

                    2018-01-05T14:33:04.482-05:00 info StatsMonitor[7FC82B787780] [Originator@6876 sub=WorkQueue.vmacoreDefaultIOQueue] Created: WorkQueue.vmacoreDefaultIOQueue, type = fair, priority = 8 , itemWeight = 1

                    2018-01-05T14:33:04.482-05:00 info StatsMonitor[7FC82B787780] [Originator@6876 sub=FairScheduler] Priority level 16 is now active.

                    2018-01-05T14:33:04.482-05:00 info StatsMonitor[7FC82B787780] [Originator@6876 sub=WorkQueue.vmacoreDefaultLongTaskQueue] Created: WorkQueue.vmacoreDefaultLongTaskQueue, type = fair, priority = 16 , itemWeight = 1

                    2018-01-05T14:33:04.482-05:00 info StatsMonitor[7FC82B787780] [Originator@6876 sub=ThreadPool] Registered foreign worker - allocated: 1, idle: 0

                    2018-01-05T14:33:04.482-05:00 info StatsMonitor[7FC82B787780] [Originator@6876 sub=ThreadPool] Registered foreign worker - allocated: 2, idle: 0

                    2018-01-05T14:33:04.482-05:00 info StatsMonitor[7FC82B787780] [Originator@6876 sub=ThreadPool] Registered foreign worker - allocated: 3, idle: 0

                    2018-01-05T14:33:04.482-05:00 info StatsMonitor[7FC82B787780] [Originator@6876 sub=ThreadPool] Registered foreign worker - allocated: 4, idle: 0

                    2018-01-05T14:33:04.482-05:00 info StatsMonitor[7FC82B745700] [Originator@6876 sub=ThreadPool] Thread enlisted

                    2018-01-05T14:33:04.482-05:00 info StatsMonitor[7FC82B745700] [Originator@6876 sub=ThreadPool] Entering worker thread loop

                    2018-01-05T14:33:04.482-05:00 info StatsMonitor[7FC82B682700] [Originator@6876 sub=ThreadPool] Thread enlisted

                    2018-01-05T14:33:04.482-05:00 info StatsMonitor[7FC82B704700] [Originator@6876 sub=ThreadPool] Thread enlisted

                    2018-01-05T14:33:04.482-05:00 info StatsMonitor[7FC82B786700] [Originator@6876 sub=ThreadPool] Thread enlisted

                    2018-01-05T14:33:04.482-05:00 info StatsMonitor[7FC82B682700] [Originator@6876 sub=ThreadPool] Entering IO thread loop

                    2018-01-05T14:33:04.482-05:00 info StatsMonitor[7FC82B704700] [Originator@6876 sub=ThreadPool] Entering worker thread loop

                    2018-01-05T14:33:04.482-05:00 info StatsMonitor[7FC82B786700] [Originator@6876 sub=ThreadPool] Entering worker thread loop

                    2018-01-05T14:33:04.482-05:00 info StatsMonitor[7FC82B6C3700] [Originator@6876 sub=ThreadPool] Thread enlisted

                    2018-01-05T14:33:04.482-05:00 info StatsMonitor[7FC82B6C3700] [Originator@6876 sub=ThreadPool] Entering worker thread loop

                    2018-01-05T14:33:04.482-05:00 info StatsMonitor[7FC82B787780] [Originator@6876 sub=ThreadPool] Thread pool fair initial threads spawned. IO: 2, Min workers: 4, Max workers: 13, Reservation ratio: 9

                    2018-01-05T14:33:04.482-05:00 info StatsMonitor[7FC82B787780] [Originator@6876 sub=ThreadPool] Thread enlisted

                    2018-01-05T14:33:04.482-05:00 info StatsMonitor[7FC82B641700] [Originator@6876 sub=ThreadPool] Thread enlisted

                    2018-01-05T14:33:04.482-05:00 info StatsMonitor[7FC82B787780] [Originator@6876 sub=Default] Syscommand enabled: true

                    2018-01-05T14:33:04.482-05:00 info StatsMonitor[7FC82B641700] [Originator@6876 sub=ThreadPool] Entering IO thread loop

                    2018-01-05T14:33:04.482-05:00 info StatsMonitor[7FC82B787780] [Originator@6876 sub=Default] ReaperManager Initialized

                    2018-01-05T14:33:04.482-05:00 info StatsMonitor[7FC82B600700] [Originator@6876 sub=ThreadPool] Thread enlisted

                    2018-01-05T14:33:04.482-05:00 info StatsMonitor[7FC82B787780] [Originator@6876 sub=StatsMonitor] Setting up signal handlers

                    2018-01-05T14:33:04.483-05:00 info StatsMonitor[7FC82B600700] [Originator@6876 sub=ThreadPool] Entering fair thread loop

                    2018-01-05T14:33:04.483-05:00 error StatsMonitor[7FC82B787780] [Originator@6876 sub=StatsMonitor] Failed to register handler for signal: 0

                    2018-01-05T14:33:04.483-05:00 info StatsMonitor[7FC82B787780] [Originator@6876 sub=StatsMonitor] Initializing

                    2018-01-05T14:33:04.483-05:00 info StatsMonitor[7FC82B787780] [Originator@6876 sub=LinuxStatsProvider(94432600755760)] Registered 162 stats (sources:114, derivatives:48)

                    2018-01-05T14:33:04.483-05:00 info StatsMonitor[7FC82B787780] [Originator@6876 sub=StatsMonitor] Found 12 file systems and 7 dirs in config for monitoring.

                    2018-01-05T14:33:04.483-05:00 info StatsMonitor[7FC82B787780] [Originator@6876 sub=LinuxStorageStatsProvider(94432600923648)] Registered 3 stats

                    2018-01-05T14:33:04.483-05:00 info StatsMonitor[7FC82B787780] [Originator@6876 sub=StatsMonitor] SqliteStorageEngine using SQLite version: 3.17.0

                    2018-01-05T14:37:04.450-05:00 info StatsMonitor[7FC82B787780] [Originator@6876 sub=StatsMonitor] Received signal 15

                    2018-01-05T14:37:04.450-05:00 info StatsMonitor[7FC82B787780] [Originator@6876 sub=StatsMonitor] Shutting down

                    • 7. Re: vCenter Server Appliance > Backup Appliance Fails
                      daphnissov Champion
                      Community WarriorsvExpert

                      Hmm, it doesn't appear to have much information in it. You might want to open an SR (if you have support) to see what other logs to pull.

                      • 8. Re: vCenter Server Appliance > Backup Appliance Fails
                        Kajetan Lurker

                        Yeah, I don't think our license includes support.