1 2 Previous Next 15 Replies Latest reply on Jun 25, 2020 2:05 AM by mfirth

    VUM stopped service , VCSA fail

    gabomedina83 Lurker

      Hi , I have a VCSA 6.7 with A few days ago, VUM service began to fail and shows the following error, the service is stopped and when I try to start it, the following error is observed

       

      could someone please give me some idea what is causing the failure???, thanks

       

      0.PNG

      2.png

      error.PNG

       

      ERROR TEXT

       

      root@vcenter [ ~ ]# service-control --start vmware-updatemgr

      Operation not cancellable. Please wait for it to finish...

      Performing start operation on service updatemgr...

      Error executing start on service updatemgr. Details {

          "detail": [

              {

                  "localized": "An error occurred while starting service 'updatemgr'",

                  "translatable": "An error occurred while starting service '%(0)s'",

                  "id": "install.ciscommon.service.failstart",

                  "args": [

                      "updatemgr"

                  ]

              }

          ],

          "componentKey": null,

          "problemId": null,

          "resolution": null

      }

      Service-control failed. Error: {

          "detail": [

              {

                  "localized": "An error occurred while starting service 'updatemgr'",

                  "translatable": "An error occurred while starting service '%(0)s'",

                  "id": "install.ciscommon.service.failstart",

                  "args": [

                      "updatemgr"

                  ]

              }

          ],

          "componentKey": null,

          "problemId": null,

          "resolution": null

      }

        • 1. Re: fail with VCSA and VUM , stopped service
          vijayrana968 Master
          vExpert

          Did you make any changes to certificates ! Or any recent activity performed !

          • 2. Re: fail with VCSA and VUM , stopped service
            gabomedina83 Lurker

            Hello, export the virtual machine and pass it to another esxi host, after making this migration I do not know if it started to fail

             

            I have this error in  vmware-vum-server.log

             

            root@vcenter [ ~ ]# vim /var/log/vmware/vmware-updatemgr/vum-server/vmware-vum-server.log

             

            2019-03-05T11:09:06.052-05:00 info vmware-vum-server[34463] [Originator@6876 sub=ThreadPool] Thread enlisted^M

            2019-03-05T11:09:06.052-05:00 info vmware-vum-server[34463] [Originator@6876 sub=ThreadPool] Entering worker thread loop^M

            2019-03-05T11:09:40.853-05:00 error vmware-vum-server[33772] [Originator@6876 sub=Ufa.HTTPService] Failed to read request; stream: <SSL(<io_obj p:0x00007fe8c4002a30, h:-1, <TCP '127.0.0.1 : 8084'>, <TCP '127.0.0.1 : 49154'> FD Closed>)>, error: N7Vmacore16TimeoutExceptionE(Operation timed out: Stream: <SSL(<io_obj p:0x00007fe8c4002a30, h:-1, <TCP '127.0.0.1 : 8084'>, <TCP '127.0.0.1 : 49154'> FD Closed>)>, duration: 00:00:45.793412 (hh:mm:ss.us))^M

            --> [context]zKq7AVECAQAAAOkXmwALdm13YXJlLXZ1bS1zZXJ2ZXIAAJR/NWxpYnZtYWNvcmUuc28AANQ9HACLShgAhGQbAJhCLADx8CgAYXEoAEV1KAAn8jUB1HMAbGlicHRocmVhZC5zby4wAAKdjA5saWJjLnNvLjYA[/context]^M

            2019-03-05T11:17:46.101-05:00 error vmware-vum-server[34463] [Originator@6876 sub=Ufa] Method integrity.VcIntegrity.retrieveVcIntegrityContent threw undeclared fault of type vim.fault.InvalidState^M

            2019-03-05T11:18:06.100-05:00 error vmware-vum-server[33771] [Originator@6876 sub=Ufa] Method integrity.VcIntegrity.retrieveVcIntegrityContent threw undeclared fault of type vim.fault.InvalidState^M

            2019-03-05T11:18:26.109-05:00 error vmware-vum-server[33823] [Originator@6876 sub=Ufa] Method integrity.VcIntegrity.retrieveVcIntegrityContent threw undeclared fault of type vim.fault.InvalidState^M

            2019-03-05T11:18:46.098-05:00 error vmware-vum-server[33820] [Originator@6876 sub=Ufa] Method integrity.VcIntegrity.retrieveVcIntegrityContent threw undeclared fault of type vim.fault.InvalidState^M

            2019-03-05T11:19:06.100-05:00 error vmware-vum-server[33770] [Originator@6876 sub=Ufa] Method integrity.VcIntegrity.retrieveVcIntegrityContent threw undeclared fault of type vim.fault.InvalidState^M

            2019-03-05T11:19:26.103-05:00 error vmware-vum-server[34389] [Originator@6876 sub=Ufa] Method integrity.VcIntegrity.retrieveVcIntegrityContent threw undeclared fault of type vim.fault.InvalidState^M

            2019-03-05T11:19:46.095-05:00 error vmware-vum-server[33772] [Originator@6876 sub=Ufa] Method integrity.VcIntegrity.retrieveVcIntegrityContent threw undeclared fault of type vim.fault.InvalidState^M

            2019-03-05T11:20:00.257-05:00 info vmware-vum-server[39217] [Originator@6876 sub=Default] [shutdownMgr,92] Ufa shutdown manager stopping... first make sure this thread is enlisted.^M

            2019-03-05T11:20:00.257-05:00 info vmware-vum-server[39217] [Originator@6876 sub=ThreadPool] Thread enlisted^M

            2019-03-05T11:20:00.257-05:00 info vmware-vum-server[39217] [Originator@6876 sub=Default] [shutdownMgr,97] Ufa shutdown manager stopping... Total number of callbacks: 1^M

            2019-03-05T11:20:00.362-05:00 info vmware-vum-server[39217] [Originator@6876 sub=Default] [shutdownMgr,109] Ufa shutdown manager stopped^M

            2019-03-05T11:20:00.362-05:00 info vmware-vum-server[33775] [Originator@6876 sub=ThreadPool] Thread delisted^M

            2019-03-05T11:20:00.362-05:00 info vmware-vum-server[33774] [Originator@6876 sub=ThreadPool] Thread delisted^M

            2019-03-05T11:20:00.362-05:00 info vmware-vum-server[33773] [Originator@6876 sub=ThreadPool] Thread delisted^M

            2019-03-05T11:20:00.362-05:00 info vmware-vum-server[33820] [Originator@6876 sub=ThreadPool] Thread delisted^M

            2019-03-05T11:20:00.362-05:00 info vmware-vum-server[33772] [Originator@6876 sub=ThreadPool] Thread delisted^M

            2019-03-05T11:20:00.362-05:00 info vmware-vum-server[33823] [Originator@6876 sub=ThreadPool] Thread delisted^M

            2019-03-05T11:20:00.362-05:00 info vmware-vum-server[33822] [Originator@6876 sub=ThreadPool] Thread delisted^M

            2019-03-05T11:20:00.362-05:00 info vmware-vum-server[33776] [Originator@6876 sub=ThreadPool] Thread delisted^M

            2019-03-05T11:20:00.362-05:00 info vmware-vum-server[33770] [Originator@6876 sub=ThreadPool] Thread delisted^M

            2019-03-05T11:20:00.362-05:00 info vmware-vum-server[34463] [Originator@6876 sub=ThreadPool] Thread delisted^M

            2019-03-05T11:20:00.362-05:00 info vmware-vum-server[33819] [Originator@6876 sub=ThreadPool] Thread delisted^M

            2019-03-05T11:20:00.362-05:00 info vmware-vum-server[34389] [Originator@6876 sub=ThreadPool] Thread delisted^M

            2019-03-05T11:20:00.362-05:00 info vmware-vum-server[33771] [Originator@6876 sub=ThreadPool] Thread delisted^M

            2019-03-05T11:20:01.362-05:00 info vmware-vum-server[33591] [Originator@6876 sub=Ufa] Plugin stopped^M

            2019-03-05T11:20:01.362-05:00 info vmware-vum-server[33591] [Originator@6876 sub=ThreadPool] Thread delisted^M

            2019-03-05T11:20:01.362-05:00 info vmware-vum-server[33591] [Originator@6876 sub=Default] [main,416] VCI service stopped.^M

            • 3. Re: VUM stopped service , VCSA fail
              mfirth Novice

              Hi,

               

              I'm having the same issue - the VUM service is refusing to start on my vCenter.

               

              We are running with an official SSL certificate for the GUI, to avoid users having to click through SSL errors to access vCenter.

               

              The SSL certificate was able to successfully install in January, so VUM must have started after the installation.

               

              However, we now need to update the SSL certificate again, and can't because VUM is in a failed state (and so any attempt to install a new certificate results in the vCenter rolling back because VUM fails to start).

               

              The problem appears to be that /usr/lib/vmware-updatemgr/bin/refreshCerts-utility.py is failing with a certificate error - from its log file:

               

              ----------------

              Can't update VUM service info: (vmodl.fault.InvalidArgument) {

                 dynamicType = <unset>,

                 dynamicProperty = (vmodl.DynamicProperty) [],

                 msg = '',

                 faultCause = <unset>,

                 faultMessage = (vmodl.LocalizableMessage) [],

                 invalidProperty = 'Invalid certificate'

              }

              Traceback (most recent call last):

                File "/usr/lib/vmware-updatemgr/bin/vumUtils.py", line 370, in updateServiceInfo

                  lsClient.ReRegister(vumServiceInfo.serviceId, setSpec)

                File "/usr/lib/vmware-updatemgr/bin/vumUtils.py", line 813, in addSecurityctxToRequests

                  reqMethod(self, *args)

                File "/usr/lib/vmware-updatemgr/bin/vumUtils.py", line 823, in ReRegister

                  sr.Set(svcId, newSpec)

                File "/usr/lib/vmware/site-packages/pyVmomi/VmomiSupport.py", line 557, in <lambda>

                  self.f(*(self.args + (obj,) + args), **kwargs)

                File "/usr/lib/vmware/site-packages/pyVmomi/VmomiSupport.py", line 363, in _InvokeMethod

                  return self._stub.InvokeMethod(self, info, args)

                File "/usr/lib/vmware/site-packages/pyVmomi/SoapAdapter.py", line 1418, in InvokeMethod

                  raise obj # pylint: disable-msg=E0702

              pyVmomi.VmomiSupport.vmodl.fault.InvalidArgument: (vmodl.fault.InvalidArgument) {

                 dynamicType = <unset>,

                 dynamicProperty = (vmodl.DynamicProperty) [],

                 msg = '',

                 faultCause = <unset>,

                 faultMessage = (vmodl.LocalizableMessage) [],

                 invalidProperty = 'Invalid certificate'

              }

              Unable to update VUM service info

              ----------------

               

              From what little I have been able to work out from the files involved, it looks like it is submitting a certificate from /usr/lib/vmware-updatemgr/bin/ssl/rui.crt to gain access to some service point (I haven't worked out the URL for what it is trying to access), and that is being rejected.

               

              From what little I can see in the vCenter GUI, that certificate file seems to match the certificate for "vpxd-extension" shown in the vCenter administration section.

               

              Is it possible to find logs from the "server" end of the connection, to see why it is having a problem?

               

              Is this someone that is being seen more widely? - it seems to be the same issue as is being described in this thread, which shows that at least some other people are having the problem.

              • 4. Re: VUM stopped service , VCSA fail
                mfirth Novice

                Hi,

                 

                I did a bit more digging, and (for my case) it seems that the old certificate was still being retrieved from somewhere (using "LookupServiceClient().GetServiceInfoList()") by vumUtils.py as the "sslTrust" field of the objects it was trying to re-register, and whatever it was trying to register them with was then rejecting the request because that certificate had expired.

                 

                I hacked the script to replace that field with the certificate that should have been present (which was already being used by vumUtils.py), and I was then able to get the update service to start, and it seems to be working.

                 

                I haven't yet checked to see whether the "LookupServiceClient().GetServiceInfoList()" call is still returning the old certificate, or whether the "re-registration" updated it to the correct one.

                 

                Attached is a patch with what I changed, in case it is useful to anyone else.

                2 people found this helpful
                • 5. Re: VUM stopped service , VCSA fail
                  irvingpop_chef Novice

                  thank you mfirth this patch fixed my VUM!!

                  • 6. Re: VUM stopped service , VCSA fail
                    icorbitt Lurker

                    Thanks a ton mfirth, got me out of a pinch.

                    • 7. Re: VUM stopped service , VCSA fail
                      Nacer_Bell Lurker

                      Hi,

                      I'm having exactly the same issue in my VCSA 6.5, I did not find vumUtils.py so I'm not sure how can I apply this patch?

                       

                      Appreciate your help !! 

                      • 8. Re: VUM stopped service , VCSA fail
                        psyko1337 Lurker

                        Thank you!

                         

                        worked in VCSA 6.7.0.42000

                        • 9. Re: VUM stopped service , VCSA fail
                          Amin Masoudifard Expert

                          I have a similar problem and upgrading the VCSA solves the problem. Can you try this? (provide a full backup of your VCSA, if you decide to upgrade)

                          Please mark my comment as the Correct Answer if this solution resolved your problem
                          • 10. Re: VUM stopped service , VCSA fail
                            luckyloveu Lurker

                            vumUtils.patch.zip how to use this patch

                            • 11. Re: VUM stopped service , VCSA fail
                              AlbertoCallegari Lurker

                              Hi mfirth... How can I use your patch?

                              • 12. Re: VUM stopped service , VCSA fail
                                mfirth Novice

                                Hi - I'm not sure if the file is the same or in the same place on 6.5 - the original poster and me are both using 6.7

                                 

                                On 6.7, the file is in /usr/lib/vmware-updatemgr/bin

                                • 13. Re: VUM stopped service , VCSA fail
                                  mfirth Novice

                                  Hi

                                   

                                  If you are using VMware V6.7, then the process is:

                                   

                                  Extract vumUtils.patch from the ZIP (on a PC, though it could be done on the VCSA if you really wanted)

                                  Log in to the VCSA and get a root Bash shell (just "shell" if you log in as root, a little more complex if you log in as a non-root user - there must be instructions somewhere on here!)

                                  Cd into /usr/lib/vmware-updatemgr/bin

                                  Copy the patch file into that directory on the VCSA (I did it by using "sftp" out from the VCSA to a Linux machine - there are probably other ways)

                                  Run "patch -p0 < vumUtils.patch" to patch the vumUtils.py file

                                  You should now be able to start the VUM service, either with "service-control --start vmware-updatemgr" from the VMware shell, or by restarting the VCSA

                                  • 14. Re: VUM stopped service , VCSA fail
                                    mfirth Novice

                                    Hi,

                                     

                                    See the reply I posted to luckyloveu - hopefully that helps

                                    1 2 Previous Next