gabomedina83
Contributor
Contributor

VUM stopped service , VCSA fail

Hi , I have a VCSA 6.7 with A few days ago, VUM service began to fail and shows the following error, the service is stopped and when I try to start it, the following error is observed

could someone please give me some idea what is causing the failure???, thanks

0.PNG

2.png

error.PNG

ERROR TEXT

root@vcenter [ ~ ]# service-control --start vmware-updatemgr

Operation not cancellable. Please wait for it to finish...

Performing start operation on service updatemgr...

Error executing start on service updatemgr. Details {

    "detail": [

        {

            "localized": "An error occurred while starting service 'updatemgr'",

            "translatable": "An error occurred while starting service '%(0)s'",

            "id": "install.ciscommon.service.failstart",

            "args": [

                "updatemgr"

            ]

        }

    ],

    "componentKey": null,

    "problemId": null,

    "resolution": null

}

Service-control failed. Error: {

    "detail": [

        {

            "localized": "An error occurred while starting service 'updatemgr'",

            "translatable": "An error occurred while starting service '%(0)s'",

            "id": "install.ciscommon.service.failstart",

            "args": [

                "updatemgr"

            ]

        }

    ],

    "componentKey": null,

    "problemId": null,

    "resolution": null

}

17 Replies
vijayrana968
Virtuoso
Virtuoso

Did you make any changes to certificates ! Or any recent activity performed !

0 Kudos
gabomedina83
Contributor
Contributor

Hello, export the virtual machine and pass it to another esxi host, after making this migration I do not know if it started to fail

I have this error in  vmware-vum-server.log

root@vcenter [ ~ ]# vim /var/log/vmware/vmware-updatemgr/vum-server/vmware-vum-server.log

2019-03-05T11:09:06.052-05:00 info vmware-vum-server[34463] [Originator@6876 sub=ThreadPool] Thread enlisted^M

2019-03-05T11:09:06.052-05:00 info vmware-vum-server[34463] [Originator@6876 sub=ThreadPool] Entering worker thread loop^M

2019-03-05T11:09:40.853-05:00 error vmware-vum-server[33772] [Originator@6876 sub=Ufa.HTTPService] Failed to read request; stream: <SSL(<io_obj p:0x00007fe8c4002a30, h:-1, <TCP '127.0.0.1 : 8084'>, <TCP '127.0.0.1 : 49154'> FD Closed>)>, error: N7Vmacore16TimeoutExceptionE(Operation timed out: Stream: <SSL(<io_obj p:0x00007fe8c4002a30, h:-1, <TCP '127.0.0.1 : 8084'>, <TCP '127.0.0.1 : 49154'> FD Closed>)>, duration: 00:00:45.793412 (hh:mm:ss.us))^M

--> [context]zKq7AVECAQAAAOkXmwALdm13YXJlLXZ1bS1zZXJ2ZXIAAJR/NWxpYnZtYWNvcmUuc28AANQ9HACLShgAhGQbAJhCLADx8CgAYXEoAEV1KAAn8jUB1HMAbGlicHRocmVhZC5zby4wAAKdjA5saWJjLnNvLjYA[/context]^M

2019-03-05T11:17:46.101-05:00 error vmware-vum-server[34463] [Originator@6876 sub=Ufa] Method integrity.VcIntegrity.retrieveVcIntegrityContent threw undeclared fault of type vim.fault.InvalidState^M

2019-03-05T11:18:06.100-05:00 error vmware-vum-server[33771] [Originator@6876 sub=Ufa] Method integrity.VcIntegrity.retrieveVcIntegrityContent threw undeclared fault of type vim.fault.InvalidState^M

2019-03-05T11:18:26.109-05:00 error vmware-vum-server[33823] [Originator@6876 sub=Ufa] Method integrity.VcIntegrity.retrieveVcIntegrityContent threw undeclared fault of type vim.fault.InvalidState^M

2019-03-05T11:18:46.098-05:00 error vmware-vum-server[33820] [Originator@6876 sub=Ufa] Method integrity.VcIntegrity.retrieveVcIntegrityContent threw undeclared fault of type vim.fault.InvalidState^M

2019-03-05T11:19:06.100-05:00 error vmware-vum-server[33770] [Originator@6876 sub=Ufa] Method integrity.VcIntegrity.retrieveVcIntegrityContent threw undeclared fault of type vim.fault.InvalidState^M

2019-03-05T11:19:26.103-05:00 error vmware-vum-server[34389] [Originator@6876 sub=Ufa] Method integrity.VcIntegrity.retrieveVcIntegrityContent threw undeclared fault of type vim.fault.InvalidState^M

2019-03-05T11:19:46.095-05:00 error vmware-vum-server[33772] [Originator@6876 sub=Ufa] Method integrity.VcIntegrity.retrieveVcIntegrityContent threw undeclared fault of type vim.fault.InvalidState^M

2019-03-05T11:20:00.257-05:00 info vmware-vum-server[39217] [Originator@6876 sub=Default] [shutdownMgr,92] Ufa shutdown manager stopping... first make sure this thread is enlisted.^M

2019-03-05T11:20:00.257-05:00 info vmware-vum-server[39217] [Originator@6876 sub=ThreadPool] Thread enlisted^M

2019-03-05T11:20:00.257-05:00 info vmware-vum-server[39217] [Originator@6876 sub=Default] [shutdownMgr,97] Ufa shutdown manager stopping... Total number of callbacks: 1^M

2019-03-05T11:20:00.362-05:00 info vmware-vum-server[39217] [Originator@6876 sub=Default] [shutdownMgr,109] Ufa shutdown manager stopped^M

2019-03-05T11:20:00.362-05:00 info vmware-vum-server[33775] [Originator@6876 sub=ThreadPool] Thread delisted^M

2019-03-05T11:20:00.362-05:00 info vmware-vum-server[33774] [Originator@6876 sub=ThreadPool] Thread delisted^M

2019-03-05T11:20:00.362-05:00 info vmware-vum-server[33773] [Originator@6876 sub=ThreadPool] Thread delisted^M

2019-03-05T11:20:00.362-05:00 info vmware-vum-server[33820] [Originator@6876 sub=ThreadPool] Thread delisted^M

2019-03-05T11:20:00.362-05:00 info vmware-vum-server[33772] [Originator@6876 sub=ThreadPool] Thread delisted^M

2019-03-05T11:20:00.362-05:00 info vmware-vum-server[33823] [Originator@6876 sub=ThreadPool] Thread delisted^M

2019-03-05T11:20:00.362-05:00 info vmware-vum-server[33822] [Originator@6876 sub=ThreadPool] Thread delisted^M

2019-03-05T11:20:00.362-05:00 info vmware-vum-server[33776] [Originator@6876 sub=ThreadPool] Thread delisted^M

2019-03-05T11:20:00.362-05:00 info vmware-vum-server[33770] [Originator@6876 sub=ThreadPool] Thread delisted^M

2019-03-05T11:20:00.362-05:00 info vmware-vum-server[34463] [Originator@6876 sub=ThreadPool] Thread delisted^M

2019-03-05T11:20:00.362-05:00 info vmware-vum-server[33819] [Originator@6876 sub=ThreadPool] Thread delisted^M

2019-03-05T11:20:00.362-05:00 info vmware-vum-server[34389] [Originator@6876 sub=ThreadPool] Thread delisted^M

2019-03-05T11:20:00.362-05:00 info vmware-vum-server[33771] [Originator@6876 sub=ThreadPool] Thread delisted^M

2019-03-05T11:20:01.362-05:00 info vmware-vum-server[33591] [Originator@6876 sub=Ufa] Plugin stopped^M

2019-03-05T11:20:01.362-05:00 info vmware-vum-server[33591] [Originator@6876 sub=ThreadPool] Thread delisted^M

2019-03-05T11:20:01.362-05:00 info vmware-vum-server[33591] [Originator@6876 sub=Default] [main,416] VCI service stopped.^M

0 Kudos
mfirth
Enthusiast
Enthusiast

Hi,

I'm having the same issue - the VUM service is refusing to start on my vCenter.

We are running with an official SSL certificate for the GUI, to avoid users having to click through SSL errors to access vCenter.

The SSL certificate was able to successfully install in January, so VUM must have started after the installation.

However, we now need to update the SSL certificate again, and can't because VUM is in a failed state (and so any attempt to install a new certificate results in the vCenter rolling back because VUM fails to start).

The problem appears to be that /usr/lib/vmware-updatemgr/bin/refreshCerts-utility.py is failing with a certificate error - from its log file:

----------------

Can't update VUM service info: (vmodl.fault.InvalidArgument) {

   dynamicType = <unset>,

   dynamicProperty = (vmodl.DynamicProperty) [],

   msg = '',

   faultCause = <unset>,

   faultMessage = (vmodl.LocalizableMessage) [],

   invalidProperty = 'Invalid certificate'

}

Traceback (most recent call last):

  File "/usr/lib/vmware-updatemgr/bin/vumUtils.py", line 370, in updateServiceInfo

    lsClient.ReRegister(vumServiceInfo.serviceId, setSpec)

  File "/usr/lib/vmware-updatemgr/bin/vumUtils.py", line 813, in addSecurityctxToRequests

    reqMethod(self, *args)

  File "/usr/lib/vmware-updatemgr/bin/vumUtils.py", line 823, in ReRegister

    sr.Set(svcId, newSpec)

  File "/usr/lib/vmware/site-packages/pyVmomi/VmomiSupport.py", line 557, in <lambda>

    self.f(*(self.args + (obj,) + args), **kwargs)

  File "/usr/lib/vmware/site-packages/pyVmomi/VmomiSupport.py", line 363, in _InvokeMethod

    return self._stub.InvokeMethod(self, info, args)

  File "/usr/lib/vmware/site-packages/pyVmomi/SoapAdapter.py", line 1418, in InvokeMethod

    raise obj # pylint: disable-msg=E0702

pyVmomi.VmomiSupport.vmodl.fault.InvalidArgument: (vmodl.fault.InvalidArgument) {

   dynamicType = <unset>,

   dynamicProperty = (vmodl.DynamicProperty) [],

   msg = '',

   faultCause = <unset>,

   faultMessage = (vmodl.LocalizableMessage) [],

   invalidProperty = 'Invalid certificate'

}

Unable to update VUM service info

----------------

From what little I have been able to work out from the files involved, it looks like it is submitting a certificate from /usr/lib/vmware-updatemgr/bin/ssl/rui.crt to gain access to some service point (I haven't worked out the URL for what it is trying to access), and that is being rejected.

From what little I can see in the vCenter GUI, that certificate file seems to match the certificate for "vpxd-extension" shown in the vCenter administration section.

Is it possible to find logs from the "server" end of the connection, to see why it is having a problem?

Is this someone that is being seen more widely? - it seems to be the same issue as is being described in this thread, which shows that at least some other people are having the problem.

0 Kudos
mfirth
Enthusiast
Enthusiast

Hi,

I did a bit more digging, and (for my case) it seems that the old certificate was still being retrieved from somewhere (using "LookupServiceClient().GetServiceInfoList()") by vumUtils.py as the "sslTrust" field of the objects it was trying to re-register, and whatever it was trying to register them with was then rejecting the request because that certificate had expired.

I hacked the script to replace that field with the certificate that should have been present (which was already being used by vumUtils.py), and I was then able to get the update service to start, and it seems to be working.

I haven't yet checked to see whether the "LookupServiceClient().GetServiceInfoList()" call is still returning the old certificate, or whether the "re-registration" updated it to the correct one.

Attached is a patch with what I changed, in case it is useful to anyone else.

irvingpop_chef
Contributor
Contributor

thank you mfirth​ this patch fixed my VUM!!

0 Kudos
icorbitt
Contributor
Contributor

Thanks a ton mfirth, got me out of a pinch.

0 Kudos
Nacer_Bell
Contributor
Contributor

Hi,

I'm having exactly the same issue in my VCSA 6.5, I did not find vumUtils.py so I'm not sure how can I apply this patch?

Appreciate your help !! 

0 Kudos
psyko1337
Contributor
Contributor

Thank you!

worked in VCSA 6.7.0.42000

0 Kudos
NathanosBlightc
Commander
Commander

I have a similar problem and upgrading the VCSA solves the problem. Can you try this? (provide a full backup of your VCSA, if you decide to upgrade)

Please mark my comment as the Correct Answer if this solution resolved your problem
0 Kudos
luckyloveu
Contributor
Contributor

vumUtils.patch.zip how to use this patch

0 Kudos
AlbertoCallegar
Contributor
Contributor

Hi mfirth​... How can I use your patch?

0 Kudos
mfirth
Enthusiast
Enthusiast

Hi - I'm not sure if the file is the same or in the same place on 6.5 - the original poster and me are both using 6.7

On 6.7, the file is in /usr/lib/vmware-updatemgr/bin

0 Kudos
mfirth
Enthusiast
Enthusiast

Hi

If you are using VMware V6.7, then the process is:

Extract vumUtils.patch from the ZIP (on a PC, though it could be done on the VCSA if you really wanted)

Log in to the VCSA and get a root Bash shell (just "shell" if you log in as root, a little more complex if you log in as a non-root user - there must be instructions somewhere on here!)

Cd into /usr/lib/vmware-updatemgr/bin

Copy the patch file into that directory on the VCSA (I did it by using "sftp" out from the VCSA to a Linux machine - there are probably other ways)

Run "patch -p0 < vumUtils.patch" to patch the vumUtils.py file

You should now be able to start the VUM service, either with "service-control --start vmware-updatemgr" from the VMware shell, or by restarting the VCSA

mfirth
Enthusiast
Enthusiast

Hi,

See the reply I posted to luckyloveu​ - hopefully that helps

0 Kudos
mfirth
Enthusiast
Enthusiast

It looks to me that the certificate is installed right the first time, but then VUM keeps retrieving the original certificate, rather than updating it when it changes.

So a re-install will fix the problem for however long your certificate is valid, but I don't think it is a permanent fix.

So far, I've just kept reapplying my patch whenever I update the VCSA if I have to - it seems to revert it on some updates but not others. I haven't had a problem doing that, so either the patch is still needed or it is at least doing no harm to the newer versions

For every version of VCSA through to 6.7.0.44100 the original vumUtils.py was binary the same as the version I created my patch against, so either VMware still haven't looked into this, or they have fixed it in a different way.

0 Kudos
tranquilnet
Contributor
Contributor

This fixed my issue, thanks! Thankfully I came across this post after already trying multiple "fixes" over several hours. Question: Do you think spinning up a new vCenter, and importing a backup created from the appliance GUI would help solve the issue more permanently? 

0 Kudos
Krexis
Contributor
Contributor

Many thanks and kudos @mfirth 

Worked like a charm and put an end to the hours and hours of finding and trying fixes that didn't make the grade.

0 Kudos