7 Replies Latest reply on Jun 9, 2020 10:52 AM by bbalido9

    Vmware site recovery manager server service is unable to start

    tcng889 Novice

      Observing the below error when we start the service. Can someone provide a solution to this

      From vmware-dr.log file:

       

      2020-01-30T05:06:47.905-08:00 warning vmware-dr[03604] [SRM@6876 sub=LocalSite.LocalSsoServer] Failed to connect:

      --> class SsoClient::InvalidCredentialsException Authentication failed: Invalid credentials

      --> [context]zKq7AVMEAAgAAFaTwQATdm13YXJlLWRyAACxPwJ2bWFjb3JlLmRsbAABOmECc3NvQ2xpZW50LmRsbAABaDoCAadKAgGczwEBGnYBAXuKAQH+cgEBo4gBAScFAgFUvwEBMIMBAOt+GwBLjhsAyYghAn9PAk1TVkNSMTIwLmRsbAACJlECA9ITAEtFUk5FTDMyLkRMTAAE5FQBbnRkbGwuZGxsAA==[/context]

      --> [backtrace begin] product: VMware vCenter Site Recovery Manager, version: 8.1.2, build: build-12686166, tag: vmware-dr, cpu: x86_64, os: windows, buildType: release

      --> backtrace[03] vmacore.dll[0x00023FB1]

      --> backtrace[04] ssoClient.dll[0x0002613A]

      --> backtrace[05] ssoClient.dll[0x00023A68]

      --> backtrace[06] ssoClient.dll[0x00024AA7]

      --> backtrace[07] ssoClient.dll[0x0001CF9C]

      --> backtrace[08] ssoClient.dll[0x0001761A]

      --> backtrace[09] ssoClient.dll[0x00018A7B]

      --> backtrace[10] ssoClient.dll[0x000172FE]

      --> backtrace[11] ssoClient.dll[0x000188A3]

      --> backtrace[12] ssoClient.dll[0x00020527]

      --> backtrace[13] ssoClient.dll[0x0001BF54]

      --> backtrace[14] ssoClient.dll[0x00018330]

      --> backtrace[15] vmacore.dll[0x001B7EEB]

      --> backtrace[16] vmacore.dll[0x001B8E4B]

      --> backtrace[17] vmacore.dll[0x002188C9]

      --> backtrace[18] MSVCR120.dll[0x00024F7F]

      --> backtrace[19] MSVCR120.dll[0x00025126]

      --> backtrace[20] KERNEL32.DLL[0x000013D2]

      --> backtrace[21] ntdll.dll[0x000154E4]

      --> [backtrace end]

      2020-01-30T05:07:13.048-08:00 verbose vmware-dr[02788] [SRM@6876 sub=LocalSite.LocalVersionReg] Timeout event broadcasted.

      2020-01-30T05:07:13.048-08:00 verbose vmware-dr[01744] [SRM@6876 sub=LocalSite.LocalVersionReg] Timeout event broadcasted.

      2020-01-30T05:07:13.079-08:00 verbose vmware-dr[04036] [SRM@6876 sub=ServerSingletonCache] Timeout event broadcasted.

      2020-01-30T05:07:13.079-08:00 verbose vmware-dr[04056] [SRM@6876 sub=StubFactory] Timeout event broadcasted.

      2020-01-30T05:07:13.079-08:00 info vmware-dr[01980] [SRM@6876 sub=ThreadPool] Spawning additional worker - allocated: 13, idle: 2

      2020-01-30T05:07:13.079-08:00 verbose vmware-dr[04028] [SRM@6876 sub=ServerSingletonCache] Timeout event broadcasted.

      2020-01-30T05:07:13.079-08:00 verbose vmware-dr[03604] [SRM@6876 sub=StubFactory] Timeout event broadcasted.

      2020-01-30T05:07:13.079-08:00 verbose vmware-dr[01980] [SRM@6876 sub=StubFactory] Timeout event broadcasted.

      2020-01-30T05:07:13.079-08:00 verbose vmware-dr[04036] [SRM@6876 sub=ServerSingletonCache] Timeout event broadcasted.

      2020-01-30T05:07:13.079-08:00 info vmware-dr[04056] [SRM@6876 sub=ThreadPool] Spawning additional worker - allocated: 14, idle: 3

      2020-01-30T05:07:13.079-08:00 verbose vmware-dr[00952] [SRM@6876 sub=ServerSingletonCache] Timeout event broadcasted.

      2020-01-30T05:07:13.079-08:00 verbose vmware-dr[04036] [SRM@6876 sub=StubFactory] Timeout event broadcasted.

      2020-01-30T05:07:13.079-08:00 verbose vmware-dr[03544] [SRM@6876 sub=StubFactory] Timeout event broadcasted.

      2020-01-30T05:07:13.079-08:00 verbose vmware-dr[03096] [SRM@6876 sub=StubFactory] Timeout event broadcasted.

      2020-01-30T05:07:13.079-08:00 info vmware-dr[03788] [SRM@6876 sub=ThreadPool] Thread enlisted

      2020-01-30T05:07:13.079-08:00 info vmware-dr[03788] [SRM@6876 sub=ThreadPool] Entering worker thread loop

      2020-01-30T05:07:13.079-08:00 verbose vmware-dr[04056] [SRM@6876 sub=ServerSingletonCache] Timeout event broadcasted.

      2020-01-30T05:07:13.079-08:00 verbose vmware-dr[00952] [SRM@6876 sub=ServerSingletonCache] Timeout event broadcasted.

      2020-01-30T05:07:13.079-08:00 info vmware-dr[03396] [SRM@6876 sub=ThreadPool] Thread enlisted

      2020-01-30T05:07:13.079-08:00 info vmware-dr[03396] [SRM@6876 sub=ThreadPool] Entering worker thread loop

      2020-01-30T05:07:13.079-08:00 info vmware-dr[04036] [SRM@6876 sub=ServerSingletonCache] Dr::TranslateExnToFault: Failed to obtain MoRef/ServiceInstanceContent, context not available within timeout period

      2020-01-30T05:07:13.079-08:00 warning vmware-dr[04036] [SRM@6876 sub=EventDomain] Failed to get service instance content. Retry after 60 seconds: (dr.fault.ConnectionDownFault) {

      -->    faultCause = (dr.fault.Timedout) {

      -->       faultCause = (vmodl.MethodFault) null,

      -->       faultMessage = <unset>,

      -->       timeout = 300

      -->       msg = ""

      -->    },

      -->    faultMessage = <unset>

      -->    msg = ""

      --> }

      --> [context]zKq7AVMEAAgAAFaTwQAUdm13YXJlLWRyAAAqPwJ2bWFjb3JlLmRsbAAB9LEHZHItdHlwZXMuZGxsAAFJOwoC7AsDZHItdm1vbWkuZGxsAANXmxRjb25uZWN0aW9uLWJhc2UuZGxsAAMNejIEIGkGTVNWQ1IxMjAuZGxsAARt4wUFYyoJbnRkbGwuZGxsAAPInhQGqfQLY29ubmVjdGlvbi12Yy5kbGwABvoPEAb8KQsGHzELBtcEFAat8gsG2YwWBq3yCwZcmbAG2YwW[/context]

      --> [backtrace begin] product: VMware vCenter Site Recovery Manager, version: 8.1.2, build: build-12686166, tag: vmware-dr, cpu: x86_64, os: windows, buildType: release

      --> backtrace[03] vmacore.dll[0x00023F2A]

      --> backtrace[04] dr-types.dll[0x0007B1F4]

      --> backtrace[05] dr-types.dll[0x000A3B49]

      --> backtrace[06] dr-vmomi.dll[0x00030BEC]

      --> backtrace[07] connection-base.dll[0x00149B57]

      --> backtrace[08] connection-base.dll[0x00327A0D]

      --> backtrace[09] MSVCR120.dll[0x00066920]

      --> backtrace[10] MSVCR120.dll[0x0005E36D]

      --> backtrace[11] ntdll.dll[0x00092A63]

      --> backtrace[12] connection-base.dll[0x00149EC8]

      --> backtrace[13] connection-vc.dll[0x000BF4A9]

      --> backtrace[14] connection-vc.dll[0x00100FFA]

      --> backtrace[15] connection-vc.dll[0x000B29FC]

      --> backtrace[16] connection-vc.dll[0x000B311F]

      --> backtrace[17] connection-vc.dll[0x001404D7]

      --> backtrace[18] connection-vc.dll[0x000BF2AD]

      --> backtrace[19] connection-vc.dll[0x00168CD9]

      --> backtrace[20] connection-vc.dll[0x000BF2AD]

      --> backtrace[21] connection-vc.dll[0x00B0995C]

      --> backtrace[22] connection-vc.dll[0x00168CD9]

      --> [backtrace end]

      2020-01-30T05:07:24.518-08:00 verbose vmware-dr[01080] [SRM@6876 sub=SamlTokenFactory connID=pbm-admin-5624] TokenAcquisition associated with ListenerToken 2 is being cancelled.

      2020-01-30T05:07:24.518-08:00 warning vmware-dr[01080] [SRM@6876 sub=LocalSite.LocalPbmServer connID=pbm-admin-5624] Failed to connect:

      --> class Dr::TimeoutException Operation timed out after '300.000000' seconds

      --> [context]zKq7AVMEAAgAAFaTwQAMdm13YXJlLWRyAAAqPwJ2bWFjb3JlLmRsbAABw4EKY29ubmVjdGlvbi1iYXNlLmRsbAABBYsYAl0SB2NvbW1vbi5kbGwAAGzLHADrfhsAS44bAMmIIQN/TwJNU1ZDUjEyMC5kbGwAAyZRAgTSEwBLRVJORUwzMi5ETEwABeRUAW50ZGxsLmRsbAA=[/context]

      --> [backtrace begin] product: VMware vCenter Site Recovery Manager, version: 8.1.2, build: build-12686166, tag: vmware-dr, cpu: x86_64, os: windows, buildType: release

      --> backtrace[03] vmacore.dll[0x00023F2A]

      --> backtrace[04] connection-base.dll[0x000A81C3]

      --> backtrace[05] connection-base.dll[0x00188B05]

      --> backtrace[06] common.dll[0x0007125D]

      --> backtrace[07] vmacore.dll[0x001CCB6C]

      --> backtrace[08] vmacore.dll[0x001B7EEB]

      --> backtrace[09] vmacore.dll[0x001B8E4B]

      --> backtrace[10] vmacore.dll[0x002188C9]

      --> backtrace[11] MSVCR120.dll[0x00024F7F]

      --> backtrace[12] MSVCR120.dll[0x00025126]

      --> backtrace[13] KERNEL32.DLL[0x000013D2]

      --> backtrace[14] ntdll.dll[0x000154E4]

      --> [backtrace end]

      2020-01-30T05:07:24.518-08:00 verbose vmware-dr[03096] [SRM@6876 sub=HttpConnectionPool-000001] [RemoveConnection] Connection removed; cnx: <SSL(<io_obj p:0x000000855a373368, h:-1, <TCP '172.25.71.248 : 49517'>, <TCP '172.25.71.237 : 443'> FD Closed>)>; pooled: 2

      2020-01-30T05:07:24.596-08:00 verbose vmware-dr[00952] [SRM@6876 sub=LocalSite.LocalDsServer connID=ds-admin-51] Dr::Connection::PersistentConnectionBase::ConnectLocked: Attempting to connect

      2020-01-30T05:07:24.596-08:00 verbose vmware-dr[00952] [SRM@6876 sub=LocalSite.LocalCisLicenseServer connID=license-admin-e2ca] Dr::Connection::PersistentConnectionBase::ConnectLocked: Attempting to connect

      2020-01-30T05:07:24.596-08:00 verbose vmware-dr[00952] [SRM@6876 sub=LocalSite.LocalAuthzServer connID=authz-admin-f2dd] Dr::Connection::PersistentConnectionBase::ConnectLocked: Attempting to connect

      2020-01-30T05:07:24.596-08:00 verbose vmware-dr[00952] [SRM@6876 sub=LocalSite.LocalDsServer connID=ds-admin-51] Failed to connect. Not initialized: DsConnectionHandler SSO

      2020-01-30T05:07:24.596-08:00 verbose vmware-dr[03788] [SRM@6876 sub=LocalSite.LocalAuthzServer connID=authz-admin-f2dd] Failed to connect. Not initialized: AuthzConnectionHandler SSO

      2020-01-30T05:07:24.596-08:00 verbose vmware-dr[00952] [SRM@6876 sub=LocalSite.LocalCisLicenseServer connID=license-admin-e2ca] Failed to connect. Not initialized: CisLicenseConnectionHandler SSO

      2020-01-30T05:07:37.441-08:00 verbose vmware-dr[01744] [SRM@6876 sub=HttpConnectionPool-000001] [RemoveConnection] Connection removed; cnx: <SSL(<io_obj p:0x000000855a373488, h:-1, <TCP '172.25.71.248 : 49518'>, <TCP '172.25.71.241 : 443'> FD Closed>)>; pooled: 1

      2020-01-30T05:07:44.551-08:00 verbose vmware-dr[01980] [SRM@6876 sub=LocalLkpServer connID=lkp-admin-d62c] Dr::LkpConnectionHandler::DoPing: Ping succeeded

      2020-01-30T05:07:47.739-08:00 verbose vmware-dr[01080] [SRM@6876 sub=LocalSite.LocalVcServer] Dr::Connection::PersistentConnectionBase::ConnectLocked: Attempting to connect

      2020-01-30T05:07:47.739-08:00 info vmware-dr[00952] [SRM@6876 sub=LocalSite.LocalVersionReg] Compatible VMODL version found: 'vim.version.version10'

      2020-01-30T05:07:47.739-08:00 verbose vmware-dr[00952] [SRM@6876 sub=LocalSite.LocalVcServer] Using VMODL version 'vim.version.version10'

      2020-01-30T05:07:47.739-08:00 verbose vmware-dr[00952] [SRM@6876 sub=StubFactory] Event broadcasted

      2020-01-30T05:07:47.755-08:00 verbose vmware-dr[03544] [SRM@6876 sub=LocalSite.LocalVcServer] Failed to connect. Not initialized: VcConnectionHandler SSO

      2020-01-30T05:07:47.864-08:00 verbose vmware-dr[01744] [SRM@6876 sub=HttpConnectionPool-000001] [RemoveConnection] Connection removed; cnx: <SSL(<io_obj p:0x000000855a4ca308, h:-1, <TCP '172.25.71.248 : 49519'>, <TCP '172.25.71.237 : 443'> FD Closed>)>; pooled: 2

      2020-01-30T05:07:47.926-08:00 verbose vmware-dr[02668] [SRM@6876 sub=LocalSite.LocalSsoServer] Dr::Connection::PersistentConnectionBase::ConnectLocked: Attempting to connect

      2020-01-30T05:07:47.926-08:00 verbose vmware-dr[04028] [SRM@6876 sub=LocalSite.LocalSsoServer.ConnHandler] Connecting to SSO server.

      2020-01-30T05:07:47.926-08:00 verbose vmware-dr[03604] [SRM@6876 sub=LocalSite.LocalSsoServer.ConnHandler] Using VMODL version 'sso.version.version1_5'

      2020-01-30T05:07:47.926-08:00 verbose vmware-dr[03604] [SRM@6876 sub=StubFactory] Event broadcasted

      2020-01-30T05:07:47.958-08:00 verbose vmware-dr[00952] [SRM@6876 sub=SamlTokenFactory ctxID=3542813a] Setting signing certificates and broadcasting.

      2020-01-30T05:07:48.005-08:00 error vmware-dr[01080] [SRM@6876 sub=LocalSite.LocalSsoServer.ConnHandler] Dr::SsoConnectionHandler::GetTokenComplete: Unable to retrieve token from SSO; no token. Exception:

       

       

       

      From Event Viewer Logs::

       

      VMware vCenter Site Recovery Manager application error.

       

       

      Application error: class Vmacore::Exception Failed to connect to remote services on ["psc.scdpode.local", "vcenter.scdpode.local"]

      [context]zKq7AVMEAAgAAFaTwQAUdm13YXJlLWRyAAAKox52bWFjb3JlLmRsbAAAkA0GAA41BgCxPwIBrS4mdm13YXJlLWRyLmV4ZQABbQcmAq2rCmRyLWF1dGhvcml6YXRpb24uZGxsAAGkpSUBSIQlAf3RIwEtAiYBjsYbA96lAHdpbmRvd3NTZXJ2aWNlLmRsbAAExwgCZHItdm1hY29yZS5kbGwABFeIAgQFhwID8K4ABdpaAHNlY2hvc3QuZGxsAAbSEwBLRVJORUwzMi5ETEwAB+RUAW50ZGxsLmRsbAA=[/context]

      [backtrace begin] product: VMware vCenter Site Recovery Manager, version: 8.1.2, build: build-12686166, tag: vmware-dr, cpu: x86_64, os: windows, buildType: release

      backtrace[00] vmacore.dll[0x001EA30A]

      backtrace[01] vmacore.dll[0x00060D90]

      backtrace[02] vmacore.dll[0x0006350E]

      backtrace[03] vmacore.dll[0x00023FB1]

      backtrace[04] vmware-dr.exe[0x00262EAD]

      backtrace[05] vmware-dr.exe[0x0026076D]

      backtrace[06] dr-authorization.dll[0x000AABAD]

      backtrace[07] vmware-dr.exe[0x0025A5A4]

      backtrace[08] vmware-dr.exe[0x00258448]

      backtrace[09] vmware-dr.exe[0x0023D1FD]

      backtrace[10] vmware-dr.exe[0x0026022D]

      backtrace[11] vmware-dr.exe[0x001BC68E]

      backtrace[12] windowsService.dll[0x0000A5DE]

      backtrace[13] dr-vmacore.dll[0x000208C7]

      backtrace[14] dr-vmacore.dll[0x00028857]

      backtrace[15] dr-vmacore.dll[0x00028705]

      backtrace[16] windowsService.dll[0x0000AEF0]

      backtrace[17] sechost.dll[0x00005ADA]

      backtrace[18] KERNEL32.DLL[0x000013D2]

      backtrace[19] ntdll.dll[0x000154E4]

      [backtrace end]