4 Replies Latest reply on Nov 24, 2015 5:54 AM by dartron

    Unable to add an ESXi host to a VirtualCenter

    iceman94400 Lurker
    Visibility: Open to anyone

      Hi everyone,

       

       

      First of all, some details about my infrastucture :

      VMware ESXi version: VMware ESXi 5.0.0 (VMKernel Release Build 914586)

      VMware VirtualCenter version: v5.0.0.16964

      License: still valid

       

      After an undetermined issue, one of our ESXi hosts was marked as Disconnected in our VirtualCenter.

       

      I did the following diagnosis :

      • reconnected the ESXi using the contextual menu (Connect) from the vCenter : message Unable to correctly read current respool tree on host <IP ADDRESS>
      • netcat from vCenter to ESXi host on TCP/443 : OK
      • connection to the ESXi host using the vSphere Client : OK
      • ran diagnosis on ESXi host using the dcui utility > Test Management Network : ESXi host still disconnected
      • restarted management agents (hostd and vpxa) on the ESXi host : ESXi host still disconnected
      • tested if the ESXi host is in All-Paths-Down (APD) condition : no issue

      ~ # esxcfg-mpath -b |grep -C 1 dead

       

      Extract from the vCenter log :

       

      2015-10-07T14:02:35.837+02:00 [26000 error 'Default' opID=2C4FA4DF-00000044] Unable to correctly read current respool tree on host [vim.HostSystem:host-175,10.10.10.3], received (vpxapi.ResourcePoolChangeInfo) {

      --> dynamicType = <unset>,

      --> fullSync = false,

      --> rootResourcePool = (vpxapi.ResourcePoolSpec) null,

      --> }

      2015-10-07T14:02:35.837+02:00 [26000 error 'Default' opID=2C4FA4DF-00000044] [VpxdInvtHostCnx] Connect2 caught failure reading host resource pool tree.

      2015-10-07T14:02:35.852+02:00 [26000 info 'vmomi.soapStub[10]' opID=2C4FA4DF-00000044] Resetting stub adapter for server TCP:10.10.10.3:443 : Closed

      2015-10-07T14:02:35.852+02:00 [26000 error 'MoHost' opID=2C4FA4DF-00000044] [VpxdMoHost::Reconnect] Got method fault: vim.fault.ReadHostResourcePoolTreeFailed

      2015-10-07T14:02:35.852+02:00 [26000 error 'MoHost' opID=2C4FA4DF-00000044] [VpxdMoHost::Reconnect] Backtrace: backtrace[00] rip 000000018013d40a (no symbol)

      --> backtrace[01] rip 00000001800ffa38 (no symbol)

      --> backtrace[02] rip 00000001800fffee (no symbol)

      --> backtrace[03] rip 000000018008794b (no symbol)

      --> backtrace[04] rip 00000000007c83dc (no symbol)

      --> backtrace[05] rip 0000000000a720c6 (no symbol)

      --> backtrace[06] rip 000000013fd1b6c4 (no symbol)

      --> backtrace[07] rip 000000013fd2afd5 (no symbol)

      --> backtrace[08] rip 000000013fd2d78d (no symbol)

      --> backtrace[09] rip 000000013fd2df71 (no symbol)

      --> backtrace[10] rip 000000013fe357d0 (no symbol)

      --> backtrace[11] rip 0000000000849f89 (no symbol)

      --> backtrace[12] rip 00000000003efab0 (no symbol)

      --> backtrace[13] rip 000000013fa16e76 (no symbol)

      --> backtrace[14] rip 000000013fa007cb (no symbol)

      --> backtrace[15] rip 000000013fa094ea (no symbol)

      --> backtrace[16] rip 0000000180153dad (no symbol)

      --> backtrace[17] rip 00000001801552d4 (no symbol)

      --> backtrace[18] rip 000000018014dc65 (no symbol)

      --> backtrace[19] rip 0000000074222fdf (no symbol)

      --> backtrace[20] rip 0000000074223080 (no symbol)

      --> backtrace[21] rip 0000000076e0f56d (no symbol)

      --> backtrace[22] rip 0000000077043281 (no symbol)

      -->

      2015-10-07T14:02:35.852+02:00 [26000 error 'Default' opID=2C4FA4DF-00000044] (Log recursion level 2) vim.fault.ReadHostResourcePoolTreeFailed

       

      Management agents logs when attempting to reconnect an ESXi server from a vCenter :

       

      ~ # tail -f /var/log/vpxa.log

      2015-10-07T10:59:35.523Z [64A4FB90 warning 'Libs'] SSL_VerifyX509: Certificate verification is disabled, so connection will proceed despite the error

      2015-10-07T10:59:35.523Z [64A4FB90 warning 'Libs'] SSL_VerifyX509: Certificate verification is disabled, so connection will proceed despite the error

      2015-10-07T10:59:35.524Z [64A4FB90 warning 'Libs'] SSL_VerifyX509: Certificate verification is disabled, so connection will proceed despite the error

      2015-10-07T10:59:35.540Z [64AF4B90 warning 'Libs'] SSL_VerifyX509: Certificate verification is disabled, so connection will proceed despite the error

      2015-10-07T10:59:35.540Z [64AF4B90 warning 'Libs'] SSL_VerifyX509: Certificate verification is disabled, so connection will proceed despite the error

      2015-10-07T10:59:35.540Z [64AF4B90 warning 'Libs'] SSL_VerifyX509: Certificate verification is disabled, so connection will proceed despite the error

      2015-10-07T10:59:38.866Z [64A2EB90 error 'Default' opID=448E62A1-00000066-8c] [VpxaClientAdapter::InvokeCommon] Re-throwing method-fault 'vmodl.fault.ManagedObjectNotFound' received while invoking GetChildConfiguration on vim.ResourcePool:ha-root-pool

      2015-10-07T10:59:38.866Z [64A2EB90 error 'VpxaHalResourcePool' opID=448E62A1-00000066-8c] [GetResPoolInfo] Failed to get resource pool tree. vmodl.fault.ManagedObjectNotFound

       

      ~ # tail -f /var/log/hostd.log

      pam_per_user: create_subrequest_handle(): doing map lookup for user "vpxuser"

      pam_per_user: create_subrequest_handle(): creating new subrequest (user="vpxuser", service="system-auth-local")

      Accepted password for user vpxuser from 10.10.10.80

      pam_per_user: create_subrequest_handle(): doing map lookup for user "vpxuser"

      pam_per_user: create_subrequest_handle(): creating new subrequest (user="vpxuser", service="system-auth-local")

      Accepted password for user vpxuser from 127.0.0.1

      2015-10-07T10:59:36.911Z [64C40B90 warning 'Locale' opID=448E62A1-00000066-8c] No message string to format object vim.option.OptionDef.

      -->

      2015-10-07T10:59:38.864Z [64C81B90 info 'Vmomi' opID=448E62A1-00000066-8c] Activation [N5Vmomi10ActivationE:0x651e1988] : Invoke done [GetChildConfiguration] on [vim.ResourcePool:ha-root-pool]

      2015-10-07T10:59:38.865Z [64C81B90 info 'Vmomi' opID=448E62A1-00000066-8c] Throw vmodl.fault.ManagedObjectNotFound

      2015-10-07T10:59:38.865Z [64C81B90 info 'Vmomi' opID=448E62A1-00000066-8c] Result:

      --> (vmodl.fault.ManagedObjectNotFound) {

      --> dynamicType = <unset>,

      --> faultCause = (vmodl.MethodFault) null,

      --> obj = 'vim.VirtualMachine:15',

      --> msg = "",

      --> }

      2015-10-07T10:59:42.368Z [FFE3BAD0 warning 'Locale'] No message string to format object vim.option.OptionDef.

      -->

      2015-10-07T10:59:42.374Z [FFE3BAD0 warning 'PropertyCollector'] ComputeGUReq took 2662776 microSec

      snmpsvc: bora/vim/hostd/snmpsvc/proto/agt_engine.c(967): cannot add objectId in varbind

       

      Ultimately, I tried to delete one of the ESXi host using the contextual menu and readd it but I am faced w/ the same message as before (only this time, the ESXi host is not readded to the vCenter).

       

      Do you have any idea what actions I can try at this point ?

       

      Thanks in advance.