1 2 3 Previous Next 32 Replies Latest reply on Aug 6, 2020 10:05 AM by pareshgk

    Adding ESXi host to vCenter fails

    Flammi Novice

      Hi All,

       

      I've spend more than two days on this already and I'm absolutely clueless by now. Was hoping someone here can help.

       

      I have one vCenter in my home lab with one ESX. Bought a second ESX and I'm trying to add it to my vCenter without success.

      I already deployed a new vCenter installed different versions of ESX no luck.

       

      Right now I have a vCenter 7 and the ESXi host I'm trying to add is running 6.7.

      When I try to add the host to the vCenter I see the folloing in the vmauthd.log of the ESX host:

       

      2020-07-20T13:59:17Z vmauthd[2102937]: Msg_SetLocaleEx: HostLocale=UTF-8 UserLocale=NULL

      2020-07-20T13:59:17Z vmauthd[2102937]: Could not expand environment variable HOME.

      2020-07-20T13:59:17Z vmauthd[2102937]: Could not expand environment variable HOME.

      2020-07-20T13:59:17Z vmauthd[2102937]: DictionaryLoad: Cannot open file "/usr/lib/vmware/config": No such file or directory.

      2020-07-20T13:59:17Z vmauthd[2102937]: DictionaryLoad: Cannot open file "~/.vmware/config": No such file or directory.

      2020-07-20T13:59:17Z vmauthd[2102937]: DictionaryLoad: Cannot open file "~/.vmware/preferences": No such file or directory.

      2020-07-20T13:59:17Z vmauthd[2102937]: lib/ssl: OpenSSL using FIPS_drbg for RAND

      2020-07-20T13:59:17Z vmauthd[2102937]: lib/ssl: protocol list tls1.2

      2020-07-20T13:59:17Z vmauthd[2102937]: lib/ssl: protocol list tls1.2 (openssl flags 0x17000000)

      2020-07-20T13:59:17Z vmauthd[2102937]: lib/ssl: cipher list ECDHE+AESGCM:RSA+AESGCM:ECDHE+AES:RSA+AES

      2020-07-20T13:59:17Z vmauthd[2102937]: lib/ssl: curves list prime256v1:secp384r1:secp521r1

      2020-07-20T13:59:17Z vmauthd[2102937]: Connect from remote socket (192.168.2.45:58542).

      2020-07-20T13:59:17Z vmauthd[2102937]: Connect from 192.168.2.45

      2020-07-20T13:59:17Z vmauthd[2102937]: recv() FAIL: 1.

      2020-07-20T13:59:17Z vmauthd[2102937]: VMAuthdSocketRead: read failed.  Closing socket for reading.

      2020-07-20T13:59:17Z vmauthd[2102937]: Read failed.

       

      ON the vCenter side I see those errors in the vpxd.log

       

      2020-07-20T15:59:27.291+02:00 info vpxd[14503] [Originator@6876 sub=InvtHostCnx opID=kcuf2q8s-2242-auto-1qc-h5:70001655-75-StartHostSync-61f79812] Connect in progress; [vim.HostSystem:host-2021,esxi-nuc.flammi.home]

      2020-07-20T15:59:27.299+02:00 error vpxd[13388] [Originator@6876 sub=TrustedInfrastructure.HostConfig] [VapiBaseClientwrapper] failed: Error:

      -->    system_error

      --> Messages:

      -->    vapi.invalid.result.code

      -->

      2020-07-20T15:59:27.300+02:00 warning vpxd[13499] [Originator@6876 sub=TrustedInfrastructure.HostConfig opID=kcuf2q8s-2242-auto-1qc-h5:70001655-75] [AcquireJWToken] Host 'esxi-nuc.flammi.home' is missing trusted VAPI service. Exception: N4Vpxd7Langley29BadVapiConfigurationExceptionE(Error:

      -->    system_error

      --> Messages:

      -->    vapi.invalid.result.code

      --> )

      -->

       

      Any ideas?

      Thanks a lot fellas!

        • 1. Re: Adding ESXi host to vCenter fails
          harry89 Enthusiast

          Is the host added and in disconnected state .

          Are you able to login directly to the host through host client

           

          Send output of service-control --status --all on VC.

           

          Also endpoint.log  snippet as well during the operation

          • 2. Re: Adding ESXi host to vCenter fails
            Flammi Novice

            Hi harr89,

             

            thanks for the help. Really appreciate the reply.

             

            The host is not added at all.

            It appears in the list of hosts as disconnected for about two seconds and then disappears again.

             

            I can login to the host just fine to manage it directly.

             

            root@vcenter [ ~ ]# service-control --status --all

            Running:

            applmgmt lookupsvc lwsmd observability observability-vapi pschealth vlcm vmafdd vmcad vmdird vmonapi vmware-analytics vmware-certificateauthority vmware-certifp

            Stopped:

            vmcam vmware-imagebuilder vmware-netdumper vmware-rbd-watchdog vmware-vcha

             

            There is nothing added to the endpoint.log of the vCenter during the operation.

             

            Pierre

            • 3. Re: Adding ESXi host to vCenter fails
              harry89 Enthusiast

              on vc ssh session

              Go to cd /var/log/vmware

              Go to folder observability  or observability vapi and check the logs

              this is related to something with trust relationship

               

              From vc run below command

              curl telnet://hostip:443 --- whats the ouput at the bottom

              • 4. Re: Adding ESXi host to vCenter fails
                nachogonzalez Enthusiast

                Hey Flammi, hope you are doing fine

                 

                 

                let me ask you a few questions:

                 

                if you log in via SSH to the VCSA can you ping the new host?
                can you log in to host web console?
                can you log in to host via ssh?

                 

                Can you please share host's vmkernel.log output?

                 

                Looking forward to hearing form you

                 

                Warm regards

                • 5. Re: Adding ESXi host to vCenter fails
                  Flammi Novice

                  Hi Harry,

                   

                  I'm with you. I also assume it's something trust related. Just can't get my head around it and have no experience with this at all.

                   

                  I didn't fine anything suspicious in the logs you mentioned.

                  observability-vapi.log -> nothing added during the process

                  observability-runtime.log.stderr -> nothing added during the process

                  observability-runtime.log.stdout -> nothing added during the process

                  The only logfile where something is added is the  observability.log. I couldn't attache it for some reason so you can have a look at it here.

                  Dropbox - observability.log

                  I tried to add the server at 12:16:25

                   

                  If I run curl telnet://hostip:443 nothing happens. No errors, no output.

                  • 6. Re: Adding ESXi host to vCenter fails
                    Flammi Novice

                    Hi Nachogonzalez,

                     

                    thanks a lot for your reply.

                     

                    Yes, all the other logins work just fine.

                    I have a VM running on the server already.

                    Ping is also possible. IP or DNS name.

                     

                    Kernel log is here: Dropbox - vmkernel.log

                    I couldn't find anything in there the helps.

                    I tried to add the host again today at 12:16 CET.

                    So in the kernel log it's 10:16 (UTC).

                     

                    Thanks a lot

                    Pierre

                    • 7. Re: Adding ESXi host to vCenter fails
                      harry89 Enthusiast

                      Trust me this will be some bug with product VC 7.0

                      • 8. Re: Adding ESXi host to vCenter fails
                        nachogonzalez Enthusiast

                        Hey Flammi

                        I was just reviewing the logs

                         

                        2020-07-20T15:01:00.096Z cpu1:2097180)ScsiDeviceIO: 3435: Cmd(0x459a40d64180) 0x1a, CmdSN 0x9c5 from world 0 to dev "mpx.vmhba32:C0:T0:L0" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.

                        2020-07-20T15:03:33.768Z cpu7:2097675)DVFilter: 5963: Checking disconnected filters for timeouts

                        2020-07-20T15:07:03.768Z cpu1:2097619)NMP: nmp_ResetDeviceLogThrottling:3580: Error status H:0x0 D:0x2 P:0x0 Sense Data: 0x5 0x20 0x0 from dev "mpx.vmhba32:C0:T0:L0" occurred 1 times(of 1 commands)

                        2020-07-20T15:13:33.768Z cpu5:2097675)DVFilter: 5963: Checking disconnected filters for timeouts

                        2020-07-20T15:23:33.768Z cpu0:2097675)DVFilter: 5963: Checking disconnected filters for timeouts

                        2020-07-20T15:33:33.768Z cpu6:2097675)DVFilter: 5963: Checking disconnected filters for timeouts

                        2020-07-20T15:36:10.774Z cpu7:2097186)ScsiDeviceIO: 3435: Cmd(0x459a40cdafc0) 0x85, CmdSN 0x15 from world 2099077 to dev "mpx.vmhba32:C0:T0:L0" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0.

                        2020-07-20T15:37:03.768Z cpu0:2097619)NMP: nmp_ResetDeviceLogThrottling:3580: Error status H:0x0 D:0x2 P:0x0 Sense Data: 0x5 0x20 0x0 from dev "mpx.vmhba32:C0:T0:L0" occurred 1 times(of 1 commands)

                        2020-07-20T15:43:33.768Z cpu6:2097675)DVFilter: 5963: Checking disconnected filters for timeouts

                        2020-07-20T15:53:33.768Z cpu5:2097675)DVFilter: 5963: Checking disconnected filters for timeouts

                        2020-07-20T16:03:33.770Z cpu0:2097675)DVFilter: 5963: Checking disconnected filters for timeouts

                        2020-07-20T16:07:03.768Z cpu5:2097619)NMP: nmp_ResetDeviceLogThrottling:3580: Error status H:0x0 D:0x2 P:0x0 Sense Data: 0x5 0x20 0x0 from dev "mpx.vmhba32:C0:T0:L0" occurred 1 times(of 1 commands)

                        2020-07-20T16:12:19.189Z cpu7:2097186)ScsiDeviceIO: 3435: Cmd(0x459a40a5af80) 0x1a, CmdSN 0xaaf from world 0 to dev "mpx.vmhba32:C0:T0:L0" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.

                        2020-07-20T16:13:33.768Z cpu0:2097675)DVFilter: 5963: Checking disconnected filters for timeouts

                        2020-07-20T16:23:33.768Z cpu4:2097675)DVFilter: 5963: Checking disconnected filters for timeouts

                        2020-07-20T16:33:33.768Z cpu4:2097675)DVFilter: 5963: Checking disconnected filters for timeouts

                        2020-07-20T16:37:03.768Z cpu5:2097619)NMP: nmp_ResetDeviceLogThrottling:3580: Error status H:0x0 D:0x2 P:0x0 Sense Data: 0x5 0x20 0x0 from dev "mpx.vmhba32:C0:T0:L0" occurred 1 times(of 1 commands)

                        2020-07-20T16:43:33.768Z cpu4:2097675)DVFilter: 5963: Checking disconnected filters for timeouts

                        2020-07-20T16:53:33.768Z cpu4:2097675)DVFilter: 5963: Checking disconnected filters for timeouts

                        2020-07-20T16:54:15.047Z cpu3:2097182)ScsiDeviceIO: 3435: Cmd(0x459a40b3dd80) 0x1a, CmdSN 0xb2f from world 0 to dev "mpx.vmhba32:C0:T0:L0" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.

                        2020-07-20T17:03:33.767Z cpu4:2097675)DVFilter: 5963: Checking disconnected filters for timeouts

                        2020-07-20T17:07:03.768Z cpu3:2097619)NMP: nmp_ResetDeviceLogThrottling:3580: Error status H:0x0 D:0x2 P:0x0 Sense Data: 0x5 0x20 0x0 from dev "mpx.vmhba32:C0:T0:L0" occurred 1 times(of 1 commands)

                        2020-07-20T17:13:33.767Z cpu0:2097675)DVFilter: 5963: Checking disconnected filters for timeouts

                        2020-07-20T17:23:33.768Z cpu0:2097675)DVFilter: 5963: Checking disconnected filters for timeouts

                        2020-07-20T17:32:19.577Z cpu0:2097179)ScsiDeviceIO: 3435: Cmd(0x459a40ce8940) 0x1a, CmdSN 0xba9 from world 0 to dev "mpx.vmhba32:C0:T0:L0" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.

                        2020-07-20T17:33:33.767Z cpu0:2097675)DVFilter: 5963: Checking disconnected filters for timeouts

                        2020-07-20T17:37:03.767Z cpu5:2097619)NMP: nmp_ResetDeviceLogThrottling:3580: Error status H:0x0 D:0x2 P:0x0 Sense Data: 0x5 0x20 0x0 from dev "mpx.vmhba32:C0:T0:L0" occurred 1 times(of 1 commands)

                        2020-07-20T17:43:33.767Z cpu0:2097675)DVFilter: 5963: Checking disconnected filters for timeouts

                        2020-07-20T17:53:33.767Z cpu1:2097675)DVFilter: 5963: Checking disconnected filters for timeouts

                        2020-07-20T18:03:33.767Z cpu1:2097675)DVFilter: 5963: Checking disconnected filters for timeouts

                        2020-07-20T18:06:10.862Z cpu1:2097180)ScsiDeviceIO: 3435: Cmd(0x459a40d313c0) 0x85, CmdSN 0x1e from world 2099077 to dev "mpx.vmhba32:C0:T0:L0" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0.

                        2020-07-20T18:07:03.767Z cpu0:2097619)NMP: nmp_ResetDeviceLogThrottling:3580: Error status H:0x0 D:0x2 P:0x0 Sense Data: 0x5 0x20 0x0 from dev "mpx.vmhba32:C0:T0:L0" occurred 1 times(of 1 commands)

                         

                        You are having check conditions on the storage array

                         

                        Might want to check over there.


                        Also try rebooting host's management services (services.sh restart) or simply reboot the host.

                        • 9. Re: Adding ESXi host to vCenter fails
                          Flammi Novice

                          I also tried it with VC6.7 doesn't work.

                          I tried ESX 6.5 & 6.7 both didn't work.

                          It's totally bizarre!

                          • 10. Re: Adding ESXi host to vCenter fails
                            Flammi Novice

                            Hi Nachogonzalez,

                             

                            I don't have any storage devices connected. Just a local NVMe M.2 SSD. That's it.

                             

                            Thanks

                            Pierre

                            • 11. Re: Adding ESXi host to vCenter fails
                              harry89 Enthusiast

                              check esxi thouroghly ... all the services ... ram size  df -h

                              • 12. Re: Adding ESXi host to vCenter fails
                                nachogonzalez Enthusiast

                                have you tried restarting management services?

                                • 13. Re: Adding ESXi host to vCenter fails
                                  Flammi Novice

                                  Thank you both. I didn't only restart mgmt service or checked services on the host, I reinstalled ESXi probably 10 times by now with different versions.

                                  will try ESXi7 tomorrow....

                                  • 14. Re: Adding ESXi host to vCenter fails
                                    harry89 Enthusiast

                                    Are there any core dumps on vc

                                     

                                    cd /var/core

                                    1 2 3 Previous Next