Adding ESXi host to vCenter fails

[Flammi]

Hi All,

I've spend more than two days on this already and I'm absolutely clueless by now. Was hoping someone here can help.

I have one vCenter in my home lab with one ESX. Bought a second ESX and I'm trying to add it to my vCenter without success.

I already deployed a new vCenter installed different versions of ESX no luck.

Right now I have a vCenter 7 and the ESXi host I'm trying to add is running 6.7.

When I try to add the host to the vCenter I see the folloing in the vmauthd.log of the ESX host:

2020-07-20T13:59:17Z vmauthd[2102937]: Msg_SetLocaleEx: HostLocale=UTF-8 UserLocale=NULL

2020-07-20T13:59:17Z vmauthd[2102937]: Could not expand environment variable HOME.

2020-07-20T13:59:17Z vmauthd[2102937]: Could not expand environment variable HOME.

2020-07-20T13:59:17Z vmauthd[2102937]: DictionaryLoad: Cannot open file "/usr/lib/vmware/config": No such file or directory.

2020-07-20T13:59:17Z vmauthd[2102937]: DictionaryLoad: Cannot open file "~/.vmware/config": No such file or directory.

2020-07-20T13:59:17Z vmauthd[2102937]: DictionaryLoad: Cannot open file "~/.vmware/preferences": No such file or directory.

2020-07-20T13:59:17Z vmauthd[2102937]: lib/ssl: OpenSSL using FIPS_drbg for RAND

2020-07-20T13:59:17Z vmauthd[2102937]: lib/ssl: protocol list tls1.2

2020-07-20T13:59:17Z vmauthd[2102937]: lib/ssl: protocol list tls1.2 (openssl flags 0x17000000)

2020-07-20T13:59:17Z vmauthd[2102937]: lib/ssl: cipher list ECDHE+AESGCM:RSA+AESGCM:ECDHE+AES:RSA+AES

2020-07-20T13:59:17Z vmauthd[2102937]: lib/ssl: curves list prime256v1:secp384r1:secp521r1

2020-07-20T13:59:17Z vmauthd[2102937]: Connect from remote socket (192.168.2.45:58542).

2020-07-20T13:59:17Z vmauthd[2102937]: Connect from 192.168.2.45

2020-07-20T13:59:17Z vmauthd[2102937]: recv() FAIL: 1.

2020-07-20T13:59:17Z vmauthd[2102937]: VMAuthdSocketRead: read failed.  Closing socket for reading.

2020-07-20T13:59:17Z vmauthd[2102937]: Read failed.

ON the vCenter side I see those errors in the vpxd.log

2020-07-20T15:59:27.291+02:00 info vpxd[14503] [Originator@6876 sub=InvtHostCnx opID=kcuf2q8s-2242-auto-1qc-h5:70001655-75-StartHostSync-61f79812] Connect in progress; [vim.HostSystem:host-2021,esxi-nuc.flammi.home]

2020-07-20T15:59:27.299+02:00 error vpxd[13388] [Originator@6876 sub=TrustedInfrastructure.HostConfig] [VapiBaseClientwrapper] failed: Error:

-->    system_error

--> Messages:

-->    vapi.invalid.result.code

-->

2020-07-20T15:59:27.300+02:00 warning vpxd[13499] [Originator@6876 sub=TrustedInfrastructure.HostConfig opID=kcuf2q8s-2242-auto-1qc-h5:70001655-75] [AcquireJWToken] Host 'esxi-nuc.flammi.home' is missing trusted VAPI service. Exception: N4Vpxd7Langley29BadVapiConfigurationExceptionE(Error:

-->    system_error

--> Messages:

-->    vapi.invalid.result.code

--> )

-->

Any ideas?

Thanks a lot fellas!

[harry89]

Is the host added and in disconnected state .

Are you able to login directly to the host through host client

Send output of service-control --status --all on VC.

Also endpoint.log  snippet as well during the operation

[Flammi]

Hi harr89,

thanks for the help. Really appreciate the reply.

The host is not added at all.

It appears in the list of hosts as disconnected for about two seconds and then disappears again.

I can login to the host just fine to manage it directly.

root@vcenter [ ~ ]# service-control --status --all

Running:

applmgmt lookupsvc lwsmd observability observability-vapi pschealth vlcm vmafdd vmcad vmdird vmonapi vmware-analytics vmware-certificateauthority vmware-certifp

Stopped:

vmcam vmware-imagebuilder vmware-netdumper vmware-rbd-watchdog vmware-vcha

There is nothing added to the endpoint.log of the vCenter during the operation.

Pierre

[harry89]

on vc ssh session

Go to cd /var/log/vmware

Go to folder observability  or observability vapi and check the logs

this is related to something with trust relationship

From vc run below command

curl telnet://hostip:443 --- whats the ouput at the bottom

[nachogonzalez]

Hey Flammi, hope you are doing fine

let me ask you a few questions:

if you log in via SSH to the VCSA can you ping the new host?
can you log in to host web console?
can you log in to host via ssh?

Can you please share host's vmkernel.log output?

Looking forward to hearing form you

Warm regards

[Flammi]

Hi Harry,

I'm with you. I also assume it's something trust related. Just can't get my head around it and have no experience with this at all.

I didn't fine anything suspicious in the logs you mentioned.

observability-vapi.log -> nothing added during the process

observability-runtime.log.stderr -> nothing added during the process

observability-runtime.log.stdout -> nothing added during the process

The only logfile where something is added is the  observability.log. I couldn't attache it for some reason so you can have a look at it here.

Dropbox - observability.log

I tried to add the server at 12:16:25

If I run curl telnet://hostip:443 nothing happens. No errors, no output.

[Flammi]

Hi Nachogonzalez,

thanks a lot for your reply.

Yes, all the other logins work just fine.

I have a VM running on the server already.

Ping is also possible. IP or DNS name.

Kernel log is here: Dropbox - vmkernel.log

I couldn't find anything in there the helps.

I tried to add the host again today at 12:16 CET.

So in the kernel log it's 10:16 (UTC).

Thanks a lot

Pierre

[harry89]

Trust me this will be some bug with product VC 7.0

[nachogonzalez]

Hey Flammi​

I was just reviewing the logs

2020-07-20T15:01:00.096Z cpu1:2097180)ScsiDeviceIO: 3435: Cmd(0x459a40d64180) 0x1a, CmdSN 0x9c5 from world 0 to dev "mpx.vmhba32:C0:T0:L0" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.

2020-07-20T15:03:33.768Z cpu7:2097675)DVFilter: 5963: Checking disconnected filters for timeouts

2020-07-20T15:07:03.768Z cpu1:2097619)NMP: nmp_ResetDeviceLogThrottling:3580: Error status H:0x0 D:0x2 P:0x0 Sense Data: 0x5 0x20 0x0 from dev "mpx.vmhba32:C0:T0:L0" occurred 1 times(of 1 commands)

2020-07-20T15:13:33.768Z cpu5:2097675)DVFilter: 5963: Checking disconnected filters for timeouts

2020-07-20T15:23:33.768Z cpu0:2097675)DVFilter: 5963: Checking disconnected filters for timeouts

2020-07-20T15:33:33.768Z cpu6:2097675)DVFilter: 5963: Checking disconnected filters for timeouts

2020-07-20T15:36:10.774Z cpu7:2097186)ScsiDeviceIO: 3435: Cmd(0x459a40cdafc0) 0x85, CmdSN 0x15 from world 2099077 to dev "mpx.vmhba32:C0:T0:L0" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0.

2020-07-20T15:37:03.768Z cpu0:2097619)NMP: nmp_ResetDeviceLogThrottling:3580: Error status H:0x0 D:0x2 P:0x0 Sense Data: 0x5 0x20 0x0 from dev "mpx.vmhba32:C0:T0:L0" occurred 1 times(of 1 commands)

2020-07-20T15:43:33.768Z cpu6:2097675)DVFilter: 5963: Checking disconnected filters for timeouts

2020-07-20T15:53:33.768Z cpu5:2097675)DVFilter: 5963: Checking disconnected filters for timeouts

2020-07-20T16:03:33.770Z cpu0:2097675)DVFilter: 5963: Checking disconnected filters for timeouts

2020-07-20T16:07:03.768Z cpu5:2097619)NMP: nmp_ResetDeviceLogThrottling:3580: Error status H:0x0 D:0x2 P:0x0 Sense Data: 0x5 0x20 0x0 from dev "mpx.vmhba32:C0:T0:L0" occurred 1 times(of 1 commands)

2020-07-20T16:12:19.189Z cpu7:2097186)ScsiDeviceIO: 3435: Cmd(0x459a40a5af80) 0x1a, CmdSN 0xaaf from world 0 to dev "mpx.vmhba32:C0:T0:L0" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.

2020-07-20T16:13:33.768Z cpu0:2097675)DVFilter: 5963: Checking disconnected filters for timeouts

2020-07-20T16:23:33.768Z cpu4:2097675)DVFilter: 5963: Checking disconnected filters for timeouts

2020-07-20T16:33:33.768Z cpu4:2097675)DVFilter: 5963: Checking disconnected filters for timeouts

2020-07-20T16:37:03.768Z cpu5:2097619)NMP: nmp_ResetDeviceLogThrottling:3580: Error status H:0x0 D:0x2 P:0x0 Sense Data: 0x5 0x20 0x0 from dev "mpx.vmhba32:C0:T0:L0" occurred 1 times(of 1 commands)

2020-07-20T16:43:33.768Z cpu4:2097675)DVFilter: 5963: Checking disconnected filters for timeouts

2020-07-20T16:53:33.768Z cpu4:2097675)DVFilter: 5963: Checking disconnected filters for timeouts

2020-07-20T16:54:15.047Z cpu3:2097182)ScsiDeviceIO: 3435: Cmd(0x459a40b3dd80) 0x1a, CmdSN 0xb2f from world 0 to dev "mpx.vmhba32:C0:T0:L0" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.

2020-07-20T17:03:33.767Z cpu4:2097675)DVFilter: 5963: Checking disconnected filters for timeouts

2020-07-20T17:07:03.768Z cpu3:2097619)NMP: nmp_ResetDeviceLogThrottling:3580: Error status H:0x0 D:0x2 P:0x0 Sense Data: 0x5 0x20 0x0 from dev "mpx.vmhba32:C0:T0:L0" occurred 1 times(of 1 commands)

2020-07-20T17:13:33.767Z cpu0:2097675)DVFilter: 5963: Checking disconnected filters for timeouts

2020-07-20T17:23:33.768Z cpu0:2097675)DVFilter: 5963: Checking disconnected filters for timeouts

2020-07-20T17:32:19.577Z cpu0:2097179)ScsiDeviceIO: 3435: Cmd(0x459a40ce8940) 0x1a, CmdSN 0xba9 from world 0 to dev "mpx.vmhba32:C0:T0:L0" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.

2020-07-20T17:33:33.767Z cpu0:2097675)DVFilter: 5963: Checking disconnected filters for timeouts

2020-07-20T17:37:03.767Z cpu5:2097619)NMP: nmp_ResetDeviceLogThrottling:3580: Error status H:0x0 D:0x2 P:0x0 Sense Data: 0x5 0x20 0x0 from dev "mpx.vmhba32:C0:T0:L0" occurred 1 times(of 1 commands)

2020-07-20T17:43:33.767Z cpu0:2097675)DVFilter: 5963: Checking disconnected filters for timeouts

2020-07-20T17:53:33.767Z cpu1:2097675)DVFilter: 5963: Checking disconnected filters for timeouts

2020-07-20T18:03:33.767Z cpu1:2097675)DVFilter: 5963: Checking disconnected filters for timeouts

2020-07-20T18:06:10.862Z cpu1:2097180)ScsiDeviceIO: 3435: Cmd(0x459a40d313c0) 0x85, CmdSN 0x1e from world 2099077 to dev "mpx.vmhba32:C0:T0:L0" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0.

2020-07-20T18:07:03.767Z cpu0:2097619)NMP: nmp_ResetDeviceLogThrottling:3580: Error status H:0x0 D:0x2 P:0x0 Sense Data: 0x5 0x20 0x0 from dev "mpx.vmhba32:C0:T0:L0" occurred 1 times(of 1 commands)

You are having check conditions on the storage array

Might want to check over there.

Also try rebooting host's management services (services.sh restart) or simply reboot the host.

[Flammi]

I also tried it with VC6.7 doesn't work.

I tried ESX 6.5 & 6.7 both didn't work.

It's totally bizarre!

[Flammi]

Hi Nachogonzalez,

I don't have any storage devices connected. Just a local NVMe M.2 SSD. That's it.

Thanks

Pierre

[harry89]

check esxi thouroghly ... all the services ... ram size  df -h

[nachogonzalez]

have you tried restarting management services?

[Flammi]

Thank you both. I didn't only restart mgmt service or checked services on the host, I reinstalled ESXi probably 10 times by now with different versions.

will try ESXi7 tomorrow....

[harry89]

Are there any core dumps on vc

cd /var/core

[RajeevVCP4]

Vcenter server appliances or windows based ?

Are you able to ping ESXi host from vcenter server

Are you able to connect ESXi by root ( ui)

is esxi have valid license

[harry89]

sub=TrustedInfrastructure.HostConfig

module is something which is newly added in VC 7.0.

IT could be something related  on the hardware as well , if there is any secure boot or any other TPM level feature on the host as well .

[Flammi]

Hi Rajeev,

yes all that standard stuff is working just fine.

I see the the vCenter is trying to communicate to the ESX hosts. But for some reason the login fails.

192.168.2.45 is my vCenter.

2020-07-20T19:13:32Z vmauthd[2105865]: Msg_SetLocaleEx: HostLocale=UTF-8 UserLocale=NULL

2020-07-20T19:13:32Z vmauthd[2105865]: Could not expand environment variable HOME.

2020-07-20T19:13:32Z vmauthd[2105865]: Could not expand environment variable HOME.

2020-07-20T19:13:32Z vmauthd[2105865]: DictionaryLoad: Cannot open file "/usr/lib/vmware/config": No such file or directory.

2020-07-20T19:13:32Z vmauthd[2105865]: DictionaryLoad: Cannot open file "~/.vmware/config": No such file or directory.

2020-07-20T19:13:32Z vmauthd[2105865]: DictionaryLoad: Cannot open file "~/.vmware/preferences": No such file or directory.

2020-07-20T19:13:32Z vmauthd[2105865]: lib/ssl: OpenSSL using FIPS_drbg for RAND

2020-07-20T19:13:32Z vmauthd[2105865]: lib/ssl: protocol list tls1.2

2020-07-20T19:13:32Z vmauthd[2105865]: lib/ssl: protocol list tls1.2 (openssl flags 0x17000000)

2020-07-20T19:13:32Z vmauthd[2105865]: lib/ssl: cipher list ECDHE+AESGCM:RSA+AESGCM:ECDHE+AES:RSA+AES

2020-07-20T19:13:32Z vmauthd[2105865]: lib/ssl: curves list prime256v1:secp384r1:secp521r1

2020-07-20T19:13:32Z vmauthd[2105865]: Connect from remote socket (192.168.2.45:55718).

2020-07-20T19:13:32Z vmauthd[2105865]: Connect from 192.168.2.45

2020-07-20T19:13:32Z vmauthd[2105865]: recv() FAIL: 1.

2020-07-20T19:13:32Z vmauthd[2105865]: VMAuthdSocketRead: read failed.  Closing socket for reading.

2020-07-20T19:13:32Z vmauthd[2105865]: Read failed.

2020-07-20T19:13:34Z vmauthd[2105958]: Msg_SetLocaleEx: HostLocale=UTF-8 UserLocale=NULL

2020-07-20T19:13:34Z vmauthd[2105958]: Could not expand environment variable HOME.

2020-07-20T19:13:34Z vmauthd[2105958]: Could not expand environment variable HOME.

2020-07-20T19:13:34Z vmauthd[2105958]: DictionaryLoad: Cannot open file "/usr/lib/vmware/config": No such file or directory.

2020-07-20T19:13:34Z vmauthd[2105958]: DictionaryLoad: Cannot open file "~/.vmware/config": No such file or directory.

2020-07-20T19:13:34Z vmauthd[2105958]: DictionaryLoad: Cannot open file "~/.vmware/preferences": No such file or directory.

2020-07-20T19:13:34Z vmauthd[2105958]: lib/ssl: OpenSSL using FIPS_drbg for RAND

2020-07-20T19:13:34Z vmauthd[2105958]: lib/ssl: protocol list tls1.2

2020-07-20T19:13:34Z vmauthd[2105958]: lib/ssl: protocol list tls1.2 (openssl flags 0x17000000)

2020-07-20T19:13:34Z vmauthd[2105958]: lib/ssl: cipher list ECDHE+AESGCM:RSA+AESGCM:ECDHE+AES:RSA+AES

2020-07-20T19:13:34Z vmauthd[2105958]: lib/ssl: curves list prime256v1:secp384r1:secp521r1

2020-07-20T19:13:34Z vmauthd[2105958]: Connect from remote socket (192.168.2.45:55738).

2020-07-20T19:13:34Z vmauthd[2105958]: Connect from 192.168.2.45

2020-07-20T19:13:34Z vmauthd[2105958]: recv() FAIL: 1.

2020-07-20T19:13:34Z vmauthd[2105958]: VMAuthdSocketRead: read failed.  Closing socket for reading.

2020-07-20T19:13:34Z vmauthd[2105958]: Read failed.

Thanks

Pierre

[harry89]

Hey Flammi ,

Please try below step on esxi

• Connect to the ESXi using Host Client
•    Select Manage Tab
•    Select Advanced Settings
•    Locate the option "Config.HostAgent.ssl.keyStore.allowSelfSigned"
•    Edit the value from false to true

Cheers !

[nachogonzalez]

Hi @flammi
can you provide vCenter and ESXi actual versions?

warm regards