Hi All,
I've spend more than two days on this already and I'm absolutely clueless by now. Was hoping someone here can help.
I have one vCenter in my home lab with one ESX. Bought a second ESX and I'm trying to add it to my vCenter without success.
I already deployed a new vCenter installed different versions of ESX no luck.
Right now I have a vCenter 7 and the ESXi host I'm trying to add is running 6.7.
When I try to add the host to the vCenter I see the folloing in the vmauthd.log of the ESX host:
2020-07-20T13:59:17Z vmauthd[2102937]: Msg_SetLocaleEx: HostLocale=UTF-8 UserLocale=NULL
2020-07-20T13:59:17Z vmauthd[2102937]: Could not expand environment variable HOME.
2020-07-20T13:59:17Z vmauthd[2102937]: Could not expand environment variable HOME.
2020-07-20T13:59:17Z vmauthd[2102937]: DictionaryLoad: Cannot open file "/usr/lib/vmware/config": No such file or directory.
2020-07-20T13:59:17Z vmauthd[2102937]: DictionaryLoad: Cannot open file "~/.vmware/config": No such file or directory.
2020-07-20T13:59:17Z vmauthd[2102937]: DictionaryLoad: Cannot open file "~/.vmware/preferences": No such file or directory.
2020-07-20T13:59:17Z vmauthd[2102937]: lib/ssl: OpenSSL using FIPS_drbg for RAND
2020-07-20T13:59:17Z vmauthd[2102937]: lib/ssl: protocol list tls1.2
2020-07-20T13:59:17Z vmauthd[2102937]: lib/ssl: protocol list tls1.2 (openssl flags 0x17000000)
2020-07-20T13:59:17Z vmauthd[2102937]: lib/ssl: cipher list ECDHE+AESGCM:RSA+AESGCM:ECDHE+AES:RSA+AES
2020-07-20T13:59:17Z vmauthd[2102937]: lib/ssl: curves list prime256v1:secp384r1:secp521r1
2020-07-20T13:59:17Z vmauthd[2102937]: Connect from remote socket (192.168.2.45:58542).
2020-07-20T13:59:17Z vmauthd[2102937]: Connect from 192.168.2.45
2020-07-20T13:59:17Z vmauthd[2102937]: recv() FAIL: 1.
2020-07-20T13:59:17Z vmauthd[2102937]: VMAuthdSocketRead: read failed. Closing socket for reading.
2020-07-20T13:59:17Z vmauthd[2102937]: Read failed.
ON the vCenter side I see those errors in the vpxd.log
2020-07-20T15:59:27.291+02:00 info vpxd[14503] [Originator@6876 sub=InvtHostCnx opID=kcuf2q8s-2242-auto-1qc-h5:70001655-75-StartHostSync-61f79812] Connect in progress; [vim.HostSystem:host-2021,esxi-nuc.flammi.home]
2020-07-20T15:59:27.299+02:00 error vpxd[13388] [Originator@6876 sub=TrustedInfrastructure.HostConfig] [VapiBaseClientwrapper] failed: Error:
--> system_error
--> Messages:
--> vapi.invalid.result.code
-->
2020-07-20T15:59:27.300+02:00 warning vpxd[13499] [Originator@6876 sub=TrustedInfrastructure.HostConfig opID=kcuf2q8s-2242-auto-1qc-h5:70001655-75] [AcquireJWToken] Host 'esxi-nuc.flammi.home' is missing trusted VAPI service. Exception: N4Vpxd7Langley29BadVapiConfigurationExceptionE(Error:
--> system_error
--> Messages:
--> vapi.invalid.result.code
--> )
-->
Any ideas?
Thanks a lot fellas!
Hi All,
I found the problem.
I had a unsupported NVMe drive in the Intel NUC.
Guess I have to replace it 😉
Thanks so much for all your help.
I do appreciate it!
KR
Pierre
Is the host added and in disconnected state .
Are you able to login directly to the host through host client
Send output of service-control --status --all on VC.
Also endpoint.log snippet as well during the operation
Hi harr89,
thanks for the help. Really appreciate the reply.
The host is not added at all.
It appears in the list of hosts as disconnected for about two seconds and then disappears again.
I can login to the host just fine to manage it directly.
root@vcenter [ ~ ]# service-control --status --all
Running:
applmgmt lookupsvc lwsmd observability observability-vapi pschealth vlcm vmafdd vmcad vmdird vmonapi vmware-analytics vmware-certificateauthority vmware-certifp
Stopped:
vmcam vmware-imagebuilder vmware-netdumper vmware-rbd-watchdog vmware-vcha
There is nothing added to the endpoint.log of the vCenter during the operation.
Pierre
on vc ssh session
Go to cd /var/log/vmware
Go to folder observability or observability vapi and check the logs
this is related to something with trust relationship
From vc run below command
curl telnet://hostip:443 --- whats the ouput at the bottom
Hey Flammi, hope you are doing fine
let me ask you a few questions:
if you log in via SSH to the VCSA can you ping the new host?
can you log in to host web console?
can you log in to host via ssh?
Can you please share host's vmkernel.log output?
Looking forward to hearing form you
Warm regards
Hi Harry,
I'm with you. I also assume it's something trust related. Just can't get my head around it and have no experience with this at all.
I didn't fine anything suspicious in the logs you mentioned.
observability-vapi.log -> nothing added during the process
observability-runtime.log.stderr -> nothing added during the process
observability-runtime.log.stdout -> nothing added during the process
The only logfile where something is added is the observability.log. I couldn't attache it for some reason so you can have a look at it here.
I tried to add the server at 12:16:25
If I run curl telnet://hostip:443 nothing happens. No errors, no output.
Hi Nachogonzalez,
thanks a lot for your reply.
Yes, all the other logins work just fine.
I have a VM running on the server already.
Ping is also possible. IP or DNS name.
Kernel log is here: Dropbox - vmkernel.log
I couldn't find anything in there the helps.
I tried to add the host again today at 12:16 CET.
So in the kernel log it's 10:16 (UTC).
Thanks a lot
Pierre
Trust me this will be some bug with product VC 7.0
Hey Flammi
I was just reviewing the logs
2020-07-20T15:01:00.096Z cpu1:2097180)ScsiDeviceIO: 3435: Cmd(0x459a40d64180) 0x1a, CmdSN 0x9c5 from world 0 to dev "mpx.vmhba32:C0:T0:L0" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2020-07-20T15:03:33.768Z cpu7:2097675)DVFilter: 5963: Checking disconnected filters for timeouts
2020-07-20T15:07:03.768Z cpu1:2097619)NMP: nmp_ResetDeviceLogThrottling:3580: Error status H:0x0 D:0x2 P:0x0 Sense Data: 0x5 0x20 0x0 from dev "mpx.vmhba32:C0:T0:L0" occurred 1 times(of 1 commands)
2020-07-20T15:13:33.768Z cpu5:2097675)DVFilter: 5963: Checking disconnected filters for timeouts
2020-07-20T15:23:33.768Z cpu0:2097675)DVFilter: 5963: Checking disconnected filters for timeouts
2020-07-20T15:33:33.768Z cpu6:2097675)DVFilter: 5963: Checking disconnected filters for timeouts
2020-07-20T15:36:10.774Z cpu7:2097186)ScsiDeviceIO: 3435: Cmd(0x459a40cdafc0) 0x85, CmdSN 0x15 from world 2099077 to dev "mpx.vmhba32:C0:T0:L0" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0.
2020-07-20T15:37:03.768Z cpu0:2097619)NMP: nmp_ResetDeviceLogThrottling:3580: Error status H:0x0 D:0x2 P:0x0 Sense Data: 0x5 0x20 0x0 from dev "mpx.vmhba32:C0:T0:L0" occurred 1 times(of 1 commands)
2020-07-20T15:43:33.768Z cpu6:2097675)DVFilter: 5963: Checking disconnected filters for timeouts
2020-07-20T15:53:33.768Z cpu5:2097675)DVFilter: 5963: Checking disconnected filters for timeouts
2020-07-20T16:03:33.770Z cpu0:2097675)DVFilter: 5963: Checking disconnected filters for timeouts
2020-07-20T16:07:03.768Z cpu5:2097619)NMP: nmp_ResetDeviceLogThrottling:3580: Error status H:0x0 D:0x2 P:0x0 Sense Data: 0x5 0x20 0x0 from dev "mpx.vmhba32:C0:T0:L0" occurred 1 times(of 1 commands)
2020-07-20T16:12:19.189Z cpu7:2097186)ScsiDeviceIO: 3435: Cmd(0x459a40a5af80) 0x1a, CmdSN 0xaaf from world 0 to dev "mpx.vmhba32:C0:T0:L0" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2020-07-20T16:13:33.768Z cpu0:2097675)DVFilter: 5963: Checking disconnected filters for timeouts
2020-07-20T16:23:33.768Z cpu4:2097675)DVFilter: 5963: Checking disconnected filters for timeouts
2020-07-20T16:33:33.768Z cpu4:2097675)DVFilter: 5963: Checking disconnected filters for timeouts
2020-07-20T16:37:03.768Z cpu5:2097619)NMP: nmp_ResetDeviceLogThrottling:3580: Error status H:0x0 D:0x2 P:0x0 Sense Data: 0x5 0x20 0x0 from dev "mpx.vmhba32:C0:T0:L0" occurred 1 times(of 1 commands)
2020-07-20T16:43:33.768Z cpu4:2097675)DVFilter: 5963: Checking disconnected filters for timeouts
2020-07-20T16:53:33.768Z cpu4:2097675)DVFilter: 5963: Checking disconnected filters for timeouts
2020-07-20T16:54:15.047Z cpu3:2097182)ScsiDeviceIO: 3435: Cmd(0x459a40b3dd80) 0x1a, CmdSN 0xb2f from world 0 to dev "mpx.vmhba32:C0:T0:L0" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2020-07-20T17:03:33.767Z cpu4:2097675)DVFilter: 5963: Checking disconnected filters for timeouts
2020-07-20T17:07:03.768Z cpu3:2097619)NMP: nmp_ResetDeviceLogThrottling:3580: Error status H:0x0 D:0x2 P:0x0 Sense Data: 0x5 0x20 0x0 from dev "mpx.vmhba32:C0:T0:L0" occurred 1 times(of 1 commands)
2020-07-20T17:13:33.767Z cpu0:2097675)DVFilter: 5963: Checking disconnected filters for timeouts
2020-07-20T17:23:33.768Z cpu0:2097675)DVFilter: 5963: Checking disconnected filters for timeouts
2020-07-20T17:32:19.577Z cpu0:2097179)ScsiDeviceIO: 3435: Cmd(0x459a40ce8940) 0x1a, CmdSN 0xba9 from world 0 to dev "mpx.vmhba32:C0:T0:L0" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
2020-07-20T17:33:33.767Z cpu0:2097675)DVFilter: 5963: Checking disconnected filters for timeouts
2020-07-20T17:37:03.767Z cpu5:2097619)NMP: nmp_ResetDeviceLogThrottling:3580: Error status H:0x0 D:0x2 P:0x0 Sense Data: 0x5 0x20 0x0 from dev "mpx.vmhba32:C0:T0:L0" occurred 1 times(of 1 commands)
2020-07-20T17:43:33.767Z cpu0:2097675)DVFilter: 5963: Checking disconnected filters for timeouts
2020-07-20T17:53:33.767Z cpu1:2097675)DVFilter: 5963: Checking disconnected filters for timeouts
2020-07-20T18:03:33.767Z cpu1:2097675)DVFilter: 5963: Checking disconnected filters for timeouts
2020-07-20T18:06:10.862Z cpu1:2097180)ScsiDeviceIO: 3435: Cmd(0x459a40d313c0) 0x85, CmdSN 0x1e from world 2099077 to dev "mpx.vmhba32:C0:T0:L0" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0.
2020-07-20T18:07:03.767Z cpu0:2097619)NMP: nmp_ResetDeviceLogThrottling:3580: Error status H:0x0 D:0x2 P:0x0 Sense Data: 0x5 0x20 0x0 from dev "mpx.vmhba32:C0:T0:L0" occurred 1 times(of 1 commands)
You are having check conditions on the storage array
Might want to check over there.
Also try rebooting host's management services (services.sh restart) or simply reboot the host.
I also tried it with VC6.7 doesn't work.
I tried ESX 6.5 & 6.7 both didn't work.
It's totally bizarre!
Hi Nachogonzalez,
I don't have any storage devices connected. Just a local NVMe M.2 SSD. That's it.
Thanks
Pierre
check esxi thouroghly ... all the services ... ram size df -h
have you tried restarting management services?
Thank you both. I didn't only restart mgmt service or checked services on the host, I reinstalled ESXi probably 10 times by now with different versions.
will try ESXi7 tomorrow....
Are there any core dumps on vc
cd /var/core
Vcenter server appliances or windows based ?
Are you able to ping ESXi host from vcenter server
Are you able to connect ESXi by root ( ui)
is esxi have valid license
sub=TrustedInfrastructure.HostConfig
module is something which is newly added in VC 7.0.
IT could be something related on the hardware as well , if there is any secure boot or any other TPM level feature on the host as well .
Hi Rajeev,
yes all that standard stuff is working just fine.
I see the the vCenter is trying to communicate to the ESX hosts. But for some reason the login fails.
192.168.2.45 is my vCenter.
2020-07-20T19:13:32Z vmauthd[2105865]: Msg_SetLocaleEx: HostLocale=UTF-8 UserLocale=NULL
2020-07-20T19:13:32Z vmauthd[2105865]: Could not expand environment variable HOME.
2020-07-20T19:13:32Z vmauthd[2105865]: Could not expand environment variable HOME.
2020-07-20T19:13:32Z vmauthd[2105865]: DictionaryLoad: Cannot open file "/usr/lib/vmware/config": No such file or directory.
2020-07-20T19:13:32Z vmauthd[2105865]: DictionaryLoad: Cannot open file "~/.vmware/config": No such file or directory.
2020-07-20T19:13:32Z vmauthd[2105865]: DictionaryLoad: Cannot open file "~/.vmware/preferences": No such file or directory.
2020-07-20T19:13:32Z vmauthd[2105865]: lib/ssl: OpenSSL using FIPS_drbg for RAND
2020-07-20T19:13:32Z vmauthd[2105865]: lib/ssl: protocol list tls1.2
2020-07-20T19:13:32Z vmauthd[2105865]: lib/ssl: protocol list tls1.2 (openssl flags 0x17000000)
2020-07-20T19:13:32Z vmauthd[2105865]: lib/ssl: cipher list ECDHE+AESGCM:RSA+AESGCM:ECDHE+AES:RSA+AES
2020-07-20T19:13:32Z vmauthd[2105865]: lib/ssl: curves list prime256v1:secp384r1:secp521r1
2020-07-20T19:13:32Z vmauthd[2105865]: Connect from remote socket (192.168.2.45:55718).
2020-07-20T19:13:32Z vmauthd[2105865]: Connect from 192.168.2.45
2020-07-20T19:13:32Z vmauthd[2105865]: recv() FAIL: 1.
2020-07-20T19:13:32Z vmauthd[2105865]: VMAuthdSocketRead: read failed. Closing socket for reading.
2020-07-20T19:13:32Z vmauthd[2105865]: Read failed.
2020-07-20T19:13:34Z vmauthd[2105958]: Msg_SetLocaleEx: HostLocale=UTF-8 UserLocale=NULL
2020-07-20T19:13:34Z vmauthd[2105958]: Could not expand environment variable HOME.
2020-07-20T19:13:34Z vmauthd[2105958]: Could not expand environment variable HOME.
2020-07-20T19:13:34Z vmauthd[2105958]: DictionaryLoad: Cannot open file "/usr/lib/vmware/config": No such file or directory.
2020-07-20T19:13:34Z vmauthd[2105958]: DictionaryLoad: Cannot open file "~/.vmware/config": No such file or directory.
2020-07-20T19:13:34Z vmauthd[2105958]: DictionaryLoad: Cannot open file "~/.vmware/preferences": No such file or directory.
2020-07-20T19:13:34Z vmauthd[2105958]: lib/ssl: OpenSSL using FIPS_drbg for RAND
2020-07-20T19:13:34Z vmauthd[2105958]: lib/ssl: protocol list tls1.2
2020-07-20T19:13:34Z vmauthd[2105958]: lib/ssl: protocol list tls1.2 (openssl flags 0x17000000)
2020-07-20T19:13:34Z vmauthd[2105958]: lib/ssl: cipher list ECDHE+AESGCM:RSA+AESGCM:ECDHE+AES:RSA+AES
2020-07-20T19:13:34Z vmauthd[2105958]: lib/ssl: curves list prime256v1:secp384r1:secp521r1
2020-07-20T19:13:34Z vmauthd[2105958]: Connect from remote socket (192.168.2.45:55738).
2020-07-20T19:13:34Z vmauthd[2105958]: Connect from 192.168.2.45
2020-07-20T19:13:34Z vmauthd[2105958]: recv() FAIL: 1.
2020-07-20T19:13:34Z vmauthd[2105958]: VMAuthdSocketRead: read failed. Closing socket for reading.
2020-07-20T19:13:34Z vmauthd[2105958]: Read failed.
Thanks
Pierre
Hey Flammi ,
Please try below step on esxi
Cheers !
Hi @flammi
can you provide vCenter and ESXi actual versions?
warm regards