Adding ESXi host to vCenter fails

[Flammi]

Hi All,

I've spend more than two days on this already and I'm absolutely clueless by now. Was hoping someone here can help.

I have one vCenter in my home lab with one ESX. Bought a second ESX and I'm trying to add it to my vCenter without success.

I already deployed a new vCenter installed different versions of ESX no luck.

Right now I have a vCenter 7 and the ESXi host I'm trying to add is running 6.7.

When I try to add the host to the vCenter I see the folloing in the vmauthd.log of the ESX host:

2020-07-20T13:59:17Z vmauthd[2102937]: Msg_SetLocaleEx: HostLocale=UTF-8 UserLocale=NULL

2020-07-20T13:59:17Z vmauthd[2102937]: Could not expand environment variable HOME.

2020-07-20T13:59:17Z vmauthd[2102937]: Could not expand environment variable HOME.

2020-07-20T13:59:17Z vmauthd[2102937]: DictionaryLoad: Cannot open file "/usr/lib/vmware/config": No such file or directory.

2020-07-20T13:59:17Z vmauthd[2102937]: DictionaryLoad: Cannot open file "~/.vmware/config": No such file or directory.

2020-07-20T13:59:17Z vmauthd[2102937]: DictionaryLoad: Cannot open file "~/.vmware/preferences": No such file or directory.

2020-07-20T13:59:17Z vmauthd[2102937]: lib/ssl: OpenSSL using FIPS_drbg for RAND

2020-07-20T13:59:17Z vmauthd[2102937]: lib/ssl: protocol list tls1.2

2020-07-20T13:59:17Z vmauthd[2102937]: lib/ssl: protocol list tls1.2 (openssl flags 0x17000000)

2020-07-20T13:59:17Z vmauthd[2102937]: lib/ssl: cipher list ECDHE+AESGCM:RSA+AESGCM:ECDHE+AES:RSA+AES

2020-07-20T13:59:17Z vmauthd[2102937]: lib/ssl: curves list prime256v1:secp384r1:secp521r1

2020-07-20T13:59:17Z vmauthd[2102937]: Connect from remote socket (192.168.2.45:58542).

2020-07-20T13:59:17Z vmauthd[2102937]: Connect from 192.168.2.45

2020-07-20T13:59:17Z vmauthd[2102937]: recv() FAIL: 1.

2020-07-20T13:59:17Z vmauthd[2102937]: VMAuthdSocketRead: read failed.  Closing socket for reading.

2020-07-20T13:59:17Z vmauthd[2102937]: Read failed.

ON the vCenter side I see those errors in the vpxd.log

2020-07-20T15:59:27.291+02:00 info vpxd[14503] [Originator@6876 sub=InvtHostCnx opID=kcuf2q8s-2242-auto-1qc-h5:70001655-75-StartHostSync-61f79812] Connect in progress; [vim.HostSystem:host-2021,esxi-nuc.flammi.home]

2020-07-20T15:59:27.299+02:00 error vpxd[13388] [Originator@6876 sub=TrustedInfrastructure.HostConfig] [VapiBaseClientwrapper] failed: Error:

-->    system_error

--> Messages:

-->    vapi.invalid.result.code

-->

2020-07-20T15:59:27.300+02:00 warning vpxd[13499] [Originator@6876 sub=TrustedInfrastructure.HostConfig opID=kcuf2q8s-2242-auto-1qc-h5:70001655-75] [AcquireJWToken] Host 'esxi-nuc.flammi.home' is missing trusted VAPI service. Exception: N4Vpxd7Langley29BadVapiConfigurationExceptionE(Error:

-->    system_error

--> Messages:

-->    vapi.invalid.result.code

--> )

-->

Any ideas?

Thanks a lot fellas!

[harry89]

After doing some testing in my lab

I figured that this issue occurs if there is self signed certificate in trusted store .

This causes problem with TLS channel communication .

Please try the action plan as suggested earlier .

Please try below step on esxi

• Connect to the ESXi using Host Client
•    Select Manage Tab
•    Select Advanced Settings
•    Locate the option "Config.HostAgent.ssl.keyStore.allowSelfSigned"
•    Edit the value from false to true

cheers!!

[Flammi]

Hi Harry, Hi Nachogonzales,

thanks for your continuous support. I do appreciate it!

@Harry: I tried that already. Didn't help. Also tried to set vpxd.certmgmt.mode to thumbprint. Also no change.

@Nacho: ESX: 6.7.0 Update 3 (Build 15160138) vCenter: 6.7.0 (Build 9433931)

Didn't have time to install vCenter 7 and ESXi 7 today. Will do that tomorrow morning.

KR

Pierre

[nachogonzalez]

Are you trying to connect a vSphere 7 ESXi host to a vCenter 6.7?

[Flammi]

No, ESX: 6.7.0 Update 3 (Build 15160138) vCenter: 6.7.0 (Build 9433931)

I will install a vCenter 7 tomorrow and install ESXi 7 on the NUC.

But I'm starting to believe it's a hardware problem.

Almost impossible to have the same problems with all the different versions i've tested it with.

[nachogonzalez]

I know what's going on:

The vCenter is on a "old" version

VMware Knowledge Base

And the ESXi you are trying to connect is on a newer version

VMware Knowledge Base

Try upgrading your VCSA to version 6.7 u2b or later.
Another option would be to install a version of ESXi earlier than ESXi 6.7 U1

Let me know if this works

[SrVMoussa]

Is your vCenter higher than the ESXi? if so are you sure that you don't have any comparability issues  HW/Software ?

Is your HW HPE?

[Flammi]

Hi All,

and again thanks for trying to help me.

Unfortunately nothing of the suggested works.

Deployed the latest version of vCenter 6.7 (Build 16046713) and also tried it with vCenter 7 (again).

I tried every possible combination of Config.HostAgent.ssl.keyStore.allowSelfSigned set to true and vpxd.certmgmt.mode (vmca & thumbprint).

Always getting the same error in the logs:

2020-07-23T09:09:17.419Z info vpxd[03944] [Originator@6876 sub=HostAccess opID=kcyjqlyk-571-auto-fx-h5:70000385-7b] Using vpxapi.version.version14 to communicate with vpxa at host esxinuc.flammi.home

2020-07-23T09:09:17.482Z info vpxd[04127] [Originator@6876 sub=vpxLro opID=kcyjqlyk-f0-h5:70000103-26] [VpxLRO] -- FINISH lro-2535

2020-07-23T09:09:17.511Z info vpxd[03944] [Originator@6876 sub=HostAccess opID=kcyjqlyk-571-auto-fx-h5:70000385-7b] Using vpxapi.version.version14 to communicate with vpxa at host esxinuc.flammi.home

2020-07-23T09:09:17.517Z info vpxd[03944] [Originator@6876 sub=HostAccess opID=kcyjqlyk-571-auto-fx-h5:70000385-7b] Using vpxapi.version.version14 to communicate with vpxa at host esxinuc.flammi.home

2020-07-23T09:09:17.623Z info vpxd[03935] [Originator@6876 sub=vpxLro opID=vb-168:auto-name-43:01-e] [VpxLRO] -- BEGIN lro-2660 -- ResourceModel -- cis.data.provider.ResourceModel.query -- 526ef062-6849-e5e0-574d-c1c740b4c262(52d5c693-d4ce-07df-5f85-a39c2dc4d9a8)

2020-07-23T09:09:17.623Z info vpxd[03935] [Originator@6876 sub=vpxLro opID=vb-168:auto-name-43:01-e] [VpxLRO] -- FINISH lro-2660

2020-07-23T09:09:17.664Z error vpxd[03944] [Originator@6876 sub=Vmomi opID=kcyjqlyk-571-auto-fx-h5:70000385-7b] [VpxaClientAdapter] error logging into host esxinuc.flammi.home: N7Vmacore4Http24MalformedHeaderExceptionE(Server closed connection after 0 response bytes read; <SSL(<io_obj p:0x00007f7d9815c4c0, h:46, <TCP '192.168.2.45 : 51194'>, <TCP '192.168.2.6 : 443'>>)>)

--> [context]zKq7AVECAAAAAHna9AAPdnB4ZAAA4AArbGlidm1hY29yZS5zbwAAWCUbABWWGABrJhwAb48dAF+SHQByJh4A+oUaAEi1JQCpoiMAcW8jADpyIwCdVisB1HMAbGlicHRocmVhZC5zby4wAALdjg5saWJjLnNvLjYA[/context]

2020-07-23T09:09:17.664Z info vpxd[03935] [Originator@6876 sub=InvtHostCnx opID=kcyjqlyk-571-auto-fx-h5:70000385-7b-StartHostSync-2e48ec63] Connect in progress;[vim.HostSystem:host-32,esxinuc.flammi.home]

2020-07-23T09:09:17.664Z error vpxd[03944] [Originator@6876 sub=Vmomi opID=kcyjqlyk-571-auto-fx-h5:70000385-7b] Got vmacore exception: N7Vmacore4Http24MalformedHeaderExceptionE(Server closed connection after 0 response bytes read; <SSL(<io_obj p:0x00007f7d9815c4c0, h:46, <TCP '192.168.2.45 : 51194'>, <TCP '192.168.2.6 :

443'>>)>)

-->[context]zKq7AVECAAAAAHna9AAPdnB4ZAAA4AArbGlidm1hY29yZS5zbwAAWCUbABWWGABrJhwAb48dAF+SHQByJh4A+oUaAEi1JQCpoiMAcW8jADpyIwCdVisB1HMAbGlicHRocmVhZC5zby4wAALdjg5s

aWJjLnNvLjYA[/context]

2020-07-23T09:09:17.664Z info vpxd[03935] [Originator@6876 sub=InvtHostCnx opID=kcyjqlyk-571-auto-fx-h5:70000385-7b-StartHostSync-35976f52] Connect in progress;

[vim.HostSystem:host-32,esxinuc.flammi.home]

2020-07-23T09:09:17.670Z error vpxd[03944] [Originator@6876 sub=InvtHost opID=kcyjqlyk-571-auto-fx-h5:70000385-7b] Failed to reconnect to cleanup before host removal: N3Vim5Fault14SSLVerifyFault9ExceptionE(Fault cause: vim.fault.SSLVerifyFault)

vpxa & hostd on the ESXi are running. Also deleted /etc/vmware/vpxa/vpxa.conf and verified that the IP in the newly created vpxa.conf matches the IP of the host.

Also Regenerated the certificate on the ESXi. Nothing....

I think I'm giving up now...

[pareshgk]

We are trying to add ESXI 6.5.0 (build 14320405) to VMware VSphere (version 6.5.0.20000 build 9451637) but getting following error.

ESXI 6.5.0 (build 14320405) host is otherwise accessible on web browser and we are able to login to VM using same root credentials.

Also we are getting proper ping response to/from esxi host IP/FQDN and  VMware VSphere IP/FQDN.

In addition to that we were able to successfully install and add 2 ESXI 6.5.0 hosts with same version from other locations to VMware VSphere (version 6.5.0) same version.

we do not suspect any issues with Network. Any assistance to fix this issue would be great help.

[nachogonzalez]

Hello, you are trying to connect a newer version of esxi to an older version of vCenter.

Try upgrading the vCenter or downgrading the ESXi host.

Are all the other esxi hosts in the same version?

Do you have a firewall in the middle?

[pareshgk]

Thanks for your reply.

Answering to your queries,

Try upgrading the vCenter or downgrading the ESXi host. - Its won't be possible to upgrade vCenter, let me see if I can downgrade ESXI.

Are all the other esxi hosts in the same version? - Yes, all the Esxi Successfully added to vCenter are on the same version as failed one. It's strange that only this ESXI host particularly failed. 

Do you have a firewall in the middle? - Nope. Both these ESXI host and Vcenter are on Corporate LAN.

[harry89]

• DNS
• Host Services hostd,vpxa
• firewall

Please check the above things

[Flammi · Accepted solution]

Hi All,

I found the problem.

I had a unsupported NVMe drive in the Intel NUC.

Guess I have to replace it 😉

Thanks so much for all your help.

I do appreciate it!

KR

Pierre

[pareshgk]

Hello All,

First of all thanks a lot for your valuable suggestions and troubleshooting steps.

Wish to inform you that the my issue has been resolved as well.

Looks like it was duplicate IP issue, we have changed the IP address assigned to esxi host 6.5 and also modified DNS entry for the host and issue has been resolved.

Note :- we have just used another free IP address in the same subnet.

Thank you again.