Host IP Address Conflict Detected Alarm is displayed after connection of ESXi 6.0 machine to VCSA 6.5 but no IP address Conflict is Present on the Network

[stoda05]

Hi,

I'm receiving an error when I try to add a vmhost to a vCenter instance.  The error is "Host IP Address Conflict Alarm" which makes it sound as though the IP address is a duplicate but it's not.  I can see the arp cache for that IP address matches the MAC address for the ESXi host and there's no instability when connecting to it.

I'm just wondering how VMware determines that one of it's hosts has a duplicate IP address?  I'm thinking it's in the vCenter DB somewhere (I'm using VCSA 6.5 and ESXi 6.0 in this case).  The only other issue I can see is looking at the file /etc/vmware/vpxa/vpxa.cfg the entry for <serverPort> is -1, whereas it's 902 on the other ESXi instances that are using this vCenter server.  If I restart the host, it goes back to a value of -1.

I've managed to get it stable with 902 by editing the /etc/vmware/vpxa/vpxa.cfg file and restarting vpxa only by running /etc/init.d/vpxa start and the Vcenter console is no longer complaining that the host has a conflicting IP address, but it's disconnecting from my ESXi host on a regular basis

If I clear all the warnings (including the warning: Host connection and power state) I can reconnect the host from the GUI for a period of about 30 seconds, before it goes back to a state of Not Responding.  While it's connected it shows the error: cannot synchronize host.

If you could help, I'd really appreciate it!

Cheers,

Daniel

[parmarr]

I believe this is related to the entries still present in the DB. For more information, see https://kb.vmware.com/s/article/1003893.

[alphajfg]

I'm in the same situation. In our case, this started happening after our old vCenter Server from which we migrated was started by mistake. It ran for several days before we noticed it and it's now been stopped. However, half of our hosts are now disconnected and nothing seems to bring them back.

@parmarr, is it possible you didn't link to the right KB? It's talking about virtual machines networking issues, not hosts. I'd be really interested to know how to fix those bad DB entries.

Thanks!

[Yuva_1990]

Hi

This seems to be an issue with vcenter server where its detecting an invalid ip address from the esxi.It would be great if you can attach the vpxd logs from the affected Vcenter along with a time frame for further investigation.

Regards

Yuvaraj

[alphajfg]

Quick follow up, I noticed that one of our server running ESXi 6.5 stayed connected throughout. I proceeded to upgrade one that was running 6.0 (and kept disconnecting) to 6.5 and since then it's been 100% stable.

I'll upgrade all of my instances to 6.5 as it seems to be the solution for us.

[Daisukey1380]

Hi

I have got a similar problem and I saw vCenter server Missed 12 heartbeats before the disconnection of the host from vCenter Server.

Here I paste my vpxd.log grepped by opID=HeartbeatStartHandler-7c09267..

----------

2018-08-22T10:18:48.887+09:00 info vpxd[7FBE7B56A700] [Originator@6876 sub=InvtHostCnx opID=HeartbeatStartHandler-7c09267] [VpxdIntHost] Missed 12 heartbeats for host test-esxi01.example.local

2018-08-22T10:18:48.887+09:00 info vpxd[7FBE7B56A700] [Originator@6876 sub=InvtHostCnx opID=HeartbeatStartHandler-7c09267] Connect in progress; [vim.HostSystem:host-66,test-esxi01.example.local]

2018-08-22T10:18:58.897+09:00 info vpxd[7FBE7B56A700] [Originator@6876 sub=InvtHostCnx opID=HeartbeatStartHandler-7c09267] Connect in progress; [vim.HostSystem:host-66,test-esxi01.example.local]

2018-08-22T10:19:08.917+09:00 info vpxd[7FBE7B56A700] [Originator@6876 sub=InvtHostCnx opID=HeartbeatStartHandler-7c09267] Connect in progress; [vim.HostSystem:host-66,test-esxi01.example.local]

2018-08-22T10:19:18.937+09:00 info vpxd[7FBE7B56A700] [Originator@6876 sub=InvtHostCnx opID=HeartbeatStartHandler-7c09267] Connect in progress; [vim.HostSystem:host-66,test-esxi01.example.local]

2018-08-22T10:19:29.491+09:00 warning vpxd[7FBE7B56A700] [Originator@6876 sub=InvtHostCnx opID=HeartbeatStartHandler-7c09267] Host IP address conflict detected. test-esxi01.example.local is trying to change IP from test-esxi01.example.local to 172.16.30.21. New IP is already used by test-esxi01.example.local

----------

Hope this helps further investigation...

Regards

[F_Jonasson]

I have a similar row in my vpxd log:

2019-05-27T15:23:03.149+02:00 warning vpxd[7F865181B700] [Originator@6876 sub=InvtHostCnx opID=HeartbeatStartHandler-72e666b4] Host IP address conflict detected. esxi3.domain.local is trying to change IP from 172.16.32.74 to 172.16.4.1. New IP is already used by esxi1.domain.local

The from IP-address is the correct address that it should use. Neither host should use 172.16.4.1 and I can't find any reference of esxi1 using that address.

172.16.4.1 in my case is a CARP-VIP for two virtual firewalls(pfSense) running on esxi1 and esxi2.

[F_Jonasson]

Issue is now resolved for us.

The fix to move vCenter (VCSA) to same subnet(vlan) as ESXi's.

[AFerrit]

I actually hit this same issue with pfsense where my vcenter is on a different subnet.  The problem is that your pfsense box was doing outbound NAT.  When the ESXI box attempted to phone home to the vcenter, its source packets got rewritten by pfsense to be the NAT IP (which is by default the default outside interface).  

Vcenter suddenly said "Hey, this box was on this interface but now a request with this name is coming from this new IP"

Pfsense: Firewall --> NAT --> Outbound and disable it.  (or at least, disable it for hosts INSIDE your network).