vCenter automatically changes Host name/loses connection

[jhammer2000]

I am acknowledging that I know vCenter 6.0 is way out of date, due to non-technical issues we are over 6 months from being able to start the move to 7.0 for this environment.

So, we have a vCenter 6.0 (ver 6.0.0 2776510) appliance running, hosting 14 hosts and 60 VMs, has been fairly solid over the years.  We just started having an issue with a single ESXi Host (ver 6.0.0 3620759) on a HPE DL360 G9, all the other servers are the same generation of HPE and same ESXi version.  The infrastructure does not use DNS so the vCenter and Hosts are connected by IP only

Recently the host in question started showing as disconnected with the wrong IP showing in vCenter.  We removed it and reconnected with the correct IP, it connects, shows the correct info and then suddenly the name/IP of the host changes back to the incorrect one.  The Events tab for the Datacenter view shows the host being connected and such then "IP address changed from x.x.x.x to y.y.y.y".  This is clearly happening on the vcenter side but I cannot find what is causing this nor can I find any matching google search results.

Does anyone have any ideas?

[ptarnawski]

Strange, but maybe the IP was already taken or there is some issue with the lease form DHCP. I don't think it is a vCenter issue. More something on your network. 

Do you use NSX ? If so, is the host added to NSX? 

Do you use any other firewalls, where hosts need to be added to? 

Also check port configuration for that hosts as ACL or config could change. 

[jhammer2000]

There is no DHCP on that network at all, everything is static and it also does not have NSX.  It is a flat network and the firewall is only doing basic firewall tasks, nothing fancy, it is basically a single subnet on its own VLAN not even DNS is configured on the ESX host or vCenter

Attached is a screen cap showing the vcenter events

[ptarnawski]

can you see who made a change? 

[jhammer2000]

No one, I have repeated this many times, confirmed no one else is connected to the vcenter and when I add it the event log shown in my prev response show it going through the normal process of adding a host and then it changes the IP.  That is what is so confusing here, I have never heard of a vcenter itself changing this info

[ptarnawski]

On the screen where you showed task changing IP, there should be info who made a change 

[jhammer2000]

I get ya now, the user field is empty, replicated the task and attached new capture. I can say for certainty that no one logged in and made any changes in that that 8 second window of the host being added and the IP change being shown

[robc_yk]

There is the ability within the platform when changing an IP that it will revert back if it thinks there is no connectivity. I wonder if something is hung up in this process, thinking that it needs to revert. Does it always go to and from the same IP? How long after does it with back after you "correct" the IP?

Can you confirm that the address you would like to use it infact correct in regards to any VLAN? Does it have connectivity when it is configured the way you want it?

Since it is an HPE server, set the correct IP through the ILO, confirm connectivity of assigned IP, when the host is disconnected, remove it from your vSphere inventory, let settle for a few moments and then connect back. Does the switch happen again?

---

Helpful? Let others know by adding a Kudos and / or accepting the solution.

[jhammer2000]

The 128.x IP is the correct IP and is what the actual Host is running, I can also connect to the Host directly via the 128.x IP with no issues. The change is always to the same 138.x IP.  The VLAN tagging is handled at the router/switch level so that the hosts and guests have no VLAN tagging enabled

[a_p_]

That's indeed a strange behavior, that I haven't seen yet.

Do the ESXi host's log files (e.g. the vmkernel.log) contain related entries?

André

[jhammer2000]

The only thing in the vmkernal log for that time (and before) is an series of events regarding the local storage (not boot) that we know of and the other hosts on this environment also have.

[a_p_]

Can you - just for troubleshooting purposes - configure the ESXi host with an unused IP address in the ESXi hosts' subnet, just to see what happens.

If it switches back to the incorrect IP address, then it's got to be something on the host itself, which causes this. If it doesn't, the issue is likely with vCenter Server.

André

[a_p_]

... there's no host profile associated with that host?

André

[jhammer2000]

Can't change the IP due to strict change control policies in place, would love to be able to but don't wont the IP police hunting me down 😀

We do not have any Host Profiles enabled.  Also I did create a completely new 'DataCenter' in vCenter and added the host there so no IP pools or other oddities on that level

[King_SHA]

my answer is not an answer for your real question but you must used DNS base on best practice, when you don't do simple things you faced with problem that cant find the source of it. 

every one known, name of the host couldn't change automatically. you must search for what happened caused that.

[a_p_]

>>> Can't change the IP due to strict change control policies in place ...
Not even for temporary troubleshooting purposes?

Well, since the wrong IP address shows up in vCenter, which would most likely not be possible if the change is initiated by the host itself (how could it report the wrong IP address without connectivity?), it's got to be something in vCenter Server.

It may be like searching the needle in a haystack, but what I'd do next is to search all vCenter Server log files for related entries.

André

[ptarnawski]

How come, you cannot change your IP? You need to investigate the issue and this is one of the steps. Open the case, or create a change case or SOP. The policy cannot prevent you from resolving the issue. 

Another thought: Was any 3rd party VIB installed on the ESXi host? 

Use this command: esxcli software vib list 

[jhammer2000]

Been digging through the logs, but not able to do so with much focus.  Hopefully will be able to do so this week.

[jhammer2000]

Certain IP space is strictly controlled internally, this system falls in that category.  The process to make any changes is time consuming and painful.  Since we have a plan to move this system to 7.0 onto an new IP space in about 6 or so months going through the internal process to get that authorized is not worth it.  Bypassing policy is not always possible

[ptarnawski]

I was not talking about bypassing policy, but you have to have a process to make a change in the environment. The host OP has been changed anyway. A bit confusing situation.