Virtual Center crash when trying to add ESX host

[jhunter11]

Virtual Center was managing the server just fine. I changed the IP of the server and after rebooting and having the changes take affect, I can manage the server through the VI Client by pointing directly to it.

If I point to the Virtual Center Server it logs in just fine, but when I try to readd the host it pops up an error box that says: "The VI Client has lost the connection to "virtual.center.server" server. Do you want to return to the login dialog? Click NO to exit the VI Client."

If I press yes or no, it crashes the VMware VirtualCenter Server service and I have to restart it.

I have verified that the new IP of the ESX host can be seen from the VirtualCenter Server.

Anybody have any ideas?

[VBarak]

hi,

i would look for some remains of the last vpx agent on the esx.

if you find anything, be sure to forcly remove any "dirt" and try to readd the esx server to the VC

[jbrazio]

Stop the vCenter service.

Open a session to the vCenter server, Start > Run: cmd

On the console write: "C:\Program Files\VMware\Infrastructure\VirtualCenter Server\vpxd.exe" -s

Please pay attention if you have a different path to the vpxd to change it.

Running like this will allow you to see in "debug" mode wat vCenter is doing. Then try to connect with the client to see what makes vCenter crash.

-

---

If you find this post helpful please don't forget to award points to it.

[vistaphotos]

- First wild guess is that there's something in the inventory of the ESX that the VC doesn't like. Anything new added to the inventory of the ESX since it was last in VC?

- Second guess is that version of ESX may not be compatible with VC. ESX and VC should be same updates e.g. ESX 3.5 Update 3 with VC 2.5 Update 3 or later.

- What IP address was changed? The IP address of an ESX server managed by VirtualCenter or the IP address of the VC server itself? My guess is that it was the IP address of an ESX server so my comments below are made on that basis.

- You can connect your VI client to the VirtualCenter server OK but you get a problem when you try add the ESX host back into the inventory.

- Was the ESX completely removed from Inventory or are you trying to reconnect an entry that already exists in the inventory for that ESX? If you haven't already tried it I'd suggest completely removing the ESX from the inventory on VC and adding it back again.

- If you want to completely remove the vpxa package on ESX , "rpm -qa|grep vpx" to find the package version name on the ESX and "rpm -e <package>" to remove. Package will be pushed to ESX when connected in VC.

All suggestions offered in good faith, but proceed at your own risk

[jhunter11]

Okay, I have a quick update on this. It no longer only affects one ESX server. I tried removing and readding another ESX host to virtual center and it does the same thing.

There have been lots of VM Changes on the original ESX host in question, but the second one that I removed/readded has not been touched since adding it to virtual center.

That makes me believe it is a problem with virtual center.

Any more ideas?

[Troy_Clavell]

check the bottom of this KB http://kb.vmware.com/kb/5850444 under NOTE:

[jhunter11]

The VirtualCenter IP has not changed. Only the IP of the first ESX host, but it crashes on an ESX server that has been untouched as well, so I belive it is a VirtualCenter problem and not an ESX host problem. I updated that IP in DNS as well as the ESX host file. I also tried removing the VPX agent from the original ESX host to see if that would help. It still does the same thing. I try to add it in Virtual Center and it crashes Virtual Center.

[Troy_Clavell]

nothing in the event logs? Is you DB seperate? You have confirmend there are no disk space issues.

I know this may be a pain, but if you have any existing DB, you may want to shutdown your existing vCenter Server bring up a new vCenter instance as a VM and connect it to your DB and see what happens.

[jhunter11]

There is no log where there should be. Under the event list after I restart the server it says it connects to the host as I connect it, then fails to reconnect and that's when it crashes. It also has an error about licenses, but if I go to Administartion, there are plenty of licenses available.

Here is the license error it gives: "Host XX did not provide the information needed to acquire the correct set of licenses."

I will probably try to create a VM tomorrow and throw up the latest and greatest version of VCenter and point it to the existing database and see what happens.

Any other ideas? you've all been very helpful, thanks!

James

[jbrazio]

I tend to ask silly questions.. but did you restarted the management agent inside the hosts ?

1. service mgmt-vmware restart

-

---

If you find this post helpful please don't forget to award points to it.

[jhunter11]

Yes, I did restart the management agent as well as the VPX agent and it still does the same thing. Today, I'm going to try to install VCenter on a VM and see if it can add hosts to see if that might be the problem.

If you guys have any more ideas please shoot them my way! Thanks for all the help!

James

[vistaphotos]

Running VC in a VM isn't a bad idea as a troubleshooting step. What's the build number of the ESX software? What's the build number of the VC software?

Confirm that you can ping the ESX by IP address, short hostname and FQDN from the VC

Confirm that you can ping the VC server by IP address, short hostname and FQDN from the ESX

Confirm that hostname, hostname -i, hostname -s, hostname -f are correct on the ESX and match the entries in /etc/hosts, /etc/sysconfig/network /etc/vmware/esx.conf (path may not be correct, filename is) as well as the output of "esxcfg-vswif -l" and ifconfig

When attempting to add the ESX to VC, run "tail -f /var/log/vmware/hostd.log" on the ESX and look at the messaging. Check the C:\Document and Settings\All Users\Application Data\VMware\VirtualCenter Server\Logs\vpxd.log to see what's going on at the VC side.

[jhunter11]

vistaphotos,

Build Number of ESX Software: ESX 3.5.0, Build 82663.

Build Number of VC Software: VirtualCenter 2.5.0, Build 84767

ESX can ping IP Address, short hostname and FQDN of VC.

VC can ping IP Address, short hostname and FQDN of ESX.

I see some warnings in the vpxd.log that say the following:

SSLVerifyCertAgainstSystemStore: The remote host certificate has these problems:* A certificate in the host's chain is based on an untrusted root.

SSLVerifyCertAgainstSystemStore: Certificate verification is disabled, so connection will proceed despite the error

I also see some errors in the vpxd log that say the following:

cached HostConfigInfo has null configInfo

Unable to recover from 42000:7139

Unable to recover from 01000:3621

Unable to recover from 42000:7139

Unable to recover from 01000:3621

Failed to persist host ConfigInfo offline

Exception Invalid argument : /vpx/actv/lro/vim.HostSystem.reconnect//errors/numSamples thrown while publishing ctr /vpx/actv/lro/vim.HostSystem.reconnect//errors

An unrecoverable problem has occurred, stopping the VMware VirtualCenter service. Check database connectivity before restarting. Error: Error[VdbODBCError] (-1) "ODBC error: (42000) - [ODBC SQL Server Driver][SQL Server]Length of text, ntext, or image data (142028) to be replicated exceeds configured maximum 65536." is returned when executing SQL statement "INSERT INTO VPX_HOST_VM_CONFIG_OPTION (HOST_ID, CONFIG_OPTION_VER, DATA, ARRAY_INDEX) VALUES (?, ?, ?, ?)"

Forcing shutdown of VMware VirtualCenter now

Maybe this stuff can shed some light on the situation? Thanks. Let me know!

James

[Troy_Clavell]

you may try to delte the rui.* files under /etc/vmware/ssl on the ESX Host which is crashing your vCenter Server. Once you delete the files, restart hostd (service mgmt-vmware restart) and then try to add it back in.

[jhunter11]

I tried deleting the RUI files and readding and the same thing happened with the same errors. We are going to try updating VC tomorrow so I will keep you all posted on what that does.

Thanks for all your help, guys.

James

[satishgte]

Hi

I assume this is the problem with your Database server. You can just restart your database server as well as agent services and check.

Than ks

[jhunter11]

After uninstalling and reinstalling all of Virtual Center Server, including making brand new databases, everything works fine.

[Troy_Clavell]

glad to see you got it resolved... Sometimes it's just better to "start over" Good Deal!

[Aladen]

Are you by chance replicating your vmware database?

I just had the same problem, and this worked for me.

[hicksj]

Build Number of VC Software: VirtualCenter 2.5.0, Build 84767

When you rebuilt, did you update to a later version? We ran into a very similar problem with this build... Every time we would attempt a manual VMotion, VC would crash. (DRS seemed to be working fine) Perhaps the same would have happened to us when adding a host - I don't know as I don't believe we had any additions during that period. All attempts to resolve the issue through a Support Request failed. We then decided to upgrade VC and the problem went away.

Now that I think of it, I believe we also readdressed two hosts around the same time. Appears to have been a bug in that version.