Hosts continuously disconnecting and reconnection in Virtual Center

[PSchutte]

My Virtual Center server is running as a VM. I tried to use storage vmotion to relocate this VM onto another datastore; this unfortunately resulted in corruption of my c: drive and I decided to rebuild the VM from scratch. Fortunately the VCMS database is located on another machine and remained intact. After the re-installation, virtual center come up with everything intact as expected, however when you want to perform any administrative function on a VM an error “vim.fault.InvalidLogin” is returned. Doing research on the fault, it look like the ssl certificates on the ESX hosts are not what virtual center is expecting and I found this community article http://communities.vmware.com/thread/81593 explaining how to fix it. You need to disconnect the host and reconnect it again from virtual center. I have done this and also restarting the mgmt-vmware service on all 5 esx hosts. It seems to have solved the “vim.fault.InvalidLogin” problem, but it has created a new one. On an individual and intermittent basis all 5 hosts will disconnect and reconnect from VC. There is constantly at least one (sometimes more) host disconnected at any one time. It will stay disconnected for approx 1 min and reconnected again, almost on a polling basis. Please help!!!

[azn2kew]

We experienced this before all ESX hosts in a cluster is showing "disconnected" status every 1-3 minutes but the virtual machines themselves still functional. It really shocked when we first saw that status but nothing affected. We spent 12 hours with VMware Support finally realize it was SSL related issue with other ESX hosts not working nicely. Look at the log and see which ESX hosts showing up in the log and disconnect and remove from cluster and re-add it and it would be quicker to remove and re-add them if you can't find the culprit.

I don't have a record of my SR# otherwise it would be quickly to reference it.

If you found this information useful, please consider awarding points for "Correct" or "Helpful". Thanks!!!

Regards,

Stefan Nguyen

iGeek Systems Inc.

VMware, Citrix, Microsoft Consultant

[admin]

How long between the connected and disconnected states?

Are there any backtrace entries in the VC logs? Can you attach the last 2 log files?

They're usually located in C:\Documents & Settings\All Users\Application data\VMware\VirtualCenter server\logs\

[PSchutte]

Hi

Attached are the last two log files from VC. I think we on the right track looking at the ssl certificates. The 5 hosts in the cluster are affected, disconnecting between 1 to 3 minutes at a time. I also have two "standalone" hosts not part of any cluster, there connections are fine. I have disconnected and reconnect all 5 hosts already, shall I try to remove the hosts out of the cluster and re-add them?

[admin · Accepted solution]

Yeah from the vpxd-6.log you can see that VC fails to decrypt the vpxuser password and ends up backtracing and disconnecting the server.

They show that the problem happens on the below servers but it may be happening on others so you will need to check all the log files:

10.100.142.50

10.100.50.115

To check the log files quickly you can use the following command from Linux: grep "Invalid login on host" vpxd-*.log |awk '{print($NF)}' |sort -u

Or zip and upload all the log files and I'll get the list.

Try disconnecting all servers from VC, including the 2 standalone, then reconnect them all.

If that doesn't work, try removing the "Invalid login" ones and then re-add them.

If that doesn't work, we'll need to NULL the password in the VC database and delete the user account on the ESX server. Let me know the outcome of the other steps first though.

-

---

crypto failure: error:0407106B:rsa routines:RSA_padding_check_PKCS1_type_2:block type is not 02

Failed to decrypt password: applying key to encrypted data failed (likely the wrong key)

Invalid login on host 10.100.50.115

GetChanges host:10.100.50.115 (10.100.50.115) took 2484 ms

Got unexpected exception: vim.fault.InvalidLogin

Backtrace: backtrace[00] eip 0x0122b526 ?GenerateCoreDump@System@Vmacore@@YAXXZ

backtrace[01] eip 0x01176caa ?CreateBacktrace@SystemFactoryImpl@System@Vmacore@@UAEXAAV?$Ref@VBacktrace@System@Vmacore@@@3@@Z

backtrace[02] eip 0x0115192e ??0Throwable@Vmacore@@QAE@ABV?$basic_string@DU?$char_traits@D@std@@V?$allocator@D@2@@std@@@Z

backtrace[03] eip 0x014a1509 (no symbol)

backtrace[04] eip 0x014d8b81 ?UnsetNoRxLoss@PipeBackingInfo@VirtualSerialPort@Device@Vm@Vim@@QAEXXZ

backtrace[05] eip 0x014d8d48 ?UnsetNoRxLoss@PipeBackingInfo@VirtualSerialPort@Device@Vm@Vim@@QAEXXZ

backtrace[06] eip 0x014daf48 ?Throw@InvalidLocale@Fault@Vim@@UAEXXZ

backtrace[07] eip 0x014daf7f ?Throw@InvalidLogin@Fault@Vim@@UAEXXZ

backtrace[08] eip 0x013a82ce ?CreateSoapStubAdapter@Vmomi@@YAXABV?$basic_string@DU?$char_traits@D@std@@V?$allocator@D@2@@std@@HPAVSSLContext@Ssl@Vmacore@@00PAVLogger@Service@6@AAV?$Ref@VStubAdapter@Vmomi@@@6@@Z

backtrace[09] eip 0x013a9051 ?CreateSoapStubAdapter@Vmomi@@YAXABV?$basic_string@DU?$char_traits@D@std@@V?$allocator@D@2@@std@@HPAVSSLContext@Ssl@Vmacore@@00PAVLogger@Service@6@AAV?$Ref@VStubAdapter@Vmomi@@@6@@Z

backtrace[10] eip 0x013779ce ?_Invoke_Task@StubImpl@Vmomi@@UAEXPAVManagedMethod@2@AAV?$RefVector@VAny@Vmomi@@@Vmacore@@AAV?$Ref@VAny@Vmomi@@@5@@Z

backtrace[11] eip 0x00862e56 (no symbol)

backtrace[12] eip 0x0050ff37 (no symbol)

backtrace[13] eip 0x00510985 (no symbol)

backtrace[14] eip 0x00510a38 (no symbol)

backtrace[15] eip 0x00510a8d (no symbol)

backtrace[16] eip 0x0051169f (no symbol)

backtrace[17] eip 0x00511af1 (no symbol)

backtrace[18] eip 0x013779ce ?_Invoke_Task@StubImpl@Vmomi@@UAEXPAVManagedMethod@2@AAV?$RefVector@VAny@Vmomi@@@Vmacore@@AAV?$Ref@VAny@Vmomi@@@5@@Z

backtrace[19] eip 0x008630a2 (no symbol)

backtrace[20] eip 0x0053b0e5 (no symbol)

backtrace[21] eip 0x0053bad8 (no symbol)

backtrace[22] eip 0x006d5df6 (no symbol)

backtrace[23] eip 0x006dad6e (no symbol)

backtrace[24] eip 0x006e0e04 (no symbol)

backtrace[25] eip 0x00a1c06a (no symbol)

backtrace[26] eip 0x77e64829 GetModuleHandleA

[PSchutte]

Hi

Interesting, this is the IP's of the two standalone hosts that is not

affected by the disconnections. I ran the script on all the log files

and none of the other hosts are listed. I will disconnect and reconnect

all the hosts and let you know.

Regards

[PSchutte]

Hi Appk

I have disconnected all the hosts including the standalone ones as you suggested, then reconnecting them one at a time and also restarting the mgmt-vmware and vmware-vpxa services on all hosts just before reconnecting them in VC. The environment is a 1000% more stable with only one host disconnecting twice in the last two hours. The difference is that it stays disconnected until mgmt-vmware is restarted on that host. It might be caused by something not related to the original problem.

Thank you for your assistance.

Regards

Philip

[azn2kew]

Phillip,

Points should always good for appreciation on Correct/Helpful bro! Helps us motivated helping out.

If you found this information useful, please consider awarding points for "Correct" or "Helpful". Thanks!!!

Regards,

Stefan Nguyen

iGeek Systems Inc.

VMware, Citrix, Microsoft Consultant

[admin]

It does sound unrelated.

It could be anything such as a service console memory issue or a locked file.

You can check memory and swapping issues using the commands free -m and vmstat.

If you want me to have a look at the issue then attach the hostd and vpxa logs from around the time the disconnect occurs.

hostd logs are in /var/log/vmware and start with hostd

vpxa logs are in /var/log/vmware/vpx