Skip navigation
VMware

This Question is Answered (go to answer)

1 "helpful" answer available (6 pts)
1,501 Views 8 Replies Last post: Aug 29, 2008 10:11 AM by appk RSS
PSchutte Novice 13 posts since
Jan 11, 2008
Currently Being Moderated

Aug 28, 2008 5:57 AM

Hosts continuously disconnecting and reconnection in Virtual Center

My Virtual Center server is running as a VM. I tried to use storage vmotion to relocate this VM onto another datastore; this unfortunately resulted in corruption of my c: drive and I decided to rebuild the VM from scratch. Fortunately the VCMS database is located on another machine and remained intact. After the re-installation, virtual center come up with everything intact as expected, however when you want to perform any administrative function on a VM an error “vim.fault.InvalidLogin” is returned. Doing research on the fault, it look like the ssl certificates on the ESX hosts are not what virtual center is expecting and I found this community article http://communities.vmware.com/thread/81593 explaining how to fix it. You need to disconnect the host and reconnect it again from virtual center. I have done this and also restarting the mgmt-vmware service on all 5 esx hosts. It seems to have solved the “vim.fault.InvalidLogin” problem, but it has created a new one. On an individual and intermittent basis all 5 hosts will disconnect and reconnect from VC. There is constantly at least one (sometimes more) host disconnected at any one time. It will stay disconnected for approx 1 min and reconnected again, almost on a polling basis. Please help!!!

azn2kew Champion 3,022 posts since
Jun 21, 2006
Currently Being Moderated
1. Aug 28, 2008 7:24 AM in response to: PSchutte
Re: Hosts continuously disconnecting and reconnection in Virtual Center

We experienced this before all ESX hosts in a cluster is showing "disconnected" status every 1-3 minutes but the virtual machines themselves still functional.  It really shocked when we first saw that status but nothing affected.  We spent 12 hours with VMware Support finally realize it was SSL related issue with other ESX hosts not working nicely.   Look at the log and see which ESX hosts showing up in the log and disconnect and remove from cluster and re-add it and it would be quicker to remove and re-add them if you can't find the culprit. 

 

I don't have a record of my SR# otherwise it would be quickly to reference it.

 

If you found this information useful, please consider awarding points for "Correct" or "Helpful". Thanks!!!

 

Regards,

 

Stefan Nguyen

iGeek Systems Inc.

VMware, Citrix, Microsoft Consultant

If you found this information useful, please consider awarding points for "Correct" or "Helpful". Thanks!!! Regards, Stefan Nguyen VMware vExpert 2009 iGeek Systems Inc. VMware vExpert, VCP 3 & 4, VSP, VTSP, CCA, CCEA, CCNA, MCSA, EMCSE, EMCISA
Expert 181 posts since
Apr 6, 2008
Currently Being Moderated
2. Aug 28, 2008 10:25 AM in response to: PSchutte
Re: Hosts continuously disconnecting and reconnection in Virtual Center

How long between the connected and disconnected states?

 

Are there any backtrace entries in the VC logs? Can you attach the last 2 log files?

They're usually located in C:\Documents & Settings\All Users\Application data\VMware\VirtualCenter server\logs\

Expert 181 posts since
Apr 6, 2008
Currently Being Moderated
4. Aug 29, 2008 4:01 AM in response to: PSchutte
Re: Hosts continuously disconnecting and reconnection in Virtual Center

Yeah from the vpxd-6.log you can see that VC fails to decrypt the vpxuser password and ends up backtracing and disconnecting the server.

They show that the problem happens on the below servers but it may be happening on others so you will need to check all the log files:

10.100.142.50

10.100.50.115

 

To check the log files quickly you can use the following command from Linux: grep "Invalid login on host" vpxd-*.log |awk '{print($NF)}' |sort -u

Or zip and upload all the log files and I'll get the list.

 

Try disconnecting all servers from VC, including the 2 standalone, then reconnect them all.

If that doesn't work, try removing the "Invalid login" ones and then re-add them.

If that doesn't work, we'll need to NULL the password in the VC database and delete the user account on the ESX server. Let me know the outcome of the other steps first though.

 

-


 

 

Re: Hosts continuously disconnecting and reconnection in Virtual Center Re: Hosts continuously disconnecting and reconnection in Virtual Center crypto failure: error:0407106B:rsa routines:RSA_padding_check_PKCS1_type_2:block type is not 02

Re: Hosts continuously disconnecting and reconnection in Virtual Center Re: Hosts continuously disconnecting and reconnection in Virtual Center Failed to decrypt password: applying key to encrypted data failed (likely the wrong key)

Re: Hosts continuously disconnecting and reconnection in Virtual Center Re: Hosts continuously disconnecting and reconnection in Virtual Center Invalid login on host 10.100.50.115

Re: Hosts continuously disconnecting and reconnection in Virtual Center Re: Hosts continuously disconnecting and reconnection in Virtual Center GetChanges host:10.100.50.115 (10.100.50.115) took 2484 ms

Re: Hosts continuously disconnecting and reconnection in Virtual Center Re: Hosts continuously disconnecting and reconnection in Virtual Center Got unexpected exception: vim.fault.InvalidLogin

Re: Hosts continuously disconnecting and reconnection in Virtual Center Backtrace: backtrace[00] eip 0x0122b526 ?GenerateCoreDump@System@Vmacore@@YAXXZ

backtrace[01] eip 0x01176caa ?CreateBacktrace@SystemFactoryImpl@System@Vmacore@@UAEXAAV?$Ref@VBacktrace@System@Vmacore@@@3@@Z

backtrace[02] eip 0x0115192e ??0Throwable@Vmacore@@QAE@ABV?$basic_string@DU?$char_traits@D@std@@V?$allocator@D@2@@std@@@Z

backtrace[03] eip 0x014a1509 (no symbol)

backtrace[04] eip 0x014d8b81 ?UnsetNoRxLoss@PipeBackingInfo@VirtualSerialPort@Device@Vm@Vim@@QAEXXZ

backtrace[05] eip 0x014d8d48 ?UnsetNoRxLoss@PipeBackingInfo@VirtualSerialPort@Device@Vm@Vim@@QAEXXZ

backtrace[06] eip 0x014daf48 ?Throw@InvalidLocale@Fault@Vim@@UAEXXZ

backtrace[07] eip 0x014daf7f ?Throw@InvalidLogin@Fault@Vim@@UAEXXZ

backtrace[08] eip 0x013a82ce ?CreateSoapStubAdapter@Vmomi@@YAXABV?$basic_string@DU?$char_traits@D@std@@V?$allocator@D@2@@std@@HPAVSSLContext@Ssl@Vmacore@@00PAVLogger@Service@6@AAV?$Ref@VStubAdapter@Vmomi@@@6@@Z

backtrace[09] eip 0x013a9051 ?CreateSoapStubAdapter@Vmomi@@YAXABV?$basic_string@DU?$char_traits@D@std@@V?$allocator@D@2@@std@@HPAVSSLContext@Ssl@Vmacore@@00PAVLogger@Service@6@AAV?$Ref@VStubAdapter@Vmomi@@@6@@Z

backtrace[10] eip 0x013779ce ?_Invoke_Task@StubImpl@Vmomi@@UAEXPAVManagedMethod@2@AAV?$RefVector@VAny@Vmomi@@@Vmacore@@AAV?$Ref@VAny@Vmomi@@@5@@Z

backtrace[11] eip 0x00862e56 (no symbol)

backtrace[12] eip 0x0050ff37 (no symbol)

backtrace[13] eip 0x00510985 (no symbol)

backtrace[14] eip 0x00510a38 (no symbol)

backtrace[15] eip 0x00510a8d (no symbol)

backtrace[16] eip 0x0051169f (no symbol)

backtrace[17] eip 0x00511af1 (no symbol)

backtrace[18] eip 0x013779ce ?_Invoke_Task@StubImpl@Vmomi@@UAEXPAVManagedMethod@2@AAV?$RefVector@VAny@Vmomi@@@Vmacore@@AAV?$Ref@VAny@Vmomi@@@5@@Z

backtrace[19] eip 0x008630a2 (no symbol)

backtrace[20] eip 0x0053b0e5 (no symbol)

backtrace[21] eip 0x0053bad8 (no symbol)

backtrace[22] eip 0x006d5df6 (no symbol)

backtrace[23] eip 0x006dad6e (no symbol)

backtrace[24] eip 0x006e0e04 (no symbol)

backtrace[25] eip 0x00a1c06a (no symbol)

backtrace[26] eip 0x77e64829 GetModuleHandleA

azn2kew Champion 3,022 posts since
Jun 21, 2006
Currently Being Moderated
7. Aug 29, 2008 7:34 AM in response to: PSchutte
Re: Hosts continuously disconnecting and reconnection in Virtual Center

 

Phillip,

 

 

Points should always good for appreciation on Correct/Helpful bro!  Helps us motivated helping out.

 

If you found this information useful, please consider awarding points for "Correct" or "Helpful". Thanks!!!

 

Regards,

 

Stefan Nguyen

iGeek Systems Inc.

VMware, Citrix, Microsoft Consultant

 

 

If you found this information useful, please consider awarding points for "Correct" or "Helpful". Thanks!!! Regards, Stefan Nguyen VMware vExpert 2009 iGeek Systems Inc. VMware vExpert, VCP 3 & 4, VSP, VTSP, CCA, CCEA, CCNA, MCSA, EMCSE, EMCISA
Expert 181 posts since
Apr 6, 2008
Currently Being Moderated
8. Aug 29, 2008 10:11 AM in response to: PSchutte
Re: Hosts continuously disconnecting and reconnection in Virtual Center

It does sound unrelated.

It could be anything such as a service console memory issue or a locked file.

 

You can check memory and swapping issues using the commands free -m and vmstat.

 

If you want me to have a look at the issue then attach the hostd and vpxa logs from around the time the disconnect occurs.

hostd logs are in /var/log/vmware and start with hostd

vpxa logs are in /var/log/vmware/vpx

Bookmarked By (0)

Share This Page

Communities