VMware Cloud Community
aldikan
Hot Shot
Hot Shot

Virtual Client Connection from 64bit OS crashes Virtual Center

Hi Guys,

Virtual Center 4.1U1

Running on Windows 2008R2 Standard as a VM instance. Database is MS SQL2005 SP2 on separate physical box,

50+ ESX hosts,

Upgraded from 4.0 to 4.1U1 back in March, everything was working fine for 3 weeks, and then we start having issues: every time you establish Virtual Client connection from 64 bit OS, (windows 7 or another Windows 2008R2),

Virtual Center service stopping..., and needs to be started.

There is a reference in vpxd logs about the session: (see below).

VMware SR was opened, 10 days ago, all logs submitted, no solution yet.

We have re-installed Virtual Center Server, keeping same database and problem is still there.

Anybody else seen this? Any ideas?

Thanks guys!

*************************

[2011-04-15 08:52:53.915 02272 verbose 'App'] TaskInfoChannel destroyed for session[52862505-8a2c-51db-819a-cd951ab898dd]52178de6-81ef-a9fa-abdd-a0ab496694f5
[2011-04-15 08:53:07.478 02272 info 'App'] CoreDump: Writing minidump
[2011-04-15 08:53:15.108 02272 panic 'App']

Panic: Win32 exception: Access Violation (0xc0000005)
   Read (0) at address ffffffffffffffff
   rip: 00000001800cf794 rsp: 000000002013eee0 rbp: 000000018006fbf0
   rax: 000000002013efa0 rbx: 0000000000000005 rcx: 000000002013ef28
   rdx: 500000000755d120 rdi: 0000000000000012 rsi: 000000002a84e170
   r8:  0000000000000000 r9:  0000000000000004 r10: 000000000a8787a0
   r11: 000000002013efa8 r12: 0000000028fff700 r13: 0000000000000001
   r14: 00000000000000ff r15: 0000000000000000

Backtrace:
backtrace[00] rip 000000018010a8aa Vmacore::System::Stacktrace::CaptureWork
backtrace[01] rip 00000001800e8008 Vmacore::System::SystemFactoryImpl::CreateFileWriter
backtrace[02] rip

Reply
0 Kudos
6 Replies
Troy_Clavell
Immortal
Immortal

see if the below article helps

http://kb.vmware.com/kb/1028750

aldikan
Hot Shot
Hot Shot

Thanks Troy,

We looked into this KB already and our DB shows no corrupted VMs,

also,

Virtual Center can work fine for days, that is if no Virtual Client  connections from 64bit VMs were initiated.

Following suggestion from VMware we have removed all ESX hosts from the Virtual center and added them back..,

this has been done to "refresh" Hosts entries in the Database, however did not help the issue.

Also there is a KB on issues when you have "Nexus1000v" which we don't have either.

One thing to note: VC client established from Virtual Center itself.., does not crash the server.

Edit: Also, no Antivirus is running on VC, Windows firewall turned OFF.

Thanks,

Bring on ideas, guys,


Thanks,

Alex

Reply
0 Kudos
Troy_Clavell
Immortal
Immortal

10 days you have had this SR open and it's still not resolved?  Another option would be destroy the DB and start from scratch.  However, I would lean heavily of VMware to find a resolution.  Do you have a TAM (Technical Account Manager)?

I run W7 Enterprise x64 and have not expienced your issue.

One thing to note: VC client established from Virtual Center itself.., does not crash the server.

hmmm... Can you reproduce the crash on every other client OS connecting into vCenter using the vSphere Client?  You are running the most current client on all your 64bit clients connecting into vCenter?

Reply
0 Kudos
aldikan
Hot Shot
Hot Shot

Thanks Troy,

Yeah, we are trying to avoid destroying DB and starting fresh,

Case was opened on April 7th, SR: 11057493104

VMware Support is not where we would like it to be,

although case was escalated with 2 different channels:

normal and via VMware account manager, we still did not get too far:

We were told that logs are not giving much info, until we showed VM tech entries in the logs pertaining to connection and initiated crashes to prove the point,

Last Friday communication:

"Hello Alex,

Thank you for your Support Request.

As per our conversation over the phone, the case has been escalated and we are working with engineering team to resolve the issue.

We have received the logs and would request time until mid next week so that we can provide you an update...."

Received a call today

Anyway, back to the business:

We can reproduce the issue clearly from any Windows 7 64bit desktop, as well as from the Win2K8R2 guest VM..,

all are running latest VC, downloaded and installed by going https://VC or https://esxhost.

Another thing we just noticed:


from 64 bit OS, even if you just go to https://VC, VC will crash:

Here is the vpxd entry from above operation:

[2011-04-18 12:13:29.524 05008 verbose 'SoapAdapter.HTTPService'] User agent is 'Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.1; WOW64; Trident/4.0; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729)'

[2011-04-18 12:13:29.524 05008 verbose 'SoapAdapter.HTTPService'] HTTP Response: Client: NeedsContentLength: false UnderstandsChunking: true CanKeepAlive: true (PresetContentLength 3486)

[2011-04-18 12:13:29.524 05008 verbose 'SoapAdapter.HTTPService'] HTTP Response: Auto-completing at 3486/3486 bytes

[2011-04-18 12:13:29.525 05008 verbose 'SoapAdapter.HTTPService'] HTTP Response: Complete (processed 3486 bytes)

[2011-04-18 12:13:29.525 05008 verbose 'HTTP server'] Sent response for HEAD / (from C:\ProgramData\VMware\VMware VirtualCenter\docRoot)

[2011-04-18 12:13:44.701 05008 info 'App'] CoreDump: Writing minidump

[2011-04-18 12:13:53.898 05008 panic 'App']

Panic: Win32 exception: Access Violation (0xc0000005)

   Read (0) at address ffffffffffffffff

   rip: 00000001800cf794 rsp: 000000001541f0f0 rbp: 000000018006fbf0

   rax: 000000001541f1b0 rbx: 0000000000000004 rcx: 000000001541f138

   rdx: 4547000007585480 rdi: 000000000000001e rsi: 000000002b38eab0

   r8:  0000000000000000 r9:  0000000000000000 r10: 0000000011d5fba0

   r11: 000000002aeaac60 r12: 00000000209dc600 r13: 0000000000000000

   r14: 0000000000000142 r15: 0000000000000000

Going to give Wireshark another try, will start it up and then initiate the crash. Will chat with Security guys, maybe something was done that caused all this.

Thanks,

Alex

Reply
0 Kudos
aldikan
Hot Shot
Hot Shot

Hi Guys,

For those who followed this discussion, here is update:

- Case with VMware still opened, seems to not go anywhere. As per Tech, this had been escalated to Development/Engineering department.

- With my co-worker we have done a tons of leg work and found the culprit

  • Micorsoft Direct Access server was introduced to the environment
  • 64 bit client OSs such as Windows7 and Win2K8R2 started to use ISATAP encapsulated addresses: (96bit from IPv6+32bit IPv4)
  • Attempting to establish client session with ISATAP address string will crash Virtual Center Server service right a way.
  • We created a new test environment: (brand new Virtual Center 4.1U1 Server/new SQL Express DB and a few W2K8R2 guests, and reproduced the problem every time,
  • If regular IPv6 is used: No issues
  • If IPv4 is used no issues
  • If ISATAP is used (string will look like: 2002:8e98:3e83:8000:200:5efe:10.x.x.x) this will crash Virtual Center

I am thinking we discovered the bug, and will try to address this with VMware.

Hope this helps somebody who is thinking of introducing Microsoft Direct Access server and planing to use ISATAP( Intra-Site Automatic Tunnel Addressing Protocol)

Will update, when more info available

Alex

Reply
0 Kudos
aldikan
Hot Shot
Hot Shot

This has been identified as a VMware bug,

Problem should be fixed in the future update (4.1U2).

I have been informed by Vmware Support over the phone and written confirmation should follow in the next few days.

Alex

Reply
0 Kudos