Access violation exception in gvmomi.dll (0xC0000005)

[Jelar]

Hi all!

I'm using PowerShell and VIX API to automate working with VMs on VMWare vSphere. When a lot of VM's processing for a long period of time in different threads, PowerShell script crashes with access violation exception (0xC0000005). After analyzing this issue with Debug Diagnostic Tools I have got the following report:

The module C:\Program Files\VMware\VMware  VIX\VSphere-4.1\32bit\gvmomi.dll has caused an access violation  exception (0xC0000005) when trying to read from memory location  0x00000000.

Thread 29 - System ID 4292

Entry point	vix!Vix_TranslateCryptoError+140
Create time	4/19/2011 5:36:03 PM
Time spent in user mode	0 Days 0:0:6.687
Time spent in kernel mode	0 Days 0:0:8.515

Function	Arg 1	Arg 2	Arg 3	Source
0x00000000	064e0970	00000000	00000000
gvmomi!gvmomi_session_sspi_connect+3916	064e0970	00000000	00000000
gvmomi!gvmomi_session_retrieve_contents+a8	04f10688	06558af0	11173eb2
vix!Vix_VMDBSet+21bad	683ce9a5	00000000	00000000
vix!Vix_VMDBSet+21e16	683ce9a5	00000000	00000000
vix!Vix_VMDBSet+239ce	00000000	00000000	00000000
vix!Vix_PumpEvents+167b	04ea3bb0	05f7ffec	7c80b729
vix!Vix_TranslateCryptoError+154	04ea3bb0	7c9101db	00000000
kernel32!BaseThreadStart+37	111aa1b0	04ea3bb0	00000000

Module Information
Image Name:	C:\Program Files\VMware\VMware VIX\VSphere-4.1\32bit\gvmomi.dll	Symbol Type:	Export
Base address:	0x05260000	Time Stamp:	Sat Jun 12 07:17:07 2010
Checksum:	0x00333675	Comments:
COM DLL:	False	Company Name:	VMware, Inc.
ISAPIExtension:	False	File Description:	VMware GVMOMI Library
ISAPIFilter:	False	File Version:	e.x.p build-266898
Managed DLL:	False	Internal Name:	gvmomi
VB DLL:	False	Legal Copyright:	Copyright © 1998-2009 VMware, Inc.
Loaded Image Name:	gvmomi.dll	Legal Trademarks:
Mapped Image Name:		Original filename:	gvmomi.dll
Module name:	gvmomi	Private Build:
Single Threaded:	False	Product Name:	VMware Workstation
Module Size:	3.21 MBytes	Product Version:	e.x.p build-266898
Symbol File Name:	gvmomi.dll	Special Build:	&

Recommendation   Please follow up with the vendor of this module, VMware, Inc., for  further assistance with this issue.

Anyone has idea how to resolve this issue? Maybe this gvmomi.dll will be fixed in the future?

Waiting for yours advices...

With best regards,

Jelar

[juliendufour]

Hello

I'm facing the same problem using the VIX API through a Java JNI wrapper. The issue if very sporadic but always happens when opening a disk. VDDK 1.2.0 is installed on the server.

Here is a copy of a JVM dump log :

#

# A fatal error has been detected by the Java Runtime Environment:

#

#  EXCEPTION_ACCESS_VIOLATION (0xc0000005) at pc=0x11103ae4, pid=752, tid=6272

#

# JRE version: 6.0_21-b07

# Java VM: Java HotSpot(TM) Client VM (17.0-b17 mixed mode, sharing windows-x86 )

# Problematic frame:

# C  [gvmomi.dll+0x3ae4]

#

# If you would like to submit a bug report, please visit:

#   http://java.sun.com/webapps/bugreport/crash.jsp

# The crash happened outside the Java Virtual Machine in native code.

# See problematic frame for where to report the bug.

#

---------------  T H R E A D  ---------------

Current thread (0x02f82800):  JavaThread "Thread-12" daemon [_thread_in_native, id=6272, stack(0x036e0000,0x03730000)]

siginfo: ExceptionCode=0xc0000005, reading address 0x0000000d

Registers:

EAX=0x00000001, EBX=0x00000000, ECX=0x00000001, EDX=0x11305f14

ESP=0x0372eeb0, EBP=0x0372eeb8, ESI=0x00000001, EDI=0x08b53740

EIP=0x11103ae4, EFLAGS=0x00210202

Top of Stack: (sp=0x0372eeb0)

0x0372eeb0:   00000006 08b99ac0 08b59998 065b96ac

0x0372eec0:   08b53740 08cbe670 00000000 05de4008

0x0372eed0:   00000000 05de4018 0372eefc 11103e97

0x0372eee0:   08b59998 11103aa0 00000000 05de4008

0x0372eef0:   05ede068 05de4010 05de400c 05de4008

0x0372ef00:   06755b67 05de4008 08b53b00 05eee1e8

0x0372ef10:   0372ef20 061f2064 05de4008 061f2040

0x0372ef20:   00000000 0675f492 08b53b00 05eee1e8

Instructions: (pc=0x11103ae4)

0x11103ad4:   00 00 68 b0 5e 30 11 50 e8 ad 0b 1e 00 83 c4 14

0x11103ae4:   8b 46 0c 8b 48 10 51 68 94 46 2e 11 6a 50 57 e8

Stack: [0x036e0000,0x03730000],  sp=0x0372eeb0,  free space=13b0372e9eck

Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)

C  [gvmomi.dll+0x3ae4]

Java frames: (J=compiled Java code, j=interpreted, Vv=VM code)

j  com.intrinsec.drp4it.vixdisklib.VDLConnection.vdlOpen(Ljava/lang/String;I)Lcom/intrinsec/drp4it/vixdisklib/VDLDiskHandle;+0

j  com.intrinsec.drp4it.vixdisklib.VDLConnection.access$100(Lcom/intrinsec/drp4it/vixdisklib/VDLConnection;Ljava/lang/String;I)Lcom/intrinsec/drp4it/vixdisklib/VDLDiskHandle;+3

j  com.intrinsec.drp4it.vixdisklib.VDLConnection$2.evaluate()Lcom/intrinsec/drp4it/vixdisklib/VDLDiskHandle;+12

j  com.intrinsec.drp4it.vixdisklib.VDLConnection$2.evaluate()Ljava/lang/Object;+1

j  com.intrinsec.drp4it.vixdisklib.VDLExecutor$2.compute()Lcom/trazere/util/type/Either;+4

j  com.intrinsec.drp4it.vixdisklib.VDLExecutor$2.compute()Ljava/lang/Object;+1

J  com.trazere.util.closure.ResetableClosure.evaluate()Ljava/lang/Object;

j  com.intrinsec.drp4it.vixdisklib.VDLExecutor.run()V+51

j  com.intrinsec.drp4it.vixdisklib.VDLExecutor.access$000(Lcom/intrinsec/drp4it/vixdisklib/VDLExecutor;)V+1

j  com.intrinsec.drp4it.vixdisklib.VDLExecutor$1.run()V+4

v  ~StubRoutines::call_stub

---------------  P R O C E S S  ---------------

Java Threads: ( => current thread )

  0x07037800 JavaThread "Thread-811" [_thread_blocked, id=4760, stack(0x07c70000,0x07cc0000)]

  0x07007400 JavaThread "26003458@qtp-29787757-219" [_thread_blocked, id=664, stack(0x07830000,0x07880000)]

  0x02cf7800 JavaThread "Thread-809" [_thread_blocked, id=7388, stack(0x077e0000,0x07830000)]

  0x0705f400 JavaThread "Thread-808" [_thread_blocked, id=7028, stack(0x07790000,0x077e0000)]

  0x02d18400 JavaThread "Thread-807" [_thread_blocked, id=4852, stack(0x07740000,0x07790000)]

  0x06f28800 JavaThread "15083321@qtp-29787757-218" [_thread_blocked, id=5108, stack(0x035f0000,0x03640000)]

  0x02d1fc00 JavaThread "Thread-802" [_thread_in_native, id=4248, stack(0x076f0000,0x07740000)]

  0x03144400 JavaThread "15125489@qtp-29787757-215" [_thread_blocked, id=6632, stack(0x03660000,0x036b0000)]

  0x06f07c00 JavaThread "Thread-791" [_thread_in_native, id=5824, stack(0x07920000,0x07970000)]

  0x06f18400 JavaThread "Thread-789" [_thread_blocked, id=6752, stack(0x076a0000,0x076f0000)]

  0x06f67800 JavaThread "Thread-781" [_thread_in_native, id=5596, stack(0x03950000,0x039a0000)]

  0x06f39400 JavaThread "Keep-Alive-Timer" daemon [_thread_blocked, id=356, stack(0x034e0000,0x03530000)]

=>0x02f82800 JavaThread "Thread-12" daemon [_thread_in_native, id=6272, stack(0x036e0000,0x03730000)]

  0x03002c00 JavaThread "Timer-1" daemon [_thread_blocked, id=6952, stack(0x03580000,0x035d0000)]

  0x02fd9400 JavaThread "29529080@qtp-29787757-1 - Acceptor0 SelectChannelConnector@0.0.0.0:9000" [_thread_in_native, id=6124, stack(0x03530000,0x03580000)]

  0x02f42800 JavaThread "com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#2" daemon [_thread_blocked, id=6136, stack(0x03450000,0x034a0000)]

  0x02f50800 JavaThread "com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#1" daemon [_thread_blocked, id=3508, stack(0x03400000,0x03450000)]

  0x02f64000 JavaThread "com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#0" daemon [_thread_blocked, id=4412, stack(0x033b0000,0x03400000)]

  0x02f83c00 JavaThread "Timer-0" daemon [_thread_blocked, id=7520, stack(0x03360000,0x033b0000)]

  0x029a7c00 JavaThread "Low Memory Detector" daemon [_thread_blocked, id=6960, stack(0x02c20000,0x02c70000)]

  0x029a1c00 JavaThread "CompilerThread0" daemon [_thread_blocked, id=4392, stack(0x02bd0000,0x02c20000)]

  0x029a0000 JavaThread "Attach Listener" daemon [_thread_blocked, id=4672, stack(0x02b80000,0x02bd0000)]

  0x0299ec00 JavaThread "Signal Dispatcher" daemon [_thread_blocked, id=7872, stack(0x02b30000,0x02b80000)]

  0x0295d800 JavaThread "Finalizer" daemon [_thread_blocked, id=5860, stack(0x02ae0000,0x02b30000)]

  0x0295c000 JavaThread "Reference Handler" daemon [_thread_blocked, id=5368, stack(0x02a90000,0x02ae0000)]

  0x002a7400 JavaThread "main" [_thread_blocked, id=7040, stack(0x00640000,0x00690000)]

Other Threads:

  0x0295a800 VMThread [stack: 0x02a40000,0x02a90000] [id=7372]

  0x029b2800 WatcherThread [stack: 0x02c70000,0x02cc0000] [id=3336]

VM state:not at safepoint (normal execution)

VM Mutex/Monitor currently owned by a thread: None

Heap

def new generation   total 11520K, used 6115K [0x12990000, 0x13600000, 0x1d430000)

  eden space 10304K,  56% used [0x12990000, 0x12f38c00, 0x133a0000)

  from space 1216K,  26% used [0x134d0000, 0x13520310, 0x13600000)

  to   space 1216K,   0% used [0x133a0000, 0x133a0000, 0x134d0000)

tenured generation   total 25344K, used 17749K [0x1d430000, 0x1ecf0000, 0x32990000)

   the space 25344K,  70% used [0x1d430000, 0x1e5855a0, 0x1e585600, 0x1ecf0000)

compacting perm gen  total 21760K, used 21586K [0x32990000, 0x33ed0000, 0x36990000)

   the space 21760K,  99% used [0x32990000, 0x33ea4930, 0x33ea4a00, 0x33ed0000)

    ro space 10240K,  54% used [0x36990000, 0x36f0c0f0, 0x36f0c200, 0x37390000)

    rw space 12288K,  55% used [0x37390000, 0x37a2fb80, 0x37a2fc00, 0x37f90000)

Dynamic libraries:

0x00400000 - 0x00424000      C:\bin\jdk-1.6.0_21\jre\bin\java.exe

0x7c920000 - 0x7c9e8000      C:\WINDOWS\system32\ntdll.dll

0x7c800000 - 0x7c912000      C:\WINDOWS\system32\kernel32.dll

0x77ff0000 - 0x7809e000      C:\WINDOWS\system32\ADVAPI32.dll

0x77c20000 - 0x77cc0000      C:\WINDOWS\system32\RPCRT4.dll

0x76f00000 - 0x76f13000      C:\WINDOWS\system32\Secur32.dll

0x7c340000 - 0x7c396000      C:\bin\jdk-1.6.0_21\jre\bin\msvcr71.dll

0x6d8b0000 - 0x6db57000      C:\bin\jdk-1.6.0_21\jre\bin\client\jvm.dll

0x77f30000 - 0x77fc1000      C:\WINDOWS\system32\USER32.dll

0x77bd0000 - 0x77c19000      C:\WINDOWS\system32\GDI32.dll

0x76a50000 - 0x76a7f000      C:\WINDOWS\system32\WINMM.dll

0x76240000 - 0x7625d000      C:\WINDOWS\system32\IMM32.DLL

0x71a90000 - 0x71a98000      C:\WINDOWS\system32\rdpsnd.dll

0x77980000 - 0x77991000      C:\WINDOWS\system32\WINSTA.dll

0x77b70000 - 0x77bca000      C:\WINDOWS\system32\msvcrt.dll

0x71b10000 - 0x71b67000      C:\WINDOWS\system32\NETAPI32.dll

0x76b20000 - 0x76b2b000      C:\WINDOWS\system32\PSAPI.DLL

0x6d860000 - 0x6d86c000      C:\bin\jdk-1.6.0_21\jre\bin\verify.dll

0x6d3e0000 - 0x6d3ff000      C:\bin\jdk-1.6.0_21\jre\bin\java.dll

0x6d340000 - 0x6d348000      C:\bin\jdk-1.6.0_21\jre\bin\hpi.dll

0x6d8a0000 - 0x6d8af000      C:\bin\jdk-1.6.0_21\jre\bin\zip.dll

0x6d6c0000 - 0x6d6d3000      C:\bin\jdk-1.6.0_21\jre\bin\net.dll

0x71ad0000 - 0x71ae7000      C:\WINDOWS\system32\WS2_32.dll

0x71ac0000 - 0x71ac8000      C:\WINDOWS\system32\WS2HELP.dll

0x719f0000 - 0x71a32000      C:\WINDOWS\System32\mswsock.dll

0x76e80000 - 0x76eab000      C:\WINDOWS\system32\DNSAPI.dll

0x76f20000 - 0x76f27000      C:\WINDOWS\System32\winrnr.dll

0x76ec0000 - 0x76eef000      C:\WINDOWS\system32\WLDAP32.dll

0x76f30000 - 0x76f35000      C:\WINDOWS\system32\rasadhlp.dll

0x68000000 - 0x68035000      C:\WINDOWS\system32\rsaenh.dll

0x768d0000 - 0x76994000      C:\WINDOWS\system32\USERENV.dll

0x6d610000 - 0x6d619000      C:\bin\jdk-1.6.0_21\jre\bin\management.dll

0x5d3d0000 - 0x5d42b000      C:\WINDOWS\system32\hnetcfg.dll

0x719b0000 - 0x719b8000      C:\WINDOWS\System32\wshtcpip.dll

0x6d6e0000 - 0x6d6e9000      C:\bin\jdk-1.6.0_21\jre\bin\nio.dll

0x6d840000 - 0x6d848000      C:\bin\jdk-1.6.0_21\jre\bin\sunmscapi.dll

0x76160000 - 0x761f5000      C:\WINDOWS\system32\CRYPT32.dll

0x76140000 - 0x76152000      C:\WINDOWS\system32\MSASN1.dll

0x10000000 - 0x10007000      C:\Drp4it\2.5\server\lib\VixDiskLibJni.dll

0x03740000 - 0x03830000      C:\Program Files\VMware\VMware Virtual Disk Development Kit\bin\vixDiskLib.dll

0x78130000 - 0x781cb000      C:\WINDOWS\WinSxS\x86_Microsoft.VC80.CRT_1fc8b3b9a1e18e3b_8.0.50727.3053_x-ww_B80FA8CA\MSVCR80.dll

0x77480000 - 0x775b9000      C:\WINDOWS\system32\ole32.dll

0x78aa0000 - 0x78b5e000      C:\WINDOWS\system32\MSVCR100.dll

0x77d30000 - 0x77d51000      C:\WINDOWS\system32\NTMARTA.DLL

0x7e020000 - 0x7e02f000      C:\WINDOWS\system32\SAMLIB.dll

0x03bb0000 - 0x03d0c000      C:\Program Files\VMware\VMware Virtual Disk Development Kit\plugins\disklibplugin.dll

0x71a80000 - 0x71a8a000      C:\WINDOWS\system32\WSOCK32.dll

0x03d20000 - 0x03f42000      C:\Program Files\VMware\VMware Virtual Disk Development Kit\bin\vmomi.dll

0x03f60000 - 0x04295000      C:\Program Files\VMware\VMware Virtual Disk Development Kit\bin\vmacore.dll

0x775c0000 - 0x7764b000      C:\WINDOWS\system32\OLEAUT32.dll

0x7c9f0000 - 0x7d1fc000      C:\WINDOWS\system32\SHELL32.dll

0x77cd0000 - 0x77d22000      C:\WINDOWS\system32\SHLWAPI.dll

0x7c420000 - 0x7c4a7000      C:\WINDOWS\WinSxS\x86_Microsoft.VC80.CRT_1fc8b3b9a1e18e3b_8.0.50727.3053_x-ww_B80FA8CA\MSVCP80.dll

0x042b0000 - 0x05a67000      C:\Program Files\VMware\VMware Virtual Disk Development Kit\bin\types.dll

0x77e20000 - 0x77f23000      C:\WINDOWS\WinSxS\x86_Microsoft.Windows.Common-Controls_6595b64144ccf1df_6.0.3790.4770_x-ww_05FDF087\comctl32.dll

0x064a0000 - 0x0659a000      C:\Program Files\VMware\VMware Virtual Disk Development Kit\bin\libeay32.dll

0x061a0000 - 0x061d1000      C:\Program Files\VMware\VMware Virtual Disk Development Kit\bin\ssleay32.dll

0x061f0000 - 0x06235000      C:\Program Files\VMware\VMware Virtual Disk Development Kit\bin\vixDiskLibVim.dll

0x11100000 - 0x11412000      C:\Program Files\VMware\VMware Virtual Disk Development Kit\bin\gvmomi.dll

0x065a0000 - 0x0665e000      C:\Program Files\VMware\VMware Virtual Disk Development Kit\bin\glib-2.0.dll

0x06270000 - 0x06283000      C:\Program Files\VMware\VMware Virtual Disk Development Kit\bin\intl.dll

0x06670000 - 0x06749000      C:\Program Files\VMware\VMware Virtual Disk Development Kit\bin\iconv.dll

0x06750000 - 0x0677c000      C:\Program Files\VMware\VMware Virtual Disk Development Kit\bin\gobject-2.0.dll

0x06790000 - 0x0687e000      C:\Program Files\VMware\VMware Virtual Disk Development Kit\bin\libxml2.dll

0x06880000 - 0x06893000      C:\Program Files\VMware\VMware Virtual Disk Development Kit\bin\zlib1.dll

0x068b0000 - 0x06903000      C:\Program Files\VMware\VMware Virtual Disk Development Kit\bin\libcurl.dll

0x06920000 - 0x06964000      C:\Program Files\VMware\VMware Virtual Disk Development Kit\bin\libldap_r.dll

0x06980000 - 0x069a2000      C:\Program Files\VMware\VMware Virtual Disk Development Kit\bin\liblber.dll

0x77b60000 - 0x77b68000      C:\WINDOWS\system32\VERSION.dll

0x73c10000 - 0x73c22000      C:\WINDOWS\system32\cryptnet.dll

0x721d0000 - 0x721d5000      C:\WINDOWS\system32\SensApi.dll

0x74f40000 - 0x74f59000      C:\WINDOWS\system32\Cabinet.dll

0x76550000 - 0x7665d000      C:\WINDOWS\system32\setupapi.dll

0x76b60000 - 0x76b8c000      C:\WINDOWS\system32\WINTRUST.dll

0x76bc0000 - 0x76be8000      C:\WINDOWS\system32\imagehlp.dll

VM Arguments:

jvm_args: -Dfile.encoding=utf-8 -Djava.library.path=C:\Drp4it\2.5\server\lib -Djava.endorsed.dirs=C:\Drp4it\2.5\server\lib\endorsed -Xmx512m

java_command: com.intrinsec.drp4it.server.Server C:\Drp4it\2.5\server C:\Program Files\VMware\VMware Virtual Disk Development Kit

Launcher Type: SUN_STANDARD

Environment Variables:

JAVA_HOME=C:\bin\jdk-1.6.0_21

PATH=C:\Program Files\Windows Resource Kits\Tools\;C:\Program Files\Support Tools\;C:\WINDOWS\system32;C:\WINDOWS;C:\WINDOWS\System32\Wbem;C:\WINDOWS\system32\WindowsPowerShell\v1.0;C:\bin\jdk-1.6.0_21\bin;C:\bin\apache-ant-1.8.0\bin;C:\Program Files\VMware\VMware Virtual Disk Development Kit\bin;C:\bin\postgresql-9.0.3\bin

USERNAME=drp4it

OS=Windows_NT

PROCESSOR_IDENTIFIER=x86 Family 6 Model 44 Stepping 2, GenuineIntel

---------------  S Y S T E M  ---------------

OS: Windows Server 2003 family Build 3790 Service Pack 2

CPU:total 16 (16 cores per cpu, 2 threads per core) family 6 model 44 stepping 2, cmov, cx8, fxsr, mmx, sse, sse2, sse3, ssse3, sse4.1, sse4.2, popcnt, ht

Memory: 4k page, physical 16765360k(15196288k free), swap 18592880k(17285656k free)

vm_info: Java HotSpot(TM) Client VM (17.0-b17) for windows-x86 JRE (1.6.0_21-b07), built on Jul 17 2010 01:10:15 by "java_re" with MS VC++ 7.1 (VS2003)

time: Tue Oct 25 12:22:20 2011

elapsed time: 13381 seconds