4 Replies Latest reply: May 20, 2015 5:19 AM by timwiffen RSS

    EXCEPTION_ACCESS_VIOLATION

    Steve1980 Lurker

      Hi

       

      I've been using Sigar in my Java app for months without any issue and for some reason within the last 2 weeks JVM has started crashing intermittently.

       

      Here's the error that I get:

       

      #
      # A fatal error has been detected by the Java Runtime Environment:
      #
      #  EXCEPTION_ACCESS_VIOLATION (0xc0000005) at pc=0x000000004e5d4ed4, pid=6896, t
      id=12984
      #
      # JRE version: 6.0_30-b12
      # Java VM: Java HotSpot(TM) 64-Bit Server VM (20.5-b03 mixed mode windows-amd64
      compressed oops)
      # Problematic frame:
      # C  [sigar-amd64-winnt.dll+0x14ed4]
      #
      # An error report file with more information is saved as:
      # C:\bot\hs_err_pid6896.log
      #
      # If you would like to submit a bug report, please visit:
      # The crash happened outside the Java Virtual Machine in native code.
      # See problematic frame for where to report the bug.
      #

       

       

      The operating system is Windows Server 2008 R2, I've updated Java and the operating system, but I still get the random crashes. I haven't made any major changes to the application within the last few weeks.

       

      Does anyone have any idea what might be causing the crash?

       

      Thanks

        • 1. Re: EXCEPTION_ACCESS_VIOLATION
          vishal_attivio Novice

          Hey, I'm seeing the same crash it seems in the same DLL!

           

          Look at hexadecimal offset - 0x14ed4.  It's at the very same spot as yours.

           

          #
          # A fatal error has been detected by the Java Runtime Environment:
          #
          #  EXCEPTION_ACCESS_VIOLATION (0xc0000005) at pc=0x000000000cd84ed4, pid=7904, tid=9104
          #
          # JRE version: 7.0_07-b10
          # Java VM: Java HotSpot(TM) 64-Bit Server VM (23.3-b01 mixed mode windows-amd64 compressed oops)
          # Problematic frame:
          # C  [sigar-amd64-winnt.dll+0x14ed4]  JNI_OnUnload+0xc4
          #
          # Failed to write core dump. Minidumps are not enabled by default on client versions of Windows
          #
          # If you would like to submit a bug report, please visit:
          #   http://bugreport.sun.com/bugreport/crash.jsp
          # The crash happened outside the Java Virtual Machine in native code.
          # See problematic frame for where to report the bug.
          #

           

           

          The crash we are seeing is pretty intermittent too.  So far, I haven't been able to reproduce it in an independent test harness.  Only as part of a heavyweight app.

           

          I see you have posted this almost 7 months ago.  Have you made any progress?

           

          Also, which version of SIGAR are you using?

           

          Thanks,

          Vishal

          • 2. Re: EXCEPTION_ACCESS_VIOLATION
            applepieinhd Lurker

            I got similar JVM crash in windows server 2008 & sigar-1.6.4. Here's the crash report:

             

            #
            # A fatal error has been detected by the Java Runtime Environment:
            #
            #  EXCEPTION_ACCESS_VIOLATION (0xc0000005) at pc=0x0000000010005cc4, pid=3292, tid=3600
            #
            # JRE version: 6.0_35-b10
            # Java VM: Java HotSpot(TM) 64-Bit Server VM (20.10-b01 mixed mode windows-amd64 compressed oops)
            # Problematic frame:
            # C  [sigar-amd64-winnt.dll+0x5cc4]
            #
            # If you would like to submit a bug report, please visit:
            # The crash happened outside the Java Virtual Machine in native code.
            # See problematic frame for where to report the bug.
            #
            ---------------  T H R E A D  ---------------
            Current thread (0x00000000060c1800):  JavaThread "ThreadPoolTaskExecutor-9" [_thread_in_native, id=3600, stack(0x000000000b060000,0x000000000b160000)]
            siginfo: ExceptionCode=0xc0000005, reading address 0x000000000000001c
            Registers:
            RAX=0x0000000000000000, RBX=0x00000000f21b5ac8, RCX=0x000007feff103120, RDX=0x000000000067b3b0
            RSP=0x000000000b15e900, RBP=0x000000000b15efc0, RSI=0x00000000f21b6dd0, RDI=0x0000000000000028
            R8 =0x00002e3dffff0000, R9 =0x00002e3e000e0001, R10=0x0000000000000000, R11=0x000007feff1031d0
            R12=0x0000000000000000, R13=0x00000000f21b5ac0, R14=0x000000000b15eff0, R15=0x00000000060c1800
            RIP=0x0000000010005cc4, EFLAGS=0x0000000000010246
            Top of Stack: (sp=0x000000000b15e900)
            0x000000000b15e900:   000000000a3922f0 0000000000000000
            0x000000000b15e910:   00000000f21b5ac8 000000007720a3fa
            0x000000000b15e920:   000000000b2968b0 000000000b15e938
            0x000000000b15e930:   0000000000000003 0000000000000ea8
            0x000000000b15e940:   000000000b15e950 0000000000000000
            0x000000000b15e950:   0000000200000000 0000000000000000
            0x000000000b15e960:   0000000000080006 000007fffff4d268
            0x000000000b15e970:   0000000000000008 0000000000000000
            0x000000000b15e980:   0000000000000000 00000000000a0008
            0x000000000b15e990:   00000000772b33e0 0000000000000820
            0x000000000b15e9a0:   0000005c003a0043 0000000000000000
            0x000000000b15e9b0:   0000000000000000 0000000000000000
            0x000000000b15e9c0:   0000000000000000 0000000000000000
            0x000000000b15e9d0:   0000000000000000 0000000000000000
            0x000000000b15e9e0:   0000000000000000 0000000000000000
            0x000000000b15e9f0:   0000000000000000 0000000000000000
            Instructions: (pc=0x0000000010005cc4)
            0x0000000010005ca4:   00 00 e9 13 ff ff ff 48 8b 84 24 90 02 00 00 48
            0x0000000010005cb4:   8b 80 a0 01 00 00 48 89 44 24 48 48 8b 44 24 48
            0x0000000010005cc4:   83 78 1c 00 75 46 48 8b 8c 24 98 02 00 00 e8 b9
            0x0000000010005cd4:   01 00 00 48 89 84 24 98 02 00 00 4c 8b 84 24 98
            Register to memory mapping:
            RAX=0x0000000000000000 is an unknown value
            RBX=0x00000000f21b5ac8 is an oop
            {method}
            - klass: {other class}
            RCX=0x000007feff103120 is an unknown value
            RDX=0x000000000067b3b0 is an unknown value
            RSP=0x000000000b15e900 is pointing into the stack for thread: 0x00000000060c1800
            RBP=0x000000000b15efc0 is pointing into the stack for thread: 0x00000000060c1800
            RSI=0x00000000f21b6dd0 is an oop
            {instance class}
            - klass: {other class}
            RDI=0x0000000000000028 is an unknown value
            R8 =0x00002e3dffff0000 is an unknown value
            R9 =0x00002e3e000e0001 is an unknown value
            R10=0x0000000000000000 is an unknown value
            R11=0x000007feff1031d0 is an unknown value
            R12=0x0000000000000000 is an unknown value
            R13=0x00000000f21b5ac0 is an oop
            {constMethod}
            - klass: {other class}
            - method:       0x00000000f21b5ac8 {method} 'gather' '(Lorg/hyperic/sigar/Sigar;Ljava/lang/String;)V' in 'org/hyperic/sigar/FileSystemUsage'
            - exceptions:   0x00000000f0001ef8
            R14=0x000000000b15eff0 is pointing into the stack for thread: 0x00000000060c1800
            R15=0x00000000060c1800 is a thread
            Stack: [0x000000000b060000,0x000000000b160000],  sp=0x000000000b15e900,  free space=1018k
            Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
            C  [sigar-amd64-winnt.dll+0x5cc4]  sigar_mem_get+0x4a4
            j  org.hyperic.sigar.FileSystemUsage.fetch(Lorg/hyperic/sigar/Sigar;Ljava/lang/String;)Lorg/hyperic/sigar/FileSystemUsage;+11
            j  org.hyperic.sigar.Sigar.getMountedFileSystemUsage(Ljava/lang/String;)Lorg/hyperic/sigar/FileSystemUsage;+66
            j  sun.reflect.GeneratedMethodAccessor78.invoke(Ljava/lang/Object;[Ljava/lang/Object;)Ljava/lang/Object;+40
            j  sun.reflect.DelegatingMethodAccessorImpl.invoke(Ljava/lang/Object;[Ljava/lang/Object;)Ljava/lang/Object;+6
            j  java.lang.reflect.Method.invoke(Ljava/lang/Object;[Ljava/lang/Object;)Ljava/lang/Object;+161
            j  org.hyperic.sigar.SigarProxyCache.invoke(Ljava/lang/Object;Ljava/lang/reflect/Method;[Ljava/lang/Object;)Ljava/lang/Object;+363
            j  org.hyperic.sigar.SigarInvoker.invoke([Ljava/lang/Object;Ljava/lang/String;)Ljava/lang/Object;+25
            j  org.hyperic.sigar.SigarInvoker.invoke(Ljava/lang/Object;Ljava/lang/String;)Ljava/lang/Object;+18
            j  org.hyperic.sigar.jmx.SigarInvokerJMX.invoke(Ljava/lang/String;)Ljava/lang/Object;+6

            ...

            ...

            VM Arguments:

            jvm_args: -Xms512M -Xmx1024M -Xmn100M -XX:PermSize=128M -XX:MaxPermSize=256M -Xcheck:jni -Dosgi.install.area=equinox

            java_command: org.eclipse.core.runtime.adaptor.EclipseStarter -configuration equinox

            Launcher Type: SUN_STANDARD

             

            Environment Variables:

            JAVA_HOME=C:\Program Files\Java\jre6

            PATH=C:\Program Files\Java\jre6\bin:C:\Program Files (x86)\Common Files\NetSarang;C:\Windows\system32;C:\Windows;C:\Windows\System32\Wbem;C:\condor\bin\;c:\Program Files (x86)\Microsoft SQL Server\90\Tools\binn\;c:\app\jre\bin

            USERNAME=GV245$

            OS=Windows_NT

            PROCESSOR_IDENTIFIER=Intel64 Family 6 Model 44 Stepping 2, GenuineIntel

             

             

             

            ---------------  S Y S T E M  ---------------

             

            OS: Windows Server 2008 , 64 bit Build 6001 Service Pack 1

             

            CPU:total 2 (32 cores per cpu, 2 threads per core) family 6 model 44 stepping 2, cmov, cx8, fxsr, mmx, sse, sse2, sse3, ssse3, sse4.1, sse4.2, popcnt, ht

             

            Memory: 4k page, physical 1047332k(231664k free), swap 2379660k(985228k free)

             

            vm_info: Java HotSpot(TM) 64-Bit Server VM (20.10-b01) for windows-amd64 JRE (1.6.0_35-b10), built on Aug 28 2012 16:50:10 by "java_re" with MS VC++ 8.0 (VS2005)

             

             

             

            Also, with -Xcheck:jni I got such error in log:

             

            FATAL ERROR in native method: Static field ID passed to JNI

            at org.hyperic.sigar.FileSystemUsage.gather(Native Method)

            at org.hyperic.sigar.FileSystemUsage.fetch(FileSystemUsage.java:30)

            at org.hyperic.sigar.Sigar.getMountedFileSystemUsage(Sigar.java:712)

            at sun.reflect.GeneratedMethodAccessor78.invoke(Unknown Source)

            at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)

            at java.lang.reflect.Method.invoke(Unknown Source)

            at org.hyperic.sigar.SigarProxyCache.invoke(SigarProxyCache.java:193)

            - locked <0x00000000b5001ad8> (a org.hyperic.sigar.SigarProxyCache)

            • 3. Re: EXCEPTION_ACCESS_VIOLATION
              vishal_attivio Novice

              Hey everyone,

               

              I believe I have found the cause of my crash on Windows x64 and have solved it.  This pertains to SIGAR 1.6.4.

               

               

              ROOT CAUSE

               

               

              The root cause is the following sequence of operations, code for which you can find in <SRCROOT>bindings\java\src\jni\javasigar.c :

               

              1. Storing a 64-bit wide pointer in a 32-bit wide location:

               

              static void sigar_set_pointer(JNIEnv *env, jobject obj, const void *ptr) {
                  jfieldID pointer_field;
                  jclass cls = JENV->GetObjectClass(env, obj);

              #ifdef SIGAR_POINTER_LONG
                  pointer_field = JENV->GetFieldID(env, cls, "longSigarWrapper", "J");
                  JENV->SetLongField(env, obj, pointer_field, (jlong)ptr);
              #else
                  pointer_field = JENV->GetFieldID(env, cls, "sigarWrapper", "I");
                  JENV->SetIntField(env, obj, pointer_field, (int)ptr);

              #endif

              }

               

              This is already dangerous because of potential truncation , but it gets worse!

               

               

              2. Then retrieving that pointer back from the 32-bit wide location into a 64-bit pointer:

               

              static void *sigar_get_pointer(JNIEnv *env, jobject obj) {
                  jfieldID pointer_field;
                  jclass cls = JENV->GetObjectClass(env, obj);

              #ifdef SIGAR_POINTER_LONG
                  pointer_field = JENV->GetFieldID(env, cls, "longSigarWrapper", "J");
                  return (void *)JENV->GetLongField(env, obj, pointer_field);
              #else
                  pointer_field = JENV->GetFieldID(env, cls, "sigarWrapper", "I");
                  return (void *)JENV->GetIntField(env, obj, pointer_field);

              #endif


              }

               

              When you do a widening conversion like this, sometimes you can get sign extension in the high order bit, so a pointer that was

               

              00000000A7901770   (8 bytes, or 64 bits)

              gets set as

              A7901770   (4 bytes, or 32 bits)

              and then retrieved as

              FFFFFFFFA7901770  (8 bytes, or 64 bits)

              because the high order nibble in the 32-bit value is an 'A' which means '1010' in binary
              and that causes sign extension.

              3. Then dereferencing the crazy pointer address created by the sign extending widening conversion:

               

               

              static jni_sigar_t *sigar_get_jpointer(JNIEnv *env, jobject obj) {
                  jni_sigar_t *jsigar =
                      (jni_sigar_t *)sigar_get_pointer(env, obj);

                  if (!jsigar) {
                      sigar_throw_exception(env, "sigar has been closed");
                      return NULL;
                  }

                  if (jsigar->open_status != SIGAR_OK) {
                      sigar_throw_error(env, jsigar,
                                        jsigar->open_status);
                      return NULL;
                  }

                  return jsigar;
              }

               

               

               

              Note that the intermittent nature of this crash can be explained by the stored pointer not always having its high order bit set and therefore not always being subject to sign extension.

               

               


              OUR SOLUTION

               

               

              We don't build on 32-bit platforms, so all I did to fix this was to take out the code that stored and retrieved the code from the int.

               

              So sigar_set_pointer looks like this:

               

               

               

              static void sigar_set_pointer(JNIEnv *env, jobject obj, const void *ptr) {
                  jfieldID pointer_field;
                  jclass cls = JENV->GetObjectClass(env, obj);

                  pointer_field = JENV->GetFieldID(env, cls, "longSigarWrapper", "J");
                  JENV->SetLongField(env, obj, pointer_field, (jlong)ptr);
              }

               

               

              And sigar_get_pointer looks like this:

               

              static void *sigar_get_pointer(JNIEnv *env, jobject obj) {
                  jfieldID pointer_field;
                  jclass cls = JENV->GetObjectClass(env, obj);

                  pointer_field = JENV->GetFieldID(env, cls, "longSigarWrapper", "J");
                  return (void *)JENV->GetLongField(env, obj, pointer_field);

              }

               

               

               

              Note that this a safe solution despite the signed / unsigned conversion because in those situations where either operand is unsigned,

              both are converted to unsigned.  Please refer to the discussion here:

              http://stackoverflow.com/questions/5800946/unsigned-to-signed-conversion

               

               

               


              MAYBE A BETTER SOLUTION FOR PORTABILITY

               

               


              For a more sustainable submission to the codebase that is more portable across more platforms, I was thinking something could be done at the top of

              SRCROOT\include\sigar.h

               

              where the following

               

              #if defined(_LP64)         || \
                   defined(__LP64__)      || \
                   defined(__64BIT__)     || \
                   defined(__powerpc64__) || \
                   defined(__osf__)
              #define SIGAR_64BIT
              #endif

               

              could be changed to

               

              #if defined(_LP64)         || \

                  defined(__LP64__)      || \

                  defined(__64BIT__)     || \

                  defined(__powerpc64__) || \

                  defined(_M_AMD64)      || \

                  defined(_M_IA64)       || \

                  defined(_M_X64)        || \

                  defined(__osf__)

              #define SIGAR_64BIT

              #endif

               

               

               

              The source of these macros, being predefined in the Microsoft VC compilers:

               

              http://msdn.microsoft.com/en-us/library/b0084kay.aspx

               

               

              Thoughts?  If anyone wants to contact me to discuss about how to get this (or a better solution) into a future release of SIGAR,

              my e-mail is vrao at my company's domain:

               

              http://attivio.com

               

              Thanks,

              Vishal Rao

              • 4. Re: EXCEPTION_ACCESS_VIOLATION
                timwiffen Lurker

                We have made the proposed changes and a few other attempts to fix the problem in the fork formicary/sigar at sigar-1.6-formicary_fixes · GitHub. However we still have this issue on Windows x64.

                 

                We don't get errors every time but it appears almost any method call to SIGAR is capable of causing an error and they always happen eventually.

                 

                Is anyone managing to use SIGAR on Windows x64?