11 Replies Latest reply on Mar 26, 2009 6:01 PM by nitingupta

    Vprobe-toolkit in Windows

    fixitchris Expert

      #1 Enable Vprobes on Server 2.0 per http://communities.vmware.com/thread/193662, including CYGWIN install

      #2 Install TortoiseSVN http://tortoisesvn.tigris.org/

      #3 SVN Checkout... https://vprobe-toolkit.svn.sourceforge.net/svnroot/vprobe-toolkit

      #4 Install WinHugs http://www-users.cs.york.ac.uk/~ndm/downloads/

      #5 Place winprobe script in your vprobe-toolkit\bin folder created in step #3

       

      There are some user defined constants in the winprobe script.  I would like it if somebody posted an update to the winprobe script as I am not familiar with bash scripting.

       

      User defined winprobe constants:

       

      COMPILER=/cygdrive/c/progra~1/winhugs/runhugs.exe
      VMRUNCNX=" -T server -h https://localhost:8333/sdk -u user -p pass"
      VMRUN=/cygdrive/c/progra~1/VMware/VMware~1/vmrun.exe$VMRUNCNX
      
      out=$($COMPILER -98 c:\\vprobe-toolkit\\emmett\\main.lhs -c "$cmd" $compileargs $files) || die "compile error" 2
      VMRUNOUT=$($VMRUN vprobeLoad "$VM" "$out" 2>&1)|| die "vmrun error:$VMRUNOUT" 3
      

       

      winprobe execution takes three parameters to load a probe from a VP source file (Emmet script):

       

      ./winprobe [vpsourcefile] /path/to/vm /path/to/vm_directory
      

       

      #6 Open CYGWIN shell and set environment variables to help with execution:

       

      VM1="[standard] store\Windows Server 2003 Standard Edition.vmx"
      VM1DIR="/cygdrive/d/virtua~1/store"
      

       

      You can also set these variables permanently within the Windows System Environment Variables

       

      #7 Load a probe:

      hello.emt:

       

      VMM10Hz {
         logstr("Hi\n");
      }
      

       

      ./winprobe hello.emt "$VM1" "$VM1DIR"
      

       

       

      You should see output every 100ms...

        • 1. Re: Vprobe-toolkit in Windows
          Novice

          Chris,

           

          Sorry, what exactly was the question? Which of the above steps are broken for you?

           

           

           

          Keith

           

           

          • 2. Re: Vprobe-toolkit in Windows
            fixitchris Expert

             

            Keith, I guess looking over my post I did not really have a question... The above works ok for loading Emmett Vprobes on Windows so far.

             

             

            I do have a question though... What are we probing?  The hypervisor?  The VM?  Why not just debug the VM from the inside?  What is the theory behind this magical madness?

             

             

             

             

             

            PS. Post more videos!

             

             

            • 3. Re: Vprobe-toolkit in Windows
              Novice

               

              Doing this stuff from outside the VM has some big pluses, and some big minuses. On the pro side:

               

              1. Most importantly, VProbes is safe. Using a kernel debugger or something from within a guest provides some very dangerous opportunities: e.g., you could poke a hole in a critical guest memory structure. Also, kernel debuggers usually stop the target program to allow inspection. VProbes keeps things moving along while data is harvested, so it can be used even on mission-critical, highly available systems.

              2. VProbes doesn't rely on any in-guest tools. If the target is, for instance, an appliance that doesn't even have a login shell, let alone /usr/bin/top, VProbes works no better or worse.

              3. It doesn't care what state the guest is in. If the guest is hard hung, or is swapping too hard for a shell to come up, or whatever, VProbes can be a "tool of last resort" to try to harvest information.

              4. VProbes is also immune to guest attempts to fool it. For instance, malware often tries to change in-guest instrumentation tools to make itself undetectable. Since the VM/hypervisor trust boundary protects vprobes, it is immune to this weirdness.

              5. VProbes is neutral as to the identity of the guest; since it's machine-level, you can use the same tools for Windows, Linux, Solaris, FreeBSD, MacOS, etc. To the extent the preload scripts work, even some common, but OS-specific, notions, like processes, can be captured.

               

              The big negatives all boil down to the same fundamental problem: the machine abstraction level is sometimes very far away from the application abstraction level. If it's a java application running, for instance, VProbes will sit there telling you: "java.exe is running!" Well, no kidding; you probably wanted to know, e.g., which method invocation it was in. We're thinking about ways to bridge that gap, but alas, have very little to offer for serious discussion at this time.

               

               

               

               

               

              1 person found this helpful
              • 5. Re: Vprobe-toolkit in Windows
                Novice

                 

                The processor manuals, of course, are for reference, not for reading cover-to-cover, though perhaps the protected mode memory management chapters of Intel vol. 3 are worth a thorough read.

                 

                 

                I think books about concrete operating systems are more useful than the academic texts, which tend to spend too much real estate on ideas that turned out to be dead ends. Solomon and Russinovich's Inside Windows 200X series are excellent, providing broad, deep, well-written insight into the most commercially important operating system out there. I haven't seen anything quite as great in the Linux world, but Love's Linux Kernel Development is just fine.

                 

                 

                1 person found this helpful
                • 6. Re: Vprobe-toolkit in Windows
                  fixitchris Expert

                  I'm getting a bunch of these in the log file running vprobes per my Windows implementation...

                   

                  Mar 04 16:25:29.339: vcpu-0| VirtualProtect Host_MemProtect(0x1a40000,0x4000) returns 0x57
                  Mar 04 16:25:29.339: vcpu-0| VProbe: Loading VP script
                  Mar 04 16:25:29.354: vcpu-0| VProbes VP version 0.2
                  Mar 04 16:25:29.354: vcpu-0| VirtualProtect Host_MemProtect(0x1a40000,0x4000) returns 0x57
                  Mar 04 16:25:29.370: vmx| SOCKET 24 (1312) recv error 10054: An existing connection was forcibly closed by the remote host
                  

                   

                  • 7. Re: Vprobe-toolkit in Windows
                    nitingupta Novice

                    I just got this same error when running: vmrun vprobeLoad Fedora10.vmx '(vprobe VMM1Hz (printf "hworld!\n"))'

                    on Vista 32-bit host.

                     

                    vmware.log shows:

                        Mar 20 06:11:46.117: vcpu-0| VirtualProtect Host_MemProtect(0x2580000,0x4000) returns 0x57

                        Mar 20 06:11:46.117: vcpu-0| VProbe: Loading VP script

                        Mar 20 06:11:46.170: vcpu-0| VirtualProtect Host_MemProtect(0x2580000,0x4000) returns 0x57

                        Mar 20 06:11:46.260: vmx| SOCKET 3 (1240) recv error 10054: An existing connection was forcibly closed by the remote host

                     

                    I see that VirtualProtect() is used to set protection flags for a region of memory:

                    http://msdn.microsoft.com/en-us/library/aa366898(VS.85).aspx

                     

                    This function is returning 0x57 which according to:

                    http://msdn.microsoft.com/en-us/library/ms681382(VS.85).aspx

                    means: ERROR_INVALID_PARAMETER 87 (0x57) - The parameter is incorrect.

                     

                    hmm...seems like vprobes is using some combination of protection flags which is not legal -- one such possibility is trying to set ReadWriteExec on hardware with NX (no-execute) feature enabled.

                     

                    Try disable NX feature in BIOS (Intel calls this "XD"). I will post result if I try this

                    • 8. Re: Vprobe-toolkit in Windows
                      fixitchris Expert

                      Most excellent find.

                      • 9. Re: Vprobe-toolkit in Windows
                        nitingupta Novice

                        Did it work? I disabled NX on my system and still getting same error. Maybe its due to some other problem??

                        • 10. Re: Vprobe-toolkit in Windows
                          fixitchris Expert

                          No change here.  I'm not sure what the purpose of this call is as it pertains to vprobes... Maybe it's just an exception when the memory was paged out???

                          • 11. Re: Vprobe-toolkit in Windows
                            nitingupta Novice

                            looks like this error is harmless (though I'm still not sure what it means).

                            Sample vp script:

                            vmrun vprobeLoad Fedora10.vmx '(vprobe VMM1Hz (printf "hworld!\n"))'

                            keeps printing this message in vprobe.out (in VM directory) until you issue:

                            vmrun vprobeReset Fedora10.vmx