VMware {code} Community
fixitchris
Hot Shot
Hot Shot

Vprobe-toolkit in Windows

#1 Enable Vprobes on Server 2.0 per http://communities.vmware.com/thread/193662, including CYGWIN install

#2 Install TortoiseSVN

#3 SVN Checkout...

#4 Install WinHugs

#5 Place winprobe script in your vprobe-toolkit\bin folder created in step #3

There are some user defined constants in the winprobe script. I would like it if somebody posted an update to the winprobe script as I am not familiar with bash scripting.

User defined winprobe constants:

COMPILER=/cygdrive/c/progra~1/winhugs/runhugs.exe
VMRUNCNX=" -T server -h https://localhost:8333/sdk -u user -p pass"
VMRUN=/cygdrive/c/progra~1/VMware/VMware~1/vmrun.exe$VMRUNCNX

out=$($COMPILER -98 c:\\vprobe-toolkit\\emmett\\main.lhs -c "$cmd" $compileargs $files) || die "compile error" 2
VMRUNOUT=$($VMRUN vprobeLoad "$VM" "$out" 2>&1)|| die "vmrun error:$VMRUNOUT" 3

winprobe execution takes three parameters to load a probe from a VP source file (Emmet script):

./winprobe [vpsourcefile] /path/to/vm /path/to/vm_directory

#6 Open CYGWIN shell and set environment variables to help with execution:

VM1="[standard] store\Windows Server 2003 Standard Edition.vmx"
VM1DIR="/cygdrive/d/virtua~1/store"

You can also set these variables permanently within the Windows System Environment Variables

#7 Load a probe:

hello.emt:

VMM10Hz {
   logstr("Hi\n");
}

./winprobe hello.emt "$VM1" "$VM1DIR"

You should see output every 100ms...

Reply
0 Kudos
11 Replies
admin
Immortal
Immortal

Chris,

Sorry, what exactly was the question? Which of the above steps are broken for you?

Keith

Reply
0 Kudos
fixitchris
Hot Shot
Hot Shot

Keith, I guess looking over my post I did not really have a question... The above works ok for loading Emmett Vprobes on Windows so far.

I do have a question though... What are we probing? The hypervisor? The VM? Why not just debug the VM from the inside? What is the theory behind this magical madness?

PS. Post more videos!

Reply
0 Kudos
admin
Immortal
Immortal

Doing this stuff from outside the VM has some big pluses, and some big minuses. On the pro side:

  1. Most importantly, VProbes is safe. Using a kernel debugger or something from within a guest provides some very dangerous opportunities: e.g., you could poke a hole in a critical guest memory structure. Also, kernel debuggers usually stop the target program to allow inspection. VProbes keeps things moving along while data is harvested, so it can be used even on mission-critical, highly available systems.

  2. VProbes doesn't rely on any in-guest tools. If the target is, for instance, an appliance that doesn't even have a login shell, let alone /usr/bin/top, VProbes works no better or worse.

  3. It doesn't care what state the guest is in. If the guest is hard hung, or is swapping too hard for a shell to come up, or whatever, VProbes can be a "tool of last resort" to try to harvest information.

  4. VProbes is also immune to guest attempts to fool it. For instance, malware often tries to change in-guest instrumentation tools to make itself undetectable. Since the VM/hypervisor trust boundary protects vprobes, it is immune to this weirdness.

  5. VProbes is neutral as to the identity of the guest; since it's machine-level, you can use the same tools for Windows, Linux, Solaris, FreeBSD, MacOS, etc. To the extent the preload scripts work, even some common, but OS-specific, notions, like processes, can be captured.

The big negatives all boil down to the same fundamental problem: the machine abstraction level is sometimes very far away from the application abstraction level. If it's a java application running, for instance, VProbes will sit there telling you: "java.exe is running!" Well, no kidding; you probably wanted to know, e.g., which method invocation it was in. We're thinking about ways to bridge that gap, but alas, have very little to offer for serious discussion at this time.

fixitchris
Hot Shot
Hot Shot

What documentation do you recommend to become a Vprobe expert?

besides:

Reply
0 Kudos
admin
Immortal
Immortal

The processor manuals, of course, are for reference, not for reading cover-to-cover, though perhaps the protected mode memory management chapters of Intel vol. 3 are worth a thorough read.

I think books about concrete operating systems are more useful than the academic texts, which tend to spend too much real estate on ideas that turned out to be dead ends. Solomon and Russinovich's Inside Windows 200X series are excellent, providing broad, deep, well-written insight into the most commercially important operating system out there. I haven't seen anything quite as great in the Linux world, but Love's Linux Kernel Development is just fine.

fixitchris
Hot Shot
Hot Shot

I'm getting a bunch of these in the log file running vprobes per my Windows implementation...

Mar 04 16:25:29.339: vcpu-0| VirtualProtect Host_MemProtect(0x1a40000,0x4000) returns 0x57
Mar 04 16:25:29.339: vcpu-0| VProbe: Loading VP script
Mar 04 16:25:29.354: vcpu-0| VProbes VP version 0.2
Mar 04 16:25:29.354: vcpu-0| VirtualProtect Host_MemProtect(0x1a40000,0x4000) returns 0x57
Mar 04 16:25:29.370: vmx| SOCKET 24 (1312) recv error 10054: An existing connection was forcibly closed by the remote host

Reply
0 Kudos
nitingupta
Contributor
Contributor

I just got this same error when running: vmrun vprobeLoad Fedora10.vmx '(vprobe VMM1Hz (printf "hworld!\n"))'

on Vista 32-bit host.

vmware.log shows:

Mar 20 06:11:46.117: vcpu-0| VirtualProtect Host_MemProtect(0x2580000,0x4000) returns 0x57

Mar 20 06:11:46.117: vcpu-0| VProbe: Loading VP script

Mar 20 06:11:46.170: vcpu-0| VirtualProtect Host_MemProtect(0x2580000,0x4000) returns 0x57

Mar 20 06:11:46.260: vmx| SOCKET 3 (1240) recv error 10054: An existing connection was forcibly closed by the remote host

I see that VirtualProtect() is used to set protection flags for a region of memory:

http://msdn.microsoft.com/en-us/library/aa366898(VS.85).aspx

This function is returning 0x57 which according to:

http://msdn.microsoft.com/en-us/library/ms681382(VS.85).aspx

means: ERROR_INVALID_PARAMETER 87 (0x57) - The parameter is incorrect.

hmm...seems like vprobes is using some combination of protection flags which is not legal -- one such possibility is trying to set ReadWriteExec on hardware with NX (no-execute) feature enabled.

Try disable NX feature in BIOS (Intel calls this "XD"). I will post result if I try this Smiley Happy

Reply
0 Kudos
fixitchris
Hot Shot
Hot Shot

Most excellent find.

Reply
0 Kudos
nitingupta
Contributor
Contributor

Did it work? I disabled NX on my system and still getting same error. Maybe its due to some other problem??

Reply
0 Kudos
fixitchris
Hot Shot
Hot Shot

No change here. I'm not sure what the purpose of this call is as it pertains to vprobes... Maybe it's just an exception when the memory was paged out???

Reply
0 Kudos
nitingupta
Contributor
Contributor

looks like this error is harmless (though I'm still not sure what it means).

Sample vp script:

vmrun vprobeLoad Fedora10.vmx '(vprobe VMM1Hz (printf "hworld!\n"))'

keeps printing this message in vprobe.out (in VM directory) until you issue:

vmrun vprobeReset Fedora10.vmx

Reply
0 Kudos