Using WinDbg over a named pipe between two VMs (running on an ESX)

Version 3
Visibility: Open to anyone

    There are a lot of blog and forum posts about how to use WinDbg to live debug a VM, most seem to be aimed at users of VMware Workstation, with the intent to use the Debugging Tools for Windows installed on the host operation system, against a Windows VM running on it. If you run a  Windows VM on a Windows host OS, you would create a serial port on that VM and map it to a named pipe, which is accessible by the host operating system, WinDbg could then connect to that named pipe. If you want to  simulate a nullmodem cable between two VMs running on an ESX host, you can still use named pipes that connect virtual serial ports, but it is completely transparent to the guest operating systems, which will only  see the serial ports. It is quite simple, but the myriad of tutorials  make it seem more complicated than it actually is. So here is a simple  step by step guide for the scenario described in the title.

     

    Debuggee = the VM that I want to debug

    Debugger = the VM that has WinDbg installed

     

    I'm  using ESX 4.1 GA (build 260247) and Windows 2008 R2 (build 7600), the  biggest difference is in how to configure Windows for live debugging for  NT5 vs NT6, see "Boot Parameters to Enable Debugging" for your specific of Windows version, this guide will use a default installation of W2K8 R2.

     

    Lets add and configure the serial port for our VM that is about to be debugged, AKA the "Debuggee":

    01-DE-edit.jpg

    Rightclick and "Edit Settings".

    02-DE-add.jpg

    Click "Add" under the "Hardware" tab.

    03-DE-serial.jpg

    Select "Serial Port" and click on "Next".

    04-DE-pipe.jpg

    On the next page, select "Connect to named pipe".

    05-DE-settings.jpg

    Choose any "Pipe Name" you like and remember it.

    Make sure that "Near end:" is set to "Server" and "Far end:" to "A virtual machine".

    The checkboxes for "Connect at power on" and "Yield CPU on poll" should also be checked.

    06-DE-done.jpg

    Verify your settings and power on the VM.

    07-DE-bcdedit.jpg

    Configure your Windows OS for accepting connections from a Debugger, in my case "bcdedit /set debug on".

    I also ran "bcdedit /dbgsettings" to verify the settings, refer to the msdn article "Boot Parameters to Enable Debugging" for further information.

     

    Now we will configure the VM that has WinDbg (and Symbols etc.) installed, AKA the "Debugger":

    08-DR-edit.jpg

    Rightclick and "Edit Settings".

    09-DR-settings.jpg

    The steps to set up the serial port are the same, the only difference is that you have to select "Client" for the "Near end:".

    10-DR-done.jpg

    Verify your settings and power on the VM.

    11-DR-debug.jpg

    Start WinDbg, and select "File -> Kernel Debug"

    12-DR-com.jpg

    The default settings for COM should be correct, as if you were using a physical nullmodem cable.

    13-DR-connect.jpg

    You should see that WinDbg is not connected to the Debuggee yet.

    14-DR-break.jpg

    Select "Debug -> Break" to break into the Debuggee.

    15-DR-finished.jpg

    Done! You should find your way from here.

     

    Cheers

     

    Valentin