Using WinDbg over a named pipe between two VMs (running on an ESX)

Version 3
Visibility: Open to anyone

    There are a lot of blog and forum posts about how to use WinDbg to live debug a VM, most seem to be aimed at users of VMware Workstation, with the intent to use the Debugging Tools for Windows installed on the host operation system, against a Windows VM running on it. If you run a  Windows VM on a Windows host OS, you would create a serial port on that VM and map it to a named pipe, which is accessible by the host operating system, WinDbg could then connect to that named pipe. If you want to  simulate a nullmodem cable between two VMs running on an ESX host, you can still use named pipes that connect virtual serial ports, but it is completely transparent to the guest operating systems, which will only  see the serial ports. It is quite simple, but the myriad of tutorials  make it seem more complicated than it actually is. So here is a simple  step by step guide for the scenario described in the title.


    Debuggee = the VM that I want to debug

    Debugger = the VM that has WinDbg installed


    I'm  using ESX 4.1 GA (build 260247) and Windows 2008 R2 (build 7600), the  biggest difference is in how to configure Windows for live debugging for  NT5 vs NT6, see "Boot Parameters to Enable Debugging" for your specific of Windows version, this guide will use a default installation of W2K8 R2.


    Lets add and configure the serial port for our VM that is about to be debugged, AKA the "Debuggee":


    Rightclick and "Edit Settings".


    Click "Add" under the "Hardware" tab.


    Select "Serial Port" and click on "Next".


    On the next page, select "Connect to named pipe".


    Choose any "Pipe Name" you like and remember it.

    Make sure that "Near end:" is set to "Server" and "Far end:" to "A virtual machine".

    The checkboxes for "Connect at power on" and "Yield CPU on poll" should also be checked.


    Verify your settings and power on the VM.


    Configure your Windows OS for accepting connections from a Debugger, in my case "bcdedit /set debug on".

    I also ran "bcdedit /dbgsettings" to verify the settings, refer to the msdn article "Boot Parameters to Enable Debugging" for further information.


    Now we will configure the VM that has WinDbg (and Symbols etc.) installed, AKA the "Debugger":


    Rightclick and "Edit Settings".


    The steps to set up the serial port are the same, the only difference is that you have to select "Client" for the "Near end:".


    Verify your settings and power on the VM.


    Start WinDbg, and select "File -> Kernel Debug"


    The default settings for COM should be correct, as if you were using a physical nullmodem cable.


    You should see that WinDbg is not connected to the Debuggee yet.


    Select "Debug -> Break" to break into the Debuggee.


    Done! You should find your way from here.