VMware

This Question is Answered

2 "helpful" answers available (6 pts)
8 Replies Last post: Oct 27, 2009 2:32 AM by DAJames  

Debugging kernel shutdown in a linux guest OS posted: Aug 3, 2009 10:22 AM

Click to view DAJames's profile Novice
DAJames
6 posts since
Aug 3, 2009

I'm trying to investigate the shutdown of a linux kernel. I'm using a (slightly hacked) Ubuntu Hardy 32-bit system running under VMWare Workstation 6.5 on a 64-bit Windows Vista Host (Vista x64 is the standard environment for all developers here).

I've read VMWare provides some debugging support, but I have been unable to find any hard information on the extent of that support, and how to set it up. One of the difficulties I face is that I need to be able to debug the kernel's shutdown/restart sequence, which involves (or can involve, depending on the details of the platform) switching briefly back to real mode ... as I understand it tools like kdb and kgdb aren't able to debug across mode switches (they're protect-mode kernel tools)

I'm hoping that VMWare itself can use its virtualization of the hardware to provide some debugging support (is this how replay-based debugging works?) -- but how?

Cheers,
Daniel.


Tags: debug, linux, kernel, real-mode

Re: Debugging kernel shutdown in a linux guest OS

1. Aug 3, 2009 10:34 AM in response to: DAJames
Click to view jmattson's profile Master
jmattson
1,221 posts since
Mar 29, 2006
See this blog posting.

Re: Debugging kernel shutdown in a linux guest OS

2. Aug 4, 2009 1:33 AM in response to: jmattson
Click to view DAJames's profile Novice
DAJames
6 posts since
Aug 3, 2009
Thanks -- I have seen that article (in fact, that's where I got the link to this form from).

I confess <blush> that I missed that tiny little sentence hidden in the first paragraph that just says "No gdb". This is good -- SO good that it deserves to be lit up in lights and shouted from the rooftops. If this works as the blog suggests the VMWare guys need to tout this as a major feature of VMware and provide some official doumentation. The blog is two years old now and (judging from some of the comments) some of the debugging features were a little ... experimental ... when it was written. What's the current state of the art?

Bad news, though ... my first attempt (using the Windows gdb straight from sourceforge) just led to a gdb crash and the guest OS hanging, still waiting for a debugger connection. I'll persevere with this, but I'd be grateful for any further tips to get me going quickly.

Cheers,
Daniel.

Re: Debugging kernel shutdown in a linux guest OS

3. Aug 6, 2009 4:42 AM in response to: DAJames
Click to view DAJames's profile Novice
DAJames
6 posts since
Aug 3, 2009
OK, a little more detail:

I'm running VMware Workstation 6.5.1. build 126130 on Vista x64 SP1 (VMware's "About" dialog actually says its "Microsoft Windows Server 'Longhorn' Business x64 6.0.6001, Service Pack 1" but I'm pretty sure this is really a standard Vista desktop install and not 2008 server). The hardware is a Dell dual quad-core Xeon (E5420) with 8GB. Microsoft Visual Studio (both 2005 and 2008) is installed, as is the VMware virtual debugger plug-in. I've been using VS2005 for some Windows-side development on the host, but haven't used the virtual debugger so I can't confirm that that works. Visual Studio is NOT running when I try to debug the linux VM. I know that there's a newer version of Workstation (6.5.2) but the release notes suggest that the changes are minor so I haven't yet attempted to upgrade (better the devil you know ..).


The guest OS is basically Ubuntu 8.04.2 (Hardy Heron) with a few extra packages and some kernel-level code of my own that I want to be able to debug. I have rebuilt the kernel (with standard configuration) so I can change that and add options in any way that's needed to help with debugging ... but I'm not sure what may be needed to support VMware's debugging.


The gdb I'm trying to use is GNU gdb 5.2.1 (from sourceforge). I downloaded a (32-bit) Windows binary (.exe installer) and I'm using it without change.

I have added the lines:

debugStub.listen.guest32.remote = "TRUE"
debugStub.hideBreakpoints= "TRUE"
monitor.debugOnStartGuest32 = "TRUE"

to the .vmx file for this VM. When I start the VM I get a black screen in VMware and a wait cursorfor a few seconds, which I guess is the VM starting and waiting for connection from the debugger.


When I run gdb in a Vista command window I get:


D:\VMs\Ubuntu_8.04.2>"c:\Program Files (x86)\mingw\bin\gdb.exe"
GNU gdb 5.2.1
Copyright 2002 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB. Type "show warranty" for details.
This GDB was configured as "i686-pc-mingw32".
(gdb) target remote localhost:8832
localhost:8832: No error.
(gdb) continue
The program is not being run.
(gdb)
(this is not the "gdb crashes" result that I reported before, I haven't been able to reproduce that).

As I understand it "The program is not being run." is not what I should expect to see ... but I don't know how to work out what's going on. I believe port 8832 is correct for a 32-bit guest OS (but I tried 8864and got exactly the same result).


I run VMware workstation with administrator rights (my account on my PC here is a local administrator, but we run with UAC enabled so I start VMware with "Run As Administrator" -- I've found that Windows VMs run OK without admin rights but Ubuntu doesn't ... however, I think that's a different issue) -- I have tried running gdb both with and without admin rights, but the result is the same.


What's going on here?

Re: Debugging kernel shutdown in a linux guest OS

4. Aug 6, 2009 7:48 AM in response to: DAJames
Click to view jmattson's profile Master
jmattson
1,221 posts since
Mar 29, 2006
You may want to try a more modern gdb from a Linux box or Linux VM.

Re: Debugging kernel shutdown in a linux guest OS

5. Aug 11, 2009 7:34 AM in response to: jmattson
Click to view DAJames's profile Novice
DAJames
6 posts since
Aug 3, 2009

A VM? It didn't occut to me that I could debug one VM from another ... but, of course, the debugger connection is just TCP/IP so it doesn't matter whether the debug machine is real or virtual.

OK. I have a debian VM on the same PC, which has gdb 6.8-debian. That works.

Thanks.


Re: Debugging kernel shutdown in a linux guest OS

6. Oct 23, 2009 9:44 PM in response to: DAJames
Click to view gev's profile Lurker
gev
1 posts since
Oct 23, 2009

You already have a working solution but my 2 cents wont hurt.

I ran into the same problem and got stuck til someone suggested i used a different GDB. I got gdb thats part of the cygwin package and that worked pretty well

( Running: 64 bit Vista & Vmware workstation 6.5.2 build-156735 )

Re: Debugging kernel shutdown in a linux guest OS

7. Oct 26, 2009 10:50 PM in response to: gev
Click to view fzkhan's profile Lurker
fzkhan
1 posts since
Oct 26, 2009

Hi gev,

I am also facing the same issue on my 32 bit XP machine.... I have Vmware workstation 6.5.2 build-156735 with fedora 11 as VM. I have added the following lines .vmx file for this VM.

debugStub.listen.guest32.remote = "TRUE"
debugStub.hideBreakpoints= "TRUE"
monitor.debugOnStartGuest32 = "TRUE"

When I start the VM I get a black screen in VM console(display). but when i start remote debugging using gdb i get following errors:

Microsoft Windows XP http://Version 5.1.2600
(C) Copyright 1985-2001 Microsoft Corp.

C:\Documents and Settings\iukhan>gdb
GNU gdb 5.2.1
Copyright 2002 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB. Type "show warranty" for details.
This GDB was configured as "i686-pc-mingw32".
(gdb) target remote localhost:8832
localhost:8832: No such file or directory.
(gdb)
(gdb) target remote localhost:8832
localhost:8832: No error.
(gdb) target remote localhost:8834
localhost:8834: No error.
(gdb) target remote localhost:8835
localhost:8835: No error.
(gdb) target remote localhost:8836
localhost:8836: No error.

I tried using gdb thats part of the cygwin package but that results in same issue...any help. Please suggest.

Thanks


Re: Debugging kernel shutdown in a linux guest OS

8. Oct 27, 2009 2:32 AM in response to: fzkhan
Click to view DAJames's profile Novice
DAJames
6 posts since
Aug 3, 2009

That version of Windows gdb is the same one that I used unsuccessfully (though mine was from mingw not cygwin). I think you may need a later version.

It's a while since I tried this but I IIRC you are using the wrong IP address -- you need the IP of the VMware virtual network adapter, which on my machine is 192.168.109.1 (I don't know whether that can vary). I didn't find that the IP address I used made any difference, though, I couldn't get debugging from the host to work.

What I eventually did was to create a second linux VM (Debian Lenny, as it happens, I don't think it should matter what distro you use as long as it has a recent gdb) and run gdb in that VM. That enabled me to debug my first VM (using 192.168.109.1 as the remote IP).

AIUI the debug port number will always be 8832 for 32-bit targets and 8864 for 64-bit targets. Trying 8834, etc., won't work.

I hope this helps.

Daniel.


Actions

VMware Developer

SDKs, APIs, Videos, Learn and much more in the Developer community.

Learn More

Developer Sample Code

Increase your developer productivity with VMware API sample code.

Learn More

VMworld Sessions & Labs

Online access to the latest VMworld Sessions & Labs and online services.

Learn more

Purchase PSO Credits Online

Purchase credits to redeem training and consulting services online.

Buy Now

Community Hardware Software

View reported configurations or report your own.

Learn More

VMware vSphere

Come witness the next giant leap in virtualization.

Register Today

Communities