EFI Debugging

[eg0n]

Hi,

I was hoping to use VMware Fusion for debugging some custom EFI drivers and how they interact with Mac OS X. Does the EFI firmware included with VMware Fusion contain the SourceLevelDebugPkg package from TianoCore? If not, is it possible to obtain a version of this firmware that does, or possibly swap it out for an OVMF image built from the TianoCore sources?

I've attempted to connect to a Mac OS X VM via pipe-type serial ports from a Linux VM (as per this thread: http://communities.vmware.com/message/748577#748577) with the UDK debugger tool from Intel (at boot time) and have not succeeded, leading me to believe that the EFI firmware probably doesn't have the SourceLevelDebugPkg compiled in (which is not unexpected I suppose). The serial connectivity seems to be working, as I can connect two minicom instances together on different Linux VMs with this configuration.

Egon

[dariusd]

Hi eg0n, and welcome to the VMware Communities!

What a rockin' question!

We don't include SourceLevelDebugPkg, but you can do much the same thing (and better in some ways)  using gdb and the debug stub facility available in a VMware Virtual Machine.  Add these to the .vmx file for the VM you to which you wish to attach:

   debugStub.listen.guest32 = "TRUE"
   debugStub.listen.guest64 = "TRUE"

If your debugger is on a different host, you'll need to add:

   debugStub.listen.guest32.remote = "TRUE"
   debugStub.listen.guest64.remote = "TRUE"

Now, when you power on that VM, vmware-vmx will listen  on TCP ports 8832 and 8864 (on the host) for 32-bit and 64-bit gdb connections respectively.  Those debug connections will work just like a hardware debugger on a physical machine -- when you stop execution, all cores are halted, the guest doesn't even know that time has stopped, and you can happily single-step interrupt handlers, exceptions, etc.  Connect using gdb:

   (gdb) target remote localhost:8832

or, if you are on another host and have enabled the "remote" options above,

   (gdb) target remote <hostname>:8832

or, for 64-bit EFI,

   (gdb) set architecture i386:x86-64
   (gdb) target remote localhost:8864

etc.

The debug stub, though, won't automatically give you module load address information that you could probably get easily through SourceLevelDebugPkg.

For what it's worth, we don't even use SourceLevelDebugPkg internally at VMware when we're developing/debugging our EFI firmware... we use the above debugStub with some gdb script automation to locate the PE/COFF headers and load the corresponding symbols.

To answer your next question, it is possible to substitute our ROMs with other images (the .vmx options you seek are efi32.filename and efi64.filename), but those other images will need compatible early bring-up code that is necessary to have the firmware function at all within a VMware VM -- for instance, OVMF would need to be taught how to figure the available memory and the physical memory map, and we, out of necessity, have our own mechanism to pass that information into the VM.

Hope this helps!

--

Darius

[eg0n]

Hi Darius,

I tried adding the debugStub.listen.guest* options to a freshly installed Lion VM on Fusion 4.1.1, and vmware-vmx is certainly listening on those ports, but I don't seem to be able to connect from GDB. I'm assuming it's 64-bit EFI, since I created a 64-bit VM:

gdb$ set architecture i386:x86-64

The target architecture is assumed to be i386:x86-64

gdb$ target remote localhost:8864

[New thread 1]

Remote register badly formatted: T05thread:00000001;06:203c474580ffffff;07:003c474580ffffff;10:686937817fffffff;

here: 0ffffff;07:003c474580ffffff;10:686937817fffffff;

And with the 32-bit connect:

gdb$ set architecture i386

The target architecture is assumed to be i386

gdb$ target remote localhost:8832

Ignoring packet error, continuing...

Ignoring packet error, continuing...

Ignoring packet error, continuing...

Malformed response to offset query, timeout

gdb$ target remote localhost:8832

localhost:8832: Connection refused

Any ideas?

---

EDIT: Ahh, here's the relevant snippet from vmware.log. Not too illuminating for me still...

2012-02-10T01:12:50.377+10:00| vmx| W110: Debug stub remote connection accepted

2012-02-10T01:12:50.377+10:00| vcpu-0| I120: Vix: [4525670400 vmxCommands.c:4840]: VMAutomation_Pause: pause = TRUE

2012-02-10T01:12:50.377+10:00| vcpu-0| I120: Not informing DCL we are stopping.

2012-02-10T01:12:50.378+10:00| vmx| I120: Error: Encountered Apple bug #5202831.  Disconnecting.

2012-02-10T01:12:50.378+10:00| vmx| I120: SOCKET 23 (223) recv detected client closed connection

2012-02-10T01:12:50.378+10:00| vmx| I120: SOCKET creating new listening socket on port 40994

2012-02-10T01:12:50.378+10:00| vmx| W110: VMware Fusion is listening for debug connection on port 8864.

2012-02-10T01:12:50.378+10:00| vmx| W110:     target remote localhost:8864

2012-02-10T01:12:50.378+10:00| vmx| I120: Vix: [140735304714592 vmxCommands.c:4840]: VMAutomation_Pause: pause = FALSE

---

As far as symbols go, I can probably live without them for the moment, but I certainly wouldn't object to any GDB scripts you might be able to share I can hack some stuff up otherwise. I guess if I really wanted to use SourceLevelDebugPkg I could always have a go at replacing the DXE module in the VMware image with one built with SourceLevelDebugPkg, but that sounds hard and not really worthwhile if the pseudo-hardware debugger method will work for me.

One more question - is there a flag that can be set in the vmx file or a key combo or some other way to get VMware to drop into the EFI shell instead of booting the OS, as it does when it cannot find a bootable disk? How about to hit a hardware breakpoint at a certain point in the early phases of PEI/DXE?

Thanks very much for your help!

Egon.

[dariusd · Accepted solution]

Hi eg0n,

Unless you embark on some strange and undocumented hackery, the EFI bitness will correspond to the guest OS bitness.

gdb$ target remote localhost:8864

[New thread 1]

Remote register badly formatted: T05thread:00000001;06:203c474580ffffff;07:003c474580ffffff;10:686937817fffffff;

here: 0ffffff;07:003c474580ffffff;10:686937817fffffff;

I've seen this error with an older GDB on a 32-bit OS, which was supposed to be able to handle 64-bit targets but for some reason died with this exact error when connecting to the 64-bit debug stub -- we provide 64-bit CPU registers, but gdb is only expecting 32-bit register values.  I've since upgraded to a newer 64-bit OS and haven't seen this failure in a long time. Not sure if it was the version upgrade (Debian 5 to Debian 6) or the architecture change which fixed the problem.

gdb$ set architecture i386

The target architecture is assumed to be i386

gdb$ target remote localhost:8832

Ignoring packet error, continuing...

Ignoring packet error, continuing...

Ignoring packet error, continuing...

Malformed response to offset query, timeout

The timeouts you see here indicate that you've connected to the 32-bit debug stub, we've stopped execution of the VM, but we're actually in 64-bit mode, so you should connect to the 64-bit debug stub instead (I don't think this is a concept that GDB's remote protocol can represent).  That matches with what you described. 

So basically I think you need a newer, properly 64-bit capable GDB, and then try on port 8864 again.

It wouldn't be terribly productive for me to share our own GDB scripts, since we add secret sauce to our debug-build ROMs, and the scripts largely depend on that secret sauce.  Just about the only part that would be useful on released ROMs is the reverse-search for a PE/COFF header from an arbitrary address, and that is easy enough to do (just looking for the 'MZ' and 'PE' headers is sufficient, and the 'MZ' is always page-aligned), and it'd be particularly easy in comparison to the work I'd need to do to get permission to publish our GDB scripts here... 

You should be able to move the EFI Shell to the top of the boot order in the Boot Maintenance Manager -- after powering on the VM, quickly click inside the window to "grab" keyboard into the VM, then hold the Option key while the VMware splash screen is showing.  That'll get you to the main EFI menu (yes, it's not pretty) and you can use the Boot Manager to do a one-off launch of the EFI Shell or the Boot Maintenance Manager to enable/disable the OS boot option or to reorder the Shell to the top.  There's no hotkey assigned to the Shell, and there is not yet any VMX option that will force entry to the Shell either (at least, not that I can immediately think of).  We don't provide a UI for assigning hotkeys to Boot Options, but our code should still include the hotkey processing stuff, so if you can edit the Boot#### NVRAM option for the Shell, you might be able to create a hotkey manually and it might work, even though we don't officially support that feature.

I can't think of any neat/constant/supportable way of hitting an early breakpoint.  You can always add a CpuDeadLoop() to your own driver's entry point, at which time the VM will "hang" until you connect the debugger and bust out of the deadloop.  If you need to do stuff before your own driver's entry point, then it may well be something that I shouldn't hear about... :smileydevil:

Cheers,

--

Darius

[eg0n]

Hi Darius,

I figured the delay on the connect to the 32-bit port was something along those lines.

I've seen this error with an older GDB on a 32-bit OS, which was supposed to be able to handle 64-bit targets but for some reason died with this exact error when connecting to the 64-bit debug stub -- we provide 64-bit CPU registers, but gdb is only expecting 32-bit register values.  I've since upgraded to a newer 64-bit OS and haven't seen this failure in a long time. Not sure if it was the version upgrade (Debian 5 to Debian 6) or the architecture change which fixed the problem.

I'm using the current version of Apple's fork of GDB on the Mac OS X 10.7.3 host OS (on which Fusion is running), which I have used before for debugging the XNU kernel in 64-bit mode in a VM (using Apple's KDP though, rather than this method):

GNU gdb 6.3.50-20050815 (Apple version gdb-1708) (Thu Nov  3 21:59:02 UTC 2011)

Copyright 2004 Free Software Foundation, Inc.

GDB is free software, covered by the GNU General Public License, and you are

welcome to change it and/or distribute copies of it under certain conditions.

Type "show copying" to see the conditions.

There is absolutely no warranty for GDB.  Type "show warranty" for details.

This GDB was configured as "x86_64-apple-darwin".

Is this likely to be incompatible with the debug stub or something? I'll give it a try with the current version of the main FSF version from a Linux VM, but I'd prefer to do it from OS X if possible.

It wouldn't be terribly productive for me to share our own GDB scripts, since we add secret sauce to our debug-build ROMs, and the scripts largely depend on that secret sauce...

I thought that might be the case Shouldn't be too hard to do the symbol resolution and whatnot, if I find it necessary I guess.

You should be able to move the EFI Shell to the top of the boot order in the Boot Maintenance Manager...

Boot Maintenance Manager works nicely, that will do for now, but I might explore the NVRAM option stuff as well.

I can't think of any neat/constant/supportable way of hitting an early breakpoint.  You can always add a CpuDeadLoop() to your own driver's entry point, at which time the VM will "hang" until you connect the debugger and bust out of the deadloop.  If you need to do stuff before your own driver's entry point, then it may well be something that I shouldn't hear about... :smileydevil:

Good idea, that'll probably do it for the driver debugging, and yeah, probably verging on territory that you'd be best to "LALALALA I CAN'T HEAR YOU" with the rest. Thanks very much for all the info!

Egon.

[eg0n]

OK I can connect from a Debian VM with the latest package version. I'll see if I can figure out what's going on with OS X version, but it might just be incompatible because it forked so long ago

[tetrapackage]

I will give a +1 to the first answer given. Very easy and informative. Thanks as well.

[eg0n]

I'm not sure why, but this works fine on a different OS X machine with the same version of GDB.