VMware Communities: Memory watchpoints emulation ? ...

Up to Discussions in Replay-Based Debugging

This Question is Answered

1 "correct" answer available (10 pts) 2 "helpful" answers available (6 pts)

12 Replies Last post: Apr 29, 2009 5:21 PM by Shrinand

Memory watchpoints emulation ? posted: May 14, 2008 7:31 AM

J.F.V

15 posts since
May 8, 2008

Hello !

It seems like hardware watchpoints are unsupported in vmware workstation < 6.0.3

Is this a planed feature ?

I have enabled hiddenBreakpoints in the configuration file but the gdb client keeps
receiving vmware errors as soon as a hwbp is set and the debugged program is
continued.

The context of my problem is as follow: I am running an OS-less image starting
execution at 0x7c00. Another image is then overwriting the first one, at this
same address, and starts execution at 0x7XXX (e.g. a breakpoint on 0x7c00 is not
enough to catch the start of the new image).

Given that the code is many thousands assembly instr, it would have been just easy
to set a hwbp on 0x7c00 and see when the first image is overwritten.

How can vmware-ws help me to do that ?

Thanks

-JFV

Re: Memory watchpoints emulation ?

1. May 14, 2008 10:47 AM

in response to: J.F.V

slava_malyugin

55 posts since
May 14, 2007

Hello J.F.V! Thank you for trying out the feature. The watchpoints are indeed missing in WS6.0.3, but were added in WS6.5 Beta. If you could give latest beta a try and tell if it works, I would appreciate it. Note that the image at 0x7C00 may be overwritten by the DMA transaction. They way watchpoints in WS6.5 Beta are implemented they should still trigger on DMA write, but will stop the Guest at some random location when DMA transaction completes. Caveat: per our policy, I cannot promise that released version of WS6.5 will have the watchpoints or any other functionality. I certainly hope it will

Also, if I understand your setting correctly, you could use hidden breakpoint on the entry point of second image. In worst case scenario (if both first and second images have entry points at the same address), the breakpoint will hit twice, but you can set gdb to stop only on second invocation of the breakpoint. Will this work for you?

Sincerely,

Vyacheslav (Slava) Malyugin

Re: Memory watchpoints emulation ?

2. Dec 27, 2008 7:52 AM

in response to: J.F.V

212okins

7 posts since
Dec 26, 2008

Suppose you want to set watchpoints in a kernel module or vmlinux in the guest OS.
They may do what you expect a few times but won't work all the time because the guest
OS kernel saves and restores debug register dr7 (and others) for user processes.
Suppose the save had occurred, then an interrupt is nested on top of that thread --
it means your watchpoints are no longer applicable to the interrupt thread until the
restore occurs much later...
To work around this problem you'd have to edit the sources in arch/i386/ of the guest
OS and remove all saves & restores of dr7, recompile and reinstall the kernel.
If you have missed any modifications to dr7 then setting some bit in dr4 would cause
a trap as I recall and you can find the offending code...

Re: Memory watchpoints emulation ?

3. Jan 5, 2009 6:22 AM

in response to: 212okins

slava_malyugin

55 posts since
May 14, 2007

Watchpoints in our debug stub are implemented without using DR registers, so there should be no conflicts with Linux kernel using these. Please let us know if they do not work for you.

Sincerely,

Vyacheslav (Slava) Malyugin

Re: Memory watchpoints emulation ?

4. Jan 6, 2009 4:53 PM

in response to: slava_malyugin

212okins

7 posts since
Dec 26, 2008

Slava, I am debugging corruption of the magic field in a spinlock.
Every time I "continue" my gdb crashes and the remote monitor dies too (no longer listens).
Commands other than continue, like "info thr" issued after the watch command work, only
continue seems to be causing problems.
I can suspend and resume the guest thereby reviving the vmware monitor but that doesn't solve
the watchpoint issue.

I am running vmware workstation 6.5.1-build-126130 hosted
on a 64 bit linux OS, my guest is a 32 bit linux OS.

Please provide a link to unreleased software if need be, I am willing to be a guinea pig for Vmware.

GNU gdb 6.8-debian
Copyright (C) 2008 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
(gdb) target remote localhost:8832
Remote debugging using localhost:8832
New Thread 999999
0xc0112c37 in ?? ()
(gdb) set debug remote 1
(gdb) watch *0xdf302650
Sending packet: $mdf302650,4#c7...Ack
Packet received: ad4eadde
Hardware watchpoint 1: *3744474704
(gdb) cont
Continuing.
Sending packet: $mdf302650,4#c7...Ack
Packet received: ad4eadde
Segmentation fault

Re: Memory watchpoints emulation ?

5. Jan 7, 2009 6:47 AM

in response to: 212okins

slava_malyugin

55 posts since
May 14, 2007

Interesting - gdb chockes on "read memory" response. Could you please give older version of gdb a try to see if its regression in the debugger? The latest one I have tried with debug stub was 6.6. Thank you.

Sincerely,

Vyacheslav (Slava) Malyugin

Re: Memory watchpoints emulation ?

6. Jan 9, 2009 7:55 AM

in response to: slava_malyugin

212okins

7 posts since
Dec 26, 2008

Slava,

this is the output from my gdb session, debugger version=6.3.0.0-0..31rh, the vmware
monitor doesn't die during this session which is good news.

(gdb) watch *0xddd51650
Sending packet: $mddd51650,4#fa...Ack
Packet received: ad4eadde
Hardware watchpoint 1: *3721729616
(gdb) cont
Continuing.
Sending packet: $Hg1#e0...Ack
Packet received: OK
Sending packet: $p8#a8...Ack
Packet received: a267bb00
Sending packet: $Hgf423f#14...Ack
Packet received: OK
Sending packet: $p8#a8...Ack
Packet received: 944a2dc0
Sending packet: $Hg1#e0...Ack
Packet received: OK
Sending packet: $p8#a8...Ack
Packet received: a267bb00
Sending packet: $p5#a5...Ack
Packet received: 28bae8bf
Sending packet: $mbb67a2,8#95...Ack
Packet received: c38db6000000008d
Sending packet: $mbb67a2,7#94...Ack
Packet received: c38db600000000
Sending packet: $p4#a4...Ack
Packet received: 0cb7e8bf
Sending packet: $mddd51650,4#fa...Ack
Packet received: ad4eadde
Couldn't write debug register: Operation not permitted.

Re: Memory watchpoints emulation ?

7. Jan 9, 2009 10:42 AM

in response to: 212okins

slava_malyugin

55 posts since
May 14, 2007

This sounds like a bug in gdb that we hit some time ago: it tries to access DR registers on the Host instead of DR registers on the remote target. I believe the bug only kicks in crossplatform mode, that is, 64-bit gdb talking to 32-bit target or vice versa. Could you please give 32-bit gdb a try? Thank you.

Sincerely,

Vyacheslav (Slava) Malyugin

Re: Memory watchpoints emulation ?

8. Jan 9, 2009 4:30 PM

in response to: slava_malyugin

212okins

7 posts since
Dec 26, 2008

Nope this is a 32 bit gdb hosted on rhel4 AS (guest OS), talking to the monitor, the version is shown above.
I can try gdb'ing from Suse next or better yet can you point me to some distribution or gdb version that is known
to work correctly with the vmware monitor?

Addendum: later today I tried gdb 6.6 on 32 bit suse - same problem as rhel4/AS, I also built 64 bit gdb executables,
versions 6.6, 6.7 and 6.8 from 2 sources - the GNU gdb site and the debian package repository.
None of them could continue after setting a watchpoint. Also tried building the tip of the cvs repository for gdb,
it failed to compile.
I am wondering if there is any version of gdb at all that can set a watchpoint in the vmware monitor and continue.
If there is I'd appreciate getting the gdb version and linux distribution from you, thanks.

Re: Memory watchpoints emulation ?

9. Jan 12, 2009 9:48 AM

in response to: 212okins

ecl100

48 posts since
Sep 24, 2007

212okins wrote:
Couldn't write debug register: Operation not permitted.

Hi, 212okins. I've seen this before. This is (as Slava said) because gdb is trying to access the debug registers on the host despite the fact you are doing remote debugging. It's a gdb bug, and I don't know if it's been fixed yet. I've worked around this problem by ensuring that gdb believes the host architecture for which it was compiled is different from the architecture it is debugging (note that I'm contradicting Slava's advice on this matter). Specifically, I've done this by configuring compilation of gdb with --host=i686-pc-linux-gnu --target=i386-pc-linux-gnu. My guess is that you can achieve the same thing without recompiling (by using "set architecture" in gdb to be an x86 architecture that is different from how gdb was compiled). If that doesn't work we'll have to try something else...

My understanding is that the gdb maintainers are aware of this issue, but I don't know if they've fixed it yet, so we'll have to deal with a workaround. Note that you should observe this same problem even if you are using gdb/gdbserver (i.e., it is not a vmware-specific problem). Nevertheless, my apologies for the inconvenience.

Best of success to you. And I hope you'll update us with your findings.

Thanks! E.

Re: Memory watchpoints emulation ?

10. Jan 13, 2009 10:59 AM

in response to: ecl100

212okins

7 posts since
Dec 26, 2008

Thanks ecl100 for the comments.
I built 32 gdb 6.6 and 6.7.1 for cross debugging exactly as you suggested.
In either case I was unable to interrupt the debugger after continuing.
Every time you see the words "remote_interrupt called" in the attached log file
I was trying to CTRL-C into it but it wouldn't let me.
I did not check to see if a change to the memory location would cause the
debugger to stop because gdb has little value to me if I am unable to interrupt it.
Setting remote verbose-resume-packet to on resulted in "warning: Invalid remote reply".
Any further comments, suggestions?

-----------------gdb 6.6 --host=i686-pc-linux-gnu, --target=i386-pc-linux-gnu--------------
(gdb) watch *0xc0441e20
Hardware watchpoint 1: *3225689632
(gdb) set debug remote 1
(gdb) cont
Continuing.
Sending packet: $mc02d4760,1#c4...Ack
Packet received: 6a
Sending packet: $mc02d4760,1#c4...Ack
Packet received: 6a
Sending packet: $mc02d4762,8#cd...Ack
Packet received: fc061e5055575652
Sending packet: $mc02d4760,a#f4...Ack
Packet received: 6aeffc061e5055575652
Sending packet: $mc0441e20,4#c0...Ack
Packet received: 3c343e4c
Sending packet: $Z2,c0441e20,4#0b...Ack
Packet received: OK
Packet Z2 (write-watchpoint) is supported
Sending packet: $vCont?#49...Ack
Packet received:
Packet vCont (verbose-resume) is NOT supported
Sending packet: $Hc0#db...Ack
Packet received: OK
Sending packet: $c#63...Ack

remote_interrupt called
remote_stop called
Packet received: T05thread:000f423f;05:dc3f3cc0;04:d03f3cc0;08:60472dc0;
New Thread 999999
Sending packet: $p8#a8...Ack
Packet received: 60472dc0
Sending packet: $mc0441e20,4#c0...Ack
Packet received: 3c343e4c
Sending packet: $z2,c0441e20,4#2b...Ack
Packet received: OK
Sending packet: $mc0441e20,4#c0...Ack
Packet received: 3c343e4c
Sending packet: $Z2,c0441e20,4#0b...Ack
Packet received: OK
Sending packet: $c#63...Ack
remote_interrupt called
remote_stop called
Packet received: T05thread:000f423f;05:dc3f3cc0;04:d03f3cc0;08:e83d2dc0;
Sending packet: $mc0441e20,4#c0...Ack
Packet received: 3c343e4c
Sending packet: $z2,c0441e20,4#2b...Ack
Packet received: OK
Sending packet: $mc0441e20,4#c0...Ack
Packet received: 3c343e4c
Sending packet: $Z2,c0441e20,4#0b...Ack
Packet received: OK
Sending packet: $c#63...Ack

Re: Memory watchpoints emulation ?

11. Jan 15, 2009 8:43 AM

in response to: 212okins

ecl100

48 posts since
Sep 24, 2007

Hi, 212okins. I've replicated the bug you describe (^c doesn't work when watchpoints are set). The problem is that (as you last message shows) when you hit ^c, execution stops, but gdb chooses to evaluate the watched expression(s), and continue if it hasn't changed (and it hasn't, otherwise you would have hit the watchpoint). Why does gdb do this? I believe it is because we are sending the wrong signal number ("T05") to gdb in response to the ^c. When gdb sees "T05" it assumes we've hit a watchpoint. We really should be sending "T02" in response to ^c. I've tried this out and it appears to solve the problem. I've file a bug for this, and we should fix it.

As for a workaround, we need to convince gdb not to evaluate the watchpoint. Would you be able to use an access watchpoint (awatch)? Do any gdb experts out there have any suggestions?

Thanks for pointing this out, and I apologize for the bug. E.

Re: Memory watchpoints emulation ?

12. Apr 29, 2009 5:21 PM

in response to: ecl100

Shrinand

5 posts since
Jul 2, 2008

I was running into the same issue with older versions of gdb (6.3, 6.1 and even the released version of 6.8). I just tried with gdb-6.8.50.20090130 and it seems to work. You might want to give it a try.

-Shri

Go to original post

Search VMware Communities Advanced Search

Actions

View as PDF

Up to Discussions in Replay-Based Debugging

This Question is Answered

Memory watchpoints emulation ? posted: May 14, 2008 7:31 AM

Re: Memory watchpoints emulation ?

Re: Memory watchpoints emulation ?

Re: Memory watchpoints emulation ?

Re: Memory watchpoints emulation ?

Re: Memory watchpoints emulation ?

Re: Memory watchpoints emulation ?

Re: Memory watchpoints emulation ?

Re: Memory watchpoints emulation ?

Re: Memory watchpoints emulation ?

Re: Memory watchpoints emulation ?

Re: Memory watchpoints emulation ?

Re: Memory watchpoints emulation ?

Actions

Communities

Community Resources

VMworld

Get Support

Other Links

This Question is Answered

Memory watchpoints emulation ? posted: May 14, 2008 7:31 AM

Actions

VMware Developer

Developer Sample Code

VMworld Sessions & Labs

Purchase PSO Credits Online

Community Hardware Software

VMware vSphere

Community Resources

VMworld

Get Support

Other Links