Record replay debugging feedback

[cpearce]

Ive been using record replay debugging on Firefox's Mochitest suite to try to debug intermittent test failures. Details:

First off let me say that replay debugging is pretty cool, and I think it's very promissing, and can help us a lot debugging multi threaded problems. Jason Joel asked me to post feedback, so here goes:

1. It's slow. Painfully slow. It takes several minutes to start a replay, and several minutes to seek to a saved point while replaying. I realise this is an experimental feature, but this is a real pain. It's particularly bad when reverse replaying. It takes ages to analyze the recording, can part or all of that be indexed and/or memoized? I'm on a Lenovo T61 Laptop with 3GB ram, Core2Duo T7500 @ 2.2Ghz, Vista Business SP1. I'm running everything on an external hard disk, which probably doesn't help, but still.

2. Sometimes when replaying, the recording will hang. The Workstation UI is still responsive, but when I break-all in Visual Studio, it tells me that all threads appear to be deadlocked, and aren't running code. This makes replaying very unreliable. Coupled with (1), this makes it very frustrating. I suspect that it's caused by thread creation or switching confusing it somehow.

3. When I hit a break point on a non-main thread, often a lot of the other threads (we usually have about 15-20 mostly idle threads in a healthy FF) report having different IDs, but all claim to be stopped at the same location as the break point. I know that they can't all be stopped there, it's being incorrectly reported.

4. Allowing only "about four" breakpoints is painful. When deugging multi-threaded problems, I often care about more entry points that four.

5. I found hit-count break points unreliable.

[cpearce]

I've also added a marker near the beginning of a recording, and now replay won't pass that marker. This is annoying, as the marker is before the bit that I'm interested in, and I can't remove the marker, only delete stuff on either side of the marker.

[JJoel42]

Thanks for posting this feedback. We are very interested in the experiences of developers that are using this feature and we will look into the issues that you have listed. We are aware of the performance and reliability issues which is why this feature shipped as "experimental" and our engineering team is working on improving speed and stability. I will check with the team to determine the cause of the limition on breakpoints and see if we can also increase that number.

[admin]

Hi, cpearce. Thanks for all the comments/bugs! I've filed bugs for (2), (3), and (5). Concerning performance, I agree that we're slow. Working on it! Concerning (4), visual studio is limited to 4 data breakpoints (i.e., watchpoints) because it thinks it is using the hardware debug registers to implement data breakpoints. For now, we're stuck with this limit. But you should have the ability to set closer to 25 control (i.e., normal) breakpoints. I agree that 25 is still fixed and small. I've filed a bug to raise the limit.

Again, thanks so much!