Debugging in Win32 - special requirements
The Debug Loop (of the debugger)
The Message Loop (of the debuggee)
How GoBug monitors events in the debuggee
How GoBug deals with exceptions

Debugging in Win32 - special requirements

A Win32 debugger needs to be especially designed to monitor and deal with system events, messages and actions. In particular, it must be able to monitor and report on the messages to and from the debuggee, since this is the main way the operating system controls and communicates with the debuggee. Messages may be sent by the system either directly to the debuggee or via the message queue (picked up in the message loop). When waiting for a return from GetMessage in the message loop, the debugger must be able to catch and monitor execution elsewhere in the debuggee caused by messages sent to window procedures. The debugger must also be able to debug multi-threaded applications, and monitor the activities of each thread and show the inter-action between threads. It must be able to display virtual memory and stack areas and register values and identify those areas of memory unique to the operating system. It must be able to trace execution into Dlls. It must be able to deal with exceptions caused by the debuggee allowing the debuggee to continue without crashing, and it must be able to report errors reported by the apis. Finally the debugger must be able to close the debuggee in a system-friendly manner.

The Debug loop (of the debugger)
When you choose a program to debug, GoBug creates a new thread which starts the program using CreateProcess. Then GoBug goes into a debug loop. This loop surrounds the Windows API WaitForDebugEvent and ContinueDebugEvent.

The API WaitForDebugEvent returns upon the happening of the following:- (a) after the processor executes an instruction for the debuggee (b) when a message is due to be sent to the debuggee (c) before or upon an event concerning the debuggee (for example creation of windows, change of focus etc). When WaitForDebugEvent returns, GoBug deals with it in "debug loop processing" and then calls WaitForDebugEvent again, completing the loop.

Of particular interest here are these four facts:-

1. You can see from the above that after an instruction is executed by the processor for the debuggee, the debuggee enters the debug loop. When this occurs, the debuggee is completely suspended, including all the threads of the debuggee. This is GoBug's state when waiting for user input (F5 to F9).
 Proof that all threads are suspended when a thread is in the debug loop.
2. Despite the fact that all the debuggee's threads are suspended when the debuggee is in the debug loop, in fact only one of the debuggee's threads has caused the debuggee to enter the loop. This is the thread for which the processor last executed an instruction. Further, it is possible that no up to date information is available from the system about the other threads.
3. To get into the debug loop the next instruction executed by the processor for any particular thread of the debuggee, GoBug must set the trap flag in the context of the thread in the loop. This is what GoBug does unless you press F8 (run, hook, part log) or F9 (run in the background). If either of these keys are pressed the trap flag is cleared for that thread, so that from that point on no debuggee's instructions for that thread will not come into the loop. The result of this is that once you have pressed F8 or F9, it is possible that you may not be able to regain full debug control over the thread concerned. However control can usually be regained by setting a breakpoint or by using the hot-key.
4. You can see from the above that it is possible to have different "run" states for different debuggee threads. For example the main thread might be running (because you pressed F9 when that thread was in the loop) but all the other threads might be single-stepping (trap flag set for those each time).

The Message Loop (of the debuggee)

This is a diagram of a typical Windows program which uses the Graphical User Interface (GUI). After the program starts, it initialises the main window and then goes into the message loop. In the loop the API GetMessage is called. Typically this API does not return until a key is pressed or there is a mouse click. When such an event occurs, the API DispatchMessage sends the key press or mouse click to the window procedure to be dealt with. It does this by calling the window procedure and then returning.

You can see from this that if you want to single step through the application's code, you will be able to view the code in the initialisation section, and then you will be able to see the call to GetMessage, but after that all you will do is to go around the message loop. You will miss all the processing of the message within the window procedure itself, since this will all be dealt with within the DispatchMessage API.
To compound this you will also miss the way the application processes numerous messages sent by the system itself (not sent by the message loop) to the window procedure.
Clearly to make sensible debugging possible, some way must be found to make the debugger respond to calls to the window procedure.
GoBug achieves this by the "Single-step message break". In effect, GoBug senses when a message is due to go to the window procedure and sets a breakpoint at the window procedure ready to receive that message. When the breakpoint is reached GoBug shows the window procedure in the codepane, and you can choose to ignore the message or you can single step through the window procedure.
You don't have to use the "Single-step message break". You can switch it off if you like and set your own breakpoints. You can run to a specific message, and you can specify which window procedure must receive the message before the breakpoint is triggered.

How GoBug monitors events in the debuggee
You can see from the above the GoBug needs to know when messages are being sent to the debuggee, whether by the system, or by other applications or by the debuggee itself, whether from the message loop or in some other way. In addition to this, GoBug needs to know what events are occurring in the debuggee, for example, whether new windows or threads are being created or if the debuggee is loading a Dll. Some of these events are reported to GoBug's debug loop by the system as debug events. Other events and all messages are monitored by hooks which is a very small piece of code in GoBugSpy.dll and which loads within the memory context of the debuggee when the hook is required for logging. If you look at the memory areas of the application under test you will see GoBugSpy.dll if it is there. It is unusual for a Dll run by one program to load in the memory context of another, but this is necessary for the hook to work. GoBugSpy is not loaded if you run the debuggee in the background (F9) or if the application does not rely on the system Graphical User Interface (GUI).

How GoBug closes the debuggee
In Win32 an application often needs to clear up before closing. This might involve closing files, handles and memory, unloading Dlls or writing to the registry or to an ini file. For this reason unless the debuggee is a console program, GoBug attempts to close the debuggee in a natural manner - by sending the message WM_QUIT. If this fails to close the debuggee, GoBug will try to call the API ExitProcess on the debuggee's behalf. If that fails, GoBug will terminate its debugging thread (which will in turn close the debuggee's main thread). If that fails too, then after a warning to you, GoBug will force the debuggee closed by calling TerminateProcess. This is not an optimum way to close a process, and can leave some resources open.

How GoBug deals with exceptions
If an exception occurs in a program which is being debugged, the system sends the exception firstly to the debugger. This causes the debuggee to enter the debug loop. GoBug intervenes and gives you details of the exception which occurred. For more details see Exceptions caused by the debuggee.