I am not going to explore this configuration in any detail. You normally place the source for your driver on the Host Machine, which is also your personal development machine. In addition, the driver symbols (a byproduct of the build process) will probably be on the Host as well. This simple configuration has the benefit of simplicity, but it severely limits our options for remoting the debug session. In an environment where multiple developers are working on a project, we need to ensure that both the Source and the Symbols are in a central place rather than on a development machine that isn't widely accessible. Centralizing source and symbols opens up a load of debugging possibilities.
In this scenario we are going to clear our Source and Symbols off the Host Machine and archive all of it off onto a Server. I am going to describe how to configure the Target and Host at a genteel pace, before quickly checking that it works. This setup forms the basis for the more esoteric configurations in the following sections. Figure 2 shows the Host, Target and Server.
Figure 2. Target, Host and Server.
The Target Machine has been setup with a shrink wrap XP installation. The only "tweak" is a modified boot.ini file. The boot.ini file contains a list of the alternative operating systems that you can boot from the main partition. Right after you install Windows XP, there will be just one entry in this file. We want to modify boot.ini to enable us to hookup a debugger to the serial port. The easiest way to edit this file is to open a command prompt, change the attributes for the file, and launch Notepad:
C:\>attrib -s -h -r boot.ini
C:\>notepad boot.ini
Now edit the boot settings to add /debugport and /baudrate switches in the [operating systems] section.
[boot loader] timeout=30
default=multi(0)disk(0)rdisk(0)partition(1)\WINDOWS
[operating systems]
multi(0)disk(0)rdisk(0)partition(1)\WINDOWS="Microsoft Windows XP Professional" /fastdetect /debugport=COM1 /baudrate=115200
The next time you boot the target machine, the new switches will cause the kernel to wait until it can attach to WinDbg before proceeding.
In this configuration, you install WinDbg on the host machine. I am using version 6.1.0.0017.2, but this will undoubtedly be out of date before I reach the end of this paragraph. We need to set some environment variables to ensure that the debugger can resolve both the operating system symbols and the symbols for my drivers. We also need to ensure that the Host's baudrate matches the baud rate we specified in the Target Machine's boot.ini. COM1 is the default serial port, so we don't need to specify that.
Let's suppose that we've put all the symbols and source on the Sysphus server. Here are the environment variables we need to set up on the Host Machine.
_NT_DEBUG_BAUD_RATE=115200
_NT_SYMBOL_PATH=SRV*\\Sysphus\softshared\symbols\SymStore*http://msdl.microsoft.com/download/symbols
_NT_ALT_SYMBOL_PATH=\\Sysphus\softshared\symbols\KillerApp\sym
_NT_SOURCE_PATH=\\Sysphus\softshared\symbols\KillerApp\sym
Figure 3 illustrates the Source and Symbol configuration described by these environment settings. The debugger needs to find both the symbols and the source for the binaries running on the Target. We don't particularly care how they got onto the server. They could have been transferred to the server from a build on the Host Machine, or they may have been archived here. If the debugger running on the Host needs to match up OS symbols it will first look in Symstore on the Sysphus server. If this search fails, the debugger will attempt to transfer the symbols from the Microsoft symbol server. Storing OS symbols on a local server means that other developers will not have to download symbols from the Microsoft server -- they can point their debugger at the server and retrieve the local copies. This saves both time and disk space.
Figure 3. Symbols and Source.
Let's just test that this actually works. I have my latest drivers installed on the Target Machine and WinDbg is waiting to be invoked on the Host. The hardware is waiting to be plugged in and everything's ready to roll. The target is turned off.
C:\>WinDbg -k
WinDbg should start up and display
Microsoft (R) Windows Debugger Version 6.1.0017.2
Copyright (c) Microsoft Corporation. All rights reserved.
Opened \\.\com1
Waiting to reconnect...
Connected to Windows XP 2600 x86 compatible target, ptr64 FALSE
Kernel Debugger connection established.
Symbol search path is: \\Sysphus\softshared\symbols\KillerApp\sym;SRV*\\Sysphus\softshared\symbols\SymStore*http://msdl.microsoft.com/download/symbols
Executable search path is:
Windows XP Kernel Version 2600 MP (1 procs) Checked x86 compatible
Built by: 2600.xpsp1.020828-1920
bp SaiNtHid!DriverEntry
Figure 4. Contact!
The debugger has correctly grabbed the source from the server. Now let's ponder how we can leverage the server for some real power debugging.
Adding a Remote Host
You may have wondered why I depicted the Host machine as a laptop. If you have a building full of computers, any of which is likely to have some weird interaction with your driver, using a laptop is a quick and easy way of enabling remote debugging on any machine it's attached to. We need only make a slight change to the scenario we have already examined. Our invocation of WinDbg is going to have to change to reflect a new role for the Host: we need to tell WinDbg that it has been promoted to a server.
C:\>WinDbg -server tcp:port=3001 -k
To start the debug server we need a TCP/IP socket number. Your network administrator will be able to give you a valid number, but be prepared for lengthy discussions about firewalls and network security. Doughnuts should help. The socket number I have been given for this Host Machine is 3001. If you ARE the network administrator, or are entrusted with the grave responsibility of selecting your own socket number, see the links in the Assigning a TCP/IP Socket Number Note.
You can still control your debug session from the Host computer. The role of the Host has been enhanced, rather than subverted, by setting it up as a server. A second developer can now join the debug session, as illustrated in Figure 5.
Figure 5. Adding a Remote Host.
The second developer can now join the debug session on the \\DEEPTHOUGHT Remote Host computer by issuing this command from a command prompt:
C:\>windbg -remote tcp:server=\\Jolwhqldev-xp,port=3001 -srcpath \\Sysphus\softshared\symbols\KillerApp\src
This syntax should be self explanatory, but it is worth noting that the Remote Host does not need to set up the symbols. The association between the binaries on the Target and the Symbols on the Server is set up by the "real" Host machine (the laptop in all my pictures). In addition to describing the connection to the Host with the server name and port number, we need to ensure that our session can access the source on the server. Depending on the debugging scenario we may choose to limit the role of the Host machine, see the Attack of The Drones note for more on this.
WinDbg should respond with something like the following
nt!RtlpBreakWithStatusInstruction:
80aaadcc cc int 3
DEEPTHOUGHT\jwr (tcp 10.1.2.4:3389) connected at Tue Feb 25 2003
Now that the Remote Host is connected we can break into the session and set break points in exactly the same way we did in our initial experiments.
Attack of The Drones
If the Host is not manned we can set it up as a "drone" computer. The Host does not need the source at all in this situation. If the Host is a roaming laptop, it is probably a very good idea to prevent the machine from having access to the source. In fact we could avoid the overhead of a GUI by using KD.EXE as our Host debugger instead of WinDbg. The WinDbg Session running on the Remote Host will connect to this session in exactly the same way, regardless of whether KD or WinDbg is running on the host. We can invoke KD with exactly the same environment variable settings that we used in our initial configuration of the Host. (Of course, it is not necessary to set the _NT_SOURCE_PATH variable since KD won't be accessing the source.)
You can also append the -noio flag to ensure maximum inscrutability from KD. This will ensure that the debugger generates no output on the server machine. If you are unused to the KD debugger, this is certainly the preferred operating mode.
The flags do differ slightly between WinDbg and KD. Since KD is never anything but a kernel debugger, we don't need to use the -k switch. However -k is used to specify the Target connection. The following examples should illustrate the point.
- To launch KD on the Host, using TCP port 3001 and COM1:
C:\>kd -server tcp:port=3001 -noio
- To launch KD on the Host, using port 3002 and COM2 instead (here we need the -k switch):
C:\>kd -server tcp:port=3002 -k com:port=com2 -noio
I tend to have a machine set up like this with KD invoked in the registry run section, so that a naive or occasional user can simply turn on the machine and start a debug session, to which a developer can attach from a Remote Host.
Adding Multiple Remote Hosts
You can extend the single Remote Host concept to allow Multiple Remote Hosts to join the debugging session. The scenario I presented above is geared toward the lone developer. Additional developers can join the same debug session in exactly the same way. Figure 6 shows an additional Remote Host on the network.
Figure 6. Adding Another Remote Host.
We can connect to the Real Host is exactly the same as we did in "Adding a Remote Host". You can also connect to the session using the WinDbg GUI.
Using the WinDbg GUI to connect to a Remote Session
Using the GUI to connect to the Remote Host machine is only slightly more complex and arcane than using the command line. We need to set up the Server and Source Path, using two dialogs accessible in the file menu.
Step 1. Select Connect to a Remote Session .
Step 2. Enter the Debug Server and TCP/IP socket .
Step 3. Set up the Source Path .
Adding Multiple Targets to a single Host
Adding a second Target machine is also pretty straightforward. All we need is a second com port and a second TCP/IP socket. Initially we used com1 to connect to the Target, so we will use com2 to connect up the Second Target machine. We will need to invoke a separate WinDbg for each Target. To check that this works we can start up the Host for the COM1 Target using the steps in Review of the Standard Target and Host Setup.
Here are the additional steps to prepare for debugging of a second Target:
C:\>WinDbg -k com:port=Com2
Now we have two debug sessions running on the Host.
-
To enable Remote Hosts to connect to either of these Debug Sessions We need a second TCP/IP socket. Recall that our original Target was assigned the socket 3001. I am going to associate 3002 with the second Target attached to com2.
Shut down the new, local only, debug session and type the following:-
C:\>WinDbg -server tcp: port=3002-k com:port=com2
Now multiple Remote Hosts can connect to either of these debug sessions by specifying the correct TCP/IP socket. This is an ideal scenario for a Test Lab. A single computer can host a debug session for each test machine, using multiple com ports.
Figure 7 shows a basic Multiple Target setup.
Figure 7. Multiple Targets .
Why would I want to do this?
Let's look at a concrete example of the multiple Target setup in action. Let's suppose I have some code that demonstrates an OS specific spin lock. (In fact, I'm using the SPINLOCK sample that accompanies Programming the Microsoft Windows Driver Model Second Edition (Microsoft Press 2003).) If the code is running on XP, it's going to use the more efficient in-stack queued spin lock, otherwise it's going to degrade gracefully and use the old-style spin lock. I am going to prove that this works by stepping through the code simultaneously on windows 2000 and XP.
The Target Machine on the left in Figure 7 is running Windows XP. The machine on the right has Windows 2000 installed. I'll use ports 3001 and 3002 to handle the remote debugging sessions for these machines, respectively.
Let's assume that the debuggers on the Host have been set up and are connected up correctly to com1 and com2. I am going to focus on how we get two debug sessions running on a Remote Host. Here we go....
- Install the driver on both Target machines. In this example, you use the Add Hardware wizard for a fake device whose driver has an INF file.
- Turn off both Target machines.
-
On the Remote Host, run an instance of the debugger for each Target we want to debug.
Port 3001, the XP Target machine on com1:
C:\>windbg -remote tcp:server=\\Jolwhqldev-xp,port=3001 -srcpath \\Sysphus\softshared\symbols\KillerApp\src
Port 3002, the Windows 2000 Target machine on com1:
C:\>windbg -remote tcp:server=\\Jolwhqldev-xp,port=3002 -srcpath \\Sysphus\softshared\symbols\KillerApp\src
bu spinlock!InitializeSpinLockFunctionPointers
g
-
Both debuggers will break when execution reaches the breakpoint we set. We can single step through the code and prove irrefutably that this code will indeed only try to use queued spin locks on Windows XP. figure 8 is a screen shot showing two instances of WinDbg running on my Remote Host, with execution paused at the OS-specific code.
Figure 8. Which Spin Lock ? .
The debugger correctly illustrates the different code paths on the different operating systems.