Introduction

Extending Visual Studio with managed code can be challenging. This is due in large part to the fact that Visual Studio is a COM application written in native code and its interactions with managed code happen through .NET/COM interoperability. Because of this, you can experience some really weird bugs. And by “really weird”, I mean the wacky, hair-pulling kind of bugs that have driven lesser programmers insane. Debugging these kinds of problems is difficult because, unless you set up Visual Studio appropriately, only the managed code debugger is used when run your managed extensibility project. Without the native code debugger, the vast native code base of Visual Studio remains a complete mystery and, without that knowledge, hair-pulling bugs can only be solved through trial-and-error.

This article gives an overview of how to achieve a rich-debugging experience when Visual Studio itself is the debuggee. By the time we’re through, you'll be debugging Visual Studio in mixed-mode (with both native and managed code debuggers), getting full call stacks with symbols from the IDE itself and watching exceptions that occur in the IDE but are never marshaled across to the managed world. With these tools, you will gain insights that will lead to better-informed solutions to any hair-pulling bugs that you're struggling with.

Prerequisite Setup

Before venturing into the world of native code debugging you’ll need to make sure that you have set Visual Studio up properly.

First, Visual C++ must be installed to enable native code debugging. You must have a Professional SKU (or higher) to install Visual C++. If you have a Standard or Express SKU of Visual Studio, this isn’t an option. If you chose not to install C++ when you installed Visual Studio, do so now. Just insert your installer media and it should allow you to modify your existing installation to include C++. Note that, if you aren’t planning to do any C/C++ development, you don’t need any of the extra libraries (e.g. MFC, ATL, etc.) so you can uncheck all of the sub-features and just install the bare-bones C++ language service.

Second, you should install the Microsoft Debugging Tools for Windows. These tools are necessary if you’re doing crazy low-level development (e.g. Windows driver development) and need to debug Windows at the kernel-level. However, we’re only interested in one file: symsrv.dll. After installing the Debugging Tools, browse to the install directory on your hard drive and copy symsrv.dll to the Visual Studio IDE directory, replacing the version of symsrv.dll that is already there. (NOTE: The Visual Studio .NET 2003 IDE directory is {Program Files}\Microsoft Visual Studio .NET 2003\Common7\IDE by default and the Visual Studio 2005 IDE directory is {Program Files}\Microsoft Visual Studio 8\Common7\IDE by default.) Symsrv.dll will allow Visual Studio to interact with the latest version of the Microsoft Symbol Server.

Setting up Debug Symbols

The next step is to set up Visual Studio to use Microsoft’s symbol server to download any .PDB files that it needs and to cache them on disk.

For Visual Studio .NET 2003:

1. Load any solution or create a new empty solution or project.
2. In the Solution Explorer, right-click on the solution node and choose “Properties” from the context menu.
3. In the Properties dialog, browse to the “Common Properties\Debug Symbol Files” page.
4. On the Debug Symbol Files page, click the “New Line” button (with the little folder icon). This will create a new entry in the “Search these paths for symbol files” box. For the entry, type “srv*C:\SymbolCache*http://msdl.microsoft.com/download/symbols” (without quotes). “C:\SymbolCache” is simply the directory on your hard drive where download symbols (.PDB files) will be downloaded and cached for later. You can name this to something else if you like.
5. Click the OK button to accept your changes and close the dialog.

Sometimes Visual Studio .NET 2003 doesn’t seem to keep these settings and you must add them again. Fortunately, in Visual Studio 2005, this feature is much more accessible.

For Visual Studio 2005:

1. Select “Tools | Options…” from the main menu.
2. In the Options dialog, browse to the “Debugging\Symbols” page.
3. On the Symbols page, click the “New Line” button (with the little folder icon). This will create a new entry in the “Symbol file (.pdb) locations” box. For the entry, type “http://msdl.microsoft.com/download/symbols” (without quotes)
4. In the “Cache symbols from symbols servers to this directory” box, type “C:\SymbolCache”. NOTE: You can change this to cache symbols in a different location on your hard drive.
5. Click the OK button to accept your changes and close the dialog.

Attaching to an Already-Running Visual Studio Process

At this point, everything should be set up and ready for debugging. Let’s give it a try by attaching the debugger to another instance of Visual Studio.

1. Close all instances of Visual Studio.
2. Open two instances of Visual Studio.
3. In one of the instances, choose “Tools | Attach to Process…” from the main menu (choose “Tools | Debug Processes…” in Visual Studio .NET 2003).
4. Choose the other Visual Studio process for debugging:
   a. In Visual Studio .NET 2003, choose the “devenv.exe” process and click the “Attach…” button. This will bring up the “Attach to Process” dialog. In that dialog, check the boxes next to “Common Language Runtime” and “Native”, and click the OK button. When the dialog closes (might” take some time), click the Close button on the Processes dialog.
   b. In Visual Studio 2005, choose the “devenv.exe” process and click the “Select” button. This will bring up the “Select Code Type” dialog. In that dialog, select the “Debug these code types” radio button and check the boxes next to “Managed” and “Native”, and click the OK button. When the dialog closes, click the Attach button on the “Attach to Process” dialog.
5. You might have to wait a long time after attaching to the second Visual Studio process as Visual Studio downloads the necessary symbol files from the symbol server and stores them on your hard drive. Once it is finished, you can choose “Debug | Windows | Modules” to display all of the loaded modules and ensure that symbols have been loaded for them. (NOTE: Symbols won’t be loaded for every module but they should be loaded for most of them. Check the “devenv.exe” process in the Modules window. If symbols are loaded for “devenv.exe”, you have everything set up properly.)
6. Hopefully, the symbols were loaded properly. If so, go ahead and press Ctrl+Alt+Break in the debugger to Break All. This will break Visual Studio in the middle of its normal message loop. If you bring up the Call Stack window, you should see something like this:

This is totally awesome.

If you are an x86 assembler junkie, bring up the Disassembly window to see that the disassembled instructions now include method names as well.

Exceptions

When dealing with the Visual Studio Exceptions dialog (Ctrl+Alt+E), most developers are only concerned with children of the Common Language Runtime Exceptions node. The sad truth is that, other than the Managed Debugging Assistants (MDAs) that were introduced in Visual Studio 2005, none of the other exception types have any effect in the Managed code debugger. However, now that we can load both the Native and Managed code debuggers simultaneously, we can use all of the exception types in this dialog when debugging Visual Studio.

Trapping C++ Exceptions, Native Run-Time Checks, or Win32 Exceptions can be extremely useful if your extensibility project is pushing Visual Studio in ways that it might not like to be pushed. Sometimes, an exception can be thrown and handled in Visual Studio’s code but not surfaced to your managed code. In these cases, it is possible for Visual Studio to be in a state that your code might not be expecting and seeing the exception for yourself is the only way to really solve the problem.

What’s Next?

The simple techniques presented in this article are extremely powerful and can help you to create better and more-informed code that interacts with Visual Studio. However, they are not limited to Visual Studio. I’ve used the same tricks with other Microsoft products (anyone working with Visual Studio Tools for Office?). Also, we haven’t exhausted all of the possibilities that can be achieved when debugging Visual Studio. If you have any other tips or tricks, feel free to contact me at blog@diditwith.net.