.

NET Debugging 101 with Tess Ferrandez November 5, 2009

Hanselminutes is a weekly audio talk show with noted web developer and technologist Scott Hanselman and hosted by Carl Franklin. Scott discusses utilities and tools, gives practical how-to advice, and discusses ASP.NET or Windows issues and workarounds.

Text transcript of show #186 November 5, 2009

.NET Debugging 101 with Tess Ferrandez

Scott's in Sweden this week and he sat down with master debugger and ASP.NET Escalation Engineer Tess Ferrandez. She explains .NET Debugging 101. What's a dump file? Do you need PDBs? How do you use WinDBG and what are the best ways to debug memory issues, perf problems and hangs.

(Transcription services provided by PWOP Productions)

Our Sponsors

http://www.telerik.com

http://www.nsoftware.com

http://dotnet.sys-con.com

Copyright PWOP Productions Inc.

Transcription by PWOP Productions, http://www.pwop.com

Page 1 of 12

.NET Debugging 101 with Tess Ferrandez November 5, 2009

Lawrence Ryan: From hanselminutes.com, i t 's Hanselminutes, a weekly discussion with web developer and technologist, Scott Hanselman. This is Lawrence Ryan, announcing show #186, recorded live Thursday, November 5, 2009. Support for Hanselminutes is provided by Telerik RadControls, the most comprehensive suite of components for Windows Forms and ASP.NET web applications, online at www.telerik.com. In this episode, Scott talks debugging with Tess Ferrandez. Scott Hanselman: Hi, this is Scott Hanselman and this is another episode of Hanselminutes and I'm here in Malmo, Sweden and I have the opportunity to sit down with Tess Ferrandez who is an Escalation engineer with Microsoft and a well-known ASP.NET debugger. Tess Ferrandez: Scott Hanselman: talk to me. Tess Ferrandez: Hi, Scott. Thanks for taking the time to Yeah, thank you.

Scott Hanselman: Tess Ferrandez: Scott Hanselman: Tess Ferrandez:

I see that there's usually... I think you're not alone. Really? Uh-hmm.

Scott Hanselman: I think sometimes people read your blog because they want to feel smart. Tess Ferrandez: Yes, I think that too.

Scott Hanselman: I mean, I know how to hit F5 and I know how to push F9, that's pretty much the extent of my debugging. Tess Ferrandez: Okay.

Scott Hanselman: Back in the day, I could use like SoftICE and do low level debugging, but .NET has worked so well I haven't personally needed to escalate anything. Tess Ferrandez: And I think that's the case for a lot of people. That's probably why they read it and not use it. Scott Hanselman: Hmm.

Scott Hanselman: So you are kind of world reknowned as an ASP.NET debugger. You've got a really great blog and I think you said the blog is called "If broke it is, fix it you should." Tess Ferrandez: Scott Hanselman: Yes. That's the name of your blog.

Tess Ferrandez: Or maybe they read it also because there are still like tips to be learned about how to do things to not run into issues. Scott Hanselman: Yeah, exactly. I know that every once in a while on Vista I would get a blue screen and I know that the DUMP file would appear, memory.dump... Tess Ferrandez: Yeah.

Tess Ferrandez: It's funny because this kind of a Yodaish statement. I've never seen Star Wars. Scott Hanselman: Tess Ferrandez: You've never seen Star Wars? No.

Scott Hanselman: Did you know that it was a Yoda-like statement when you said it? Tess Ferrandez: Kind of.

Scott Hanselman: And I would need to send it to someone smarter than me, usually probably you. Tess Ferrandez: Actually would be a Kernel Dump. Scott Hanselman: Oh, okay. not because that

Scott Hanselman: It sounded something like that character you've never seen before. Tess Ferrandez: Exactly.

Scott Hanselman: I thought it would be cool to kind of talk about kind of Debugging 101 because to be totally honest with you, while I really enjoy your blog and it's kind of like I'm reading a mystery novel... Tess Ferrandez: Scott Hanselman: it is I'm reading. Tess Ferrandez: Yes. I don't always understand what

Tess Ferrandez: And so that's kind of -- Kernel Dump is of the whole OS and the whole system. Scott Hanselman: Okay.

Tess Ferrandez: I work mostly with user mode dump, so dumps of processes. Scott Hanselman: Okay, so this is good. Let's do a little Debugging 101 here. So a Kernel Dump, you said, is a dump of everything.

Okay.

Transcription by PWOP Productions, http://www.pwop.com

Page 2 of 12

.NET Debugging 101 with Tess Ferrandez November 5, 2009

Tess Ferrandez:

Yes.

Scott Hanselman: It's dump of the complete memory of the system. Tess Ferrandez: Scott Hanselman: Exactly. Okay.

Scott Hanselman: Okay, and when I make the user dump, are those things going to be lifted off to disk... Tess Ferrandez: Scott Hanselman: process? Tess Ferrandez: Yeah. And then written out to the Uh-hmm, yes.

Tess Ferrandez: Including all the different processes. So a Kernel Dump contains everything you're seeing like that. Scott Hanselman: So a Kernel Dump then, when it blue screens, that's a complete snapshot in time of what's happening on the computer at that moment that it crashed. Tess Ferrandez: Yes.

Scott Hanselman: So it will reflect out the true kind of the virtual memory that I have. Tess Ferrandez: Yeah and so if you've allocated a .NET habit, it will be in the dump. Scott Hanselman: Okay. Now, I've seen you open up Task Manager and then just right click on a process and say "Create Dump." Tess Ferrandez: Scott Hanselman: to make a dump? Yes. Is that the most common way

Scott Hanselman: Okay and you work with a user mode dump which means that when a process crashes, you have a snapshot in time of that process. Tess Ferrandez: Scott Hanselman: dump file? Exactly. So what's contained within that

Tess Ferrandez: It depends a little bit on how you created them but if you create a mini dump with full options, which is normally called a Full Dump, then it contains basically all the memory that you've committed in the process, everything you've allocated. Scott Hanselman: Okay. So if I have an ASP.NET worker process and I've used up 300 megs of RAM, I'm going to get a 300 meg dump file? Tess Ferrandez: Yeah. So if you allocated and committed 300 megs, yeah. Scott Hanselman: Okay.

Tess Ferrandez: It depends on what you're trying to capture because if you want to capture a snapshot right now like for example a hang or a performance issue and see what it is doing right now, or if you want to capture a memory dump to figure out how much memory you're using, yeah, create full Userdump. If you want to capture dump, for example, on exception, you would have to use another tool, either WinDbg's Companion ADPlus, then configure that to create and get memory dumps on exceptions or in crashes. Scott Hanselman: Tess Ferrandez: Scott Hanselman: Tess Ferrandez: automating CDB. It's called ADPlus? ADPlus, Auto Dump Plus. Okay. Or actually its a VBS file that's

Tess Ferrandez: And what's in a RAM may not really reflect what you're actually using because that's just what is paged into RAM like that. Scott Hanselman: Okay. So you're saying that I've committed it but I haven't filled it up or...? Tess Ferrandez: No. So if you have multiple processes running on the same system... Scott Hanselman: Okay.

Scott Hanselman: Okay, so slow down because we have a lot of listeners hearing a lot of acronyms. Tess Ferrandez: Yes.

Scott Hanselman: So VBS file is Visual Basic Script that's going to go into automation of a what? Tess Ferrandez: CDB. line version of WinDbg. CDB is the command

Tess Ferrandez: Then they're sharing the same RAM and you may not be able to sit everything you have in the process inside your RAM and so some of it is going to be paged out to disk.

Scott Hanselman: Okay. So WinDbg is the Windows debugger that you get when you install the debugging tools for Windows.

Transcription by PWOP Productions, http://www.pwop.com

Page 3 of 12

.NET Debugging 101 with Tess Ferrandez November 5, 2009

Tess Ferrandez: It's a free debugger and it's a native only debugger. It doesn't really understand .NET at all. Scott Hanselman: Okay, really. So WinDbg doesn't understand what you call mix mode or managed code. Tess Ferrandez: Yeah.

Scott Hanselman: And it will be down for as long as it took to create that dump? Tess Ferrandez: Yeah. I mean, it depends a little bit on how big the process is, but we're targeting something like one or two seconds. Scott Hanselman: Tess Ferrandez: Really. Yeah.

Scott Hanselman: And then CDB is the command line debugger version of WinDbg. Tess Ferrandez: Yeah. So WinDbg doesn't look scary enough, there is CDB. Scott Hanselman: Okay. And then CDB is apparently so complicated that I would want to use a Visual Basic Script to automate it, it's what you're saying. Tess Ferrandez: It's not necessarily that it is so complicated. It's that you need to do it so fast that if you had to do it yourself, like for example you would then -- it would be difficult to user to use the script to go in. For example, if you would capture Dump on an exception, you can either choose a set of break point i n t h a t exception and when that happens click something and create a dump or use a script that captures that break. Scott Hanselman: Okay. Now, why would I as a developer want to either right click and create a dump or put in a tool that would capture dump versus going debug attached to process? Tess Ferrandez: Because I don't think you'll have Visual Studio installed on your ASP.NET server. Scott Hanselman: debugging? Sure. What about like remote

Scott Hanselman: So it sounds like that's a really great way for you as an escalation engineer. Tess Ferrandez: Yeah.

Scott Hanselman: It captures a moment in time and really sees what's going on. Tess Ferrandez: Absolutely, yes.

Scott Hanselman: Okay. Do you think that this is a skill that more developers should have? Tess Ferrandez: I'm not really sure because at first I did think that a lot of people should have these skills, but then I'm not sure if you should spend a whole lot of time actually learning this, learning this to a depth where you're debugging like me because this is my job. I think that most people should be able to use, open a simple Dump, run three commands in that, yeah. Scott Hanselman: Okay. Is there a real number of commands, there are three things that people should know? Tess Ferrandez: Yeah. They can choose three commands that are good. Scott Hanselman: Like I know that I felt quite proud of myself when I had a blue screen and then I open -- that was again a Kernel Mode Dump, and I opened up WinDbg and I typed in -- I think it was like, what is it? Bang, Analyze Dash V? Tess Ferrandez: Yeah.

Tess Ferrandez: So still even when you're doing remote debugging or when you're attaching, you'll stop the process, and if you have a site attached to the app, then you can just go in and stop the process and at your leisure look at some values and continue. Scott Hanselman: Okay. So using it in production, I go and attach a debugger to this process, it's really going to have that things tied up completely. Tess Ferrandez: Yeah.

Scott Hanselman: And that's like the standard I don't know what I'm doing but figure it out the best you can. Tess Ferrandez: Yes.

Scott Hanselman: But you're saying that I can right click, create dump in production and the process will just keep running. Tess Ferrandez: Yes.

Scott Hanselman: And then WinDbg will give you a general sense of what it thinks might have happened, and that was the extent of my abilities at WinDbg. Tess Ferrandez: That is good because when you run, specifically when you have a crash, a Kernel Mode crash, if you run and analyze, actually what it

Transcription by PWOP Productions, http://www.pwop.com

Page 4 of 12

.NET Debugging 101 with Tess Ferrandez November 5, 2009

will do is it will look out where you are and then it will also match-up with bucket, like Dr. Watson bucket set at Microsoft. Scott Hanselman: So let's dig into that a little bit b e c a u s e y o u 're speaking from a Microsoft perspective. Doctor Watson was a little tool that I think lived in Windows. Is it 3.1 or did it start, you know, as a little picture...? Tess Ferrandez: with error reporting. Yeah. I think it was Exchange

Scott Hanselman: repeated bug... Tess Ferrandez:

Do you look and see if this is a Yes.

Scott Hanselman: Like if they have a bucket, you go, "Oh, we already know about this." Tess Ferrandez: Uh-hmm, not necessarily from Dr. Watson's buckets, but yeah, from our own bucket system, yeah. Scott Hanselman: So yeah, I know that the team looks at these buckets and says, "Well, look, a thousand people are having this particular problem. We should really dig into that." And they would go and they would debug it. Tess Ferrandez: And if they do dig into that, then the next time you have an issue like that, you're going to say, hey, there's a reported problem, go ahead and download this driver and... Scott Hanselman: Right, and in Windows 7.0 that's what that action center says. Tess Ferrandez: Yeah.

Scott Hanselman: Yeah. It's a little man with -- it's a little like iconic icon of this little doctor with a stethoscope listening into a process. Tess Ferrandez: Okay.

Scott Hanselman: And Dr. Watson is the kind of what we say when -- we actually use it as a verb inside of Microsoft like "Oh no, my process has watsoned." Tess Ferrandez: Yeah.

Scott Hanselman: And then it goes and it sends up an error report to a service like a Cloud service in the sky, and how does it just decide this bucket idea? I've seen this notion of a bucket idea before. Tess Ferrandez: So it looks at the stack that crashed and then it compares it to other crashes that have happened. Scott Hanselman: Tess Ferrandez: will put it in the when, let's say, dumps for the investigated. So though. Okay.

Scott Hanselman: It says, "Oh, we know about this. Go click here and we'll fix it." Tess Ferrandez: Uh-hmm.

And if it fits the same stack, it same bucket. Then for example 10 people or more have sent in same bucket, then it will be I'm guessing 10, that might not be

Scott Hanselman: It's really kind of cool. So how many understands the difference between debug versus released mode, because I know a lot of people and I don't think they want to admit it but maybe the listeners would admit it to themselves who will put a lot of debug code into production because the perception is that a .NET application that's i n production, that's in release mode is hard to debug. Tess Ferrandez: Okay.

Scott Hanselman: All right, to some large numbers. Because I understand that Dr. Watson and this notion of shared buckets of identical stacks was a big part of what made Windows 7.0 a much better operating system as they really put some statistical analysis in this and I suppose that... Tess Ferrandez: Yeah.

Scott Hanselman: So they just say forget it, we'll just put this in debug mode and we'll leave it there. Tess Ferrandez: Okay. trade-offs

Scott Hanselman: What are the between debug versus release mode?

Scott Hanselman: Is it true that in your job you're looking at these kinds of things and reporting to the ASP.NET team? Tess Ferrandez: I'm not necessarily looking at Watson-buckets, that's not really what our team does. Our team works on people call in and have issues, and then we'll report back for example if they have bugs in ASP.NET itself. Transcription by PWOP Productions, http://www.pwop.com

Tess Ferrandez: So I would say because the release mode is optimized and debug mode is not in most cases, if you would actually go in with a .NET debugger like Visual Studio, yeah, it may not pop up the exact right line of code. You might be off by a couple of lines or two. But are you really going to do that? Are you really going to attach Visual Studio to it?

Page 5 of 12

.NET Debugging 101 with Tess Ferrandez November 5, 2009

Scott Hanselman:

Probably not in production.

Scott Hanselman: If you have hundreds of pages, you can have hundreds of little, tiny assemblies. Tess Ferrandez: Scott Hanselman: one. Tess Ferrandez: Yes. In a release mode, you just get Uh-hmm. Aha.

Tess Ferrandez: Yeah and for tools like WinDbg or taking memory dumps, you don't need that. So the only reason I could see, if you're really, really concern about line numbers and things, is if you yourself is recording a stack and reporting back code lines or whatever. Scott Hanselman: Uh-hmm.

Scott Hanselman: Tess Ferrandez: That doesn't matter, I don't see much of any reason to have debug mode on the server. For ASP.NET, it's even worse because a debug mode kind of implies that you want to let a .NET debugger attach to it and be able to debug it for him. Scott Hanselman: Right.

Tess Ferrandez Or rather if you have debug equals false in a web.config. Scott Hanselman: Okay.

Tess Ferrandez: So there's a bit of a difference if you say debug release mode versus debug equals true release. Scott Hanselman: That's a very interesting point. So you're saying that the dropped down list in Visual Studio is switching between debug and release as one thing. Tess Ferrandez: debug equals true. That has nothing to do with

Tess Ferrandez: And because of that timeouts are not honored because if they... Scott Hanselman: What does that mean?

Tess Ferrandez: Like for example, if you set an ASP.NET timeout, default timeout is like 1-1/2 minute for any .aspx page. Scott Hanselman: Okay. So if a page takes too long, it will shut it down. Tess Ferrandez: Yeah. You know, you'll see it like page timeout or whatever. That does not happen if you have debug mode. Scott Hanselman: Oh no.

Scott Hanselman: That tells the compiler for your code-behind pages what to do. Tess Ferrandez: Yeah, uh-hmm.

Scott Hanselman: And then saying debug equals true or false in the compiler section of the web.config. It tells the... Tess Ferrandez: compiler. Scott Hanselman: treat those pages. Tess Ferrandez: JIT debugger -- or sorry, JIT

Tess Ferrandez: And the reason of course is you attach to the debugger, the account has it checked out. Scott Hanselman: for 20 minutes. Right, you could be debugging

JIT compiler, how it should Yes. Interesting.

Tess Ferrandez: Yeah. So that doesn't happen. You don't optimize the code and that may or may not be a big performance issue for, you know, most sites really don't have that type of performance like it needs. Scott Hanselman: Right.

Scott Hanselman:

Tess Ferrandez: But one thing that does make a big difference is that if you run in debug mode, you'll create one assembly per .aspx page so they'll all be... Scott Hanselman: created. Tess Ferrandez: Also many assemblies get Yeah.

Hey everybody, this is Scott coming at you from another place and time. No doubt you probably bump into testing tasks now and then in your work and you know writing functional test is probably not your favorite thing. It's kind of difficult. It takes time and the results can be dubious. Well, get ready to start liking tests, thanks to Telerik. With the new WebAii testing framework, building web automation tests is a breeze. You've got code automation with advance ASP.NET AJAX and Silverlight applications. You can write a single test, have it execute against multiple browsers at once. You benefit from rich API, there's LINQ support, integration with Visual Studio unit testing, also NUnit, xUnit, and MbUnit, not to mention

Transcription by PWOP Productions, http://www.pwop.com

Page 6 of 12

.NET Debugging 101 with Tess Ferrandez November 5, 2009

the free wrappers for a Telerik RadControl for ASP.NET AJAX and Silverlight all shipping with Telerik's new testing tool. One of its best features, the WebAii testing framework, which is developed by ArtOfTest, is absolutely free. If you're already hooked on WebAii testing framework, start using it right away. Go to www.telerik.com for more info. Thanks a lot. So it sounds like putting debug mode in web is just a bad idea altogether. Tess Ferrandez: Scott Hanselman: to do that. Tess Ferrandez: Yeah. There's really no good reason

Tess Ferrandez: Scott Hanselman:

Yes. And then what do you do?

Tess Ferrandez: So first off, like you said go ahead and look at what you're using it for, or what objects are on the heap and everything, and then for each of them you can use -- in WinDbg, you can use commands like GC Roo and that's a command from the SOS extension. Scott Hanselman: Strike. Tess Ferrandez: Okay. So SOS is called Son of Yeah.

No, none really. Scott Hanselman: Which really means nothing. This is the funny thing about Microsoft... Tess Ferrandez: Scott Hanselman: Tess Ferrandez: Scott Hanselman: Tess Ferrandez: example. Scott Hanselman: apologize. It does mean something. Does it really? Yeah. Oh, tell me please. So let me just get back to that

Scott Hanselman: But if something bad happens other than really good logging and tracing, I'm pretty much limited to dumps done at that point. Am I not? Tess Ferrandez: Yeah.

Scott Hanselman: So what are some of the classifications of things that could go wrong in ASP.NET that could cause me to want to take a dump? I see that's funny, the way I say that "take a dump." That makes everybody laugh. Tess Ferrandez: Yeah. In fact, yesterday I was called down on that and they were like please say "take a memory dump or create a memory dump." So most of the cases I get are either high memory usage, performance issues, or my process crashed, or I have this really weird exception that I have not clearly quite have. Scott Hanselman: Okay. So memory issues, this is mostly leaks like I'm using too much memory? Tess Ferrandez: Yeah. So really in .NET there's like this feeling that I should not be able to use up too much because I have the garbage collector. Scott Hanselman: Right.

Oh

yeah,

that's

okay.

Tess Ferrandez: So you would run a command to figure out why it's still being referenced so that's what you would do to look at... Scott Hanselman: Also it will tell you, so you say I've got these objects... Tess Ferrandez: Scott Hanselman: Tess Ferrandez: Yeah. I want to find out why. Who is still referencing them.

Tess Ferrandez: Well, the garbage collector will only collect garbage but it doesn't collect things that -well, how does it know it's garbage, you're not using it anymore, you're not referencing it anymore. Scott Hanselman: Right.

Scott Hanselman: It will actually walk it back and tell you who allocated that. Tess Ferrandez: Absolutely, yeah.

Scott Hanselman: And when I say who, is that an object or is that a call set? Tess Ferrandez: No, it's an object so it's either a static object that has a reference to something that has a reference to something. Scott Hanselman: Okay.

Tess Ferrandez: So most memory issues in .NET applications are really you're referencing things that you may not even know that you're referencing. Scott Hanselman: Okay. So that means that if you took a memory dump and look at it, you would find a lot of objects of a certain type and then you'd say why do I have a million strings. Transcription by PWOP Productions, http://www.pwop.com

Page 7 of 12

.NET Debugging 101 with Tess Ferrandez November 5, 2009

Tess Ferrandez: Like for example, a static object has a reference to the cache. In your cache, you have such invariables, your such invariables are holding on to this object. Scott Hanselman: Tess Ferrandez: stack. Ah, I see. So you'll see the object not to

Scott Hanselman: Tess Ferrandez: Scott Hanselman:

Right, the codename for... Yeah. Or the core runtime.

Tess Ferrandez: Exactly. So they have this debugging called Debugging Extension called the Strike as in Strike the lightning. Scott Hanselman: Oh, a lightning strike, okay.

Scott Hanselman: When I see these things on your blog, I see small abbreviation commands, like you said it was a GC Root to command it. Tess Ferrandez: Yeah.

Tess Ferrandez: Yeah and this is kind of an offspring of that so this is Son of Strike. Scott Hanselman: Okay and now you say SOS. Yeah. Okay, interesting. It's not just a call for help. That was a long time ago. Yeah. That was like nine years ago. Yeah, I know. How long have you worked for A long time.

Scott Hanselman: Are there dozens of these or hundreds of these things I have to memorize? I mean, I'm trying to take it from Analyze Dash V to get somewhere in your direction of ability. Tess Ferrandez: Okay. Some would say basically there maybe five commands that you really want to know, three or five, like you want to know what the stacks of the threads are, that's one command that's called CLR stack. Scott Hanselman: Okay.

Tess Ferrandez: Scott Hanselman: Tess Ferrandez: Scott Hanselman: Tess Ferrandez: Scott Hanselman: Tess Ferrandez: Scott Hanselman: Microsoft? Tess Ferrandez:

Tess Ferrandez: You want to know the object someday that are in memory, the Dump keeps that. Scott Hanselman: Okay.

Tess Ferrandez: And you want to know, for example, looking at the specific object, dumped object, and then GC Root to figure out who's holding on to memory, and that's really like with those commands if you don't want to go any deeper, you'll get a lot, lot of information just using them. Scott Hanselman: So using those five commands will probably get me a lot of what I need. Tess Ferrandez: Scott Hanselman: Tess Ferrandez: may. Yeah. Okay. So that's memory.

Scott Hanselman: Wow. So the memory thing makes sense. The idea that I can take a snapshot in time and I can wander around the snapshot and I can see these objects and figure that out, that makes sense to me. But on a Perf, using a dump a moment in time to identify a Perf problem, how do I do that? Tess Ferrandez: Well, you can either just look and see when it comes down to a Perf issue it's basically what am I doing and why is it taking so long. Scott Hanselman: Okay.

So back to Son of Lightning if I Tess Ferrandez: That's really what it comes down to. So looking at the threads like to figure out what they're doing, so what the .NET calls actually are, and preferably if you see it's the Perf issue, if something is taking a long time, then maybe taking a couple of memory dumps during that time to figure out if things are moving along or not or if things are really locked. Scott Hanselman: I see. Now, if something is "taking a long time," a r e n 't there kind of two categories of what could be causing that? One would

Scott Hanselman: Oh yes, I'm sorry. So this is an extension to WinDbg. Tess Ferrandez: Scott Hanselman: Yes. Okay.

Tess Ferrandez: So the name comes from in the very beginning the CLR was called lightning, or that was the working name for the CLR, for the core...

Transcription by PWOP Productions, http://www.pwop.com

Page 8 of 12

.NET Debugging 101 with Tess Ferrandez November 5, 2009

b e I 'm waiting for something in the sense that if there's a concurrency issue, we're in line. Tess Ferrandez: the Idle Hang. So that's basically what we call

Scott Hanselman: It's really is a shame. A profiler is just an amazing thing in .NET that you instrument so much. Tess Ferrandez: Yeah, absolutely. In Visual Studio, there are -- even in 2008, my feeling is that it has gotten even better in 2010 in the way profiling works. There is, for Busy Hangs, you would want to do the CPU profiling. Scott Hanselman: Uh-hmm.

Scott Hanselman: Okay, Idle Hang. So a bunch of threads are waiting for one thread to let them in. Tess Ferrandez: Yeah, or an external resource or something like that. Scott Hanselman: And then I would think that the other one would be I have an inefficient algorithm and I think that would be difficult to tell it from... Tess Ferrandez: Yeah, so that's pretty much where you have to take multiple dumps to see maybe how many durations you're doing in this particular algorithm, or some of the more common ones with Busy Hangs are high CPUs and garbage collection. When you allocate a lot of large objects or if you do make a lot of full garbage collections, that causes high CPU and garbage collection, or something like using efficient regular expressions. Scott Hanselman: So an Idle Hang is waiting in line for someone to let me in. Tess Ferrandez: Yeah.

Tess Ferrandez: And for Idle Hangs, you would want to do the instrumented profiling. So the CPU profiling basically goes in and looks at -- every now and then it goes in and looks at the stack and see where you're spending most of your time, whereas instrument profiling is more elapsed time, so how long it took for a person who punched into. Scott Hanselman: Now, I could be spending my time in my own code, or I could be spending my time in Microsoft code, and when I'm looking at a call stack, if I've got no -- correct me if I'm wrong here, if I've got no symbols I'm just going to see offsets in my stack. Tess Ferrandez: Yeah, for the native like for the stuff that's not your code. Or rather, use the native even for a .NET code. You don't need symbols. Scott Hanselman: So this is the part I'm confused about. What do I need, and what is a PDB for? Because I've heard you say this before, that I don't need symbols. Tess Ferrandez: Yeah.

Scott Hanselman: And then a Busy Hang is I'm just doing a lot of work that I don't need to be. Tess Ferrandez: Yeah.

Scott Hanselman: And I could be doing a Busy Hang that is my code wait doing a lot of work. Tess Ferrandez: Yes.

Scott Hanselman: But what are these PDBs for? Do I need them or not? Tess Ferrandez: Yes. So for natives, a non.NET stuff, you do need them because there's nothing in that method or nothing in that object that tells you what the function name is. So if you don't have symbols, the symbols basically is a database that says this address means this function. Scott Hanselman: Okay.

Scott Hanselman: Or it could be indirect in the sense that I'm making a lot of work for .NET. Tess Ferrandez: Exactly. So I would say if you could reproduce a performance issue, then you're better off doing some kind of profiling in Visual Studio for example, but if you can't reproduce it on a test machine, then you would have to take some of... Scott Hanselman: So it's a Busy Hang that only happens in production, then I would want to do multiple dumps. Tess Ferrandez: Scott Hanselman: profilers. Tess Ferrandez: Yeah. It's funny how few people use Oh, yeah.

Tess Ferrandez: And the debugger goes in and says, okay, I saw him at this address, where would I be? Oh, at this function plus some offset. Scott Hanselman: Okay.

Tess Ferrandez: For a .NET code, you don't need that because that's all built-in to metadata in the objects themselves. Scott Hanselman: at all in .NET? Okay. So do I not need PDBs

Transcription by PWOP Productions, http://www.pwop.com

Page 9 of 12

.NET Debugging 101 with Tess Ferrandez November 5, 2009

Tess Ferrandez: Scott Hanselman:

No. What are they created for?

Tess Ferrandez: Yeah. Oh, there is one but I'm wondering do they have things for example System.Web. Scott Hanselman: I think they do. Okay.

Tess Ferrandez: Well, they're not needed for debugging with WinDbg, but for example for debugging with Visual Studio they'll need them because they contain information about filenames, where your code is like my numbers, where dysfunction is, that kind of stuff. Scott Hanselman: Tess Ferrandez: I see. Okay. Parameters.

Tess Ferrandez:

Scott Hanselman: I think if I go tools options and I go into the debugger section, there's a series of checkboxes and I can go and say use the public symbol server. Tess Ferrandez: Yeah, but I'm thinking like, for example, for WinDbg, I don't recall if they showed these -- they do show the complaints? Scott Hanselman: Tess Ferrandez: I'm pretty sure that they do. Okay, yeah.

Scott Hanselman: So on managed code, I need the PDBs not only for line numbers and filenames but for offsets to method name in mapping. Tess Ferrandez: Yeah and depending on what PDBs you have, like if you have private or public ones, they may or may not contain typed names, parameter names, that kind of stuff. Scott Hanselman: Tess Ferrandez: Name of locals.

Scott Hanselman: And I think what it does is, at least for the mix mode stuff, is it will bring down the PDBs for System.Web and for all those different things. Tess Ferrandez: Okay, yeah.

Yeah. Scott Hanselman: And then you're saying that with the symbol server, it doesn't just contain .NET stuff. It contains... Tess Ferrandez: It contains native.

Scott Hanselman: Okay. So I could be in my code, or I could be in Microsoft's code, it seems like now that I can look at the Microsoft's source code for .NET framework, while I've got the function names of... Tess Ferrandez: Yeah.

Scott Hanselman: You know, like I could be in a regular expression, I could be in System.RegEx or something like that, I still would like to get the line numbers then. So I want to hook up a symbol server, is that correct? Tess Ferrandez: Scott Hanselman: Yes. To bring those PDBs down?

Scott Hanselman: So if I get down into ntdll.dll or something low level Kernel32, it will bring those down to it and that gives me the benefit of having a call stack that shows me method names all the way up the stack, not just to a point. Tess Ferrandez: Scott Hanselman: Exactly, yes. Okay.

Tess Ferrandez: Yeah and .NET symbols are usually not up there so that's mostly for native. Scott Hanselman: Okay. So where do I get those symbols? Are those already on my system? I'm pretty sure that they are up there actually. Tess Ferrandez: so then I... God, I've never -- yeah, okay,

Tess Ferrandez: And the symbol server basically is, and so if you have your own symbols you already maybe has put them in directory and set up that as part of your symbol pack. Scott Hanselman: And that was my next question. If I run a large enterprise and we get this at my last job, I want to set up a share such that when I have a continuous integration server that's doing my builds, I want to copy all those PDBs over there. Tess Ferrandez: Yeah. So there's a difference like you can either put them in directory and then it will be recognized by name. But for a symbol server, the symbol server is basically kind of like a folder structure where in my time, you know, multiple PDBs for MS core works or for ntdll depending on the

Scott Hanselman: I think that there's a Microsoft Public Symbol server.

Transcription by PWOP Productions, http://www.pwop.com

Page 10 of 12

.NET Debugging 101 with Tess Ferrandez November 5, 2009

operating system or depending on framework version of the thing. Scott Hanselman: Right, and there's a certain structure that those folders need to be. They include a hash of the files... Tess Ferrandez: Yeah.

switch to Visual Studio 2010 Beta 2.0, you can't look at .NET based on a heap. Scott Hanselman: Okay.

Tess Ferrandez: But it's got a very nice way of showing like the stacks moving to source code and with the recording of that to... Scott Hanselman: Uh-hmm, for Channel 9. We'll put up a video of this for Channel 9 to demonstrate that, but in a nutshell for Busy Hangs and Idle Hangs... Tess Ferrandez: Yeah. your beautiful

Scott Hanselman: So that you can have multiple versions side-by-side. Tess Ferrandez: Uh-hmm, and that's so that you can debug dumps from everywhere on the same machine or debug processes with different framework versions and stuff like that and you can set that up for your own company too. You can set up symbol server that works like if you have different sources. Scott Hanselman: So you can have multiple pass that you can search. Tess Ferrandez: Yes.

Scott Hanselman: This gives visualization. I think it's a WPF...

Tess Ferrandez: Yeah, so you just double it up with Visual Studio, it looked at something called parallel stacks that will show you all the stacks of all the threads and yeah, it's awesome. I can't even describe it in words, yeah. Scott Hanselman: I mean, this seems like a large portion from what you've showed me and again we'll have a link up on Channel 9 with the Hanselminutes on 9 section. This is going to make a huge chunk of your job, less text mode... Tess Ferrandez: Scott Hanselman: Tess Ferrandez: stick with WinDbg. Yes. And more graphical. I would say for me I'll probably

Scott Hanselman: You can search the Microsoft Symbol Server and then your own. Tess Ferrandez: Yeah, absolutely.

Scott Hanselman: Where is a good place for you to get like start with debugging if they're in IIS and ASP.NET basically? Tess Ferrandez: Yes, so apart from WinDbg, there's also this tool called DebugDiag. DebugDiag 1.1 is the one that's out now and that's actually released with a 64-bit version. Scott Hanselman: Okay, cool.

Scott Hanselman: Really? Why? Because this is all amazing. You showed me this, isn't it? Tess Ferrandez: Scott Hanselman: mode? Yeah. But is it the speed of text

Tess Ferrandez: What it does is it tells you not only to gather memory dumps in a very easy way where you can configure it to take dumps on certain exceptions or crashes or whatever, but it also helps you analyze memory dumps. So it has a hang analyzer, it has a memory analyzer, you can write your own scripts. So I wrote a script on my blog, blogs.msdn.com/tess, where it's a script for debugging but not memory leaks. That shows the most common things and we use it basically to open your dumped app with that script and this gives you information you need without even opening Windump. Scott Hanselman: Now I understand that there are some improvements though with specifically run dump files in 2010 that's going to make some of these things, maybe not obsolete, but certainly less often used. Tess Ferrandez: Yeah, absolutely. So not so much in the memory space. You can't completely

Tess Ferrandez: Yeah. It's the speed and it's the fact that sometimes you need to go in and look at native objects that you can't really access to and... Scott Hanselman: You know, what would be interesting would be to figure if you could, because you can create extensions now to Visual Studio, if you could write something that would bring that WinDbg command line in... Tess Ferrandez: Scott Hanselman: Yeah. And do a split screen.

Transcription by PWOP Productions, http://www.pwop.com

Page 11 of 12

.NET Debugging 101 with Tess Ferrandez November 5, 2009

Tess Ferrandez: So you can write a lot of the commands, for example, for SOS, you can use them in the immediate window in Visual Studio. Scott Hanselman: Really? And are there stuff on your blog about how to use SOS in the immediate window? Tess Ferrandez: Scott Hanselman: Tess Ferrandez: that a while back. Scott Hanselman: stuff. Tess Ferrandez: Yeah, uh-hmm. Very cool. So I think you wrote a post on I may have. I've written a lot of Maybe a couple of years ago.

Scott Hanselman: Cool, yeah. Well, your blog is a great research for this kind of stuff. You've got all sorts of stories about adventures in debugging, you've got scripts and resources that people can check out. Tess Ferrandez: And labs if you want to try debugging hangs, or memory leaks, or things of that sort. Scott Hanselman: Right and I've noticed that you did this recently. You put up a series of labs where we can download a dump and we can go through it and try to learn ourselves. Tess Ferrandez: website that has it... Scott Hanselman: Tess Ferrandez: I actually downloaded a

That will make a dump. Yeah.

Scott Hanselman: Yeah, yeah. Very cool. Well, thank you so much, Tess, for sitting down with me. Tess Ferrandez: Well, thank you.

Scott Hanselman: Cool, and be sure to check out Tess's blog at blogs.msdn.com/tess. I'll put links for all of that, as well as the tools that we've talked about up on the show site. This has been another episode of Hanselminutes and I'll see you again next week.

Transcription by PWOP Productions, http://www.pwop.com

Page 12 of 12