windbg :: analysing a memory leak in w3wp.exe

As the scope of my role increases, I find that I am conducting a lot more deep diagnostics and analysis myself. I find it hard to be the guy that just ‘finds’ issues and then passes it on to the developer to do the digging. The least I could do is give them a little more guidance on where the performance issue exists, right ? Maybe, maybe not, but secretly, I like it.
So, we had this memory leak, it was a small one, but still, a leak is a leak. I managed to run the performance test for 2 days straight, and take quite a few memory/crash dumps along the way to analyse them to see what was leaking. I thought, if I could show the dev’s where the leak was, they could just go and fix it…at least, that was my thought process. The only issue, is that WinDbg is quite a scary tool if you don’t know what you’re doing, so I’ll try and cover 0.1% of using it here.

First thing, you’ll need some crash/memory dumps. I wont go through that process, Google is your friend. Once you have at least two dumps, fire up WinDbg and import the crash dump.
First thing you’ll need to run is

lmDvmclr

This will display the file version of .NET that you’re using. It shows you the Image path of your .NET framework, mine being :

Image path: C:\Windows\Microsoft.NET\Framework64\v4.0.30319\clr.dll

Jump into that directory, and grab a copy of SOS.DLL which is a managed code extension library used for analysis of the memory dump. If you happen to have the same version on your client machine, use the SOS.dll in the C:\Windows\Microsoft.NET\Framework64\v4.0.30319\, otherwise, you can download it from your Web App server.
Simply navigate to the C:\Windows\Microsoft.NET\Framework64\v4.0.30319\ and download the SOS.dll file, place it into the same directory as the DMP file you are analyzing.

Once you have the DLL in the same directory as the dump file, run the following command

.load D:\DumpFiles\SOS.dll

Now you can run a few different things. I like to look at the dump heap, which will give you a list of the objects in the dump, and how much memory is allocated (It will be a big list).

!dumpheap -stat


Another handy one is to look at the Large Object heap (Objects larger than 85,000 bytes or arrays larger than 20,000 items.

!dumpheap -min 85000

You can then compare the two dumps you have (or maybe more) to see if something is growing…
You can also use Debug Diag analysis to compare the dump which gives you a really nice GUI breakdown of the entire dump in a HTML page.

Ok, so now that we have this detail, lets take a look a little deeper. We want to find out which actual object is consuming all of this space.
First I check the heaps, and grab the addresses of them. You can get this info by running the following command:

 !eeheap -gc

That gives you all the heaps, and their addresses:

Number of GC Heaps: 4
------------------------------
Heap 0 (0000000e96050990)
generation 0 starts at 0x0000000e974033a8
generation 1 starts at 0x0000000e973d18f0
generation 2 starts at 0x0000000e96361000
ephemeral segment allocation context: none
segment     begin allocated  size
0000000e96360000  0000000e96361000  0000000e9757ea08  0x121da08(18995720)
Large object heap starts at 0x0000001296361000
segment     begin allocated  size
0000001296360000  0000001296361000  0000001296da4c48  0xa43c48(10763336)
Heap Size:               Size: 0x1c61650 (29759056) bytes.
------------------------------
Heap 1 (0000000e96078f90)
generation 0 starts at 0x0000000f97236c80
generation 1 starts at 0x0000000f97091e08
generation 2 starts at 0x0000000f96361000
ephemeral segment allocation context: none
segment     begin allocated  size
0000000f96360000  0000000f96361000  0000000f97315410  0xfb4410(16466960)
Large object heap starts at 0x00000012a6361000
segment     begin allocated  size
00000012a6360000  00000012a6361000  00000012a65a2890  0x241890(2365584)
Heap Size:               Size: 0x11f5ca0 (18832544) bytes.
------------------------------
Heap 2 (0000000e96081580)
generation 0 starts at 0x0000001096c64768
generation 1 starts at 0x0000001096c2d9e8
generation 2 starts at 0x0000001096361000
ephemeral segment allocation context: none
segment     begin allocated  size
0000001096360000  0000001096361000  0000001096eb1930  0xb50930(11864368)
Large object heap starts at 0x00000012b6361000
segment     begin allocated  size
00000012b6360000  00000012b6361000  00000012b69b2788  0x651788(6625160)
Heap Size:               Size: 0x11a20b8 (18489528) bytes.
------------------------------
Heap 3 (0000000e96089b70)
generation 0 starts at 0x0000001196e60b58
generation 1 starts at 0x0000001196e0d890
generation 2 starts at 0x0000001196361000
ephemeral segment allocation context: none
segment     begin allocated  size
0000001196360000  0000001196361000  0000001196ed4618  0xb73618(12006936)
Large object heap starts at 0x00000012c6361000
segment     begin allocated  size
00000012c6360000  00000012c6361000  00000012c6b60ae8  0x7ffae8(8387304)
Heap Size:               Size: 0x1373100 (20394240) bytes.
------------------------------
GC Heap Size:            Size: 0x536c4a8 (87475368) bytes.

There’s 4 GC heaps, we’re interested in Heap 0, specifically the LOH for that heap. Address 0×0000001296361000
You may not know which one you need, so it may be required to look at the contents of each one, I’ve already done this, but this is the process.
Dump the contents out of the heap to see what’s in it:

!dumpheap 0x0000001296361000

You will get a list of all the objects, and then a summary. I haven’t put the whole list, as its huge, but it looks something like this

............
00000012c6a9daf8 00007ffbfd1a46f0   188210     
00000012c6acba30 0000000e9602cdf0       30 Free
00000012c6acba50 00007ffbfd1a46f0   189084     
00000012c6af9cf0 0000000e9602cdf0       30 Free
00000012c6af9d10 00007ffbfd1a46f0   189098     
00000012c6b27fc0 0000000e9602cdf0       30 Free
00000012c6b27fe0 00007ffbfd1ae840   131096     
00000012c6b47ff8 0000000e9602cdf0       30 Free
00000012c6b48018 00007ffbfd1ae840   101066 
 
Statistics:
              MT    Count    TotalSize Class Name
00007ffbfe3229d0        1       134728 System.Collections.Generic.HashSet`1+Slot[[System.Object, mscorlib]][]
00007ffbfd337860        6       792960 System.Collections.Hashtable+bucket[]
00007ffbfd15ba38       21       826624 System.Object[]
0000000e9602cdf0      135      2444746      Free
00007ffbfd1ae840       17      3143894 System.Byte[]
00007ffbfd1abf40       24      5547536 System.Char[]
00007ffbfd1a46f0       62     15250336 System.String
Total 266 objects

So now I can see that in this heap, I have all of the above objects, their addresses and the count of them. This is identical to what we saw further up in this blog, so I know I’m looking at the right LOH.
Now we need to go deeper, I want to see all of the objects assigned to system.string as I knew that this contained the most amount of memory. There’s 62 objects in there, and I want to see what the actual string is.

With this info, I take the entire list of objects and their addresses, and I throw it in excel, sort by size, and delete all the “Free” objects. Then I start from the largest object, and have a look in it, working my up/down the list.

This is how you dump each object.

!dumpobj 00000012c6b48018

and this is what you can expect to see. I haven’t pasted the whole string, as its huge (536324 bytes to be exact), which is actually very large for a string, but now I can pass this up to the dev’s to show them what’s using up all the space in the LOH.

Name:        System.String
MethodTable: 00007ffbfd1a46f0
EEClass:     00007ffbfd188528
Size:        536324(0x82f04) bytes
File:        C:\Windows\Microsoft.Net\assembly\GAC_64\mscorlib\v4.0_4.0.0.0__b77a5c561934e089\mscorlib.dll
String:      <Case CaseID="2261421" ProcessID="NEWAPPLY............................

That’s as deep as I’ll go now, and whilst this doesn’t give you the stack trace, its alot of insight into the heap and what is holding onto memory.

enjoy!

Leave a Reply

Your email address will not be published. Required fields are marked *

*

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>