A memory leak analysis of a. Net his back end service


1: Background

1. Tell a story

The day before yesterday, that his elder brother came to see me again. Last time, because ofCPU burstI’ve solved the problem. It seems that I trust you very much. This time another program encountered a memory leak. I hope I can help diagnose it.

In fact, this elder brother’s technology is still very good. Since he can give me dump, it’s really a very difficult problem. I have to be prepared. After communication, the memory of the program will slowly expand until it self destructs. The problem is just like this. Next, I will sacrifice my housekeeping tool WinDbg.

2: WinDbg analysis

1. Where did it leak?

As I have said in many previous articles, when encountering this kind of memory leak, the first thing to do is to find out what the problem isManaged heapstillUnmanaged heapWhat’s your problem? If it’s the latter, most of the time you have to raise your hand to surrender, because the water is too deep… Don’t look at the casesAllocHGlobalMethod to allocate unmanaged memory, and then use! The pediatrics that heap went to look for, the reality is much more complicated than this…

Next, use it first!address -summaryTake a look at the committed memory of the current process.

0:000> !address -summary

--- Usage Summary ---------------- RgnCount ----------- Total Size -------- %ofBusy %ofTotal
Free                                    345     7dfd`ca3ca000 ( 125.991 TB)           98.43%
                             37399      201`54dbf000 (   2.005 TB)  99.83%    1.57%
Heap                                  29887        0`d179b000 (   3.273 GB)   0.16%    0.00%
Image                                  1312        0`0861b000 ( 134.105 MB)   0.01%    0.00%
Stack                                   228        0`06e40000 ( 110.250 MB)   0.01%    0.00%
Other                                    10        0`001d8000 (   1.844 MB)   0.00%    0.00%
TEB                                      76        0`00098000 ( 608.000 kB)   0.00%    0.00%
PEB                                       1        0`00001000 (   4.000 kB)   0.00%    0.00%

--- Type Summary (for busy) ------ RgnCount ----------- Total Size -------- %ofBusy %ofTotal
MEM_MAPPED                              352      200`00a40000 (   2.000 TB)  99.57%    1.56%
MEM_PRIVATE                           67249        2`2cbcb000 (   8.699 GB)   0.42%    0.01%
MEM_IMAGE                              1312        0`0861b000 ( 134.105 MB)   0.01%    0.00%

--- State Summary ---------------- RgnCount ----------- Total Size -------- %ofBusy %ofTotal
MEM_FREE                                345     7dfd`ca3ca000 ( 125.991 TB)           98.43%
MEM_RESERVE                           11805      200`22ae8000 (   2.001 TB)  99.60%    1.56%
MEM_COMMIT                            57108        2`1313e000 (   8.298 GB)   0.40%    0.01%

From the perspective of hexagram, the process submits memoryMEM_COMMIT = 8.2GThen let’s look at the managed heap size and use the!eeheap -gcOrders.

0:000> !eeheap -gc
Number of GC Heaps: 1
generation 0 starts at 0x0000027795928060
generation 1 starts at 0x000002779572F0D0
generation 2 starts at 0x000002763DCE1000

Total Size:              Size: 0xcd28c510 (3442001168) bytes.
GC Heap Size:    Size: 0xcd28c510 (3442001168) bytes.

As you can see from the last line, the current GC heapSize= 3442001168 /1024/1024/1024 =3.2GIn other words:8.2G - 3.2G = 5GI lost my memory… Nima, typicalUnmanaged memory leakIf you don’t know which pot to open, you may have to plant it…

2. Look for unmanaged memory leaks

In addition to GC heap, there is also a loader heap in the process. There are many things in it, such as high-frequency heap, low-frequency heap, stub heap, JIT heap, etc., which store AppDomain, module, method descriptor, method table, eeclass and other related information. From experience, this loader heap is for investigationUnmanaged leakagePriority places, to view, can use!eeheap -loaderOrders.

0:000> !eeheap -loader
Module 00007ffe2b1b6ca8: Size: 0x0 (0) bytes.
Module 00007ffe2b1b7e80: Size: 0x0 (0) bytes.
Module 00007ffe2b1b9058: Size: 0x0 (0) bytes.
Module 00007ffe2b1ba230: Size: 0x0 (0) bytes.
Module 00007ffe2b1bb408: Size: 0x0 (0) bytes.
Module 00007ffe2b1bc280: Size: 0x0 (0) bytes.
Module 00007ffe2b1bd458: Size: 0x0 (0) bytes.
Module 00007ffe2b1be630: Size: 0x0 (0) bytes.
Module 00007ffe2b1bf808: Size: 0x0 (0) bytes.
Module 00007ffe2b1f0a50: Size: 0x0 (0) bytes.
Module 00007ffe2b1f1c28: Size: 0x0 (0) bytes.
Module 00007ffe2b1f2aa0: Size: 0x0 (0) bytes.
Total size:      Size: 0x0 (0) bytes.
Total LoaderHeap size:   Size: 0xc0fb9000 (3237711872) bytes total, 0x5818000 (92372992) bytes wasted.

It’s a good thing that the command didn’t lose. I was shocked when I lost. It took several minutes for the WinDbg interface to stop… Two information can be obtained from the output

  • Total occupation of loader heap:3237711872 /1024/1024/1024 = 3.01G

  • There are a lot of modules generated, I estimate there are tens of thousands…

In order to satisfy my curiosity, I decided to write a small script to see how many modules are there???

Let me go. There are 19 w modules. No wonder they take up more than 3 gigabytes. I feel that they are not far away from the truth. The next question is what are these modules and where do they come from???

3. Find the source of module

To find the source, you can think about it carefully. The nesting relationship of modules should be as follows:Module -> Assembly -> AppdomainSo checking AppDomain may give us more information. Next, use!DumpDomainExporting all the application domains of the current process takes a few minutes, ah… The screenshot is as follows:

It can be seen from the figure that there are a lot ofDynamicType of assembly, you must want to ask what does this mean? Yes, this is the Assembly created dynamically by the code, which is as high as 19W… The next problem to be solved is: how are these assemblies created???

4. Export module content

Old readers should know how I export the problem code from the module. Yes, I’m looking for the startaddress of the module. Here I choose one of the modules: 00007ffe2b1f2aa0.

2:2:152> !dumpmodule 00007ffe2b1f2aa0
Name: Unknown Module
Attributes:              Reflection SupportsUpdateableMethods IsDynamic IsInMemory 
Assembly:                000002776c1d8470
BaseAddress:             0000000000000000
PEFile:                  000002776C1D8BF0
ModuleId:                00007FFE2B1F2EB8
ModuleIndex:             00000000000177CF
LoaderHeap:              0000000000000000
TypeDefToMethodTableMap: 00007FFE2B1EE8C0
TypeRefToMethodTableMap: 00007FFE2B1EE8E8
MethodDefToDescMap:      00007FFE2B1EE910
FieldDefToDescMap:       00007FFE2B1EE960
MemberRefToDescMap:      0000000000000000
FileReferencesMap:       00007FFE2B1EEA00
AssemblyReferencesMap:   00007FFE2B1EEA28

I’ll go. It’s unfortunate that baseaddress has no address. That means that you can’t export the module. It’s right to think about it. After all, it’s dynamically generated. Maybe people who write code can’t figure out what is in the module? Is there really no way? But as the saying goes, there is no way out!dumpmoduleThere is a MT (method table) parameter in the command, which is used to display the types in the current module. This is a major clue.

||2:2:152> !dumpmodule -mt 00007ffe2b1f2aa0 
Name: Unknown Module
Attributes:              Reflection SupportsUpdateableMethods IsDynamic IsInMemory 
Assembly:                000002776c1d8470

Types defined in this module

              MT          TypeDef Name
00007ffe2b1f3168 0x02000002 
00007ffe2b1f2f60 0x02000003 

Types referenced in this module

              MT            TypeRef Name
00007ffdb9f70af0 0x02000001 System.Object
00007ffdbaed3730 0x02000002 Castle.DynamicProxy.IProxyTargetAccessor
00007ffdbaec8f98 0x02000003 Castle.DynamicProxy.ProxyGenerationOptions
00007ffdbaec7fe8 0x02000004 Castle.DynamicProxy.IInterceptor

You can see that there are two definitions in the moduletype, all of which have their method table addressesmtIn exchange formd(method descriptor) to get the final module content.

It’s finally clear here that this old man used castle to make an AOP function. It should be that he didn’t use AOP correctly, resulting in the generation of AOP19w +It’s no wonder that the memory will eventually explode… The root is finally found. How to modify it???

5. Modify Castle AOP problem code

I’m embarrassed. After all, I really haven’t played castle, but it’s the old rule. Go to Bing and have a lookThe end of the worldThere are some articles about memory leak caused by Castle AOPCastle Windsor Interceptor memory leakThe solution is also provided. The screenshot is as follows:

Quickly throw this link to my brother, I feel that I can only help him here, the rest can only see the nature.

3: Summary

It’s really nature. I finished the self-test that night.

I asked my brother how to change it. I didn’t hesitate to release the source code. As expected, I will change it according to the foreigner’s suggestionProxyGeneratorSet it to static… Otherwise, there will be a new assembly. Take a look at the code before the change. The screenshot is as follows:

After solving these two difficult problems, do you feel like you are going to send me a small trophy?

More high quality dry goods: see my GitHub:dotnetfly