Implementation principle of leakcanary2.0 (kotlin version) for Android memory leak detection

Time:2021-5-12

This paper introduces the implementation principle of the open source Android memory leak monitoring tool leakcanary version 2.0, and introduces the implementation principle of the new hprof file parsing module, including the hprof File protocol format, part of the implementation source code and so on.

1、 Overview

LeakCanaryIs a very common memory leak detection tool. After a series of changes and upgrades, leakcanary came to version 2.0. The basic principle of memory monitoring in version 2.0 is not different from that in previous versions. The more important change is that version 2.0 uses its own hprof file parser and no longer relies on haha. The language used in the whole tool is also switched from Java to kotlin. In this paper, combined with the source code, the basic principle of memory leak monitoring in version 2.0 and the implementation principle of hprof file parser are briefly analyzed and introduced.

Official link to leakcanary:https://square.github.io/leakcanary/

one point one   The difference between the old and the new

1.1.1  . Access method

New edition:  Just configure it in gradle.

dependencies {
  // debugImplementation because LeakCanary should only run in debug builds.
  debugImplementation 'com.squareup.leakcanary:leakcanary-android:2.5'
}

Old version:  1) Gradle configuration; 2)Application   Initialization in   LeakCanary.install(this)  。

Knock on the blackboard:
1) The initialization of leakcanary2.0 is automatically completed when the app process is pulled up;
2) Initialization source code:

internal sealed class AppWatcherInstaller : ContentProvider() {
 
  /**
   * [MainProcess] automatically sets up the LeakCanary code that runs in the main app process.
   */
  internal class MainProcess : AppWatcherInstaller()
 
  /**
   * When using the `leakcanary-android-process` artifact instead of `leakcanary-android`,
   * [LeakCanaryProcess] automatically sets up the LeakCanary code
   */
  internal class LeakCanaryProcess : AppWatcherInstaller()
 
  override fun onCreate(): Boolean {
    val application = context!!.applicationContext as Application
    AppWatcher.manualInstall(application)
    return true
  }
  //....
}

3) Principle: oncreate of ContentProvider is executed before oncreate of application, so it will be executed automatically when app process is pulled up   AppWatcherInstaller   The oncreate life cycle based on Android can be automatically initialized;
4) expand: ContentProvider’s onCreate method is invoked in the main process, so do not perform time-consuming operations, otherwise it will slow down the startup speed of App.

1.1.2   Overall function

Leakcanary version 2.0 is an open source function module of hprof file parsing and leak reference chain searching (named shark). The following chapters will focus on the implementation principle of this part.

one point two   Overall structure
Implementation principle of leakcanary2.0 (kotlin version) for Android memory leak detection
In leakcanary 2.0, shark is added.

2、 Source code analysis

Steps of leakcanary automatic detection:

  1. Detect the objects that may leak;
  2. Heap snapshot to generate hprof file;
  3. Analyze hprof file;
  4. Classify leaks.

two point one   Detection implementation

There are four types of objects detected automatically

  • Destroyed activity instance
  • Destroyed fragment instance
  • View instance destroyed
  • Cleared ViewModel instance

In addition, leakcanary will also detect  AppWatcher  Monitored objects:

AppWatcher.objectWatcher.watch(myDetachedView, "View was detached")

2.1.1   Leakcanary initialization

Implementation principle of leakcanary2.0 (kotlin version) for Android memory leak detection

AppWatcher.config : It contains switches for monitoring instances such as activity and fragment;

Life cycle monitoring of activity: Registration  Application.ActivityLifecycleCallbacks ;

Lifecycle monitoring of fragment: again, registerFragmentManager.FragmentLifecycleCallbacks , But fragment is more complex, because there are three kinds of fragment, namely android.app.fragment, android.x.fragment.app.fragment, and android.support.v4.app.fragment. Therefore, it is necessary to register the fragmentmanager.fragmentlifecyclecallbacks under their respective packages;

Monitoring of ViewModel: because ViewModel is also a feature under Android x, it relies on monitoring of Android x.fragment.app.fragment;

Monitor the visibility of application: trigger heapdump when it is not visible to check whether there is leakage in the surviving object. If an activity triggers onactivitystarted, the program will be visible. If an activity triggers onactivitystopped, the program will not be visible. Therefore, monitoring visibility is also registered  Application.ActivityLifecycleCallbacks  To achieve.

//Internalappwatcher initialization
fun install(application: Application) {

    ......

    val configProvider = { AppWatcher.config }
    ActivityDestroyWatcher.install(application, objectWatcher, configProvider)
    FragmentDestroyWatcher.install(application, objectWatcher, configProvider)
    onAppWatcherInstalled(application)
  }
 
//Internaleakcanary initialization
override fun invoke(application: Application) {
    _application = application
    checkRunningInDebuggableBuild()
 
    AppWatcher.objectWatcher.addOnObjectRetainedListener(this)
 
    val heapDumper = AndroidHeapDumper(application, createLeakDirectoryProvider(application))
 
    val gcTrigger = GcTrigger.Default
 
    val configProvider = { LeakCanary.config }
    //Asynchronous threads perform time-consuming operations
    val handlerThread = HandlerThread(LEAK_CANARY_THREAD_NAME)
    handlerThread.start()
    val backgroundHandler = Handler(handlerThread.looper)
 
    heapDumpTrigger = HeapDumpTrigger(
        application, backgroundHandler, AppWatcher.objectWatcher, gcTrigger, heapDumper,
        configProvider
    )
    //Application visibility monitoring
    application.registerVisibilityListener { applicationVisible ->
      this.applicationVisible = applicationVisible
      heapDumpTrigger.onApplicationVisibilityChanged(applicationVisible)
    }
    registerResumedActivityListener(application)
    addDynamicShortcut(application)
 
    disableDumpHeapInTests()
  }

2.1.2   How to detect leakage

1) Objectwatcher, the listener of the object
ObjectWatcher   The key code of the system is as follows:

@Synchronized fun watch(
    watchedObject: Any,
    description: String
  ) {
    if (!isEnabled()) {
      return
    }
    removeWeaklyReachableObjects()
    val key = UUID.randomUUID()
        .toString()
    val watchUptimeMillis = clock.uptimeMillis()
    val reference =
      KeyedWeakReference(watchedObject, key, description, watchUptimeMillis, queue)
    SharkLog.d {
      "Watching " +
          (if (watchedObject is Class<*>) watchedObject.toString() else "instance of ${watchedObject.javaClass.name}") +
          (if (description.isNotEmpty()) " ($description)" else "") +
          " with key $key"
    }
 
    watchedObjects[key] = reference
    checkRetainedExecutor.execute {
      moveToRetained(key)
    }
  }

Key class keyedweakreference: weak reference to the combination of WeakReference and ReferenceQueue, referring to the parent class of keyedweakreference

The construction method of WeakReference.
This use can realize that if the object associated with the weak reference is recycled, the weak reference will be added to the queue, and this mechanism can be used to determine whether the object is recycled in the future.

2) Detect retained objects

private fun checkRetainedObjects(reason: String) {
    val config = configProvider()
    // A tick will be rescheduled when this is turned back on.
    if (!config.dumpHeap) {
      SharkLog.d { "Ignoring check for retained objects scheduled because $reason: LeakCanary.Config.dumpHeap is false" }
      return
    }
 
    //Remove unreachable objects for the first time
    var retainedReferenceCount = objectWatcher.retainedObjectCount
 
    if (retainedReferenceCount > 0) {
        //Take the initiative to start GC
      gcTrigger.runGc()
        //Remove unreachable objects for the second time
      retainedReferenceCount = objectWatcher.retainedObjectCount
    }
 
    //Judge whether there are remaining listening objects alive, and whether the number of surviving objects exceeds the threshold
    if (checkRetainedCount(retainedReferenceCount, config.retainedVisibleThreshold)) return
 
    ....
 
    SharkLog.d { "Check for retained objects found $retainedReferenceCount objects, dumping the heap" }
    dismissRetainedCountNotification()
    dumpHeap(retainedReferenceCount, retry = true)
  }

The main steps are as follows

  • Remove unreachable objects for the first time: removeReferenceQueue  Recorded inKeyedWeakReference  Object (refers to the listening object instance);
  • Active trigger GC: Reclaim unreachable objects;
  • Remove unreachable objects for the second time: after a GC, only the objects held by WeakReference can be recycled, so remove them againReferenceQueue  Recorded inKeyedWeakReference   Object;
  • Judge whether there are remaining listening objects alive, and whether the number of surviving objects exceeds the threshold;
  • If the above conditions are met, the hprof file is crawled, and the actual call is Android nativeDebug.dumpHprofData(heapDumpFile.absolutePath) ;
  • Start asynchronousHeapAnalyzerService  Analyze the hprof file to find the leaked gcroot link, which is also the main content of the following.
//HeapDumpTrigger
private fun dumpHeap(
    retainedReferenceCount: Int,
    retry: Boolean
  ) {

   ....

    HeapAnalyzerService.runAnalysis(application, heapDumpFile)
  }

two point two   Hprof   File analysis

Parsing entry:

//HeapAnalyzerService
private fun analyzeHeap(
    heapDumpFile: File,
    config: Config
  ): HeapAnalysis {
    val heapAnalyzer = HeapAnalyzer(this)
 
    val proguardMappingReader = try {
        //Resolving obfuscated files
      ProguardMappingReader(assets.open(PROGUARD_MAPPING_FILE_NAME))
    } catch (e: IOException) {
      null
    }
    //Analyzing hprof files
    return heapAnalyzer.analyze(
        heapDumpFile = heapDumpFile,
        leakingObjectFinder = config.leakingObjectFinder,
        referenceMatchers = config.referenceMatchers,
        computeRetainedHeapSize = config.computeRetainedHeapSize,
        objectInspectors = config.objectInspectors,
        metadataExtractor = config.metadataExtractor,
        proguardMapping = proguardMappingReader?.readProguardMapping()
    )
  }

For the parsing details of hprof files, we need to involve the hprof binary file protocol

http://hg.openjdk.java.net/jdk6/jdk6/jdk/raw-file/tip/src/share/demo/jvmti/hprof/manual.html#mozTocId848088

After reading the protocol document, the binary file structure of hprof is as follows:

Implementation principle of leakcanary2.0 (kotlin version) for Android memory leak detection

Analysis process:
Implementation principle of leakcanary2.0 (kotlin version) for Android memory leak detection

fun analyze(
   heapDumpFile: File,
   leakingObjectFinder: LeakingObjectFinder,
   referenceMatchers: List<ReferenceMatcher> = emptyList(),
   computeRetainedHeapSize: Boolean = false,
   objectInspectors: List<ObjectInspector> = emptyList(),
   metadataExtractor: MetadataExtractor = MetadataExtractor.NO_OP,
   proguardMapping: ProguardMapping? = null
 ): HeapAnalysis {
   val analysisStartNanoTime = System.nanoTime()
 
   if (!heapDumpFile.exists()) {
     val exception = IllegalArgumentException("File does not exist: $heapDumpFile")
     return HeapAnalysisFailure(
         heapDumpFile, System.currentTimeMillis(), since(analysisStartNanoTime),
         HeapAnalysisException(exception)
     )
   }
 
   return try {
     listener.onAnalysisProgress(PARSING_HEAP_DUMP)
     Hprof.open(heapDumpFile)
         .use { hprof ->
           Val graph = hprofheapgraph.indexhprof (hprof, proguardmapping) // create a graph
           val helpers =
             FindLeakInput(graph, referenceMatchers, computeRetainedHeapSize, objectInspectors)
           Helpers. Analyzegraph (// analysis graph
               metadataExtractor, leakingObjectFinder, heapDumpFile, analysisStartNanoTime
           )
         }
   } catch (exception: Throwable) {
     HeapAnalysisFailure(
         heapDumpFile, System.currentTimeMillis(), since(analysisStartNanoTime),
         HeapAnalysisException(exception)
     )
   }
 }

When creating an object instance graph, leakcanary mainly resolves the following tags:

TAG meaning content
STRING character string Character id, string content
LOAD CLASS Loaded classes Serial number, class object ID, stack serial number, class name string ID
CLASS DUMP Class snapshot Class object ID, stack sequence number, parent object ID, class loader object ID, signers   object   ID、protection   domain   object   ID, 2 reserved, object size (byte), constant pool, static field, instance field
INSTANCE DUMP Object instance snapshot Object ID, stack serial number, class object ID, byte of instance field, value of each field of instance
OBJECT ARRAY DUMP Object array snapshot Array object ID, stack serial number, number of elements, array class object ID, ID of each element object
PRIMITIVE ARRAY DUMP Original type array snapshot Array object ID, stack serial number, number of elements, element type, each element
Each gcroot

The gcroot objects involved are as follows:

TAG remarks content
ROOT UNKNOWN Object ID
ROOT JNI GLOBAL Global variables in JNI Object ID, object ID referenced by JNI global variable
ROOT JNI LOCAL Local variables and parameters in JNI Object ID, thread serial number, stack frame number
ROOT JAVA FRAME Java   Stack frame Object ID, thread serial number, stack frame number
ROOT NATIVE STACK In and out parameters of native method Object ID, thread serial number
ROOT STICKY CLASS Viscosity class Object ID
ROOT THREAD BLOCK Thread block Object ID, thread serial number
ROOT MONITOR USED Objects that are called wait() or notify() or synchronized Object ID
ROOT THREAD OBJECT Thread started without stop Thread object ID, thread sequence number, stack sequence number

2.2.1   Build memory index (graph content index)

Leakcanary builds an hprofheapgraph based on the hprof file   Object that records the following member variables:

interface HeapGraph {
  val identifierByteSize: Int
  /**
   * In memory store that can be used to store objects this [HeapGraph] instance.
   */
  val context: GraphContext
  /**
   * All GC roots which type matches types known to this heap graph and which point to non null
   * references. You can retrieve the object that a GC Root points to by calling [findObjectById]
   * with [GcRoot.id], however you need to first check that [objectExists] returns true because
   * GC roots can point to objects that don't exist in the heap dump.
   */
  val gcRoots: List<GcRoot>
  /**
   * Sequence of all objects in the heap dump.
   *
   * This sequence does not trigger any IO reads.
   */
  Val objects: sequence < heapobject > // the sequence of all objects, including class object, instance object, object array and primitive type array
 
  Val classes: sequence < heapclass > // class object sequence
 
  Val instances: sequence < heapinstance > // instance object array
 
  Val objectarrays: sequence < heapobjectarray > // object array sequence

  Val primitivearrays: sequence < heapprimitivearray > // original type array sequence
}

In order to quickly locate the corresponding object in the hprof file, leakcanary provides a memory index hprofinmemoryindex  :

  1. Index stringshprofStringCache(key value): key is the character id and value is the string;

effect:   You can query the character id according to the class name, or you can query the class name according to the character ID.

  1. Index class namesclassNames(key value): key is the class object ID and value is the class string ID;

effect:   Query the class string ID according to the class object ID.

  1. Create instance index * * instanceindex (* * key value): key is the instance object ID, and value is the location of the object in the hprof file and the class object ID;

effect:   Quickly locate the location of the instance to facilitate the analysis of the value of the instance field.

  1. Index class objectsclassIndex(key value): key is the class object ID, and value is the binary combination of other fields (parent class ID, instance size, etc.);

effect:   It can quickly locate the location of class object, and it is convenient to analyze the class field type.

  1. Index object arrayobjectArrayIndex(key value): key is the class object ID, and value is the binary combination of other fields (hprof file location, etc.);

effect:   Quickly locate the position of the object array to facilitate the parsing of the objects referenced by the object array.

  1. Index the original arrayprimitiveArrayIndex(key value): key is the class object ID, and value is the binary combination of other fields (hprof file location, element type, etc.);

2.2.2   Find the leaking object

1) Because the object to be detected is

com.squareup.leakcanary.KeyedWeakReference   So it can be based on

com.squareup.leakcanary.KeyedWeakReference   Query the class name to the class object ID;

2)   Analyze the instance domain of the corresponding class, find the field name and the referenced object ID, that is, the leaked object ID;

2.2.3 find the shortest gcroot reference chain

According to the parsed gcroot object and the leaked object, the shortest reference chain is searched in the graph

//PathFinder
private fun State.findPathsFromGcRoots(): PathFindingResults {
    enqueueGcRoots()//1
 
    val shortestPathsToLeakingObjects = mutableListOf<ReferencePathNode>()
    [email protected] while (queuesNotEmpty) {
      val node = poll()//2
 
      if (checkSeen(node)) {//2
        throw IllegalStateException(
            "Node $node objectId=${node.objectId} should not be enqueued when already visited or enqueued"
        )
      }
 
      if (node.objectId in leakingObjectIds) {//3
        shortestPathsToLeakingObjects.add(node)
        // Found all refs, stop searching (unless computing retained size)
        if (shortestPathsToLeakingObjects.size == leakingObjectIds.size) {//4
          if (computeRetainedHeapSize) {
            listener.onAnalysisProgress(FINDING_DOMINATORS)
          } else {
            [email protected]
          }
        }
      }
 
      when (val heapObject = graph.findObjectById(node.objectId)) {//5
        is HeapClass -> visitClassRecord(heapObject, node)
        is HeapInstance -> visitInstance(heapObject, node)
        is HeapObjectArray -> visitObjectArray(heapObject, node)
      }
    }
    return PathFindingResults(shortestPathsToLeakingObjects, dominatedObjectIds)
  }

1) All gcroot objects are queued;

2) The objects in the queue are out of the queue in order to judge whether the objects have been accessed. If they have been accessed, an exception will be thrown. If they have not been accessed, they will continue;

3) Determine whether the object ID of the team is the object to be detected, record if it is, and continue if it is not;

4) Judge whether the number of recorded object IDS is equal to the number of leaked objects. If the number is equal, the search ends, otherwise, the search continues;

5) According to the object type (class object, instance object, object array object), access the object in different ways, resolve the object referenced in the object and join the queue, and repeat 2).

The queued elements have a corresponding data structure, referencepathnode  , The principle is linked list, which can be used to reverse the reference chain.

3、 Summary

The biggest change of leakcanary2.0 is that it is implemented by kotlin and the hprof parsing code is open source. The general idea is to parse the content of the file into a graph data structure according to the binary protocol of the hprof file. Of course, this structure needs a lot of detailed design. This paper does not cover all aspects, and then traverses the graph to find the shortest path, The path starts with the gcroot object and ends with the leaked object. As for the identification principle of the leaked object, there is no difference with the previous version.

Author: vivo   Internet client team Li   Peidong

Recommended Today

Tcaplusdb | end of May Day, holiday rework

After the relaxed and joyful May 15 holiday, today is another day full of vitality! Turn off the wake-up alarm, drink morning coffee, adjust the state, tcaplus DB people rework~ Long vacation rest body endlessly heart, strive to rise step by step! Tcapsusdb staff quickly switched from “holiday state” to “work mode”, continued to devote […]