Exploration of. Hmap files derived from meituan’s article “a tool that can improve the compilation speed of large IOS projects by 50%” (Part I)

Time:2021-12-3
Exploration of. Hmap files derived from meituan's article

preface

Some time ago, Ben Xin read an article by meituan: “a tool that can improve the compilation speed of large IOS projects by 50%. He was surprised at the title. Why? Because of itIt can prompt the compilation speed by 50% and is not realized through component binarization, our dailyImprove compilation speedIs toComponents are compiled into binary files and imported into the project。 Based on the principle of understanding what is unclear, let’s see how to achieve it.

explore

Compilation time-consuming reason

In the project, we will introduce header files, such as the following figure:The header file of person is introduced in viewcontroller

Exploration of. Hmap files derived from meituan's article

In usImport header fileAt the time of the introduction ofName of header file personSo how does Xcode find the actual location of the person file? That’s what we need to mentionHeader search path configured in the project

Exploration of. Hmap files derived from meituan's article

XcodestaycompileoftimemeetingRead the address of header search path, andSplicingFuck usImported header file name

Which means we importedHeader fileDivideTwo parts

  • 1.First halfThe file directory where the header file is located
  • 2.Second halfHeader file name

This is why when we set header search path,Just set the directory where the header file is locatedYes.

Question: because weThere are many documents in the project, then we willSet many directories in header search pathBut for finding us aboveIntroduce a header file person, he needsFind traverse allofFile directory, come onFind this class。 thisThe process has more and more classes with the projectlookupoftimeWillLonger and longer, yesMore and more time-consuming。 For example, our project has hundreds of components and tens of thousands of classes, so the time-consuming of this process is obvious.

terms of settlement

We know the reason why the project compilation takes time, so how to solve this problem? The answer given in meituan’s article isUsing hmap

hmap

What is hmap? Meituan article said itIt is the entity of header map, similar to aForm of key valueKey valueyesHeader fileofnameValueyesHeader fileofActual physical pathActually, this thingIt always exists, but we didn’t notice.

  • Think about it,for the first timeWhen you run a project or compile, you will findVery slowBut oncefunctionperhapsCompilation succeededAfter,RecompileperhapsfunctionWillSoon, why not?
  • actuallyAfter the first compilationXcodeJustmeetingHelp usGenerate some. Hmap filesRecompileWhen willDirect usetheseFind the. Hmap file quicklyCorrespondingHeader fileTherefore, the compilation speed will be much faster
Exploration of. Hmap files derived from meituan's article

We seegenerateYesMany. Hmap filesXcodeyesGenerate by categoryofArrow fingerWhat’s important isOur main project worksof. hmap file, if weClean up Xcode, so these. hmap fileAlsoBe clearedAnd then we’ll find out,Compilation is slowHe got up.

Through the above explanation, we know.hmapActuallyIt's a container, itinsidesurecontainYesPersonofFile directory, then it willGive WayWeXcodestayFind personofHeader fileTimeFasterCome on, then there’s a problem, ourselvesHow to generate. Hmap filesAnd?.hmapofBottom structureWhat is it?

Explore. Hmap files

We compile a project, check the compilation process, and find the viewcontroller. M file

Exploration of. Hmap files derived from meituan's article

I useEnclosed in red [], we can see that it uses-I parametergoAn. Hmap file is introduced, we also know aboveXcodemeetingGenerate multiple. Hmaps, in order for everyone to understand, we needRead the next. Hmap file

. hmap file structure analysis

Look at the project directory first

Exploration of. Hmap files derived from meituan's article

Let’s take another lookThe. Hmap generated by this projectWhat is the file format

Exploration of. Hmap files derived from meituan's article

We found this insideContains all. H in the project, let’s take a look.hmapExactly?What kind of data structure is it

  • data structure

We can find relevant content through llvm

Exploration of. Hmap files derived from meituan's article

We saw aThe structure is called hmapaheader, there’s another oneThe structure is called hmapbucket, there are two sentences in the red box: 1An hmapbucket object array of numbuckets follows this header file。2.There is a string following hmapbucket, which is in stringsofset

From the above, we can guess the structure of. Hmap

Exploration of. Hmap files derived from meituan's article

  • 1.The top hmapaheader records some necessary information
  • 2.In the middle of the hmapbucket, there will be as many hmapbuckets as there are header files, which will be packaged into hmapbuckets
  • 3.The string contains the first half of the path of the header file and the second half of the class name

Process: PassRead hmapaheaderGet the number of buckets saved by. Hmap, that’s it. hmap saveYesHow many?individualHeader file path, andSave in bucketThisThe offset of the header file in the following stringThen you canFrom the bottom string
inRead the path to the header file

First of all, as a developer, it is particularly important to have a learning atmosphere and an exchange circle. This is my IOS development exchange group:130 595 548, whether you are Xiaobai or Daniel, welcome to settle in. Let’s make progress and develop together! (the group will provide some free learning books and materials collected by the group owners and hundreds of interview questions and answer documents for free!)

Read. Hmap file

How do we read the. Hmap information? aboveFrom llvmIn usFind information about the structure of the hmap, is there any access related content in llvm?

  • Up there we knowStructure informationyesUnder Lex fileIf yes, is the read information also in lex
  • Finally I found oneHeaderMapTestoffile, the feeling isTest files for headermap
Exploration of. Hmap files derived from meituan's article

We areError reading hmap, need to useUpper structure

Let’s use the information obtained by llvm to write an examplePlug in for reading headermap(we write in the main file)

Hmap read

We write the following code in the main function:

  • Assertion macro
Exploration of. Hmap files derived from meituan's article

HMAP_HeaderMagicNumberyesString flipBecauseIn the hmapaheader structureinThere is an attribute magic to represent byte order, which meansIf the current magic = hmap_ SwappedMagic, which meansThe byte order is reversedYes, that’s itThe next byte order needs to be swapped

  • 2. Parameter judgment abnormal file
Exploration of. Hmap files derived from meituan's article

When the parameter is less than two(indicating that nothing was passed on) it’s timeDeemed invalidof

  • 3. Normal documents
Exploration of. Hmap files derived from meituan's article

Loop to export header map through dump method

Dump method

This method I amUse CTo write, because it feels likeC is more convenient when dealing with fetching files

Exploration of. Hmap files derived from meituan's article

What came in wasFile path

  • 1.Resolution path
Exploration of. Hmap files derived from meituan's article

analysisofPath length is less than 0Description pathabnormal

  • 2.Get the mapheader size and judge
Exploration of. Hmap files derived from meituan's article

GetMapheader size, if<0beDescription mapheader exception, ifLess than the actual mapheader size, thenreadofData exception

  • 3.Judge whether the string is flipped and read the header
Exploration of. Hmap files derived from meituan's article

  • 4.Gets the number of buckets
Exploration of. Hmap files derived from meituan's article

  • 5.Gets an array of buckets(pointer offset)
Exploration of. Hmap files derived from meituan's article

  • 6.Get string list(pointer offset)
Exploration of. Hmap files derived from meituan's article

  • 7.Traversal get bucket, thenTake out the bucketofprefixandSuffix splicing
Exploration of. Hmap files derived from meituan's article

We’ll take one upRead the code of. Hmap and write itNow, put the. Hmap code of the previous project into the project directory, and then set it in the following figure

Exploration of. Hmap files derived from meituan's article

Run project, break point

  • 1. Main function breakpoint
Exploration of. Hmap files derived from meituan's article

The first isCurrent executableofroute, the second isThe. Hmap path you just configured

  • 2. Check the number of barrels
Exploration of. Hmap files derived from meituan's article

Print yes16 barrels, butNot all header addresses(due toData alignment(reason for)

  • 3. View print data
Exploration of. Hmap files derived from meituan's article

The string table has 9 dataThere are 16 buckets

  • 4. View results
Exploration of. Hmap files derived from meituan's article

summary

Through the above reading and printing, we can confirm the following points:

  • 1. As mentioned above. hmap is a key value formKey is the header file name
  • 2.Prefix saves the first half of the header file path
  • 3.Suffix saves the second half of the header file path(header file name)
  • 4.. hmap is a stack of header files stored according to corresponding rules

It also proves that our conjecture above is correct

extend

The code written above can generate a tool. We putTool addTo ourLldb execute commandIn this way, we don’t need to read the. Hmap file in the above way, and we canThe terminal uses the same command to read

Exploration of. Hmap files derived from meituan's article

Generate your own. Hmap file

It saysxcodeYou canactiveHelp usGenerate. Hmap file, then why do we have to write it ourselves? As stated in meituan’s article, let me briefly say here:

  • 1. Our projects generallyManage third parties through cocoapodsFor example, the swift project I have nothing to write before introduces the following third-party libraries
Exploration of. Hmap files derived from meituan's article

  • 2. We found thatHeader file in the form of #import "classA. H", willHit. Hmap fileOtherwise, the relevant path will be found through header search path
Exploration of. Hmap files derived from meituan's article

The problem of directory has been mentioned aboveIt will find a header file in multiple directories, which is time-consumingWell, if I putPut a file path into a. Hmap file, then go backMuch faster。 At this time, ifMore components and third parties are introduced, thenIt will inevitably lead to slow compilation speed

Write code to generate your own. Hmap file

This part is also a difficulty. I also checked the above-mentionedHeadermaptest.cpp file in llvm, I looked at the code carefully and found that there were someGenerate. Hmap code, I wrote it myselfThe code is relatively simple, just toExplain how. Hmap is generatedof

  • 1. Introduction above. hmap fileSpeaking of, insideContains many buckets, so we have toMr. Cheng bucket
Exploration of. Hmap files derived from meituan's article

Create mapfile container maker, maker containsMapfiles one by one, that isBucketMapfile is a structure(as in headermaptest.cpp, whereHow many buckets does 8 represent750 is the size of the generated buffer

  • 2. Core codeClass nameandPath in bucketofForm preservation
    • Method overview

      Exploration of. Hmap files derived from meituan's article

*Addstring method
Exploration of. Hmap files derived from meituan's article

*Addbucket method
Exploration of. Hmap files derived from meituan's article

The above methods areFound in headermaptest.cpp of llvm

  • 3. Export the file to the specified location
    • Method overview

      Exploration of. Hmap files derived from meituan's article

*Getbuffer method
Exploration of. Hmap files derived from meituan's article

  • 4. Operation project
Exploration of. Hmap files derived from meituan's article

Generated aTestapp.hmap file, let’s goreadDownload this file to see if it is the same as that generated by Xcode

  • 5. Read the generated testapp.hmap
Exploration of. Hmap files derived from meituan's article

And Xcode generated. Hmap

Exploration of. Hmap files derived from meituan's article

We foundgenerateofresultyesequally, let’s use this self generated. Hmap

  • 6. Use self generated. Hmap
Exploration of. Hmap files derived from meituan's article

Set use header maps to no,Set the header search paths path to the. Hmap path we generated。 becauseThe project written is too small to measure much difference

summary

The reading and writing methods of. Hmap are described above. After reading it, you will have a clear understanding of. Hmap,The idea of solving the compilation speed of meituan article is worth learning, I’m up thereGenerate. HmapofmethodactuallyUnable to landYes, just to tell youHow to generate an. HmapMeituan said in the articleofCocoapods hmap prebuilt plug-in, IPersonal feeling is a scriptTraversal header file script。 It saysThe. Hmap method cannot be generated, if you let itCan land, yesWrite a scriptgoTraversal projectas well asCoapods managementofThird party LibraryofHeader fileExtract header fileUse itofmethod, finallygenerateOne. hmap file, soTo land。 This part is also used as a technical exploration. We will share the results later

supplement

. hmap has been implemented. You can see the next updated article “the implementation of. Hmap file (Part 2) hmap introduced by meituan article” a tool that can improve the compilation speed of large IOS projects by 50% “. The plug-in link will be put at the end of the article