Webpack principle series 8: product translation and packaging logic

The full text is 6000 words. Let’s talk about the packaging closed loop. Welcome to like and pay attention to forwarding.

Review the previous article《A little difficult webpack knowledge: deep parsing of dependency graph》I’ve talked about it. AfterBuild phaseAfter, webpack parses:

• modulecontent
• moduleAndmoduleDependency graph between

And enterGenerate（seal）StageAfter that, webpack first calculates the chunk graph according to the dependency of the module, module characteristics, entry configuration, etc., and determines the quantity and content of the final product. This part of the principle is described above《A little difficult knowledge: detailed explanation of webpack chunk subcontracting rules》It is also described in detail in.

This article continues to talk about the process from module translation to module merging and packaging after chunk graph. The general process is as follows:

For ease of understanding, I horizontally divide the packaging process into three stages:

• entrance: refers to starting from webpack to callingcompilation.codeGenerationAll previous pre operations
• Module translation: traversalmodulesArray, complete the translation operation of all modules, and store the results incompilation.codeGenerationResultsobject
• Module merging and packaging: under a specific context framework, combine business modules and runtime modules, merge and package them into bundles, and callcompilation.emitAssetOutput products

What’s said hereBusiness moduleRefers to the project code written by the developer;Runtime moduleIt refers to the runtime code dynamically injected by webpack to support various features after analyzing the business module. In the previous articleWebpack principle Series 6: thoroughly understand webpack runtimeIt has been explained in detail and will not be repeated here.

As you can see, webpack willmodulesTranslate into module products one by one——Module translationThen splice the module products into bundles——Module merging and packaging, we will discuss the principles of these two processes separately according to this logic.

Module translation principle

1.1 INTRODUCTION

Let’s first review webpack products:

The above example is provided byindex.js / name.jsIt consists of two business files. The corresponding webpack configuration is shown in the lower left corner of the above figure; The webpack build product is shown on the rightmain.jsAs shown in the document, it contains three parts, from top to bottom:

• name.jsTranslation product corresponding to module, function form
• Runtime code injected by webpack on demand
• index.jsThe translation product corresponding to the module, Iife (immediate execution function) form

Among them, the function of runtime code and generation logic are described in the previous articleWebpack principle Series 6: thoroughly understand webpack runtimeIt has been introduced in detail; The other two arename.js 、index.jsAfter building the product, you can see that the semantics and functions of the product are the same as those of the source code, but the form of expression has changed greatly, such asindex.jsContents before and after compilation:

On the right side of the figure above is the corresponding code in the webpack compilation product. Compared with the source code on the left, there are the following changes:

• The whole module is wrapped in Iife (immediate execution function)
• add to__webpack_require__.r(__webpack_exports__);Statement to adapt to the ESM specification
• In the source codeimportThe sentence is translated into__webpack_require__function call
• Source codeconsoleStatementnameThe variable is translated into_name__WEBPACK_IMPORTED_MODULE_0__.default
• Add comments

So how do you perform these transformations in webpack?

1.2 core process

Module translationOperation frommodule.codeGenerationCall start, corresponding to the above flowchart:

Summarize the key steps:

• callJavascriptGeneratorObject ofgenerateMethod, method internal:

  • Traversal moduledependenciesAndpresentationalDependenciesarray
  • Execute each array itemdependenyObject’s correspondingtemplate.applyMethod, inapplyModify module code or updateinitFragmentsarray
• After traversing, callInitFragment.addToSourceStatic method, which will be generated by the previous operationsourceObject andinitFragmentsMerge arrays into module products

In short, it is to traverse dependencies and modify them in dependent objectsmoduleCode, and finally merge all changes into the final product. Key points:

• stayTemplate.applyFunction, how to update the module code
• stayInitFragment.addToSourceIn static methods, how toTemplate.applyThe resulting side effects are combined into the final product

The logic of these two parts is relatively complex, which will be explained separately below.

1.3 Template. Apply function

In the above process,JavascriptGeneratorClass is undoubtedly a C-bit role, but it is not directly modifiedmoduleInstead, after several layers of detour, it is entrusted toTemplateType implementation.

In the webpack 5 source code,JavascriptGenerator.generateThe function will traverse the moduledependenciesArray, calling the corresponding array of dependent objectsTemplateSubclassapplyThe method updates the module content, which is a little windy, and the original code is more Rao, so I extract the important steps into the following pseudo code:

class JavascriptGenerator {
    generate(module, generateContext) {
        //First take out the original code content of the module
        const source = new ReplaceSource(module.originalSource());
        const { dependencies, presentationalDependencies } = module;
        const initFragments = [];
        for (const dependency of [...dependencies, ...presentationalDependencies]) {
            //Find the template corresponding to dependency
            const template = generateContext.dependencyTemplates.get(dependency.constructor);
            //Call template Apply, pass in source and initfragments
            //In the apply function, you can directly modify the source content or change the initfragments array to affect the subsequent translation logic
            template.apply(dependency, source, {initFragments})
        }
        // after the traversal is completed, call InitFragment.. Addtosource merges source and initfragments
        return InitFragment.addToSource(source, initFragments, generateContext);
    }
}

//Dependency subclass
class xxxDependency extends Dependency {}

//Dependency subclass对应的 Template 定义
const xxxDependency.Template = class xxxDependencyTemplate extends Template {
    apply(dep, source, {initFragments}) {
        // 1.  Directly operate the source to change the module code
        source.replace(dep.range[0], dep.range[1] - 1, 'some thing')
        // 2.  Supplement the code by adding an initfragment instance
        initFragments.push(new xxxInitFragment())
    }
}

As can be seen from the above pseudo code,JavascriptGenerator.generateLogical comparison of functions:

1. Initialize a series of variables
2. ergodicmoduleObject, find eachdependencyCorrespondingtemplateObject, callingtemplate.applyFunction modify module content
3. callInitFragment.addToSourceMethod, mergesourceAndinitFragmentsArray to generate the final result

The point here isJavascriptGenerator.generateFunctions do not operatemoduleSource code, which only provides an execution framework, and the logic for processing module content translation isxxxDependencyTemplateObjectapplyFunction implementation, lines 24-28 in the pseudo code of the above example.

eachDependencySubclasses are mapped to a uniqueTemplateSubclasses, and usually these two classes will be written in the same file, for exampleConstDependencyAndConstDependencyTemplate；NullDependencyAndNullDependencyTemplate。 The webpack build phase will passDependencySubclasses record the dependencies between modules in different situations; Go to the seal stage and passTemplateSubclass modificationmodulecode.

To sum upModule、JavascriptGenerator、Dependency、TemplateThe four classes form the following interactive relationship:

TemplateObjects can be updated in two waysmoduleCode of:

• Direct operationsourceObject to directly modify the module code. The initial content of the object is equal to the source code of the module. After multipleTemplate.applyAfter the function flows, it is gradually replaced with a new code form
• operationinitFragmentsArray, insert supplementary code fragments outside the module source code

The side effects generated by these two operations will eventually be passed inInitFragment.addToSourceFunction to synthesize the final result. Let’s simply add some details.

1.3.1 change code using source

SourceIt is a set of tool system for editing strings in webpack, which provides a series of string operation methods, including:

• String merging, replacement, insertion, etc
• Module code cache, sourcemap mapping, hash calculation, etc

Many plug-ins and loaders within webpack and in the community will be usedSourceLibrary editing code content, including those described aboveTemplate.applyIn the system, logically, when starting the module code generation process, webpack will first initialize with the original content of the moduleSourceObject, i.e.:

const source = new ReplaceSource(module.originalSource());

After that, it’s differentDependencySubclasses are changed in order and as neededsourceContent, e.gConstDependencyTemplateCore code in:

ConstDependency.Template = class ConstDependencyTemplate extends (
  NullDependency.Template
) {
  apply(dependency, source, templateContext) {
    // ...
    if (typeof dep.range === "number") {
      source.insert(dep.range, dep.expression);
      return;
    }

    source.replace(dep.range[0], dep.range[1] - 1, dep.expression);
  }
};

aboveConstDependencyTemplateIn, the apply function is called according to parameter conditionssource.insertInsert a piece of code, or callsource.replaceReplace a piece of code.

1.3.2 update code with initfragment

Except direct operationsourceOutside,Template.applyYou can also operateinitFragmentsArray to achieve the effect of modifying module products.initFragmentsArray items are usuallyInitFragmentSubclass instances, which usually have two functions:getContent、getEndContent, which are used to get the head and tail of the code fragment respectively.

for exampleHarmonyImportDependencyTemplateofapplyIn function:

HarmonyImportDependency.Template = class HarmonyImportDependencyTemplate extends (
  ModuleDependency.Template
) {
  apply(dependency, source, templateContext) {
    // ...
    templateContext.initFragments.push(
        new ConditionalInitFragment(
          importStatement[0] + importStatement[1],
          InitFragment.STAGE_HARMONY_IMPORTS,
          dep.sourceOrder,
          key,
          runtimeCondition
        )
      );
    //...
  }
 }

1.4 code consolidation

aboveTemplate.applyAfter processing, the translated data will be generatedsourceObjects and code snippetsinitFragmentsArray, and then you need to callInitFragment.addToSourceFunction combines the two into a module product.

addToSourceThe core code of is as follows:

class InitFragment {
  static addToSource(source, initFragments, generateContext) {
    //Order first
    const sortedFragments = initFragments
      .map(extractFragmentIndex)
      .sort(sortFragmentWithIndex);
    // ...

    const concatSource = new ConcatSource();
    const endContents = [];
    for (const fragment of sortedFragments) {
        //Merge fragment Getcontent fetched fragment content
      concatSource.add(fragment.getContent(generateContext));
      const endContent = fragment.getEndContent(generateContext);
      if (endContent) {
        endContents.push(endContent);
      }
    }

    //Merge source
    concatSource.add(source);
    //Merge fragment Getendcontent fetched fragment content
    for (const content of endContents.reverse()) {
      concatSource.add(content);
    }
    return concatSource;
  }
}

As you can see,addToSourceLogic of function:

• ergodicinitFragmentsArrays, merging in orderfragment.getContent()Product of
• mergesourceobject
• ergodicinitFragmentsArrays, merging in orderfragment.getEndContent()Product of

Therefore, the module code merging operation mainly usesinitFragmentsThe array wraps the module code layer by layersource, and bothTemplate.applyLevel maintenance.

1.5 example: custom banner plug-in

afterTemplate.applyTranslation andInitFragment.addToSourceAfter merging, the module completes the transformation from user code form to product form, in order to deepen the understanding of the aboveModule translationNext, we try to develop a banner plug-in to automatically insert a string in front of each module.

In terms of implementation, plug-ins mainly involveDependency、Template、hooksObject, code:

const { Dependency, Template } = require("webpack");

class DemoDependency extends Dependency {
  constructor() {
    super();
  }
}

DemoDependency.Template = class DemoDependencyTemplate extends Template {
  apply(dependency, source) {
    const today = new Date().toLocaleDateString();
    source.insert(0, `/* Author: Tecvan */
/* Date: ${today} */
`);
  }
};

module.exports = class DemoPlugin {
  apply(compiler) {
    compiler.hooks.thisCompilation.tap("DemoPlugin", (compilation) => {
      //Call dependencytemplates to register the mapping from dependency to template
      compilation.dependencyTemplates.set(
        DemoDependency,
        new DemoDependency.Template()
      );
      compilation.hooks.succeedModule.tap("DemoPlugin", (module) => {
        //After the module is built, insert the demodependency object
        module.addDependency(new DemoDependency());
      });
    });
  }
};

Key steps of the sample plug-in:

• to writeDemoDependencyAndDemoDependencyTemplateClass, whereDemoDependencyIt is only used as an example and has no actual function;DemoDependencyTemplateThen in itsapplyCall insource.insertInsert a string, as shown in lines 10-14 of the sample code
• usecompilation.dependencyTemplatesregisterDemoDependencyAndDemoDependencyTemplateMapping relationship of
• usethisCompilationHook acquisitioncompilationobject
• usesucceedModuleHook subscriptionmoduleBuild complete event and callmodule.addDependencyMethod additionDemoDependencyrely on

After completing the above operations,moduleThe product of the object will be called during the generation processDemoDependencyTemplate.applyFunction to insert the string defined by us. The effect is as follows:

Interested readers can also directly read the following documents of webpack 5 warehouse to learn more use cases:

• lib/dependencies/ConstDependency. JS, a simple example, can learnsourceMore ways to operate
• lib/dependencies/HarmonyExportSpecifierDependencyTemplate. JS, a simple example, can learninitFragmentsMore usage of arrays
• lib/dependencies/HarmonyImportDependencyTemplate. JS, a more complex but highly used example, can be learned comprehensivelysource、initFragmentsArray usage

Two. Module merging and packing principle

2.1 introduction

After talking about the translation process of a single module, let’s return to the flow chart:

In the flowchart,compilation.codeGenerationAfter the function is executed – that is, after the module translation phase is completed, the module translation results will be saved tocompilation.codeGenerationResultsObject, and then a new execution process will be started——Module merging and packaging。

Module merging and packagingThe process will insert the module and runtimemodule corresponding to chunk according to the rulesTemplate frameFinally, merge and output into a complete bundle file. For example, in the above example:

In the bundle file on the right side of the example, the part in the red box is the product generated by the user code file and the runtime module, and the rest supports a running framework in the form of Iife, namelyTemplate frame, that is:

(() => { // webpackBootstrap
    "use strict";
    var __webpack_modules__ = ({
        "module-a": ((__unused_webpack_module, __webpack_exports__, __webpack_require__) => {
            // !  Module code,
        }),
        "module-b": ((__unused_webpack_module, __webpack_exports__, __webpack_require__) => {
            // !  Module code,
        })
    });
    // The module cache
    var __webpack_module_cache__ = {};
    // The require function
    function __webpack_require__(moduleId) {
        // !  Implementation of webpack CMD
    }
    /************************************************************************/
    // !  Various runtime
    /************************************************************************/
    var __webpack_exports__ = {};
    // This entry need to be wrapped in an IIFE because it need to be isolated against other modules in the chunk.
    (() => {
        // !  Entry module
    })();
})();

After reviewing the logic here, the operation framework includes the following key parts:

• The outermost layer is wrapped by an Iife
• A recorded DivisionentryOther module codes__webpack_modules__Object whose key is the module identifier; The value is the translated code of the module
• An extremely simplified CMD implementation:__webpack_require__function
• Finally, a packageentryIife function of code

Module translationWill bemoduleTranslated into the form of code that can run on the host environment such as browser; andModule mergingThese operations are connected in seriesmodulesTo make it meet the development expectations as a whole and run the whole application logic normally. Next, we will reveal the generation principle of this part of the code.

2.2 core process

staycompilation.codeGenerationAfter execution, that is, after all user code modules and runtime modules have completed the translation operation,sealfunction callcompilation.createChunkAssetsFunction, triggerrenderManifestHook,JavascriptModulesPluginAfter listening to the hook message, the plug-in starts to assemble the bundle. Pseudo code:

// Webpack 5
// lib/Compilation.js
class Compilation {
  seal() {
    //First translate the codes of all modules and get ready
    this.codeGenerationResults = this.codeGeneration(this.modules);
    // 1.  Call createchunkassets
    this.createChunkAssets();
  }

  createChunkAssets() {
    //Traverse chunks and perform render operation for each chunk
    for (const chunk of this.chunks) {
      // 2.  Trigger rendermanifest hook
      const res = this.hooks.renderManifest.call([], {
        chunk,
        codeGenerationResults: this.codeGenerationResults,
        ...others,
      });
      //Submit assembly results
      this.emitAsset(res.render(), ...others);
    }
  }
}

// lib/javascript/JavascriptModulesPlugin.js
class JavascriptModulesPlugin {
  apply() {
    compiler.hooks.compilation.tap("JavascriptModulesPlugin", (compilation) => {
      compilation.hooks.renderManifest.tap("JavascriptModulesPlugin", (result, options) => {
          //The javascriptmoduleplugin plug-in returns the assembly function render through the rendermanifest hook
          const render = () =>
            //According to the contents of the chunk in render, choose to use the template 'rendermain' or 'renderchunk'`
            // 3.  Listen to the hook and return the packing function
            this.renderMain(options);

          result.push({ render /* arguments */ });
          return result;
        }
      );
    });
  }

  renderMain() {/*  */}

  renderChunk() {/*  */}
}

The core logic here is,compilationwithrenderManifestPublish bundle packaging requirements through hooks;JavascriptModulesPluginListen to this hook and call different packaging functions according to the content characteristics of chunk.

The above is only for webpack 5. In webpack 4, the packaging logic focuses onMainTemplateDone.

JavascriptModulesPluginBuilt in packaging functions include:

• renderMain: used when packaging the main chunk
• renderChunk: it is used to make a package chunk, such as the asynchronous module chunk

The logic of the two packaged functions is similar, and each module is spliced in order. The following is a brief introductionrenderMainImplementation of.

2.3 renderMainfunction

renderMainThe function involves a lot of scene judgment, and the original code is very long and winding. I picked up several key steps:

class JavascriptModulesPlugin {
  renderMain(renderContext, hooks, compilation) {
    const { chunk, chunkGraph, runtimeTemplate } = renderContext;

    const source = new ConcatSource();
    // ...
    // 1.  First calculate the core code of bundle CMD, including:
    //      - "var __webpack_module_cache__ = {};"  sentence
    //      - "__webpack_require__"  function
    const bootstrap = this.renderBootstrap(renderContext, hooks);

    // 2.  Calculate the codes of other modules except entry under the current chunk
    const chunkModules = Template.renderChunkModules(
      renderContext,
      inlinedModules
        ? allModules.filter((m) => !inlinedModules.has(m))
        : allModules,
      (module) =>
        this.renderModule(
          module,
          renderContext,
          hooks,
          allStrict ? "strict" : true
        ),
      prefix
    );

    // 3.  Calculate the runtime module code
    const runtimeModules =
      renderContext.chunkGraph.getChunkRuntimeModulesInOrder(chunk);

    // 4.  Here's the point. Start splicing the bundle
    //4.1 first, merge the core CMD implementation, that is, the bootstrap code above
    const beforeStartup = Template.asString(bootstrap.beforeStartup) + "\n";
    source.add(
      new PrefixSource(
        prefix,
        useSourceMap
          ? new OriginalSource(beforeStartup, "webpack/before-startup")
          : new RawSource(beforeStartup)
      )
    );

    //4.2 merge runtime module code
    if (runtimeModules.length > 0) {
      for (const module of runtimeModules) {
        compilation.codeGeneratedModules.add(module);
      }
    }
    //4.3 merge other module codes except entry
    for (const m of chunkModules) {
      const renderedModule = this.renderModule(m, renderContext, hooks, false);
      source.add(renderedModule)
    }

    //4.4 merge entry module code
    if (
      hasEntryModules &&
      runtimeRequirements.has(RuntimeGlobals.returnExportsFromRuntime)
    ) {
      source.add(`${prefix}return __webpack_exports__;\n`);
    }

    return source;
  }
}

The core logic is:

• First calculate the bundle CMD code, i.e__webpack_require__function
• Calculate the codes of other modules except entry under the current chunkchunkModules
• Calculate the runtime module code

• Start the merge operation. The sub steps are:

  • Merge CMD codes
  • Merge runtime module code
  • ergodicchunkModulesVariable, merge other module codes except entry
  • Merge entry module code
• Return results

Summary: first calculate the product forms of different components, then splice and pack them in order, and output the combined version.

So far, webpack has completed the translation and packaging process of the bundle, and subsequent callscompilation.emitAsset, just output the product to FS according to the context, and the webpack compilation and packaging process is over in a single time.

3、 Summary

This article goes deep into the webpack source code and discusses in detail the implementation logic of the latter half of the packaging process – from the generation of chunk graph to the final output product, focusing on:

• First, traverse all modules in chunk, perform translation operation for each module, and output module level products
• According to the type of chunk, select different structural frames, assemble module products one by one in order, and package them into the final bundle

In retrospect, we:

• In《[ten thousand words summary] understand the core principles of webpack》It discusses the workflow of webpack from front to back in a high degree, so as to help readers have a more abstract understanding of the implementation principle of webpack;
• In《[source code interpretation] in depth explanation of webpack plug-in architecture》The implementation principle of webpack plug-in mechanism is introduced in detail to help readers deeply understand the architecture of webpack and the design of hook;
• In《A little difficult webpack knowledge: deep parsing of dependency graph》The vague language is introduced in detailModule dependency graphConcept to help readers understand the process of dependency discovery and dependency construction in webpack
• In《A little difficult knowledge: detailed explanation of webpack chunk subcontracting rules》The basic logic and implementation method of chunk subcontracting are introduced in detail to help readers understand the principle of product segmentation
• In《Webpack principle Series 6: thoroughly understand webpack runtime》This paper introduces in detail the origin and function of other runtime codes in the bundle except the business module, so as to help readers understand the operation logic of the product
• Finally, it comes to the module translation and merge packaging logic introduced in this paper

So far, the main process of webpack compilation and packaging has been well connected. I believe readers will have a deep understanding and mutual encouragement of front-end packaging and engineering by carefully comparing the source code and learning patiently along the context of this article.