How to implement a bundler packaging mechanism of webpack?

Time:2021-5-7

preface

How to implement a bundler packaging mechanism of webpack?

I think these two years should be the most obvious period when “webpack” has been impacted. The previous “snowpack” is based on native browserES ModuleIt is proposed that the rapid development of “vite” standing on the shoulder of “vue3” is really the wave behind pushing the wave ahead, the wave ahead

In addition, “vite” is the main implementation technologyIt’s not a little bit newThe typical point is to use “esbuild” as the interpreter of “typescript”, which is different from that in the current communityMost packaging toolsIt’s different.

In the next article, I will introduce what esbuild is and the value it brings.

However, although the wave is really strong, at least in the past two years, the status of “webpack” is still very strongNot to be shakenYes. Therefore, a better understanding of webpack related principles can enhance our personal competitiveness.

So, back to today’s topic, let’s implement a webpack from scratchBundlerPackaging mechanism.

1 bundler packaging background

BundlerPackaging background, what is it?BundlerPackaging means that we can pass modular code throughBuild module dependency graphParsing codeExecution codeAnd a series of means to aggregate modular code intoExecutable code

In normal development, what we often use isES ModuleReference between modules in the form of. So, in order to achieve aBundlerTo package, we are going to give an example:

catalogue

|—— src
    |-- person.js
    |-- introduce.js
    |--Index.js # entry
|-- bundler.js # bundler packaging mechanism

code

// person.js
export const person = 'my name is wjc'
// introduce.js
import { person } from "./person.js";

const introduce = `Hi, ${person}`;
export default introduce;
// index.js
import introduce from "./introduce.js";

console.log(introduce);

Apart frombundler.jsIn addition, we have created three files, which are referenced between modules respectively, and will be used in the endBundlerThe packaging mechanism parses and generates executable code.

Next, we’ll do it step by stepBundlerPackaging mechanism.

2 single module analysis

BundlerIn the first step, we need to know the code in each module, and then do dependency analysis and code transformation to ensure the normal execution of the code.

First, from the entry fileindex.jsTo start, get the contents of the file (code)

const fs = require("fs")

const moduleParse = (file = "") => {
  const rawCode = fs.readFileSync(file, 'utf-8')
}

After getting the code of the module, we need to know which modules it depends on? At this time, we need to use two toolsbabelNew tools:@babel/parserand@babel/traverse. The former is responsible for transforming the code into an abstract syntax tree ast, while the latter can build dependencies based on the references of modules.

@babel/parserParse the code of the module into “abstract syntax tree ast”:

const rawCode = fs.readFileSync(file, 'utf-8')
const ast = babelParser(rawCode, {
  sourceType: "module"
})

@babel/traverseAccording to the reference identifier of the moduleImportDeclarationTo build dependencies:

const dependencies = {};
traverse(ast, {
  ImportDeclaration({ node }) {
    const dirname = path.dirname(file);
    const absoulteFile = `./${path
      .join(dirname, node.source.value)
      .replace("\", "/")}`;
    dependencies[node.source.value] = absoulteFile;
  },
});

Here, we go through@babel/traverseHere’s the entranceindex.jsDependent modulesdependenciesMedium:

// dependencies
{ './intro.js' : './src/intro.js' }

But at this pointastThe code in is still initialES6So we need the help of@babel/preset-envTo turn it intoES5Code:

const { code } = babel.transformFromAst(ast, null, {
  presets: ["@babel/preset-env"],
});

index.jsConverted code:

"use strict";
var _introduce = _interopRequireDefault(require("./introduce.js "));
function _interopRequireDefault(obj) { 
  return obj && obj.__esModule ?
    obj : {
        "default": obj
    };
}
console.log(_introduce["default"]);

At this point, we have finished the taskAnalysis of single moduleThe complete code is as follows:

const moduleParse = (file = "") => {
  const rawCode = fs.readFileSync(file, "utf-8");
  const ast = babelParser.parse(rawCode, {
    sourceType: "module",
  });
  const dependencies = {};
  traverse(ast, {
    ImportDeclaration({ node }) {
      const dirname = path.dirname(file);
      const absoulteFile = `./${path
        .join(dirname, node.source.value)
        .replace("\", "/")}`;
      dependencies[node.source.value] = absoulteFile;
    },
  });
  const { code } = babel.transformFromAst(ast, null, {
    presets: ["@babel/preset-env"],
  });

  return {
    file,
    dependencies,
    code,
  };
};

Next, we start to build the module dependency graph.

2. Build module dependency graph

As we all know, the packaging process of “webpack” will build a module dependency graph. Its formation is nothing more than starting from the entry file, entering the module through its reference module, continuing the parsing of a single module, and repeating the process. The general logic diagram is as follows:

How to implement a bundler packaging mechanism of webpack?

Therefore, at the code level, we need to start from the entry file and call themoduleParse()Parse it, and then traverse to get its corresponding dependencydependencies, and callmoduleParse()

const buildDependenceGraph = (entry) => {
  const entryModule = moduleParse(entry);
  const rawDependenceGraph = [entryModule];
  for (const module of rawDependenceGraph) {
    const { dependencies } = module;
    if (Object.keys(dependencies).length) {
      for (const file in dependencies) {
        rawDependenceGraph.push(moduleParse(dependencies[file]));
      }
    }
  }
  //Optimize dependency graph
  const dependenceGraph = {};
  rawDependenceGraph.forEach((module) => {
    dependenceGraph[module.file] = {
      dependencies: module.dependencies,
      code: module.code,
    };
  });

  return dependenceGraph;
};

Finally, the module dependency graph we built will be placed in thedependenceGraph. Now, for our example, the built dependency graph will look like this:

{ 
  './src/index.js':
   { 
     dependencies: { './introduce.js': './src/introduce.js' },
     code: '"use strict";\n\nvar...'     
    },
  './src/introduce.js':{ 
    dependencies: { 
      './person.js': './src/person.js' 
    },
    code: '"use strict";\n\nObject.defineProperty(exports,...' 
  },
  './src/person.js':
   { 
     dependencies: {},
     code: '"use strict";\n\nObject.defineProperty(exports,...' 
    } 
}

3 generate executable code

After building the module dependency graph, we need to convert the module code into executable code according to the dependency graph.

because@babel/preset-envThe processed code uses two non-existent variablesrequireandexports. So we need to define these two variables.

requireWe mainly do these two things

  • According to the module name, get the corresponding code and execute.
eval(dependenceGraph[module].code)
  • Processing module name, because the reference time is relative path, here we need to convert to absolute path, and recursively execute the dependent module code
function _require(relativePath) {
  return require(dependenceGraph[module].dependencies[relativePath]);
}

andexportIs used to store defined variables, so we define an object to store. Complete code generation functiongenerateCodedefinition:

const generateCode = (entry) => {
  const dependenceGraph = JSON.stringify(buildDependenceGraph(entry));
  return `
  (function(dependenceGraph){
    function require(module) {
      function localRequire(relativePath) {
        return require(dependenceGraph[module].dependencies[relativePath]);
      };
      var exports = {};
      (function(require, exports,  code) {
        eval(code);
      })(localRequire, exports, dependenceGraph[module].code);
      return exports;
    }
    require('${entry}');
  })(${dependenceGraph});
  `;
};

4 complete bundler packaging mechanism implementation code

completeBunlderPackage implementation code:

const fs = require("fs");
const path = require("path");
const babelParser = require("@babel/parser");
const traverse = require("@babel/traverse").default;
const babel = require("@babel/core");

const moduleParse = (file = "") => {
  const rawCode = fs.readFileSync(file, "utf-8");
  const ast = babelParser.parse(rawCode, {
    sourceType: "module",
  });
  const dependencies = {};
  traverse(ast, {
    ImportDeclaration({ node }) {
      const dirname = path.dirname(file);
      const absoulteFile = `./${path
        .join(dirname, node.source.value)
        .replace("\", "/")}`;
      dependencies[node.source.value] = absoulteFile;
    },
  });
  const { code } = babel.transformFromAst(ast, null, {
    presets: ["@babel/preset-env"],
  });

  return {
    file,
    dependencies,
    code,
  };
};

const buildDependenceGraph = (entry) => {
  const entryModule = moduleParse(entry);
  const rawDependenceGraph = [entryModule];
  for (const module of rawDependenceGraph) {
    const { dependencies } = module;
    if (Object.keys(dependencies).length) {
      for (const file in dependencies) {
        rawDependenceGraph.push(moduleParse(dependencies[file]));
      }
    }
  }
  //Optimize dependency graph
  const dependenceGraph = {};
  rawDependenceGraph.forEach((module) => {
    dependenceGraph[module.file] = {
      dependencies: module.dependencies,
      code: module.code,
    };
  });
  return dependenceGraph;
};

const generateCode = (entry) => {
  const dependenceGraph = JSON.stringify(buildDependenceGraph(entry));
  return `
  (function(dependenceGraph){
    function require(module) {
      function localRequire(relativePath) {
        return require(dependenceGraph[module].dependencies[relativePath]);
      };
      var exports = {};
      (function(require, exports,  code) {
        eval(code);
      })(localRequire, exports, dependenceGraph[module].code);
      return exports;
    }
    require('${entry}');
  })(${dependenceGraph});
  `;
};

const code = generateCode("./src/index.js");

In the end, we got itcodenamelyBundlerGenerated after packagingExecutable code. Next, we can copy it directly to the browser’sdevtoolTo view the results.

Write at the end

Although, thisBundlerThe implementation of the packaging mechanism is only a simple version, which only roughly implements the whole “webpack”BundlerPackaging process is not suitable for all use cases. However, in my opinion, learning a lot of things should be from easy to difficult, so the absorption efficiency is the highest.

Review of previous articles

Deep understanding of vue3 source code | component creation process

Deep understanding of vue3 source code | what is the origin of built-in component teleport?

Deep understanding of the patch process of vue3 source code | compile and runtime

❤️ Three strokes of love

Writing is not easy, if you think there is harvest, you can love three combo!!!

How to implement a bundler packaging mechanism of webpack?