Webpack packaging principle, write your own bundler

Time:2021-11-25

Packaging tools such as webpack can help us package the code organized with esmodule into a JS file and run it in the browser. Realize the modularization of front-end projects, and optimize the number of requests, file size, etc.

Not much to say, we will implement a similar bundler, package the modular front-end code and output JS files that can run in the browser.

preparation

Let’s take a look at how the projects we want to deal with are organized. We put a SRC folder with index.js, hello.js and word.js. The contents of each file are as follows

//index.js

import hello from "./hello.js"
console.log(hello)
//hello.js

import word from './word.js'
export default `hello ${word}`
//word.js

 const word = "word";
 export default word;

What you want to do is also very simple. You can use esmodule to finally assemble a console.log (‘Hello word ‘) in index.js, execute this JS in the browser, and print a’ hello word ‘on the console.

Then we will create a bundle.js at the same level of the SRC folder to help us package the code and enter the executable JS.

Parse entry file

We know that webpack uses an entry to enter the entry of the file to be packaged. Similarly, we also want to tell our bundler which file to package as the entry by entering the file access address.
Let’s start with the code:

const fs = require('fs')
const path = require('path')
const paser = require('@babel/parser')
const traverse = require('@babel/traverse').default
const { transformFromAst } = require('@babel/core')
const moduleAnalyser = (filename) => {
    const content = fs.readFileSync(filename, 'utf-8');	//{1}
    const ast = paser.parse(content,{			//{2}
        sourceType: 'module'
    })
    const dependencies = {};
    traverse(ast, {					//{3}
        ImportDeclaration({node}){
            const dirname = path.dirname(filename);
            const newFile = './' + path.join(dirname, node.source.value)
            dependencies[node.source.value] = newFile
        }
    })
    const { code } = transformFromAst(ast, null, {	//{4}
        presets: ["@babel/preset-env"]
    })
    return {
        filename,
        dependencies,
        code
    }
}
1. File reading

We define a moduleanalyzer method to analyze the module. Since we want to analyze the file, we need to use the FS module of node to read the file. So at {1}, we read the file in.

2. Generate abstract syntax tree

After getting the contents of the file, we need to parse it. Just as @ Babel / parser provided by Babel can help me parse the file and generate an abstract syntax tree, so we parse the file obtained by FS at {2} and generate ast. As follows:

{
  type: 'File',
  start: 0,
  end: 50,
  loc: SourceLocation {
    start: Position { line: 1, column: 0 },
    end: Position { line: 3, column: 18 },
    filename: undefined,
    identifierName: undefined
  },
  errors: [],
  program: Node {
    type: 'Program',
    start: 0,
    end: 50,
    loc: SourceLocation {
      start: [Position],
      end: [Position],
      filename: undefined,
      identifierName: undefined
    },
    sourceType: 'module',
    interpreter: null,
    body: [ [Node], [Node] ],
    directives: []
  },
  comments: []
}

We focus on program.body. There are two objects in it, which are actually two statements in index.js. Print it and you can see the following:

[
  Node {
    type: 'ImportDeclaration',
    start: 0,
    end: 30,
    loc: SourceLocation {
      start: [Position],
      end: [Position],
      filename: undefined,
      identifierName: undefined
    },
    specifiers: [ [Node] ],
    source: Node {
      type: 'StringLiteral',
      start: 18,
      end: 30,
      loc: [SourceLocation],
      extra: [Object],
      value: './hello.js'
    }
  },
  Node {
    type: 'ExpressionStatement',
    start: 32,
    end: 50,
    loc: SourceLocation {
      start: [Position],
      end: [Position],
      filename: undefined,
      identifierName: undefined
    },
    expression: Node {
      type: 'CallExpression',
      start: 32,
      end: 50,
      loc: [SourceLocation],
      callee: [Node],
      arguments: [Array]
    }
  }
]
3. Get dependency

Looking at the type, you can see that the first one is actually a reference statement. It should be very sensitive to see here. We need to package the file. Of course, this reference relationship is very important. Next, we need to continue parsing. We must find the referenced file through such a reference relationship, so the import statement should be saved. Fortunately, Babel provides @ Babel / traverse (traversal) method to maintain the overall state of AST. We use it in {3} to help us find the dependent modules.

It is worth mentioning that traverse parses a relative path, but in order to facilitate our next processing, we need to convert this relative path into an absolute path. The specific method is shown in the code.

4. AST to executable code

In addition to taking dependencies, we also need to convert ast into browser executable code, and @ Babel / core and @ Babel / preset env provided by Babel can just do this, so we did this step in {4}.

So far, we have completed the analysis of a module. Let’s see what results we will get:

{
  filename: './src/index.js',
  dependencies: { './hello.js': './src\\hello.js' },
  code: '"use strict";\n' +
    '\n' +
    'var _hello = _interopRequireDefault(require("./hello.js"));\n' +
    '\n' +
    'function _interopRequireDefault(obj) { return obj && obj.__esModule ? obj : { "default": obj }; }\n' +
    '\n' +
    'console.log(_hello["default"]);'
}

As you can see, we know who the parsed file is, what dependencies it has, and what the executable JS code is.

Get dependency map

Up to now, we have obtained the analysis of a module. To fully implement a function, we also need to process all the modules it depends on. So we need a method to help us get the whole dependency map, so we define the makedenpendenciesgraph method to help us do this.
Just look at the code first:

Const makedenpendenciesgraph = (entry) = > {// analyze all dependent modules and obtain the dependency graph
    const entryModule = moduleAnalyser(entry);
    const graph = {};
    const graphArray = [ entryModule ];
    while(graphArray.length > 0){
        [...graphArray].forEach(item => {
            graphArray.shift();
            const { dependencies } = item;
            graph[item.filename] = {
                dependencies: item.dependencies,
                code: item.code
            }
            if(dependencies) {
                for(let j in dependencies){
                    graphArray.push(moduleAnalyser(dependencies[j]))
                }  
            }
        });
    }
    return graph;
}

In fact, this part is relatively simple. We use a breadth first traversal to see if there is any dependency from the results parsed by the module analyzer. If there is any, continue to parse them and put all the parsed results together. Take a look at the generated dependency map:

{
  './src/index.js': {
    dependencies: { './hello.js': './src\\hello.js' },
    code: '"use strict";\n' +
      '\n' +
      'var _hello = _interopRequireDefault(require("./hello.js"));\n' +
      '\n' +
      'function _interopRequireDefault(obj) { return obj && obj.__esModule ? obj : { "default": obj }; }\n' +
      '\n' +
      'console.log(_hello["default"]);'
  },
  './src\\hello.js': {
    dependencies: { './word.js': './src\\word.js' },
    code: '"use strict";\n' +
      '\n' +
      'Object.defineProperty(exports, "__esModule", {\n' +
      '  value: true\n' +
      '});\n' +
      'exports["default"] = void 0;\n' +
      '\n' +
      'var _word = _interopRequireDefault(require("./word.js"));\n' +
      '\n' +
      'function _interopRequireDefault(obj) { return obj && obj.__esModule ? obj : { "default": obj }; }\n' +
      '\n' +
      'var _default = "hello ".concat(_word["default"]);\n' +
      '\n' +
      'exports["default"] = _default;'
  },
  './src\\word.js': {
    dependencies: {},
    code: '"use strict";\n' +
      '\n' +
      'Object.defineProperty(exports, "__esModule", {\n' +
      '  value: true\n' +
      '});\n' +
      'exports["default"] = void 0;\n' +
      'var word = "word";\n' +
      'var _default = word;\n' +
      'exports["default"] = _default;'
  }
}

Generate executable JS

We have obtained the dependency map. In fact, there is only the last step left to integrate the parsed contents and generate executable JS files. Upper Code:

const generateCode = (entry) => {
    const graph = makeDenpendenciesGraph(entry);
    return `(function(graph){
        function require(module){
            function localRequire(relativePath){
                return require(graph[module].dependencies[relativePath]);
            }
            var exports = {};
            (function(require, exports, code){
                eval(code)
            })(localRequire, exports, graph[module].code);
            return exports;
        };
        require('${entry}')
    })(${JSON.stringify(graph)})`;
}

In fact, we just want to put the codes in the dependency map together and return an executable JS, which is actually a JS string.

We notice that there is a require method and an exports object in the code. If we do not define these two things, JS will report an error when executing.

In the closure, we take require as the entry, and take another closure to divide each module to prevent internal variable pollution. At the same time, we note that relative paths are used in code, so we define a localrequire to convert absolute paths to find dependent modules.

So far, we have completed the packaging of the code organized by esmodule. Let’s see the results:

(function(graph){
    function require(module){
        function localRequire(relativePath){
            return require(graph[module].dependencies[relativePath]);
        }
        var exports = {};
        (function(require, exports, code){
            eval(code)
        })(localRequire, exports, graph[module].code);
        return exports;
    };
    require('./src/index.js')
 })({"./src/index.js":{"dependencies":{"./hello.js":"./src\\hello.js"},"code":"\"use strict\";\n\nvar _hello = _interopRequireDefault(require(\"./hello.js\"));\n\nfunction _interopRequireDefault(obj) { return obj && obj.__esModule ? obj : { \"default\": obj }; }\n\nconsole.log(_hello[\"default\"]);"},"./src\\hello.js":{"dependencies":{"./word.js":"./src\\word.js"},"code":"\"use strict\";\n\nObject.defineProperty(exports, \"__esModule\", {\n  value: true\n});\nexports[\"default\"] = void 0;\n\nvar _word = _interopRequireDefault(require(\"./word.js\"));\n\nfunction _interopRequireDefault(obj) { return obj && obj.__esModule ? obj : { \"default\": obj }; }\n\nvar _default = \"hello \".concat(_word[\"default\"]);\n\nexports[\"default\"] = _default;"},"./src\\word.js":{"dependencies":{},"code":"\"use strict\";\n\nObject.defineProperty(exports, \"__esModule\", {\n  value: true\n});\nexports[\"default\"] =void 0;\nvar word = \"word\";\nvar _default = word;\nexports[\"default\"] = _default;"}})

Execute this code in the browser and print out our expected ‘hello word’

The complete code is as follows:

const fs = require('fs')
const path = require('path')
const paser = require('@babel/parser')
const traverse = require('@babel/traverse').default
const { transformFromAst } = require('@babel/core')
Const moduleanalyzer = (filename) = > {// parse a module, generate an abstract syntax tree, and convert it into a manageable object
    const content = fs.readFileSync(filename, 'utf-8');
    const ast = paser.parse(content,{
        sourceType: 'module'
    })
    const dependencies = {};
    traverse(ast, {
        ImportDeclaration({node}){
            const dirname = path.dirname(filename);
            const newFile = './' + path.join(dirname, node.source.value)
            dependencies[node.source.value] = newFile
        }
    })
    const { code } = transformFromAst(ast, null, {
        presets: ["@babel/preset-env"]
    })
    return {
        filename,
        dependencies,
        code
    }
}
const makeDenpendenciesGraph = (entry) => {
    const entryModule = moduleAnalyser(entry);
    const graph = {};
    const graphArray = [ entryModule ];
    while(graphArray.length > 0){
        [...graphArray].forEach(item => {
            graphArray.shift();
            const { dependencies } = item;
            graph[item.filename] = {
                dependencies: item.dependencies,
                code: item.code
            }
            if(dependencies) {
                for(let j in dependencies){
                    graphArray.push(moduleAnalyser(dependencies[j]))
                }  
            }
        });
    }
    return graph;
}
const generateCode = (entry) => {
    const graph = makeDenpendenciesGraph(entry);
    return `(function(graph){
        function require(module){
            function localRequire(relativePath){
                return require(graph[module].dependencies[relativePath]);
            }
            var exports = {};
            (function(require, exports, code){
                eval(code)
            })(localRequire, exports, graph[module].code);
            return exports;
        };
        require('${entry}')
    })(${JSON.stringify(graph)})`;
}

const code = generateCode('./src/index.js')
console.log(code)

Recommended Today

Apache sqoop

Source: dark horse big data 1.png From the standpoint of Apache, data flow can be divided into data import and export: Import: data import. RDBMS—–>Hadoop Export: data export. Hadoop—->RDBMS 1.2 sqoop installation The prerequisite for installing sqoop is that you already have a Java and Hadoop environment. Latest stable version: 1.4.6 Download the sqoop installation […]