Webpack Series Part 6: how to write a loader

Time:2022-5-9

The full text is 5000 words, which deeply analyzes the characteristics, operation mechanism and development skills of webpack loader. Welcome to praise and pay attention. Writing is not easy. Reprinting in any form is prohibited without the consent of the author!!!

There are a lot of information about webpack loader on the Internet, which is difficult to tell. However, there is no way to bypass this point when writing webpack series blog posts, so I have read more than 20 open source projects and summarized some knowledge and skills that I need to know when writing loader as comprehensively as possible. contain:

Webpack Series Part 6: how to write a loader

Let’s go, then.

Know loader

If you want to make a summary, I think loader is a content translator with side effects!

The core of webpack loader can only be the implementation of content Converter – converting various resources into standard JavaScript content format, such as:

  • css-loaderConvert CSS to__WEBPACK_DEFAULT_EXPORT__ = ".a{ xxx }"format
  • html-loaderConvert HTML to__WEBPACK_DEFAULT_EXPORT__ = "<!DOCTYPE xxx"format
  • vue-loaderMore complicated, it will.vueThe file is converted into multiple JavaScript functions, corresponding to template, JS, CSS and custom block respectively

So why do you need to do this conversion? In essence, it is because webpack only knows the text that conforms to the JavaScript specification (other parsers are added after webpack 5): it will be called when parsing the module content in the make phaseacornConvert the text into ast object, and then analyze the code structure and module dependency; This set of logic does not work for pictures, JSON, Vue, SFC and other scenes, so loader needs to intervene to convert resources into content forms that webpack can understand.

Plugin is another set of webpack extension mechanism with stronger functions. It can insert specialized processing logic into the hooks of various objects. It can cover the whole life process of webpack, and its ability, flexibility and complexity will be much stronger than loader.

Loader Foundation

At the code level, loader is usually a function with the following structure:

module.exports = function(source, sourceMap?, data?) {
  //Source is the input of the loader, which may be the file content or the processing result of the previous loader
  return source;
};

The loader function receives three parameters:

  • source: resource input. For the first executed loader, it is the content of the resource file; Subsequent loaders are the execution results of the previous loader
  • sourceMap: optional parameter, codesourcemapstructure
  • dataOptional parameters and other information to be passed in the loader chain, such as:posthtml/posthtml-loaderThe ast object of the parameter will be passed through this parameter

amongsourceIs the most important parameter. What most loaders have to do is tosourceTranslated into another formoutputFor examplewebpack-contrib/raw-loaderCore source code of:

//... 
export default function rawLoader(source) {
  // ...

  const json = JSON.stringify(source)
    .replace(/\u2028/g, '\\u2028')
    .replace(/\u2029/g, '\\u2029');

  const esModule =
    typeof options.esModule !== 'undefined' ? options.esModule : true;

  return `${esModule ? 'export default' : 'module.exports ='} ${json};`;
}

The function of this code is to wrap the text content into a JavaScript module, for example:

// source
I am Tecvan

// output
module.exports = "I am Tecvan"

After modular packaging, this text content turns into a resource module that webpack can handle, and other modules can reference and use it.

Return multiple results

The above example passedreturnStatement returns the processing result. In addition, the loader can alsocallbackMethod to return more information for downstream loaders or webpack itself. For example, inwebpack-contrib/eslint-loaderMedium:

export default function loader(content, map) {
  // ...
  linter.printOutput(linter.lint(content));
  this.callback(null, content, map);
}

adoptthis.callback(null, content, map)Statement returns both the translated content and the sourcemap content.callbackYour complete signature is as follows:

this.callback(
    //For exception information, just pass null value when loader is running normally
    err: Error | null,
    //Translation results
    content: string | Buffer,
    //Sourcemap information of source code
    sourceMap?: SourceMap,
    //Any value that needs to be passed between loaders
    //It is often used to pass ast objects to avoid repeated parsing
    data?: any
);

Asynchronous processing

When asynchronous or CPU intensive operations are involved, the loader can also return processing results asynchronously, such aswebpack-contrib/less-loaderCore logic:

import less from "less";

async function lessLoader(source) {
  // 1.  Get asynchronous callback function
  const callback = this.async();
  // ...

  let result;

  try {
    // 2.  Call less to translate the module content into CSS
    result = await (options.implementation || less).render(data, lessOptions);
  } catch (error) {
    // ...
  }

  const { css, imports } = result;

  // ...

  // 3.  After translation, the result is returned
  callback(null, css, map);
}

export default lessLoader;

In less loader, the logic is divided into three steps:

  • callthis.asyncGet the asynchronous callback function. At this time, webpack will mark the loader as an asynchronous loader and suspend the current execution queue untilcallbackTriggered
  • calllessThe library translates less resources into standard CSS
  • Call asynchronous callbackcallbackReturn processing results

this.asyncThe returned asynchronous callback function signature is the same as that described in the previous sectionthis.callbackThe same, which will not be repeated here.

cache

Loader provides a convenient extension method for developers, but the translation operations of various resource contents performed in loader are usually CPU intensive – which may lead to performance problems in the single threaded node scenario; Or the asynchronous loader will suspend the subsequent loader queue until the asynchronous loader triggers a callback. A little carelessness may lead to too long execution time of the whole loader chain.

For this reason, by default, webpack will cache the execution results of loader until the resource or resource dependency changes. Developers need to have a basic understanding of this, which can be used if necessarythis.cachableExplicitly declare not to cache, for example:

module.exports = function(source) {
  this.cacheable(false);
  // ...
  return output;
};

Context and side effect

In addition to being a content converter, the loader running process can also limit the webpack compilation process through some context interfaces, resulting in side effects other than content conversion.

Context information can be accessed throughthisobtain,thisObject byNormolModule.createLoaderContextThe function is created before calling loader. Common interfaces include:

const loaderContext = {
    //Get the configuration information of the current loader
    getOptions: schema => {},
    //Add warning
    emitWarning: warning => {},
    //Add an error message. Note that this will not interrupt the webpack
    emitError: error => {},
    //Resolve the specific path of the resource file
    resolve(context, request, callback) {},
    //Submit files directly. The submitted files will not be processed by subsequent chunks and modules, but will be directly output to FS
    emitFile: (name, content, sourceMap, assetInfo) => {},
    //Add additional dependent files
    //In watch mode, resource recompilation will be triggered when the dependent file changes
    addDependency(dep) {},
};

Among them,addDependencyemitFileemitErroremitWarningWill have side effects on the subsequent compilation process, such asless-loaderIt contains such a code:

  try {
    result = await (options.implementation || less).render(data, lessOptions);
  } catch (error) {
    // ...
  }

  const { css, imports } = result;

  imports.forEach((item) => {
    // ...
    this.addDependency(path.normalize(item));
  });

To explain, the first call in the codelessCompile the contents of the file, and then traverse allimportStatement, that is, the above exampleresult.importsArray, call one by onethis.addDependencyThe function registers other resources imported as dependencies, and then recompilation will be triggered when these other resource files change.

Loader chain call

In use, multiple loaders can be configured for a certain resource file, and the loaders are executed from front to back and then from back to front in the order of configuration, so as to form a set of content translation workflow. For example, for the following configuration:

module.exports = {
  module: {
    rules: [
      {
        test: /\.less$/i,
        use: [
          "style-loader",
          "css-loader",
          "less-loader",
        ],
      },
    ],
  },
};

This is a typical less processing scenario for.lessThe suffix file is set with three loaders: less, CSS and style to process resource files cooperatively. According to the defined order, webpack parses the contents of less files and passes them into less loader first; The results returned by less loader are then transferred to CSS loader for processing; The result of CSS loader is then transferred to style loader; Finally, the processing result of style loader shall prevail. The simplified process is as follows:

Webpack Series Part 6: how to write a loader

In the above example, the three loaders function as follows:

  • less-loader: realize the conversion of less = > CSS and output CSS content, which cannot be directly applied in webpack system
  • css-loader: wrap CSS content likemodule.exports = "${css}"The wrapped content conforms to JavaScript syntax
  • style-loader: the simple thing to do is to package the CSS module into the require statement and call functions such as injectstyle at runtime to inject the content into the style tag of the page

The three loaders complete part of the content conversion work respectively, forming a call chain from right to left. The design of chain call has two advantages: one is to maintain the single responsibility of a single loader and reduce the complexity of the code to a certain extent; Second, fine-grained functions can be assembled into complex and flexible processing chains to improve the reusability of a single loader.

However, this is only part of the chain call. There are two problems:

  • Once the loader chain is started, all loaders need to be executed before it ends. There is no chance of interruption – unless an exception is explicitly thrown
  • In some scenarios, you don’t need to care about the specific content of the resource, but the loader needs to be executed after the source content is read out

In order to solve these two problems, webpack is superimposed on the loaderpitchThe concept of.

Loader Pitch

There are many articles about loader on the Internet, but most of them are not correctpitchExplain the common features of the pitch and why it is important to do so in depth.

In this section, I will talk about the feature of loader pitch from the three dimensions of what, how and why.

What is a pitch

Webpack allows you to mount a function namedpitchThe run-time pitch will execute earlier than the loader itself, for example:

const loader = function (source){
    console. Log ('execute after ')
    return source;
}

loader.pitch = function(requestString) {
    console. Log ('execute first ')
}

module.exports = loader

Full signature of pitch function:

function pitch(
    remainingRequest: string, previousRequest: string, data = {}
): void {
}

Contains three parameters:

  • remainingRequest: resource request string after current loader
  • previousRequest: list of loaders experienced before executing the current loader
  • data: with loader functiondataSame, used to transfer information that needs to be propagated in the loader

These parameters are not complicated, but they are closely related to requeststring. Let’s take an example to deepen our understanding:

module.exports = {
  module: {
    rules: [
      {
        test: /\.less$/i,
        use: [
          "style-loader", "css-loader", "less-loader"
        ],
      },
    ],
  },
};

css-loader.pitchThe parameters obtained in are as follows:

//Loader list and resource path after CSS loader
remainingRequest = less-loader!./xxx.less
//List of loaders before CSS loader
previousRequest = style-loader
//Default value
data = {}

Scheduling logic

Pitch translated into Chinese isThrowing, court, strength, the highest point of thingsWait, I think the reason why the pitch feature is ignored is the pot of this name. Behind it is a set of life cycle concepts of loader execution.

In terms of implementation, the loader chain execution process is divided into three stages: pitch, resource parsing and execution. In design, it is very similar to the event model of DOM, and pitch corresponds to the capture stage; Execute corresponding to bubbling stage; Between the two phases, webpack will read and parse the resource content, corresponding to the at of DOM event model\_ Target phase:

Webpack Series Part 6: how to write a loader

pitchThe phases are executed one by one from left to right in the configuration orderloader.pitchFunctions (if any), developers canpitchReturn any value to interrupt the execution of subsequent links:

Webpack Series Part 6: how to write a loader

So why design the pitch feature? After analyzing open source projects such as style loader, Vue loader and to string loader, I personally summarize two words:block

Example: style loader

First review the less loading chain mentioned earlier:

  • less-loader: convert the content of less specification to standard CSS
  • css-loader: wrap CSS content into JavaScript modules
  • style-loaderExport the result of JavaScript module with:linkstyleTags and other methods are attached to HTML, so that CSS code can run correctly on the browser

actually,style-loaderIt is not necessary for the browser to be responsible for handling the content in the specific environment, but it is not necessary for the browser to be able to run the content in the CSS environment:

// ...
//The loader itself does nothing
const loaderApi = () => {};

//Splicing module codes according to parameters in pitch
loaderApi.pitch = function loader(remainingRequest) {
  //...

  switch (injectType) {
    case 'linkTag': {
      return `${
        esModule
          ? `...`
          //Introducing the runtime module
          : `var api = require(${loaderUtils.stringifyRequest(
              this,
              `!${path.join(__dirname, 'runtime/injectStylesIntoLinkTag.js')}`
            )});
            //Introduce CSS module
            var content = require(${loaderUtils.stringifyRequest(
              this,
              `!!${remainingRequest}`
            )});

            content = content.__esModule ? content.default : content;`
      } // ...`;
    }

    case 'lazyStyleTag':
    case 'lazySingletonStyleTag': {
        //...
    }

    case 'styleTag':
    case 'singletonStyleTag':
    default: {
        // ...
    }
  }
};

export default loaderApi;

Key points:

  • loaderApiNull function, no processing
  • loaderApi.pitchIn the splicing results, the exported code includes:

    • Introducing runtime modulesruntime/injectStylesIntoLinkTag.js
    • multiplexingremainingRequestParameter, re import CSS file

The operation results are roughly as follows:

var api = require('xxx/style-loader/lib/runtime/injectStylesIntoLinkTag.js')
var content = require('!!css-loader!less-loader!./xxx.less');

Note that the pitch function of the style loader returns this paragraph. Subsequent loaders will not continue to execute. The current call chain is interrupted:

Webpack Series Part 6: how to write a loader

After that, webpack continues to parse and build the results returned by the style loader and encounters the inline loader statement:

var content = require('!!css-loader!less-loader!./xxx.less');

Therefore, from the perspective of webpack, the loader chain is actually called twice for the same file. For the first time, the pitch of the style loader is interrupted, and for the second time, the style loader is skipped according to the content of the inline loader.

Similar techniques have also appeared in other warehouses, such as Vue loader. Interested students can view my previous articles posted on bytefe official account《Webpack case — Vue loader principle analysis》, I won’t start here.

Advanced skills

development tool

Webpack provides two utilities for loader developers, which appear frequently in many open source loaders:

  • webpack/loader-utils: provides a series of tool functions such as reading configuration, serializing and deserializing requeststring, and calculating hash value
  • webpack/schema-utils: parameter verification tool

The specific interfaces of these tools have been clearly described in the corresponding readme and will not be repeated. Here are some examples often used when writing loader: how to obtain and verify user configuration; How to splice output file names.

Get and verify configuration

Loader usually provides some configuration items for developers to customize the running behavior. Users can use the configuration file of webpackuse.optionsProperty setting configuration, for example:

module.exports = {
  module: {
    rules: [{
      test: /\.less$/i,
      use: [
        {
          loader: "less-loader",
          options: {
            cacheDirectory: false
          }
        },
      ],
    }],
  },
};

Inside the loader, you need to useloader-utilsLibrarygetOptionsFunction to obtain user configurationschema-utilsLibraryvalidateFunction to verify the validity of parameters, such as CSS loader:

// css-loader/src/index.js
import { getOptions } from "loader-utils";
import { validate } from "schema-utils";
import schema from "./options.json";


export default async function loader(content, map, meta) {
  const rawOptions = getOptions(this);

  validate(schema, rawOptions, {
    name: "CSS Loader",
    baseDataPath: "options",
  });
  // ...
}

useschema-utilsWhen doing verification, you need to declare the configuration template in advance, which is usually processed into an additional JSON file, such as the one in the above example"./options.json"

Splice output file name

Webpack supports similar[path]/[name]-[hash].jsMode settingoutput.filenameThat is, the naming of the output file. This layer rule usually does not need to be paid attention to, but some scenarios, such aswebpack-contrib/file-loaderYou need to splice the results according to the file name of the asset.

file-loaderSupport the introduction of text or binary files such as PNG, JPG and SVG in JS module, and write the file to the output directory. There is a problem: if the file is calleda.jpg, after webpack processing, the output is[hash].jpg, how? You can now useloader-utilsProvidedinterpolateNamestayfile-loaderGet the path and name of the resource in. Source code:

import { getOptions, interpolateName } from 'loader-utils';

export default function loader(content) {
  const context = options.context || this.rootContext;
  const name = options.name || '[contenthash].[ext]';

  //Name of the final output of the splice
  const url = interpolateName(this, name, {
    context,
    content,
    regExp: options.regExp,
  });

  let outputPath = url;
  // ...

  let publicPath = `__webpack_public_path__ + ${JSON.stringify(outputPath)}`;
  // ...

  if (typeof options.emitFile === 'undefined' || options.emitFile) {
    // ...

    //Submit and write documents
    this.emitFile(outputPath, content, null, assetInfo);
  }
  // ...

  const esModule =
    typeof options.esModule !== 'undefined' ? options.esModule : true;

  //Return modular content
  return `${esModule ? 'export default' : 'module.exports ='} ${publicPath};`;
}

export const raw = true;

Core logic of code:

  1. According to the loader configuration, callinterpolateNameMethod to splice the full path of the target file
  2. Call contextthis.emitFileInterface, write out the file
  3. returnmodule.exports = ${publicPath}, other modules can refer to the file path

In addition to file loader, CSS loader and eslint loader all use this interface. Interested students please go to the source code by themselves.

unit testing

The benefit of writing unit tests in loader is very high. On the one hand, developers don’t have to write demos and build a test environment; On the one hand, for end users, projects with a certain test coverage usually mean higher and more stable quality.

After reading more than 20 open source projects, I summarized a set of unit testing processes commonly used in the webpack loader scenario toJest · 🃏 Delightful JavaScript TestingFor example:

  1. Create an instance of webpack and run loader
  2. Obtain the loader execution results, compare, analyze and judge whether they meet the expectations
  3. Judge whether there is an error during execution

How to run loader

There are two methods: one is to run in the node environment, call the webpack interface, and compile with code instead of the command line. Many frameworks will adopt this method, such as Vue loader, stylus loader, Babel loader, etc. the advantage is that the operation effect is closest to the end user, and the disadvantage is that the operation efficiency is relatively low (negligible).

withposthtml/posthtml-loaderFor example, it will create and run a webpack instance before starting the test:

// posthtml-loader/test/helpers/compiler. JS file
module.exports = function (fixture, config, options) {
  config = { /*...*/ }

  options = Object.assign({ output: false }, options)

  //Create webpack instance
  const compiler = webpack(config)

  //Output the build results in memoryfs mode to avoid writing to disk
  if (!options.output) compiler.outputFileSystem = new MemoryFS()

  //Execute and return the result in promise mode
  return new Promise((resolve, reject) => compiler.run((err, stats) => {
    if (err) reject(err)
    //Asynchronously return execution results
    resolve(stats)
  }))
}

Tips:
As shown in the above example, usecompiler.outputFileSystem = new MemoryFS()Statement to set webpack to output to memory, which can avoid disk writing operation and improve compilation speed.

Another method is to write a series of mock methods to build a simulated webpack running environment, such asemaphp/underscore-template-loader, the advantage is that the running speed is faster, while the disadvantage is that the development workload is large and the universality is low. Just understand it.

Comparison results

After the above example is run, theresolve(stats)Return the execution result by,statsThe object contains almost all the information about the compilation process, including time-consuming, products, modules, chunks, errors, warnings, etc. I wrote in the previous articleShare several webpack utility analysis toolsThis has been introduced in depth, and interested students can go to read it.

In the test scenario, you canstatsObject, such as the implementation of style loader:

// style-loader/src/test/helpers/readAsset. JS file
function readAsset(compiler, stats, assets) => {
  const usedFs = compiler.outputFileSystem
  const outputPath = stats.compilation.outputOptions.path
  const queryStringIdx = targetFile.indexOf('?')

  if (queryStringIdx >= 0) {
    //Resolve the output file path
    asset = asset.substr(0, queryStringIdx)
  }

  //Read file content
  return usedFs.readFileSync(path.join(outputPath, targetFile)).toString()
}

To explain, this code first calculates the file path of asset output, and then calls thereadFileMethod to read the contents of the file.

Next, there are two ways to analyze content:

  • Call jest’sexpect(xxx).toMatchSnapshot()Assertion determines whether the current running result is consistent with the previous running result, so as to ensure the consistency of the results modified many times. Many frameworks use this method a lot
  • Interpret the resource content and judge whether it meets the expectations. For example, in the unit test of less loader, the same code will be compiled twice, one executed by webpack and one called directlylessLibrary, and then analyze whether the results of the two runs are the same

Students who are interested in this are strongly recommended to have a lookless-loaderTest directory of.

Abnormal judgment

Finally, you also need to judge whether there are exceptions in the compilation process. You can also start fromstatsObject resolution:

export default getErrors = (stats) => {
  const errors = stats.compilation.errors.sort()
  return errors.map(
    e => e.toString()
  )
}

In most cases, you want to compile without errors. At this time, you just need to judge whether the result array is empty. In some cases, you may need to determine whether to throw a specific exception. At this time, you canexpect(xxx).toMatchSnapshot()Assert, compare the results before and after the update with the snapshot.

debugging

In the process of developing loader, there are some tips to improve debugging efficiency, including:

  • usendbTool to realize breakpoint debugging
  • usenpm linkLink the loader module to the test project
  • useresolveLoaderThe configuration item adds the directory where the loader is located to the test project, such as:
// webpack.config.js
module.exports = {
  resolveLoader:{
    modules: ['node_modules','./loaders/'],
  }
}

Irrelevant summary

This is the seventh article in webpack Principle Analysis series. To be honest, I didn’t expect to write so much at the beginning, and I will continue to focus in the future. In this field of front-end engineering, my goal is to save my own book. Interested students are welcome to praise and pay attention. If you feel there are omissions and doubts, you are welcome to comment and discuss.

Previous articles

Recommended Today

Front-end Weekly Issue 7

Front-end Weekly publishes weekly front-end technology-related major events, article tutorials, version updates of some frameworks, and code and tools. It is published regularly every week, and everyone is welcome to pay attention and reprint.If the external chain cannot be accessed, pay attention to the official accountfront end weekly, click here, there are solutions Big event […]