How does the CodeSandbox browser-side webpack work? Part One

Time:2019-8-14

Let’s take a look at this issue.CodeSandboxThis is a browser-side sandbox runtime environment that supports a variety of popular building boards, such ascreate-react-appvue-cliparcelWait. It can be used in rapid prototyping development, DEMO display, Bug restore and so on.

There are many similar products, such ascodepenJSFiddleWebpackBin(Abandoned)

CodeSandbox is even more powerful and can be seen as a browser-side Web pack runtime environment. Even in V3 version, it already supports VsCode mode, Vscode plug-ins, Vim mode, and themes.

In addition, CodeSandbox supports offline operation (PWA). It’s basically close to the programming experience of the local VSCode. Students with the iPad can also try to develop on it. So I usually use CodeSandbox directly for rapid prototyping.

Catalog

  • lead
  • Basic directory structure
  • Project Construction Process

    • Packager

      • WebpackDllPlugin
      • Online Packing Service
      • Back-off scheme
    • Transpilation

      • Basic Objects
      • Manager
      • TranspiledModule
      • Transpiler
      • BabelTranspiler
    • Evaluation
  • Technical Map
  • extend

lead

How does the CodeSandbox browser-side webpack work? Part One

My first impression of Code Sandbox is that it runs on the server, right? such ascreate-react-appTo run a node environment, you need to install a lot of dependencies through npm, package them through Webpack, and finally run a development server to run in the browser.

CodeSandbox packages and runs independently of the server. It’s done entirely in the browser.The general structure is as follows:

How does the CodeSandbox browser-side webpack work? Part One

  • EditorEditor. Mainly for modifying files, CodeSandbox is integrated here.VsCodeThe document will be notified when it changesSandboxTranslating. There will be articles devoted to CodeSandbox’s editor implementation.
  • SandboxCode Runner.Sandbox runs in a separate iframe, responsible for code translation and EvalationAs in the top figure, Editor is on the left and Sandbox is on the right.
  • PackagerPackage Manager. Similar to yarn and npm, responsible for pulling and caching NPM dependencies

Ives van Hoorne, author of Code Sandbox, has also tried toWebpackPorting to browser runs because almost all CLIs are built using Webpack. If you can transplant Webpack to browser, you can use the powerful ecosystem and translation mechanism of Webpack (loader/plugin), which is low-cost and compatible with all kinds of CLI.

However, the Web pack is too heavy and 3.5MB after compression, which is barely acceptable; the bigger problem is to simulate the Node running environment on the browser side, which is too expensive to pay for.

So CodeSandbox decided to build its own packer, which is lighter and optimized for the CodeSandbox platform. For example, CodeSandbox only cares about the code construction of the development environment, and its goal is to run. Compared with Webpack, it cuts out the following features:

  • Production mode. CodeSandbox only considers development mode and does not need to consider some features of production, such as

    • Code compression, optimization
    • Tree-shaking
    • performance optimization
    • Code segmentation
  • File output. No need to pack into chunk
  • Server communication. Sandbox translates and runs directly in situ, while Webpack needs to establish a long connection with the development server to receive instructions, such as HMR.
  • Static file processing (such as images). These images need to be uploaded to the CodeSandbox server
  • Plug-in mechanism and so on.

So we can think of it as followsCodeSandbox is a simplified version of Webpack, optimized for browser environments, such as parallel translation using worker

CodeSandbox’s packer uses proximityWebpack LoaderAPI, which makes it easy to migrate some loaders of Web packages. For example, here are the followingcreate-react-appImplementation (view source code):

import stylesTranspiler from "../../transpilers/style";
import babelTranspiler from "../../transpilers/babe";
// ...
import sassTranspiler from "../../transpilers/sass";
// ...

const preset = new Preset(
  "create-react-app",
  ["web.js", "js", "json", "web.jsx", "jsx", "ts", "tsx"],
  {
    hasDotEnv: true,
    setup: manager => {
      const babelOptions = {
        /*..*/
      };
      preset.registerTranspiler(
        module =>
          /\.(t|j)sx?$/.test(module.path) && !module.path.endsWith(".d.ts"),
        [
          {
            transpiler: babelTranspiler,
            options: babelOptions
          }
        ],
        true
      );
      preset.registerTranspiler(
        module => /\.svg$/.test(module.path),
        [
          { transpiler: svgrTranspiler },
          {
            transpiler: babelTranspiler,
            options: babelOptions
          }
        ],
        true
      );
      // ...
    }
  }
);

As you can see, CodeSandbox’s Reset and Webpack configurations are about the same length.However, at present, you can only use the Preset predefined by CodeSandbox, which does not support the configuration like Webpack. Personally, I think this is in line with the location of CodeSandbox. It is a rapid prototype development tool. What else do you do with Webpack?

These Presets are currently supported:

How does the CodeSandbox browser-side webpack work? Part One


Basic directory structure

The client side of CodeSandbox is open source. Otherwise, there will be no article. Its basic directory structure is as follows:

  • packages

    • appCodeSandbox Application

      • appEditor Implementation
      • embedRunning codesandbox embedded in web pages
      • sandboxRunning sandboxes, where code builds and previews are executed, is equivalent to a thumbnail version of Webpack. Running in a separate iframe

        • eval

          • preset

            • create-react-app
            • parcel
            • vue-cli
          • transpiler

            • babel
            • sass
            • vue
        • Compile.ts compiler
    • commonPlace generic components, tools, methods, resources
    • codesandbox-apiA unified protocol is encapsulated for communication between sandbox and editor (based on postmessage)
    • codesandbox-browserfsThis is a browser-side’File system’, which simulates NodeJS’s file system API and supports storing or retrieving files locally or from multiple back-end services.
    • react-sandpackCodsandbox Open SDK, which can be used to customize your own codesandbox

The source code is here.


Project Construction Process

packager -> transpilation -> evaluation

Sandbox construction is divided into three stages:

  • PackagerDuring the package loading phase, download and process all NPM module dependencies
  • TranspilationIn the translation stage, all the changed codes are translated and the module dependency graph is constructed.
  • EvaluationIn the execution phase, useevalRunning module code for preview

The following steps describe the technical points

Packager

Although NPM is a’black hole’, we still cannot live without it. In fact, a brief analysis of the front-end projectnode_modules80% of all development dependencies.

Because CodeSandbox already covers the building of code, we don’t need it.devDependenciesThat is to sayIn CodeSandbox, we only need to install all the dependencies needed to run the actual code, which can reduce hundreds of dependencies on downloads. So don’t worry about browsers being overwhelmed for the time being..

WebpackDllPlugin

CodeSandbox’s dependency packaging is affected byWebpackDllPluginInspiration: DllPlugin packages all dependencies into onedllFile, and create amanifestThe file describes the metadata of the DLL (as shown below).

Webpack translation or runtime can be indexed according to modules in manifest (for example__webpack_require__('../node_modules/react/index.js')) To load modules in dll. becauseWebpackDllPluginDependency is pre-translated before running or translation, so this part of the dependency code can be ignored in the project code translation phase, which can improve the speed of construction (the real scene of NPM dependency Dll packaging speed-up effect is not great):

How does the CodeSandbox browser-side webpack work? Part One

Manifest file

How does the CodeSandbox browser-side webpack work? Part One

Online Packing Service

Based on this idea, CodeSandbox builds its own online packaging service. Unlike Webpack Dll Plugin, CodeSandbox builds Manifest files in advance on the server side, and does not distinguish between Dll and manifest files. The specific ideas are as follows:

How does the CodeSandbox browser-side webpack work? Part One

In short, the CodeSandbox client gets itpackage.jsonAfter that, thedependenciesConvert to a dependency and version numberCombination(Identifier, for examplev1/combinations/[email protected]&[email protected]&[email protected]&[email protected]&[email protected]&[email protected]&[email protected]Then take this Combination to the server for request. The server caches the packaged results according to Combination as the cache key, and packages if no cache is hit.

Packing is actually still in useyarnTo Download all dependencies, but in order to eliminate the redundant files in NPM module, the server also traverses all dependency entry files (package. json# main), parses the require statement in AST, and recursively parses the require module. Finally, a dependency graph is formed, leaving only the necessary files..

The final output of the Manifest file, which is roughly the following structure, is equivalent to the combination of dll. JS + manifest. JSON of Web pack DllPlugin:

{
  // Module content
  "contents": {
    "/node_modules/react/index.js": {
      "Content": "'use strict';"Bibi if...", // Code content
      "Requirements": [// Other Dependent Modules
        "./cjs/react.development.js",
      ],
    },
    "/node_modules/react-dom/index.js": {/*..*/},
    "/node_modules/react/package.json": {/*...*/},
    //...
  },
  // Module Specific Installation Version Number
  "dependencies": [{name: "@babel/runtime", version: "7.3.1"}, {name: "csbbust", version: "1.0.0"},/*…*/],
  // Module aliases, such as react as an alias for preact-compat
  "dependencyAliases": {},
  // Dependency, that is, indirect dependency on information. This information can be obtained from yarn. lock
  "dependencyDependencies": {
    "object-assign": {
      "Entries": ["object-assign"], //module entry
      "Parents": ["react", "prop-types", "scheduler", "react-dom"], //parent module
      "resolved": "4.1.1",
      "semver": "^4.1.1",
    }
    //...
  }
}

Serverless Thought

It is worth mentioning that CodeSandbox’s Packer back end uses Serverless (based on AWS Lambda), which makes the Packer service more scalable and flexible to cope with high concurrency scenarios. After using Serverless, the response time of Packager has increased significantly, and the cost has gone down.

Packager is also open source. Look around.

Back-off scheme

AWS Lambda functions have limitations, such as/tmpThere can only be up to 500 MB of space. Although most dependent packaging scenarios do not exceed this limit, in order to enhance reliability (for example, the above scenario may be wrong or some modules may be missing), Packager also has a fallback scenario.

Later, the author of CodeSanbox developed a new Sandbox, which supports the placement of package management steps on the browser side, in conjunction with the packaging approach above. The principle is also simple:When translating a module, if the NPM module that the module depends on is not found, laziness is downloaded from the remoteLet’s see how it handles:

How does the CodeSandbox browser-side webpack work? Part One

In the fallback scenario, CodeSandbox does not download all the packages in package. json, but lazily loads them when module lookup fails. For example, when translating the entry file, it is found that the module react is not in the local cache module queue, then it will be downloaded remotely and then translated.

That is to say, because the dependency of the module will be analyzed statically in the translation stage, only the files that really depend on need to be downloaded, instead of the whole NPM package, which saves the cost of network transmission.

CodeSandbox passesunpkg.comorcdn.jsdelivr.netTo retrieve module information and download files, such as

  • Get package. json:https://unpkg.com/[email protected]/package.json
  • Packet directory structure acquisition:https://unpkg.com/[email protected]/?metaThis will recursively return all directory information for the package
  • Specific file download:https://unpkg.com/[email protected]/cjs/react.production.min.jsperhapshttps://cdn.jsdelivr.net/npm/@babel/[email protected]/helpers/interopRequireDefault.js

Transpilation

Now, after the Packager, let’s take a look at Transpilation, this stage.Starting from the entry file of the application, the source code is translated, AST is parsed, the lower dependency module is found, and then the recursive translation is carried out. Finally, a dependency graph is formed.:

How does the CodeSandbox browser-side webpack work? Part One

CodeSandbox’s entire translator runs in a separate iframe:

How does the CodeSandbox browser-side webpack work? Part One

Editor is responsible for changing the source code. The source code changes are passed to Compiler through postmessage, which is carried in it.Module+template

  • ModuleIt contains all source code content and module paths, including package. json. Compiler reads NPM dependencies according to package. json.
  • templateRepresents a Reset for Compiler, such ascreate-react-appvue-cliIt defines some loader rules to translate different types of files, and preset also determines the application template and entry files. As we know above, these templates are currently predefined.

Basic Objects

Before introducing Transpilation in detail, I’ll look at some basic objects to understand the relationship between them.

How does the CodeSandbox browser-side webpack work? Part One

  • ManagerThis is Sandbox’s core object, responsible for managing Preset, Manifest, and maintaining all the modules of the project (Transpiler Module)
  • ManifestAs we know from Packager above, Manifest maintains all dependent NPM module information
  • TranspiledModuleRepresents the module itself. It maintains translation results, code execution results, dependent module information, and is responsible for driving translation (calling Transpiler) and execution of specific modules.
  • PresetA project build template, such asvue-clicreate-react-appConfigure the translation rules of project files and the directory structure of the application (entry files)
  • TranspilerA loader equivalent to a Webpack is responsible for translating files of a specified type. For example, babel, typescript, pug, sass, etc.
  • WorkerTranspilerThis is a subclass of Transpiler, which schedules a Worker pool to perform translation tasks to improve translation performance.

Manager

Manager is a manager who controls the overall process of translation and execution. Now let’s look at the overall process of translation.

How does the CodeSandbox browser-side webpack work? Part One

The overall situation can be basically divided into the following four stages:

  • Configuration phaseCodeSandbox currently supports only a limited number of application templates, such as vue-cli, create-react-app. The conventions of directory structure between different templates are different, such as entry files and HTML template files. In addition, the rules of file processing are different, such as vue-cli needs to be processed..vueDocumentation.
  • Dependent on the download phaseThe Packager phase, which downloads all the dependencies of the project and generates Manifest objects
  • Change calculation stageAccording to the source code passed by Editor, the new, updated and removed modules are calculated.
  • Translation stageReally began to translate, first of all re-translate the module that needs to be updated calculated in the previous stage. Then, starting from the entry file, the new dependency graph is translated and constructed. There will be no repetition of translating unchanged modules and their sub-modules.

TranspiledModule

TranspiledModule is used to manage a specific module, which maintains the results of translation and operation, module dependency information, and drives module translation and execution:

How does the CodeSandbox browser-side webpack work? Part One

TranspiledModule retrieves the Transpiler list matching the current module from Preset, traverses the Transpiler to translate the source code, parses the AST in the process of translation, analyses the module import statements, collects new dependencies, and recursively translates the dependency list when the module translation is completed. Let’s take a look at the code in general:

async transpile(manager: Manager) {
    // Translated
    if (this.source)  return this
    // Avoid repetitive translation, one module only translates once
    if (manager.transpileJobs[this.getId()]) return this;
    manager.transpileJobs[this.getId()] = true;

    //... Reset status 

    // (ii) Obtain the Transpiler List from Preset
    const transpilers = manager.preset.getLoaders(this.module, this.query);

    // (ii) Chain call Transpiler
    for (let i = 0; i < transpilers.length; i += 1) {
      const transpilerConfig = transpilers[i];
      // (ii) Build LoaderContext, as shown below
      const loaderContext = this.getLoaderContext(
        manager,
        transpilerConfig.options || {}
      );

      // (ii) Call Transpiler to translate source code
      const {
        transpiledCode,
        sourceMap,
      } = await transpilerConfig.transpiler.transpile(code, loaderContext); // eslint-disable-line no-await-in-loop

      if (this.errors.length) {
        throw this.errors[0];
      }
    }

    this.logWarnings();

    // ...

    await Promise.all(
      this.asyncDependencies.map(async p => {
        try {
          const tModule = await p;
          this.dependencies.add(tModule);
          tModule.initiators.add(this);
        } catch (e) {
          /* let this handle at evaluation */
        }
      })
    );
    this.asyncDependencies = [];

    // (ii) Modules for recursive translation dependencies
    await Promise.all(
      flattenDeep([
        ...Array.from(this.transpilationInitiators).map(t =>
          t.transpile(manager)
        ),
        ...Array.from(this.dependencies).map(t => t.transpile(manager)),
      ])
    );

    return this;
  }

Transpiler

Transpiler is equivalent to webpack loader, and its configuration and basic API are roughly consistent with webpack, such as chain translation and loader-context.

export default abstract class Transpiler {
  initialize() {}

  dispose() {}

  cleanModule(loaderContext: LoaderContext) {}

  // (ii) Code conversion
  transpile(
    code: string,
    loaderContext: LoaderContext
  ): Promise<TranspilerResult> {
    return this.doTranspilation(code, loaderContext);
  }

  // (ii) Abstract methods, implemented by concrete subclasses
  abstract doTranspilation(
    code: string,
    loaderContext: LoaderContext
  ): Promise<TranspilerResult>;

  // ...
}

Transpiler’s interface is simple.transpileAccept two parameters:

  • codeSource code.
  • loaderContextProvided by TranspiledModule, it can be used to access translation context information, such as the configuration of Transpiler, module lookup, registration dependencies, and so on. The outline is as follows:
export type LoaderContext = {
  // (ii) Information reporting
  emitWarning: (warning: WarningStructure) => void;
  emitError: (error: Error) => void;
  emitModule: (title: string, code: string, currentPath?: string, overwrite?: boolean, isChild?: boolean) => TranspiledModule;
  emitFile: (name: string, content: string, sourceMap: SourceMap) => void;
  // (ii) Configuration information
  options: {
    context: string;
    config?: object;
    [key: string]: any;
  };
  sourceMap: boolean;
  target: string;
  path: string;
  addTranspilationDependency: (depPath: string, options?: { isAbsolute?: boolean; isEntry?: boolean; }) => void;
  resolveTranspiledModule: ( depPath: string, options?: { isAbsolute?: boolean; ignoredExtensions?: Array<string>; }) => TranspiledModule;
  resolveTranspiledModuleAsync: ( depPath: string, options?: { isAbsolute?: boolean; ignoredExtensions?: Array<string>; }) => Promise<TranspiledModule>;
    // (ii) Dependence on collection
  addDependency: ( depPath: string, options?: { isAbsolute?: boolean; isEntry?: boolean; }) => void;
  addDependenciesInDirectory: ( depPath: string, options?: { isAbsolute?: boolean; isEntry?: boolean; }) => void;
  _module: TranspiledModule;
};

Let’s start with a simple look at the JSON module’s Transpiler implementation. Each Transpiler subclass needs to implement doTranspilation, receive the source code, and return the processing results asynchronously:

class JSONTranspiler extends Transpiler {
  doTranspilation(code: string) {
    const result = `
      module.exports = JSON.parse(${JSON.stringify(code || '')})
    `;

    return Promise.resolve({
      transpiledCode: result,
    });
  }
}

BabelTranspiler

Not all modules are as simple as JSON, such as Typescript and Babel. In order to improve the efficiency of translation, Codesandbox will use Worker to perform multi-process translation, and the scheduling of multi-Worker is carried out byWorkerTranspilerCompletion, which is a subclass of Transpiler, maintains a Worker pool. Complex translation tasks such as Babel, Typescript and Sass are all based on Worker Transpiler:

How does the CodeSandbox browser-side webpack work? Part One

The typical implementation is Babel Transpiler, which forks three workers in advance when Andbox starts to improve the speed of translation start-up. Babel Transpiler will use these three workers to initialize the Worker pool first:

// Use three loaders, worker-loader fork, to handle Babel compilation
import BabelWorker from 'worker-loader?publicPath=/&name=babel-transpiler.[hash:8].worker.js!./eval/transpilers/babel/worker/index.js';

window.babelworkers = [];
for (let i = 0; i < 3; i++) {
  window.babelworkers.push(new BabelWorker());
}

The worker-loader of webpack is used to encapsulate the specified module as a Worker object. Make Worker easier to use:

// App.js
import Worker from "./file.worker.js";

const worker = new Worker();

worker.postMessage({ a: 1 });
worker.onmessage = function(event) {};

worker.addEventListener("message", function(event) {});

The specific process of Babel Tranpiler is as follows:

How does the CodeSandbox browser-side webpack work? Part One

Worker Transpiler maintainsAn idle Worker queueAnd oneTask queueIts job is to drive Workers to consume task queues. Specific translation work is carried out in Worker:

How does the CodeSandbox browser-side webpack work? Part One


Evaluation

Although called bundler, CodeSandbox does not pack, that is to say, it does not pack all modules into chunks files like Webpack.

TranspilationfromEntry fileStart translating, then analyze the file’s module import rules, recursively translate dependent modules.EvaluationAt the stage, CodeSandbox has built a completeDependency graphNow it’s time to run the app.

How does the CodeSandbox browser-side webpack work? Part One

The principle of Evaluation is relatively simple. Like Transpilation, it starts with the entry file:UseevalExecute the entry file if it is called during executionrequireThen recursively Eval dependent modules

If you know the principle of Node’s module import, you can easily understand the process:

How does the CodeSandbox browser-side webpack work? Part One

  • First, initialize HTML and findindex.htmlFile, set document. body. innerHTML to the body content of the HTML template.
  • (2) Injecting external resources. Users can customize some external static files, such as CSS and js, which need append to head
  • (3) Evaluate Entry Module
  • (4) All modules will be translated into CommonJS module specifications. So we need to simulate this module environment. Take a look at the code:

    // Implementing require method
    function require(path: string) {
      //... Intercept some special modules
    
      // Finding Modules in Manager Objects
      const requiredTranspiledModule = manager.resolveTranspiledModule(
        path,
        localModule.path
      );
    
      // Module caching, if caching exists, means no re-execution is required
      const cache = requiredTranspiledModule.compilation;
    
      return cache
        ? cache.exports
        :///(ii) Recursive evaluation
          manager.evaluateTranspiledModule(
            requiredTranspiledModule,
            transpiledModule
          );
    }
    
    // Realize require. resolve
    require.resolve = function resolve(path: string) {
      return manager.resolveModule(path, localModule.path).path;
    };
    
    // Simulate some global variables
    const globals = {};
    globals.__dirname = pathUtils.dirname(this.module.path);
    globals.__filename = this.module.path;
    
    // (ii) Place the execution result, the module object of CommonJS
    this.compilation = {
      id: this.getId(),
      exports: {}
    };
    
    // 🔴eval
    const exports = evaluate(
      this.source.compiledCode,
      require,
      this.compilation,
      manager.envVariables,
      globals
    );
  • _using Eval to execute the module. Also look at the code:

    export default function(code, require, module, env = {}, globals = {}) {
      const exports = module.exports;
      const global = g;
      const process = buildProcess(env);
      g.global = global;
      const allGlobals = {
        require,
        module,
        exports,
        process,
        setImmediate: requestFrame,
        global,
        ...globals
      };
    
      const allGlobalKeys = Object.keys(allGlobals);
      const globalsCode = allGlobalKeys.length ? allGlobalKeys.join(", ") : "";
      const globalsValues = allGlobalKeys.map(k => allGlobals[k]);
      // (ii) Encapsulate the code under a function, and the global variables are passed in as functions
      const newCode = `(function evaluate(` + globalsCode + `) {` + code + `\n})`;
      (0, eval)(newCode).apply(this, globalsValues);
    
      return module.exports;
    }

Ok! At this point, Evaluation will be explained. The actual code is much more complex than here, such as HMR (hot module replacement) support. Interested readers can see the source code of Code Sandbox by themselves.


Technical Map

I wrote another long article carelessly. It’s really a challenge to make such complicated code clear. I haven’t done enough. According to past experience, this is another article that nobody cares about, not to mention you. I don’t have much patience to read this kind of article myself. I’ll try my best to avoid it later.

  • Worker-loader: Encapsulate the specified module as Worker
  • Babel: JavaScript code translation, support ES, Flow, Typescript
  • Browserfs: Simulating Node Environment in Browsers
  • LocalForage: Client-side repository, which provides a LocalStorage-like interface with preferential use of these asynchronous storage schemes (IndexedDB or WebSQL)
  • Lru-cache: least-recent-used cache

extend

  • Creating a parallel, offline, extensible, browser based bundler for CodeSandbox
  • year of CodeSandbox – Ives van Hoorne aka @CompuIves at @ReactEurope 2018
  • How we make npm packages work in the browser
  • codesandbox/dependency-packager