RFC-013: Loader

[h-a-n-a]

• Feature Name: Loader
• Start Date: 2022-08-09
• Rspack Issue: speedy-js/rspack#516

Summary

The high level of this proposal is to support a webpack-like loader implementation on rspack. Loader implementation of webpack is a bit long. As result, this RFC is not the final version and will be implemented progressively.

Loader provides a way to support different kinds of sources to be processed with Rspack.

Motivation

The implementation of the current architecture of Rspack is the combination of resolve + load + transform, which causes some problems of common bundlers face (like Rollup). For example, due to the separation of resolve and load, we cannot easily handle the Virtual Module correctly (The id itself could be polluted by other plugins).

Another core reason for this is the architecture of Rspack is basically derived from the Webpack, which assures the architecture is sound and solid in most cases.

Guide-level explanation

Glossary

Resource related

import xxx from "./index.js?vue=true&style#some-fragment"

• resource: The absolute path to the requested file with query and fragment retained.
• resourcePath: The absolute path to the requested file without query and fragment. e.g ./index.js
• resourceQuery: The query part to the requested file. e.g ?vue=true&style
• resourceFragment: The fragment part to the requested file. e.g #some-fragment
• scheme: The starting protocol of each specifier, such as data in data:text/javascript, ....
• context: The directory of the module. Can be used as a context for resolving other stuff.

Progressive loader support

1. Stage 1: Support the basic loader API: basic loader with loaderContext loaderResult and loader composition for the user side(i.e. the module.rules part from webpack) and basic node binding. At the moment, a module.rules[number].rule can be used with type to interact with the natively-supported module types
2. Stage 2: Support built-in scheme hook for hijacking the resource reading of a scheme-based request like data: for Virtual Module and some important loader features (something like this.getResolve)
3. Stage 3: Support composition loader for loader developers, (i.e. a rust implementation of inline loader syntax and inline match resource that affect the asset module filename generation, see the section of [Prior art](#Prior art))
4. Stage 4: Support more loader features. (AST Reuse, this.importModule(MiniCSSExtractPlugin used this, but it seems not necessary for now and it's only an example), etc.)

Loader configurations

Rust option

type GlobString = String;

enum RegexOrGlob {
  Regex(regex::Regex)
  Glob(String)
}

struct ModuleRule {
  test: Option<RegexOrGlob>,
  resource_query: Option<regex::Regex>,
  resource: Option<regex::Regex>,
  module_type: Option<ModuleType>, // use internal handling with `ext` if not specified
  uses: Vec<Box<dyn Loader>> // vector of trait objects that derived from `trait Loader`
}

struct ModuleOption {
  rules: Vec<ModuleRule>
}

struct CompilerOptions {
  // other options are omitted
  module: Option<ModuleOption>
}

• CompilerOptions: The normalized options derived from node binding and JSON configs (see the source code)

Node option

module.exports = {
	module: {
    rules: [
      {
        test: /\.js$/,
        use: ["babel-loader"],
        resourceQuery: /\regex/,
        resource: /\regex/
      }
    ]
  }
}

Loader API

Rust Loader API

use rspack_core::Content; // https://github.com/speedy-js/rspack/blob/ddc04c61f6d734f36c346087cc7f75f539bee34a/crates/rspack_core/src/plugin/args.rs#L64

type Source = Content;

struct LoaderContext {
  source: Source,
  resource: String,
  resource_path: String,
  resource_fragment: String,
  resource_query: String
}

struct LoaderResult {
  content: Content
}

#[async_trait::async_trait]
trait Loader: Sync + Send {
  fn name() -> &'static str {
    "unknown-loader"
  }
  async fn load(loader_context: LoaderContext) -> anyhow::Result<Option<LoaderResult>> {
    Ok(None)
  }
}

Node Loader API

import type { LoaderContext, LoaderResult } from "@rspack/binding"

function Loader(loaderContext: LoaderContext) {
  const { source } = loaderContext;
  const code = source.getCode();
  const transformed = require("some-compiler").compile(code);
  return {
    content: transformed.code
  }
}

export default Loader;

// Type definitions
interface Source {
  getCode(): string
  getBuffer(): Buffer
  // TODO: getMap(): SourceMap
}

interface LoaderContext {
  source: Source,
  resource: string,
  resourcePath: string,
  resourceFragment: string | null,
  resourceQuery: string | null,
}

interface LoaderResult {
  content: string | Buffer,
  // TODO: handling SourceMaps
}

interface Loader {
  (loaderContext: LoaderContext): LoaderResult | Promise<LoaderResult>;
}

For each loader, the return value could either be synchronous or asynchronous. We use getCode and getBuffer pair to lazily evaluate the value that comes from the Rust side, which can be used to mitigate the communication overhead.

It's worth noting that getCode is not safe for those Buffers who are not valid UTF-8 strings, under the hood, this API will convert any Buffer that comes from the Rust side with Buffer#toString which is equal to the result of String#from_utf8_lossy as they both replaces the invalid characters with the replacement character U+FFFD.

Reference-level explanation

Due to the support of the loader being our first priority for each loader work to proceed, we have to do this in a progressive way.

Rspack

Removing resolve, load, transform hook

Under the current implementation, resolve hooks can be called from plugins and share the result with load hooks. This gives a huge benefit for writing virtual modules. However, from my perspective of view, writing virtual modules specifier that is out of the spec(which could be if you let it go with the import specifier analysis) will make the client code specialized for a certain bundler, which should be avoided. (See [#Prior art](#Prior art) to know more about the strategy webpack used to deal with this problem)

From this RFC and on, Rspack will remove the resolve / load / transform hook support. Instead, rspack will provide a defaultResolver, any other unexpected specifier will cause a program exception. For virtual modules, we will respect the spec and implement this with data: in Stage 2 of this RFC. Resolver implementations like this.getResolve (doc) will be available at LoaderContext (Stage 2) if you want to use it in a loader.

The successfully parsed specifier will be like this:

struct ResourceData {
  resource: String;
  resource_path: String;
  resource_query: Option<String>;
  resource_fragment: Option<String>;
}

Loader Runner

A new crate loader-runner will be created for running loaders for each module. This would help us to make the testing easier.

struct LoaderRunnerOptions {
  loaders: Vec<Box<dyn Loader>>;
  resource_data: ResourceData;
  plugins: Vec<Box<dyn LoaderRunnerPlugin>>
}

#[async_trait::async_trait]
trait LoaderRunnerPlugin {
  async fn process_resource(resource_data: ResourceData) -> anyhow::Result<Option<Content>>
}

Rspack will find the matching rules and sort the loaders first and pass the result to the LoaderRunner. The result for each loader runner will be a struct:

enum Content {
  String(String),
  Buffer(Vec<u8>),
  // AST // We will support AST for improving performance in later stages
}

struct LoaderRunnerResult {
  content: Content;
}

The result of each loader runner can be regarded as the transform result before.

The loader runner supports the plugin system as well to resolve the scheme-based specifier in later stages.

Interacting with natively-supported Module Types

As soon as the rspack finds the matching rules, the module type is consolidated by the module_type property in each rule. In the previous implementation of rspack, we determine the kind of module type for each module by looking up the extension of a file. After that, we trigger the corresponding parser registered for this module type to handle this with rspack core natively.

Node binding

The basic idea of the Rspack node plugin and Loader is to treat them as a whole respectively, which avoids unnecessary overheads. Under the hood, node-binding uses ThreadsafeFunction to call from any native thread (see the source code from main)

Performance optimization: lazy evaluations

We have two layers of overhead if we handle the API like the way webpack did before:

• The conversion from Buffer to checked string requires the string validation
• Communication between Native and Node.js costs a lot, especially for creating large napi objects on the Node.js side.

But It's also good to note that the communication with Buffer is not that much pain-in-the-ass. Here's the solution:

1. Avoid creating napi string directly from the native side, instead, we convert Strings to JsBuffers (source code), then from the Node side, we can create a Buffer from that.
2. Using lazy evaluation, instead of the implicit version of getter, we choose getXXX to explicitly handle the conversion. If the value passed to the JS side is binary-formatted, then there is not much overhead for reading a Buffer.

The JS loader runner basically looks like this:

// POC only
function createLoaderRunner(moduleRule /* pass in a single module rule from JS side */) {
  return function loaderWrapper({ buf /* buffer from rust side */ }) {
    const loaderContext = {
      source: {
        getCode() {
        	return buf.toString("utf-8")
      	},
      	getBuffer() {
       	 return buf
      	}
      }
    }
    
    return jsLoaders.reduceRight((prev, jsLoader) => {
       return jsLoader.apply(loaderContext, [loaderContext])
    })
  }
}

Rationale and alternatives

From the perspective of the Loader architecture, this could be replaced by resolve + load + transform , but you can see the disadvantage in the motivation part.

Prior art

For the forms of loader, see: loader plugin api design (Analysis) #315

Webpack

Basically, you can refer to State of the art for Webpack: Loader #482. But there are still some cases we missed out there and here go the details.

Built-in specifier resolving

Two types of specifier resolving

We all know that webpack internally uses enhanced-resolve to handle the resolving on the file system. Here we are talking about resolving logic for different types of specifiers. Currently, webpack supports two kinds of user-written specifiers.

import xxx from "./style.css";
import xxx from "data:text/javascript, export const foo = 123";

1. The first one is the normal import statement you can add both query and fragment to it
2. The second one is the scheme-based import statement. For data: specified scheme-based import statements, it's in the specification(spec), and supported by Nodejs as well (doc)

Implementations of the resolving logic

For the normal import statements, webpack handles this for you and it is implemented without an internal plugin (see the source code)

For scheme-based import statements, you can register a resolver like what webpack does in its internal plugin(see the source code):

// lib/schemes/DataUriPlugin.js
normalModuleFactory.hooks.resolveForScheme
  .for("data")
  .tap("DataUriPlugin", resourceData => {
  const match = URIRegEx.exec(resourceData.resource);
  if (match) {
    resourceData.data.mimetype = match[1] || "";
    resourceData.data.parameters = match[2] || "";
    resourceData.data.encoding = match[3] || false;
    resourceData.data.encodedContent = match[4] || ""; // this would directly write the match code to `resourceData`
  }
});

request and identifier

Webpack strongly relies on string parsing for inline loader syntax and inline match resource syntax.

import xx from "!./style-loader.js!./css-loader.js!./style.css"

In the proceedings of module creation, webpack uses the identifier of a module (it could be a NormalModule) to determine whether it's a newly created module or it already exists. With the inline loader syntax and inline match resource syntax, It's hard for webpack to determine the identifier, so not only the inline loader syntax and inline match resource syntax are appended to the Module#identifier, but also the stringified user-defined loader (see the Module Creation Data Module#identifier)

Parcel

Resolve

For dealing with the issue of Virtual Module, Parcel resolvers may also return code directly. This allows programmatically generating virtual modules on demand. You must still return a filePath as well, however, as this indicates where any dependencies in the code should be resolved relative to.

import {Resolver} from '@parcel/plugin';
import path from 'path';

export default new Resolver({
  async resolve({specifier}) {
    if (specifier === 'special-module') {
      return {
        filePath: path.join(__dirname, 'special-module.js'),
        code: 'export default "This is a special module!";'
      };
    }

    return null;
  }
});

Unresolved questions

Not available for Stage1

Future possibilities

This will be tracked in different stages of implementing this RFC

[hardfist]

inline loader syntax is deprecated in webpack in the future, if we didn't rely on it to implement our internal plugin, maybe we could defer it to future

[h-a-n-a]

Yes, we haven't decided on the real-world implementation of this part. Also this syntax is non-spec, we could possibly to provide an alternative to achieve the same functionality later.

[hardfist]

is rust regex syntax the same as javascript, or we may met same problem as esbuild

[h-a-n-a]

It's not. The fundamental idea of Regex is to leave them in both Nodejs and Rust side and never try to interop with them. Regex in Nodejs side will only be tested there. So a full loader match would be implemented on both rust and node.js side.