RFC-002: Asset support

[h-a-n-a]

• Start Date: 2022-06-23
• Linked PR: #413 #426 #427 #428
• Related Issue: #407 Tracking issue of asset support  #435

Summary

Make rspack handling assets in a native way.

Motivation

In favor of different loaders that transpile assets to a JavaScript module is not decent, since the module is not able to share the code generation result with other generators as we cannot separate the result and the new architecture is built to treat assets natively.

For the status quo, the internal support for css is a pain in the ass. In CSS files, developers may use url keyword to import a resource, which currently is not supported by rspack, which means we have to support it in the first place.

Detailed design

Supported module types

enum ModuleType {
   Js,
   Jsx,
   Ts,
   Tsx,
   // newly added
   InlineAsset,
   ResourceAsset,
   RawAsset,
   Asset,
   // enhancement required but not for this rfc
   Css,
}

• Css: parses css and generates css module files / plain css files
• InlineAsset: inline assets exports a data URI for the asset.
• ResourceAsset: resource assets are emitted in a separate file.
• RawAsset: resource assets export the source code of the asset.
• Asset: auto assets are determined automatically by the size limit of the source code when choosing SourceType::ResourceAsset or SourceType::InlineAsset

Module type is a native type that is internally supported by rspack. In other words, if a native type is registered then you can use this type as a target for module finalization. For example, if you set a module as ModuleType::JSON, then internally, rspack will use the internal JSON parser and JSON generators, etc to handle the corresponding type.

Parsing

Each parser will only be registered once. Each ModuleType should register its own parser, for example, when the asset type was set to ModuleType::Asset, we need to determine whether to use Inline or Resource on each real parsing job by checking the source size.

Dependency collecting

// index.js
import "index.css"

// index.css
body {
  background: url(images/rspack.png)
}

In the example above, index.js depends on index.css, which means index.css is a dependency of index.js. Renderings based on the module render itself are not enough for dealing with the import statement. If we parse the dependency with range(startIndex and endIndex, for example, the range of images/rspack.png / index.css) collected, we can modify the importing statement on the code generation stage.

trait Dependency {
  ...
  get_range(&self) -> Vec<u32> // get range of the statement
}

struct CssUrlDepedency {
  range: Vec<u32>
}

impl Depedency for CssUrlDepedency {
   get_range(&self) -> Vec<u32> {
     self.range
  }
}

Code generation

Returning different SourceTypes

enum SourceType {
   JavaScript,
   Css,
   Asset
}

SourceType is a type for module that emitted by the plugin and it's a subset for ModuleType. For Js / Jsx / Ts / Tsx, the plugin returns SourceType::JavaScript on basic use basis. For ResourceAsset, the plugin returns SourceType::Asset and SourceType::JavaScript and Rspack emits the asset to real file system, JavaScript will be handled by plugin-javascript and do the code generation as that we already did.

Since this is what Rspack based on to treat asset emission in the code generation stage, and only ModuleType::ResourceAsset(after finalization for each module) will be converted to SourceType::Asset, for other assets will only converted to JavaScript code.

Delayed generation jobs and shared code generation state

For the state of the art for Rspack, the code generation order in each chunk is unordered, which means when a CSS makes a reference to other files on the file system, with the module graph, the file may not be generated prior to the CSS module and this is very important as CSS module need to replace this when it comes to its stage of code generation.

We define modules in the same chunk as different code generation jobs. As for the example above, we have to distinguish jobs that have depends on the other job from those that don't have and run dependency jobs first:

struct CodeGenerationJobs {
  jobs: Vec<Box<dyn Module>>,
  delayed_jobs: Vec<Box<dyn Module>>
}

Also, a data structure to share code generation results between jobs.

struct CodeGenerationResult {
  sources: HashMap<AssetType, Box<dyn RspackSource::Source>>,
  hash: String,
  filename: Option<String>,
}

type URI = string;
type CodeGenerationResults = HashMap<URI, CodeGenerationResult>;

Dependency rendering

For each dependency need to impl the Dependency trait:

trait Depedency {
   fn render(&self, compilation: &compilation) -> impl RspackSources::Source
}

By achieving this, we can replace URLs in the CSS. The details of this will be stated in RFC003: CSS Support.

Assets emitting

Rspack emits assets that were generated in code generation results.

Changes of rspack-sources

rspack-sources converts the source to a string, in order to keep assets compatible with the current architecture, we need to support native buffering for assets.
Support ReplaceSource for rspack-sources in order to support dependency rendering

Alternatives

Treat assets like javascript modules, and Rspack emits them directly to the file system in the corresponding stage of code generation, return javascript module to module graph. This is a not preferable way to handle assets since only by handling assets in a native way we can handle CSS in a native way or we may need to convert CSS to a JavaScript module to invoke any other processes(like what webpack4 did) for local assets.

Prior Art

In webpack, the source type we mentioned above is called module type in modules.rule.type

Module type is a raw type that was set by users, which could also be included natively like webpack natively handles *.json as type: "json". When webpack creates a module, the module hasn't been parsed before, so webpack defines its module type in resolve stage, which is known as Source Type in rspack.

How webpack processes assets

When it comes to the parsing stage(buildModule in webpack), for each module type, like "asset", webpack will test if this asset fulfills the asset size limit and determine whether the module should be set to inline or resource. Then in the code generation stage, if an asset was priorly set to resource, webpack will generate two code generation results javascript and asset. For javascript, the asset generator generates a module that returns the corresponding URL concatenated to public URL. For asset, the asset generator emits the buffer to the real filesystem.

Below, we use module type to state the prior art of webpack.

Module type supported by webpack

This module type is firstly added in webpack 4 (full changelog):

• webpack now supports these module types:
  • javascript/auto: (The default one in webpack 3) Javascript module with all module systems enabled: CommonJS, AMD, ESM
  • javascript/esm: EcmaScript modules, all other module system are not available
  • javascript/dynamic: Only CommonJS and, EcmaScript modules are not available
  • json: JSON data, it's available via require and import
  • webassembly/experimental: WebAssembly modules (currently experimental)
• javascript/esm handles ESM more strictly compared to javascript/auto:
  • Imported names need to exist on imported module
  • Dynamic modules (non-esm, i. e. CommonJs) can only imported via default import, everything else (including namespace import) emit errors
• In .mjs modules are javascript/esm by default
• WebAssembly modules
  • can import other modules (JS and WASM)
  • Exports from WebAssembly modules are validated by ESM import
    • You'll get a warning/error when trying to import a non-existing export from WASM
  • can only be used in async chunks. They doesn't work in initial chunks (would be bad for web performance)
    • Import modules using WASM via import()
  • This is an experimental feature and subject of change

In webpack 5 with the experimental feature css enabled, webpack can thus handle css files natively. You may lookup webpack config modules.rule.type to get more of them.

In the example down below, we handle files suffixed with j with native json handler.

// webpack.config.js
{
  "modules": {
     "rules": [
         {
              test: /\.j$/,
              type: 'json',
         }
      ]
   }
}

Unresolved questions

We haven't handle the dependency that imported with import assertions and css imports with modifiers(media query / support / layer).

[hardfist]

the file may not be generated prior to the CSS module and this is very important as CSS module need to replace this when it comes to its stage of code generation

I don't get this, I know moduleA maybe depends on moduleB's state but it doesn't need to depends on moduleB's codegen result

[h-a-n-a]

the file may not be generated prior to the CSS module and this is very important as CSS module need to replace this when it comes to its stage of code generation
I don't get this, I know moduleA maybe depends on moduleB's state but it doesn't need to depends on moduleB's codegen result

If we need to support url asset in css, then we have to support this as hash generation depends on code generation result and wepack handles native css support in the same way.
But as what we’ve been discussed this afternoon, Dependency rendering might be placed after code generation so this problem has already be tackled down but still we need to think about the intangible performance regression since we have to reparse again for folding the url itself.