RFC-020: Dependency

[h-a-n-a]

• Feature Name: Dependency
• Start Date: 2022-09-20
• Rspack Issue: speedy-js/rspack#0000 Issue name should be Tracking issue of <Fill in the RFC title above>

Summary

The abstraction of dependency adds a few functionalities for make and seal internally and only a small part of the design will be exposed to the user side, like adding a dependency to the current module in the loader context, resolving a module in the loader, etc.

• In the stage of make, module dependencies will be created for creating the module graph.
• In the stage of seal, dependencies generated in the make stage will be used as a template for code generation.

Motivation

The abstraction of Dependency is designed to solve a few problems that we are facing:

1. The creation of the module graph is based on the current implementation of dependency, but it lacks the capability to support things like import assertion, etc.
2. In the stage of make, rspack is not able to add dependency in loaders, and the generation of each dependency is coupled in each module. The analysis of dependencies would be better placed in the hook or plugin tapped into plugin JavaScript. This allows dependency to transform the corresponding code of each module in the seal step(code generation step to be specific and same as how Webpack does with dependency)
3. Dependency will interfere with the code generation's priority for each module. For example, assets used in a CSS module should be replaced with the correct location of output location of that asset.
4. Lack of runtime support. Rspack does not provide a way to generate runtime requirements for each transformation. A dependency will be used for adding runtime requirements for a module, which will result in the code generation result. For example, the module with import statements should create a runtime requirement for generating rspack_require and its dependent runtimes. But now, this logic is hardcoded.

Guide-level explanation

Glossary

Most of the glossaries can be found here WIP: State of the art for Webpack: Dependency · Discussion #620. Use the list below as a compliment.

• Module Dependency: Module dependencies stand for a collection of dependencies used in the stage of make, i.e the module graph creation.

• Dependency Type: The category of a module dependency, e.g: "esm" | "commonjs"

Module resolution

Dependency type

The behavior of resolve logic varies across different dependency types. Rspack will follow the community convention.

For example: if a module is resolved in the dependency type of esm, then rspack will read the mainFields in package.json under the specificity of "browser", "module", and "main" by default.

Conditional exports are also supported by Rspack. This is firstly introduced by Node.js and later adapted by Webpack.

As a result, dependency type will have an impact on the result of a resolution.

Custom dependency type can be provided via Module.rule.resolve.byDependency, which is used to override the default rule.

High-level dependency types

Rspack is intended to support file dependency, missing dependency, and context dependency in the early stage of dependency.

A file dependency can be added in different ways:

• Calling addDependency on loader context, which accepts a String type as a parameter. This operation automatically adds the passed file as a watchable dependency.
• Calling this.resolve will add the result as a dependency to the current module.
• Calling this.getResolve will return the identical type of this.resolve, so this is the same as the second way.

A context dependency can be created via require.context or import statement with dynamic expression e.g import('./locale/' + language + '.json')

If the created dependency doesn't exist on the file system, then rspack will mark it as a missing dependency.

Progressive Dependency support

1. Finish the dependency architecture introduced in this RFC
2. Support CSS Url replacement and import assertions

Reference-level explanation

Low-level ideas

Dependency

Dependency is a primitive to describe a snippet of code on which a module is dependent. In the stage of make, i.e building a module, discovering its dependency, and making the module graph, rspack will use this dependency (module dependency to be specific) to describe a relationship between this module and its children-modules.

Template

A template is an associated function for a given dependency type.

We know that String doesn't contain any AST information. If the whole structure is driven by string concatenation and replacement, then in the later passes like optimization, in order to regain the AST, we have to do some reparse and this will jeopardize the overall performance to some extent. Also, SWC and its peripherals are strongly based on AST transformation. I decide to use AST as the first-class citizen in code transformation for the code generation step. But as always, for those types who don't have a registered AST type, then its code will be used as a fallback. Whether the ASTs or other data structures are used strongly depends on their module types.

AST

The core lib defines ASTs to maintain the relationship between module types and their corresponding AST type.

The built-in module AST type is constructed in an enum. JavaScript-like AST and CSS AST will be supported via SWC-related AST crates.

enum ModuleAST {
  JavaScript(swc_ecma_ast::Program),
  Css(swc_css_ast::Stylesheet)
}

Currently, we do not accept registering third-party ASTs from plugins and other ways.

Dependency

Currently, the implementation of dependency is coupled with the parser of each module. However, this coupling with the module makes the logic of analysis hard to be implemented on the AST level. As soon as the loaders finish their processings, the source code is not allowed to be changed. The source code will be passed to the registered parser for the module type. Dependencies will be collected when walking the generated AST. The mechanism is implemented via parser plugins.

Parser plugin

The parser plugin can be used to tap into the walking procedure of an AST.

trait ParserPlugin: Send + Sync {
  fn apply(&self, parser_context: &mut ParserContext) -> Result<()>;
}

The apply function is called before the parser's initialization. In the ParserContext, a function to register the dependency collector builders is provided since the dependency collector is module-wise.

Parser context

struct ParserContext {
  js_module_dependency_builders: Vec<Box<dyn Fn() -> Box<dyn DependencyCollector>>>,
  ...
}

impl ParserContext {
  apply_js_module_dependency_collector<T: 'static + Fn() -> Box<dyn DependencyCollector>>(
    &self,
    dep_builder: T,
  ) {
    ...
  }
  
  apply_css_...
}

Parser context is created before the parser's initialization. Currently, It has a few functions for storing dependency builders.

Dependency Collector

Dependency collectors share the same trait:

trait DependencyCollector: Send + Sync {
  fn dependencies(&self) -> Vec<Box<dyn ModuleDependency>>;
}

Rspack internally will call dependencies to get dependencies.

For JavaScript and TypeScript, swc_ecma_visit::VisitAll is used.

trait JavaScriptDependencyCollector: DependencyCollector + swc_ecma_visit::VisitAll {}

Accessing the node alias is the only allowed operation in the visitor to improve performance and avoid intangible conflicts between visitors.

#[derive(Default)]
struct JSModuleDependencyCollector {
  dependencies: Vec<ModuleDependency>
}

impl swc_ast_visit::VisitAll for JsModuleDependencyCollector {
  // your own dependency collect logic
}

impl DependencyCollector for JsModuleDependencyCollector {
  fn into_dependencies(self) -> Vec<Box<dyn ModuleDependency> {
    self.dependencies
  }
}

impl JavaScriptDependencyCollector for JSModuleDependencyCollector {}

Then, you can apply the dependency collector to the parser plugin context:

struct JsParserPlugin;
impl ParserPlugin for JsParserPlugin {
  fn apply(&self, parser_context: &mut ParserContext) -> Result<()> {
    parser_context.apply_js_module_dependency_collector(|| Box::new(JsModuleDependencyCollector::default()));
    Ok(())
  }
}

Dependency collectors stored will be built on each parse and applied to the parser.

Dependency traits

trait Dependency: Send + Sync + Hashed {
  fn parent_module(&self) -> &str;
}

struct Assertion {
  r#type: Option<ModuleType>
}

enum DependencyType {
  ESM,
  CommonJS
}

trait ModuleDependency: Dependency {
  fn request(&self) -> &str;
  fn user_request(&self) -> &str;
  fn category(&self) -> DependencyType;
  fn assertion(&self) -> &Assertion;
}

The parent module is pointing to the unique identifier of a module. Custom dependency type is currently not allowed.

make

The stage of make is where we build the module graph. It happens after the initialization of compilation. So I propose to change the existing build_start hook to the make hook and developers can utilize this hook to access compilation.

Entry

Entries are considered module dependencies. The proposed EntryPlugin is used to add an entry based on user-passed options. Multiple entry plugins will be generated if users have multiple entries passed in. In the plugin, entry dependency is created and a call to add_entry on the compilation can manually add entry dependency for compilation.

// pesudo code

impl Plugin for EntryPlugin {
  fn make(&self, compilation: &Compilation) -> Result<()> {
    compilation.compiler_option.entries.iter().map(|entry| {
      EntryDependency {
        request: entry.clone(),
        user_request: entry,
        category: DependencyType::ESM,
        assertion: Assertion::default()
      }
    });
    
    Ok(())
  }
}

Compilation

Tasks for creating modules and dependency analysis, etc will be placed to compilation and keep NormalModuleFactory as-is.

Module

trait Module: Debug + Send + Sync {
  ...
  
- fn dependencies(&self) -> Vec<Dependency>;
+ fn identifier(&self) -> &str;
}

The dependency analysis is decoupled from modules and will be replaced with the proposal described above.

The identifier of the module is composed of the dependency's unique hash and the applied loaders, which are used to uniquely identify a module. The idea behind this is that a URL alone is not enough for us to uniquely identify a resource.

Module graph

Jobs of NormalModuleFactory are created for resolving each dependency. Module dependencies of a module will always be created via dependency collectors, however, whether the dependency will create a corresponding module creation job is determined by the compilation level dependency Set based on its Hash.

In the module graph, dependencies will be moved to ModuleGraphModule and decoupled from its importer. The helper functions of the module graph will be almost the same but with some type changes.

struct ModuleGraph {
  uri_to_module: HashMap<String, ModuleGraphModule>,
- dependency_to_module_uri: HashMap<Dependency, String>,
}

struct ModuleGraphModule {
  ...
+ module_dependencies: Vec<Box<dyn ModuleDependency>>
}

Resolving

Resolving behavior is based on the module dependency's category(a.k.a dependency type).

Template

The template is an abstraction for code generation. Most of the templates will have two capabilities: modifying AST and requesting runtime requirements. Sub-traits derived from it may have special capabilities based-on its associated module types.

Base trait

enum RuntimeRequirements {
  ...
}

struct TemplateContext<'a> {
  runtime_requirements: Vec<RuntimeRequirements>,
  ...
}

trait DependencyTemplate<T> {
  fn apply(&self, ast: &mut T, template_context: &mut TemplateContext) -> Result<()>;
}

The apply fn of a dependency template will be called on the code generation step. Runtime requirements that the module required can be passed to the template context. A mutable AST is passed for developers to modify the part of the AST about which the plugin cares.

Sub-trait for JavaScript

enum FragmentStage {
  HarmonyExport = 10,
  HarmonyImport = 20
}

trait Fragment: Hashed {
  fn position(&self) -> u32 {
    0
  }
  fn stage(&self) -> FragmentStage;

  ...
}

struct Fragments {
  inner: LinkedHashSet<Box<dyn Fragment>>,
}

trait JavaScriptDependencyTemplate: DependencyTemplate<swc_ecma_ast::Program> {
  fn apply_fragments(&self, fragments: &mut Fragments) -> Result<()> {};
}

The huge difference that JavaScriptDependencyTemplate diffs from the super-trait are the fragment which is currently specialized only for JavaScript plugins. An optional apply_fragments fn can be used as a way to manually apply fragments.

Note that due to the mutability of AST in the step of code generation, analysis based on SyntaxContext will not be usable.

Fragment AST

Besides the operation given by the fn apply, fragments can have their own chance of AST modification.

To not get confused with the fn apply, AST operations for fragments will be called after the apply, and applications of fragments are strictly ordered with different stages and positions combined (higher the later).

If stages are the same and positions are not provided, then, the order of insertion will be taken into account.

trait Fragment: Hashed {
+ fn visit(&self, ast: &mut swc_ecma_ast::Program) -> Result<()> {
+   Ok(())
+ }

+ fn use_reducer(&self) -> bool {
+   false
+ }
 
+ fn reducer(&self, ast: swc_ecma_ast::Program) -> Result<swc_ecma_ast::Program> {
+   Ok(ast)
+ }
}

Both visit and reducer methods are provided for modifying the AST. The latter requires mem-move, which may do harm to performance. So it is an opt-in method. Nevertheless, both methods are useful in different cases.

Runtime requirements

Runtime requirements can be applied to template context to request runtime module creation, etc.

Code generation

struct ModuleCodeGenerationResult {
  code: Option<BoxSource>,
  ast: Option<ModuleAST>
}

trait Module: Send + Sync {
- fn render(...)
+ fn code_generation(&self, compilation: &Compilation, ...) -> ModuleCodeGenerationResult
}

ASTs have higher priority than sources.

JavaScript

JavaScript plugins would have to handle the code generation for AST types and concattenate the dependency template and its associated fragments into a Source. In order to get runtime requirements passed to core crate, Code generation results for modules would also be stored on core. This separates the render_manifest and module's code generation.

JavaScript plugin will call iterate through every modules that contains source_type JavaScript, and extract ASTs from them, then apply dependencies and its fragments, also collect the runtime requirements.

Note: this is not the final art. Runtime module will be included later.

CSS

code_generation_dependency is introduced only for CSS plugin, then for the code generation for CSS modules can be placed prior than the parent module for cases like pointing source asset filename to the correct asset filename.

Drawbacks

N/A

Rationale and alternatives

The philosophy behind the whole architecture is basically a port of the Webpack. We must admit that trait objects are pain-in-ass. Downcasting everything would bloat up the codebase and make crates hard to maintain. The overall idea behind the interop part between core and plugins changes from the concrete type of Dependency to ModuleDependency and from Source to ModuleCodeGenerationResult that contains both AST and source. This only adds a small complexity to the whole architecture, burdens are given to plugins.

Again, apparently, the architecture is all about trade-offs. Without the performance benchmarks we cannot simply assure that AST transformations are more performant than string replacements. But It will be better for SWC visitors, etc.

Prior art

Webpack

State of the art for Webpack: Dependency

Unresolved questions

• This RFC does not include the support for loaders, ideally, loaders should have unique identifier in order to create different dependency and then results to a unique module. This includes change for core and node_binding, A new RFC is going to cover this up.

Future possibilities