RFC-018: Rspack Sources

[ahabhgk]

• Feature Name: rspack sources
• Start Date: 2022-08-31
• Rspack Issue: speedy-js/rspack#0000 Issue name should be Tracking issue of <Fill in the RFC title above>

Summary

Use rspack-sources to represent source code and source map as same as webpack.

Motivation

When your source code has gone through transformations, debugging in the browser becomes a problem. Source maps solve this problem by providing a mapping between the original and the transformed source code. In addition to source compiling to JavaScript, this works for styling as well.

Guide-level explanation

The 'sources' in rspack-sources not only represents source map, but also represents source code, even source buffer.

• Create:
  • RawSource: Represents source code without source map, it will not create source map for the source code.
    • Represents source code of source map
  • OriginalSource: Represents source code, it will create source map for the source code, but the source map is created by splitting the source code at typical statement borders (;, {, }), its quality is more worse than source map created by parser which use ast internally.
  • SourceMapSource: Represents source code with source map, optionally having an additional source map for the original source (before transformations).
    • Processes loaders result
    • Minimize js code in terser-webpack-plugin
• Modify:
  • PrefixSource: Prefix every line of the decorated Source with a provided string.
    • /******/ in JavascriptModulesPlugin.renderMain
  • ConcatSource: Concatenate multiple Sources or strings to a single source.
    • Template.renderChunkModules
    • Template.renderRuntimeModules
  • ReplaceSource: Decorates a Source with replacements and insertions of source code, usally used in dependencies.
    • Replace default export
    • Replace css @url()
• Cache
  • CachedSource: It tries to reused cached results from other methods to avoid calculations, usally used after modify is finished.
    • After fileManifest.render in Compilation.createChunkAssets
    • After this.generator.generate in NormalModule.codeGeneration

For source map generation will reference the implementation of SourceMapDevToolPlugin, and EvalSourceMapDevToolPlugin will choose to implement after SourceMapDevToolPlugin is implemented.

Reference-level explanation

Source in rspack-sources will implement these methods:

• updateHash: will involve in a part of hash generation.
• size: will be used in SizeLimitPlugin and implement NormalModule.size which used in optimize and stats (SplitChunksPlugin, DefaultStatsFactoryPlugin...).
• Source.source() to emit string.
• Source.buffer() to emit buffer and string, e.g. write out assets in compilation.emitAssets.
• Source.map() to generate SourceMap.

SourceMapDevToolPlugin will iterate compilation.assets, then emit asset's source map by compilation.emitAsset(sourceMapFile, new RawSource(asset.source.map())), or append the source mapping url comment after the source code by compilation.updateAsset(file, new ConcatSource(source, comment)) at processAssets hook.

Since SourceMapDevToolPlugin gets source map from compilation.assets, so it will be very simple to generate source map once all source code use Sources to represent.

Where did webpack use Sources?

• Loader
  • NormalModule
    • normalModule.originalSource: createSource process loaders result
    • createSourceForAsset: loaderContext.emitFile
  • Loader return result
• Asset
  • compilation.assets: CompilationAssets { [index: string]: Source }
    • emitAsset
    • updateAsset
    • hooks.optimizeAssets: AsyncSeriesHook<[CompilationAssets], ProcessAssetsAdditionalOptions>
    • hooks.afterOptimizeAssets: SyncHook<[CompilationAssets]>
    • compilation.hooks.processAssets: AsyncSeriesHook<[CompilationAssets], ProcessAssetsAdditionalOptions>
    • hooks.afterProcessAssets: SyncHook<[CompilationAssets]>
    • hooks.processAdditionalAssets: AsyncSeriesHook<[CompilationAssets]>
  • compiler.hooks.assetEmitted: AsyncSeriesHook<[string, AssetEmittedInfo]>
• Code Generation
  • compilation.codeGenerationResults
    • Create in Module.codeGeneration -> Consume in renderXxx
• Render Manifest
  • compilation.renderManifest: SyncWaterfallHook<[RenderManifestEntry[], RenderManifestOptions]>
    • JavascriptModulesPlugin.renderMain/renderChunk: SyncWaterfallHook<[Source, RenderContext]>
    • CssModulesPlugin.renderChunk
    • ...
• Dependency Template
  • Module.codeGeneration -> module.generator.generate -> Dependency.Template.apply()
    • ...

Prior art

webpack-sources, webpack/lib/SourceMapDevToolPlugin

Unresolved questions

• How to implement Hash on trait object?
• JsModule.render is using ast to render js code for now, have some gap between webpack.

[h-a-n-a]

Can you elaborate on the foreseeable changes for rspack-sources and rspack?

[ahabhgk]

rspack-sources:

• Use parcel_sourcemap::SourceMap instead of sourcemap::SourceMap since we need the SourceMap is concurrent safety.
• ConcatSource will own Box<dyn Source>s instead of &mut dyn Sources since we will have lots of return ConcatSource cases in rspack.
• Source will need to implement fn size, fn buffer, and fn update_hash or Source: Hash just like webpack-sources's Source.
• &str will implement Source as a convenient way of RawSource("xxx").
• Will have more ergonomics APIs for use it in rspack, e.g. a fn boxed for SourceExt since there will be lots of Box<dyn Source> used in rspack.

rspack:

• Will use Source to represent String (source code) and Vec<u8> (buffer).
• Loader will be able to return a SourceMap.
• Will have a SourceMapDevToolPlugin to generate source map from Sources.