feat(wasm): turbopack wasm adapted to latest upstream #58
Conversation
* feat(turbopack): apply utoo patches to canary * chore: update external code --------- Co-authored-by: zoomdong <1344492820@qq.com>
* feat(turbopack): externalType support umd * fix: clippy * fix: use CompileTimeDefineValue::Evaluate --------- Co-authored-by: xusd320 <xusd320@gmail.com>
…ta types and expr evaluation
…loader runner under worker
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Summary of Changes
Hello @xusd320, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!
This pull request undertakes a substantial effort to bring Turbopack's WebAssembly build up to date with the latest upstream changes. It involves a comprehensive synchronization of Rust dependencies and the strategic introduction of platform-specific code. Key functionalities like file system interactions and asset processing are now tailored for the WASM environment, leveraging Web Workers to ensure efficient and non-blocking execution. This work is crucial for maintaining Turbopack's performance and stability when running as WebAssembly.
Highlights
- WASM Compatibility Overhaul: Extensive changes to adapt Turbopack's build and runtime for WebAssembly, aligning with the latest upstream
utoobranch. - Dependency Ecosystem Synchronization: Major updates across numerous Rust crates, including core
swc,tokio,tracing, andwasmerdependencies, ensuring compatibility within the WASM context. - Platform-Specific Code Paths: Introduction of conditional compilation (
#[cfg(not(all(target_family = "wasm", target_os = "unknown")))]) and new WASM-specific modules for file system operations (wasm_fs_offload) and asset transformations (PostCSS and Webpack loaders via Web Workers). - Optimized WASM Execution: Implementation of Web Worker-based processing for PostCSS and Webpack loaders in WASM to prevent main thread blocking and improve performance.
Using Gemini Code Assist
The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.
Invoking Gemini
You can request assistance from Gemini at any point in your pull request via creating an issue comment (i.e. comment on the pull request page) using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands.
| Feature | Command | Description |
|---|---|---|
| Code Review | /gemini review |
Performs a code review for the current pull request in its current state. |
| Pull Request Summary | /gemini summary |
Provides a summary of the current pull request in its current state. |
| Comment | @gemini-code-assist | Responds in comments when explicitly tagged, both in issue comments and review comments. |
| Help | /gemini help |
Displays a list of available commands. |
Customization
To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.
Limitations & Feedback
Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.
You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.
Footnotes
-
Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩
There was a problem hiding this comment.
Code Review
This pull request introduces significant changes to adapt Turbopack for WebAssembly (WASM) environments. The changes involve extensive use of conditional compilation (cfg) to provide platform-specific implementations for filesystem operations, process management, and networking. Key dependencies like tokio are now sourced from a fork, and many parts of the dev server and Node.js-specific logic are disabled for WASM builds. New WASM-specific implementations using Web Workers for PostCSS and Webpack loader transformations have been added.
While this is a major step towards WASM compatibility, my review has identified several critical areas where the WASM implementation is incomplete and will lead to runtime panics due to unreachable!() or todo!() macros. These are present in core functionalities like filesystem access (read_link), MDX processing, and Node.js evaluation pools. Additionally, some placeholder implementations, like for file metadata, may cause subtle bugs. It's crucial to address these incomplete implementations to ensure stability and correctness in the WASM environment.
| #[cfg(all(target_family = "wasm", target_os = "unknown"))] | ||
| let link_path: PathBuf = { | ||
| // TODO: not supported now | ||
| todo!() | ||
| }; |
There was a problem hiding this comment.
The read_link implementation for WASM is currently a todo!(), which will cause a panic at runtime if this code path is executed. This should be implemented or properly handled to avoid runtime crashes.
| #[cfg(all(target_family = "wasm", target_os = "unknown"))] | ||
| { | ||
| unreachable!() | ||
| } | ||
| #[cfg(not(all(target_family = "wasm", target_os = "unknown")))] |
There was a problem hiding this comment.
The path_to_file_url function for WASM is implemented with unreachable!(). If this function is called in a WASM environment, it will cause a panic. This needs a proper implementation or should be guarded to prevent calls on WASM.
| #[cfg(all(target_family = "wasm", target_os = "unknown"))] | ||
| { | ||
| unreachable!() | ||
| } |
There was a problem hiding this comment.
The process method for MDX transformation is not implemented for WASM and contains an unreachable!(). This will lead to a runtime panic if MDX files are processed in a WASM environment.
| #[cfg(all(target_family = "wasm", target_os = "unknown"))] | ||
| pub async fn compute( | ||
| evaluate_context: impl EvaluateContext, | ||
| sender: Vc<JavaScriptStreamSender>, | ||
| ) -> Result<Vc<()>> { | ||
| unreachable!() | ||
| } |
There was a problem hiding this comment.
The compute function for WASM is implemented with unreachable!(). This will cause a panic if this evaluation path is triggered in a WASM build.
| #[cfg(all(target_family = "wasm", target_os = "unknown"))] | ||
| async fn recv(&mut self) -> Result<Vec<u8>> { | ||
| todo!() | ||
| } |
There was a problem hiding this comment.
The recv method for NodeJsPoolProcess on WASM is a todo!(), which will cause a panic. This is a critical part of the process pool communication and needs to be implemented for WASM support. The send and wait_or_kill methods have similar issues in this file.
| #[cfg(all(target_family = "wasm", target_os = "unknown"))] | ||
| impl From<tokio_fs_ext::Metadata> for FileMeta { | ||
| fn from(meta: tokio_fs_ext::Metadata) -> Self { | ||
| FileMeta::default() | ||
| } | ||
| } |
There was a problem hiding this comment.
The WASM implementation for converting tokio_fs_ext::Metadata to FileMeta returns a default value, effectively discarding all file metadata. This could lead to incorrect behavior in parts of the system that rely on metadata, such as caching mechanisms that use modification times. This should be implemented to correctly map the metadata.
WASM adapted to lated
utoobranch.