Skip to content

Conversation

@KSDaemon
Copy link
Member

@KSDaemon KSDaemon commented Feb 14, 2025

This PR brings further improvements into data model compilation.
Cube transpilers are rewritten in Rust and are called asynchronously, moving transpilation jobs to the internal native threads under the hood. More improvements are achieved by caching the transpilation metadata between files transpile calls within one compilation phase.

This implementation supersedes other approaches taken in #9217 as it boosts performance more.

There were a bunch of iterations, research, and improvements. But long story short: now js files transpilation is 11−14 times faster than it was before!! 🚀

Here are a few numbers and charts for comparison:

Input: 300 js files with ~400 members each

We have 3 phases of compilation. That's why there are 3 messages with times for each phase.

Original master: synchronous sequential traversing in main node.js thread.

Execution Time for transpiling all 301 files: 11657.56 ms
Execution Time for transpiling all 301 files: 11871.40 ms
Execution Time for transpiling all 301 files: 12725.75 ms

In total, it's ~36 seconds.
What is more important — we observe event loop delays up to 50,000 ms!

comp1 — Original master

And here is what we have now: asynchronous parallel processing in native worker threads. (defaults to 4 libuv threads)

Execution Time for transpiling all 301 files: 222.43 ms
Execution Time for transpiling all 301 files: 190.51 ms
Execution Time for transpiling all 301 files: 2357.43 ms

In total, it's ~2,8 seconds.

Comparing to master: 36 / 2.8 = 12.8x!!!

What's also important is that I can't get rid of event loop delays totally, but now there are smaller portions of delays, and they are way shorter (5,000 VS 50,000).

comp7 — SWC async backend-native threads-2

Check List

  • Tests have been run in packages where changes made if available
  • Linter has been run for changed code
  • Tests for the changes have been added if not covered yet
  • Docs have been added / updated if required

@codecov
Copy link

codecov bot commented Feb 14, 2025

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 83.56%. Comparing base (671bcdb) to head (901a794).
Report is 3 commits behind head on master.

Additional details and impacted files
@@            Coverage Diff             @@
##           master    #9225      +/-   ##
==========================================
- Coverage   83.57%   83.56%   -0.01%     
==========================================
  Files         227      227              
  Lines       81627    81627              
==========================================
- Hits        68216    68208       -8     
- Misses      13411    13419       +8     
Flag Coverage Δ
cubesql 83.56% <ø> (-0.01%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@KSDaemon KSDaemon marked this pull request as ready for review February 15, 2025 10:41
@KSDaemon KSDaemon requested review from a team as code owners February 15, 2025 10:41
@KSDaemon KSDaemon changed the title feat(schema-compiler): Use SWC compiler instead of Babel for js models transpilation feat(schema-compiler): Boost models transpilation 12x times (using SWC instead of Babel) Feb 15, 2025
@KSDaemon KSDaemon changed the title feat(schema-compiler): Boost models transpilation 12x times (using SWC instead of Babel) feat(schema-compiler): Boost models transpilation 10-13x times (using SWC instead of Babel) Feb 15, 2025
@KSDaemon KSDaemon force-pushed the feat/compilation-swc-backend-native branch from 2de06c4 to 901a794 Compare February 17, 2025 17:45
@KSDaemon KSDaemon merged commit 2dd9a4a into master Feb 17, 2025
82 checks passed
@KSDaemon KSDaemon deleted the feat/compilation-swc-backend-native branch February 17, 2025 18:42
This was referenced Feb 18, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants