Skip to content

Conversation

@KSDaemon
Copy link
Member

@KSDaemon KSDaemon commented Mar 27, 2025

Switch to using faster xxhash for cache keys and versions of data models instead of slow crypto.createHash('md5').
Use the fastest node xxhash implementation, written in ... of course rust :)

XXH3: generates 64 or 128-bit hashes, using vectorized arithmetic.

Here are some more info for those who is interested:

Changes:

  • use xxh3 in server-core package
  • use xxh3 in query-orchestrator package
  • use xxh3 in backend-shared package
  • use xxh3 in cubestore-driver package
  • add @node-rs/xxhash package

@KSDaemon KSDaemon force-pushed the feat/xxhash branch 5 times, most recently from 64626e7 to bf7a59a Compare March 27, 2025 16:01
@KSDaemon KSDaemon marked this pull request as ready for review March 27, 2025 21:11
@KSDaemon KSDaemon requested review from a team as code owners March 27, 2025 21:11
const key = crypto.createHash('md5')
.update(args.map((v) => JSON.stringify(v)).join(','))
.digest('hex');
const key = xxh3.xxh64(args.map((v) => JSON.stringify(v)).join(',')).toString(16);
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's make a function somewhere in @cubejs-backend-shared, with a name like defaultHasherHex16 (or defaultHasher and a tiny interface over it) and use it everywhere in the code instead of defining and requiring @node-rs/xxhash in each package.

This approach is how the Rust language does it: DefaultHasher. It will allow us to change it on the fly for testing via envs or replace the implementation whenever necessary.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Absolutely agree! Great point!

@codecov
Copy link

codecov bot commented Apr 7, 2025

Codecov Report

❌ Patch coverage is 76.92308% with 3 lines in your changes missing coverage. Please review.
✅ Project coverage is 47.88%. Comparing base (452633c) to head (fc996a4).
⚠️ Report is 538 commits behind head on master.

Files with missing lines Patch % Lines
...ackages/cubejs-server-core/src/core/CompilerApi.js 40.00% 3 Missing ⚠️

❗ There is a different number of reports uploaded between BASE (452633c) and HEAD (fc996a4). Click for more details.

HEAD has 2 uploads less than BASE
Flag BASE (452633c) HEAD (fc996a4)
cubesql 2 0
Additional details and impacted files
@@             Coverage Diff             @@
##           master    #9397       +/-   ##
===========================================
- Coverage   76.40%   47.88%   -28.53%     
===========================================
  Files         400      171      -229     
  Lines      104055    21306    -82749     
  Branches     3685     3685               
===========================================
- Hits        79508    10203    -69305     
+ Misses      24112    10668    -13444     
  Partials      435      435               
Flag Coverage Δ
cube-backend 47.88% <76.92%> (+<0.01%) ⬆️
cubesql ?

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@KSDaemon KSDaemon marked this pull request as draft April 25, 2025 17:36
@KSDaemon
Copy link
Member Author

KSDaemon commented Oct 9, 2025

Just closing

@KSDaemon KSDaemon closed this Oct 9, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants