Skip to content

Comments

Improve subsource planning#31885

Merged
antiguru merged 4 commits intoMaterializeInc:mainfrom
antiguru:adapter_source_scalability
Mar 15, 2025
Merged

Improve subsource planning#31885
antiguru merged 4 commits intoMaterializeInc:mainfrom
antiguru:adapter_source_scalability

Conversation

@antiguru
Copy link
Member

@antiguru antiguru commented Mar 14, 2025

Improve subsource planning by making various parts of the adapter more scalable.

This PR includes changes to:

  • The durable transaction implementation to improve updates and validity checks. Updates are more efficient by cloning less and passing references where possible. Validity checks are more efficient by converting some that assert on name equality to a $n\cdot\log n$, where it previously was $n^2$.
  • Planning subsources is more efficient by allocating all IDs in one operation rather than one-by-one.
  • Avoiding computing trace data in the storage client when the level isn't high enough.

Takes planning 10k subsources from I-terminated-it-after-one-hour to around one minute.

Checklist

  • This PR has adequate test coverage / QA involvement has been duly considered. (trigger-ci for additional test/nightly runs)
  • This PR has an associated up-to-date design doc, is a design doc (template), or is sufficiently small to not require a design.
  • If this PR evolves an existing $T ⇔ Proto$T mapping (possibly in a backwards-incompatible way), then it is tagged with a T-proto label.
  • If this PR will require changes to cloud orchestration or tests, there is a companion cloud PR to account for those changes that is tagged with the release-blocker label (example).
  • If this PR includes major user-facing behavior changes, I have pinged the relevant PM to schedule a changelog post.

@antiguru antiguru requested review from a team as code owners March 14, 2025 16:54
@antiguru antiguru requested a review from aljoscha March 14, 2025 16:54
@antiguru antiguru force-pushed the adapter_source_scalability branch from bdc2209 to 45ae7c9 Compare March 14, 2025 16:54
Signed-off-by: Moritz Hoffmann <mh@materialize.com>
Signed-off-by: Moritz Hoffmann <mh@materialize.com>
@antiguru antiguru force-pushed the adapter_source_scalability branch from 45ae7c9 to 62a7b99 Compare March 14, 2025 19:31
@antiguru antiguru requested a review from ParkMyCar March 14, 2025 19:37
Signed-off-by: Moritz Hoffmann <mh@materialize.com>
Copy link
Contributor

@ParkMyCar ParkMyCar left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice! LGTM!

@ParkMyCar
Copy link
Contributor

If you don't mind kicking off a round of Nightlies, it would be much appreciated!

Signed-off-by: Moritz Hoffmann <mh@materialize.com>
@antiguru
Copy link
Member Author

Nightly in https://buildkite.com/materialize/nightly/builds/11470, but I couldn't trigger it through our tool for whatever reason.

@antiguru antiguru merged commit 62d2c7d into MaterializeInc:main Mar 15, 2025
245 of 251 checks passed
@antiguru antiguru deleted the adapter_source_scalability branch March 15, 2025 07:22
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants