-
Notifications
You must be signed in to change notification settings - Fork 487
[storage] Collections cleanup #34560
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
8322316 to
47a29da
Compare
|
You'll want to rebase on main to get rid of the cargo test flake. |
We report status via the clusterd protocol now, not via side-channel collections.
Awkward fit for storage collections, but this at least avoids some refactoring...
Sad to remove asserts, but the point is to no longer have StorageCollections understand the details of what it's managing.
All the remaining code exists only to keep things in sync, which we don't need.
47a29da to
5dee7c8
Compare
aljoscha
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I like it, nice cleanup in there!
| pub description: CollectionDescription<T>, | ||
|
|
||
| time_dependence: Option<TimeDependence>, | ||
| ingestion_remap_collection_id: Option<GlobalId>, |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
feels like we're taking a step backwards, with now adding a field that is specific for collections, while other parts of the PR remove specific knowledge about the collections we're managing
but also fine, I might now know all the context 🤷♂️
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Totally. We unfortunately still need it to resolve the remap collection in create_collections_for_bootstrap, though, and without adding this field we wouldn't be able to remove the DataSource
I think the next incremental step is to change the storage controller to pre-resolve this info, so that storage collections doesn't need to know about the remap business at all, and can just track it as an opaque dependency. But that's followup work...
Tease out some unnecessary code from
StorageCollections.data_sourcefield and all the code that keeps it in sync.Motivation
It's very hard to make changes to storage collections! Simplifying makes it a bit easier to experiment with details of eg. how
primarysemantics work.Breaking this out from my experiments since I think it's valuable cleanup in general and because I don't want it to bitrot as we get into the holidays.
Tips for reviewer
Very fine-grained commits, mostly for my own sanity - but it may help review as well.