DB v4 progress #2385
Draft
DB v4 progress #2385
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add test_degree_iterable as a rust test
* USe META_FILE_NAME wherever .raph is used * Iterate over edges correctly in serialise
… construct correctly still)
* attempt at fixing the graph writing logic for windows * fmt * It should never be possible for a graph to be de-cached while it is being mutated. If we allow that to happen, reinserting it doesn't fix anything as the graph might already have been re-cached as well and state is inconsistent. * fmt * initial implementation of dirty path tracking * more validation * need to clean up dirty paths during validation * Refactor writing to disk such that it writes with the new folder structure and refactor the validation logic (compiles but does not work yet) * fix writing to empty graph folder * move a lot more of the logic to GraphFolder * fix zip encoding/decoding * all the tests compile * all features compile * move the python benchmarks so they don't always run * make secondary_index the last argument so it doesn't become annoying * improved error messages * need to get the graph before creating the new folder or the new path gets cleaned up again! * don't create arbitrarily deep paths when writing graphs * better error messages * overwrite is handled internally, no need to call delete * fix doc strings * no more deserialize for Prop * load graph metadata from the parquet file instead * tidy up a lot of warnings * fmt * tidy up and add more validation for relative paths * need to bring back Prop deserialisation for the WAL * this reserve causes a race condition as it re-checks the count and is probably not really helpful anyway * flat serialisation of Prop for working with arrow * simplify serialisation for SerdeArrowArray * more refactoring of the graph path handling * avoid writing metadata without writing graph * is_reserved should not be true for files * materialize_at only works with persistent storage * need to write the metadata in decode_at variants * load metadata correctly when storage is enabled and use cached graph when available * fix send_graph * fix new_graph in graphql * improve error messages * always use / in zip paths * fmt * rearrange tests to ignore indexing/vectors * look at .raph file instead * tweak seed to get the old result as order of operations changed * hopefully more robust fast_rp test * chore: apply tidy-public auto-fixes * update graph to new format * no unzipping needed anymore * some minor cleanup * track io failure location * fix storage check * make sure all dirty paths are cleaned up properly * fix drop handling for storage-enabled graphs * use send_graph instead of save to also test the storage * remove stray println! * don't panic on errors * delete should remove the graph from cache * make sure load preserves the id type * fmt * try to fix the graphql benchmark workflow * use send_graph to test storage * add invalidation to make sure cached graphs are dropped before we delete them * rename disk_storage_enabled to disk_storage_path * rename inner metadata file to .meta and address remaining review comments --------- Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
Progress towards the new version of the underlying storage
Why are the changes needed?
Does this PR introduce any user-facing change? If yes is this documented?
How was this patch tested?
Are there any further changes required?