Skip to content

Commit c9dff5c

Browse files
authored
[ty] AST garbage collection (astral-sh#18482)
## Summary Garbage collect ASTs once we are done checking a given file. Queries with a cross-file dependency on the AST will reparse the file on demand. This reduces ty's peak memory usage by ~20-30%. The primary change of this PR is adding a `node_index` field to every AST node, that is assigned by the parser. `ParsedModule` can use this to create a flat index of AST nodes any time the file is parsed (or reparsed). This allows `AstNodeRef` to simply index into the current instance of the `ParsedModule`, instead of storing a pointer directly. The indices are somewhat hackily (using an atomic integer) assigned by the `parsed_module` query instead of by the parser directly. Assigning the indices in source-order in the (recursive) parser turns out to be difficult, and collecting the nodes during semantic indexing is impossible as `SemanticIndex` does not hold onto a specific `ParsedModuleRef`, which the pointers in the flat AST are tied to. This means that we have to do an extra AST traversal to assign and collect the nodes into a flat index, but the small performance impact (~3% on cold runs) seems worth it for the memory savings. Part of astral-sh/ty#214.
1 parent 76d9009 commit c9dff5c

File tree

824 files changed

+26966
-2527
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

824 files changed

+26966
-2527
lines changed

Cargo.lock

Lines changed: 7 additions & 0 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

Cargo.toml

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -51,6 +51,7 @@ aho-corasick = { version = "1.1.3" }
5151
anstream = { version = "0.6.18" }
5252
anstyle = { version = "1.0.10" }
5353
anyhow = { version = "1.0.80" }
54+
arc-swap = { version = "1.7.1" }
5455
assert_fs = { version = "1.1.0" }
5556
argfile = { version = "0.2.0" }
5657
bincode = { version = "2.0.0" }

crates/ruff_db/Cargo.toml

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -21,6 +21,7 @@ ruff_source_file = { workspace = true }
2121
ruff_text_size = { workspace = true }
2222

2323
anstyle = { workspace = true }
24+
arc-swap = { workspace = true }
2425
camino = { workspace = true }
2526
countme = { workspace = true }
2627
dashmap = { workspace = true }

0 commit comments

Comments
 (0)