Commit c9dff5c
authored
[ty] AST garbage collection (astral-sh#18482)
## Summary
Garbage collect ASTs once we are done checking a given file. Queries
with a cross-file dependency on the AST will reparse the file on demand.
This reduces ty's peak memory usage by ~20-30%.
The primary change of this PR is adding a `node_index` field to every
AST node, that is assigned by the parser. `ParsedModule` can use this to
create a flat index of AST nodes any time the file is parsed (or
reparsed). This allows `AstNodeRef` to simply index into the current
instance of the `ParsedModule`, instead of storing a pointer directly.
The indices are somewhat hackily (using an atomic integer) assigned by
the `parsed_module` query instead of by the parser directly. Assigning
the indices in source-order in the (recursive) parser turns out to be
difficult, and collecting the nodes during semantic indexing is
impossible as `SemanticIndex` does not hold onto a specific
`ParsedModuleRef`, which the pointers in the flat AST are tied to. This
means that we have to do an extra AST traversal to assign and collect
the nodes into a flat index, but the small performance impact (~3% on
cold runs) seems worth it for the memory savings.
Part of astral-sh/ty#214.1 parent 76d9009 commit c9dff5c
File tree
824 files changed
+26966
-2527
lines changed- crates
- ruff_db
- src
- ruff_graph/src
- ruff_linter/src
- checkers/ast
- analyze
- docstrings
- importer
- rules
- airflow/rules
- flake8_2020/rules
- flake8_annotations
- rules
- flake8_bandit/rules
- flake8_bugbear/rules
- flake8_comprehensions/rules
- flake8_pie/rules
- flake8_pyi/rules
- flake8_pytest_style/rules
- flake8_quotes/rules
- flake8_raise/rules
- flake8_return
- flake8_simplify/rules
- flake8_tidy_imports/rules
- flake8_type_checking
- flynt
- rules
- isort
- rules
- pandas_vet/rules
- perflint/rules
- pycodestyle/rules
- pydoclint/rules
- pylint/rules
- pyupgrade/rules
- pep695
- refurb
- rules
- ruff/rules
- tryceratops/rules
- ruff_python_ast_integration_tests/tests
- ruff_python_ast
- src
- visitor
- ruff_python_codegen/src
- ruff_python_formatter
- src
- comments
- expression
- module
- other
- pattern
- statement
- type_param
- tests
- ruff_python_parser
- src
- parser
- snapshots
- snapshots
- tests/snapshots
- ruff_python_semantic/src
- analyze
- model
- ty_project/src
- ty_python_semantic/src
- semantic_index
- types
Some content is hidden
Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.
824 files changed
+26966
-2527
lines changedSome generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
51 | 51 | | |
52 | 52 | | |
53 | 53 | | |
| 54 | + | |
54 | 55 | | |
55 | 56 | | |
56 | 57 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
21 | 21 | | |
22 | 22 | | |
23 | 23 | | |
| 24 | + | |
24 | 25 | | |
25 | 26 | | |
26 | 27 | | |
| |||
0 commit comments