Commit d7ad947
Tree-sitter Enhancements: TSX, TypeScript, JSON, and Markdown Support (RooCodeInc#2169)
* feat: add Tree-sitter TSX query support
Added support for parsing TSX files with Tree-sitter:
- Created TSX query patterns for React components and functions
- Fixed field name issues by using direct node matching
- Added comprehensive documentation about TSX component structure
- Created a debug tool to inspect the actual tree structure
- Added test coverage for TSX parsing
- Embedded test fixture directly in the test file
Signed-off-by: Eric Wheeler <[email protected]>
* refactor: combine TypeScript and TSX queries
Reduce duplication by:
- Import TypeScript queries from typescript.ts
- Keep only TSX-specific component and JSX queries
- Update documentation to reflect combined approach
Signed-off-by: Eric Wheeler <[email protected]>
* test: update integration for parse definitions
- Adjusted logParseResult to call the actual parse definitions function using WASM from initializeWorkingParser.
- Patched TreeSitter initialization to resolve the WASM path correctly and bypass redundant init() calls.
Signed-off-by: Eric Wheeler <[email protected]>
* test: should successfully call parseSourceCodeDefinitionsForFile
Mock loadRequiredLanguageParsers to use real parser instance from initializeTreeSitter,
ensuring proper interaction between parseSourceCodeDefinitionsForFile and its dependencies.
Signed-off-by: Eric Wheeler <[email protected]>
* test: improve test structure for parseSourceCodeDefinitions
- Combine component parsing tests into a single test case
- Update test assertions to match actual parser output
- Fix interface and component definition tests to use VSCodeCheckbox as sample
Signed-off-by: Eric Wheeler <[email protected]>
* feat: improve React component detection in tree-sitter
- Add tests for complex TSX structures (nested components, HOCs)
- Enhance TSX queries to better detect React components
- Document current limitations in test file
* feat: improve React component detection in TSX files
Make TSX/React component detection more generic and robust by:
- Implementing structural pattern matching instead of specific React wrapper functions
- Adding configurable line threshold for React component inclusion (MIN_COMPONENT_LINES)
- Adding robust HTML element filtering with regex patterns
- Improving React component name handling for nested components
- Ensuring proper context handling for multi-line React components
Also update mock captures in tree-sitter tests to span at least 4 lines
to meet the MIN_COMPONENT_LINES threshold.
Signed-off-by: Eric Wheeler <[email protected]>
* refactor: use testParseSourceCodeDefinitions in logParseResult
- Moved testParseSourceCodeDefinitions function to top level
- Updated logParseResult to use testParseSourceCodeDefinitions
- Removed duplicate function from describe block
- Added console.log for debugging output
Signed-off-by: Eric Wheeler <[email protected]>
* refactor: improve React component detection output format
- Remove individual line output for components that don't meet MIN_COMPONENT_LINES threshold
- Update tests to match the new behavior where lines < MIN_COMPONENT_LINES are skipped
- Maintain range output (e.g., 1--4) for component definitions
- Preserve context for larger definitions as ranges only
Signed-off-by: Eric Wheeler <[email protected]>
* feat: add switch/case statement support to tree-sitter TypeScript parser
- Added node patterns in typescript.ts query to capture switch statements, case clauses, and default clauses
- Modified index.ts to avoid duplicate line ranges in the output
- Added test for switch/case statements with complex case blocks
- Fixed line range tracking to prevent duplicate output
Signed-off-by: Eric Wheeler <[email protected]>
* feat: add support for enum declarations in tree-sitter TypeScript parser
Signed-off-by: Eric Wheeler <[email protected]>
* feat: add support for namespace declarations in tree-sitter TypeScript parser
Added query pattern for namespace declarations (internal_module nodes) in the TypeScript parser.
This allows the parser to identify and extract namespace declarations from TypeScript code.
- Added test case to verify namespace parsing functionality
- Added query pattern to capture namespace declarations in typescript.ts
Signed-off-by: Eric Wheeler <[email protected]>
* feat: add decorator pattern support to tree-sitter TypeScript parser
- Added support for parsing complex decorators with arguments
- Added test case for Component decorator pattern
- Enhanced TypeScript queries to capture decorator definitions
Signed-off-by: Eric Wheeler <[email protected]>
* feat: add generic type declaration support to tree-sitter TypeScript parser
- Added support for parsing generic types with constraints
- Added test case for Dictionary<K extends string | number, V> pattern
- Enhanced TypeScript queries to capture generic type definitions
Signed-off-by: Eric Wheeler <[email protected]>
* test: add conditional type support to tree-sitter TypeScript parser
- Added test case for conditional type patterns like ReturnType<T>
- Verified that conditional types with infer keyword are already supported
- Enhanced inspectTreeStructure with detailed node inspection for debugging
Signed-off-by: Eric Wheeler <[email protected]>
* test: add template literal type support to tree-sitter TypeScript parser
- Added test case for template literal type patterns like EventName<T>
- Verified that template literal types are already partially supported
- Confirmed support for complex template literal patterns in conditional types
Signed-off-by: Eric Wheeler <[email protected]>
* feat: implement tree-sitter compatible markdown processor
Adds a special case implementation for markdown files that:
- Parses markdown headers and section line ranges
- Returns captures in a format compatible with tree-sitter
- Integrates with the existing parseFile function
- Includes comprehensive tests for the implementation
Signed-off-by: Eric Wheeler <[email protected]>
* fix: markdown parser not detecting sections with horizontal rules
The markdownParser was incorrectly interpreting horizontal rules (---) as setext headers when they appeared after non-header text. This caused some sections to be missed in the output.
This fix:
- Makes setext header detection more strict by requiring at least 3 = or - characters
- Adds validation for the text line before a potential setext header
- Ensures horizontal rules are not confused with setext headers
Added a test case to verify the fix works correctly with horizontal rules.
Signed-off-by: Eric Wheeler <[email protected]>
* refactor: move helper functions to dedicated file
Move test helper functions from parseSourceCodeDefinitions.test.ts to a new helpers.ts file.
Rename test file to parseSourceCodeDefinitions.tsx.test.ts to indicate it's for TSX tests.
This improves code organization by separating test helpers from test cases.
Signed-off-by: Eric Wheeler <[email protected]>
* feat: enable JSON structure display in list_code_definitions
This change allows list_code_definitions to show JSON structures by:
- Moving JSON query patterns into the JavaScript query file
- Using the JavaScript parser for JSON files
- Removing the separate JSON parser implementation
- Adding comprehensive tests for JSON parsing
Example output for a JSON file:
# test.json
0--90 | {
1--9 | "server": {
4--8 | "ssl": {
10--45 | "database": {
11--24 | "primary": {
14--18 | "credentials": {
19--23 | "pool": {
25--44 | "replicas": [
26--43 | {
30--42 | "status": {
33--41 | "metrics": {
36--40 | "connections": {
46--73 | "features": {
47--72 | "auth": {
48--71 | "providers": {
49--53 | "local": {
54--70 | "oauth": {
56--69 | "providers": [
57--68 | {
Signed-off-by: Eric Wheeler <[email protected]>
* fix: Add compile step to CI workflow to ensure WASM files are available
The TreeSitter tests were failing in CI because the WASM files weren't being copied to the dist directory before running the tests. This adds an explicit compile step to ensure the WASM files are properly built and copied.
Signed-off-by: Eric Wheeler <[email protected]>
* fix: improve tree-sitter test type safety and debug logging
- Replace 'any' types with proper Parser types in helpers.ts
- Add centralized DEBUG flag and debugLog function in helpers.ts
- Update all console.log statements to use debugLog across all test files
- This change appeases @ellipsis-dev by improving type safety and
providing a clean way to control debug logging
Signed-off-by: Eric Wheeler <[email protected]>
---------
Signed-off-by: Eric Wheeler <[email protected]>
Co-authored-by: Eric Wheeler <[email protected]>1 parent b9f4695 commit d7ad947
File tree
14 files changed
+2530
-119
lines changed- .github/workflows
- src/services/tree-sitter
- __tests__
- queries
14 files changed
+2530
-119
lines changed| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
76 | 76 | | |
77 | 77 | | |
78 | 78 | | |
| 79 | + | |
| 80 | + | |
79 | 81 | | |
80 | 82 | | |
81 | 83 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
| 33 | + | |
| 34 | + | |
| 35 | + | |
| 36 | + | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
| 42 | + | |
| 43 | + | |
| 44 | + | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
| 48 | + | |
| 49 | + | |
| 50 | + | |
| 51 | + | |
| 52 | + | |
| 53 | + | |
| 54 | + | |
| 55 | + | |
| 56 | + | |
| 57 | + | |
| 58 | + | |
| 59 | + | |
| 60 | + | |
| 61 | + | |
| 62 | + | |
| 63 | + | |
| 64 | + | |
| 65 | + | |
| 66 | + | |
| 67 | + | |
| 68 | + | |
| 69 | + | |
| 70 | + | |
| 71 | + | |
| 72 | + | |
| 73 | + | |
| 74 | + | |
| 75 | + | |
| 76 | + | |
| 77 | + | |
| 78 | + | |
| 79 | + | |
| 80 | + | |
| 81 | + | |
| 82 | + | |
| 83 | + | |
| 84 | + | |
| 85 | + | |
| 86 | + | |
| 87 | + | |
| 88 | + | |
| 89 | + | |
| 90 | + | |
| 91 | + | |
| 92 | + | |
| 93 | + | |
| 94 | + | |
| 95 | + | |
| 96 | + | |
| 97 | + | |
| 98 | + | |
| 99 | + | |
| 100 | + | |
| 101 | + | |
| 102 | + | |
| 103 | + | |
| 104 | + | |
| 105 | + | |
| 106 | + | |
| 107 | + | |
| 108 | + | |
| 109 | + | |
| 110 | + | |
| 111 | + | |
| 112 | + | |
| 113 | + | |
| 114 | + | |
| 115 | + | |
| 116 | + | |
| 117 | + | |
| 118 | + | |
| 119 | + | |
| 120 | + | |
| 121 | + | |
| 122 | + | |
| 123 | + | |
| 124 | + | |
| 125 | + | |
| 126 | + | |
| 127 | + | |
| 128 | + | |
| 129 | + | |
| 130 | + | |
| 131 | + | |
| 132 | + | |
| 133 | + | |
| 134 | + | |
| 135 | + | |
| 136 | + | |
| 137 | + | |
| 138 | + | |
| 139 | + | |
| 140 | + | |
| 141 | + | |
| 142 | + | |
| 143 | + | |
| 144 | + | |
| 145 | + | |
| 146 | + | |
| 147 | + | |
| 148 | + | |
| 149 | + | |
| 150 | + | |
| 151 | + | |
| 152 | + | |
| 153 | + | |
| 154 | + | |
| 155 | + | |
| 156 | + | |
| 157 | + | |
| 158 | + | |
| 159 | + | |
| 160 | + | |
| 161 | + | |
| 162 | + | |
| 163 | + | |
| 164 | + | |
| 165 | + | |
0 commit comments