[BUG] Code indexing fallback broken

[SeverinBang]

Problem (one or two sentences)

I have a julia project and noticed only the Manifest.toml and Project.toml where indexed.
As a MWE, I generated two files (attached), one python file (tree-sitter supported, python wasm file available here) and one julia file (not officially tree-sitter supported, and no wasm file). I put them in a new folder, started a fresh VSCode instance in this folder and tested the codebase indexing in the following way:

1. started codebase indexing: -> 31 blocks indexed
2. created a .rooignore file and put the julia file there, cleared the index and restarted the indexing: -> 31 blocks indexed
3. cleared the index, removed the julia file from rooignore and put the python file there: -> 0 blocks indexed
4. changed the file extensions to .txt in both files and deleted the .rooignore, cleared codebase index and restarted: -> 0 blocks indexed
5. changed the file extensions to .md in both files, cleared the codebase index and restarted indexing: -> 19 blocks indexed

For me this tells me, that the fallback for not tree-sitter supported languages/files is brocken and indexing is skipped altogether.

My system:

• OS: Archlinux
• VS Code version: 1.105.1
• Roo Code version: 3.29.4
• embeding model: mxbai-embed-large:latest (via ollama, also tested with nomic-embed-text:latest)
• Ollama version: 0.12.9
• qdrant version: 1.15.2

Do you need me to provide additional information?

NOTE: i neded to rename the julia file from julia_file.jl -> julia_file.txt, otherwise I could not upload it. Please rename it back to julia_file.jl before testing.
julia_file.txt
python_file.py

Context (who is affected and when)

Affects all users with codebase indexing for non-tree-sitter supported languages.

Reproduction steps

1. create new folder with the two attached files in vscode (rename the julia file)
2. put the python file on a .rooignore file
3. start clean file indexing with your choosen setup
4. try some phrases like:
   • Search my codebase for "DataPoint struct definition"
   • Search my codebase for "function to calculate statistics mean median"
   • Search my codebase for "data filtering by value range"
   • Search my codebase for "module DataAnalytics exports"
   • Search my codebase for "Advanced analysis operations on processed data"

Expected result

Returns julia_file.jl with line number

Actual result

No results (confirming fallback is broken)

Variations tried (optional)

I tried different embedding models

App Version

3.29.4

API Provider (optional)

Ollama

Model Used (optional)

mxbai-embed-large:latest

Roo Code Task Links (optional)

No response

Relevant logs or errors (optional)

[roomote]

I'm exploring a potential fix for the code indexing fallback issue and will open a PR to share progress

[roomote]

Opened a PR with my attempt at addressing this issue. Feedback welcome!

PR: #8970

The fix adds .jl to the supported extensions list and configures it to use fallback chunking for indexing. This should resolve the issue where Julia files were not being indexed at all.

[daniel-lxs]

Thanks for the detailed report and reproduction steps!

Just to clarify - not every language is currently supported by our Tree-Sitter indexing pipeline, and we don’t have a general fallback for unsupported file types. This is intentional: it’s non-trivial to safely decide whether an arbitrary file should be treated as plain text for indexing (especially given binary formats and language-specific syntax).

So this behavior is expected rather than a bug. If you’d like Julia (or other unsupported languages) to be indexed, the right path would be to open a feature request for adding Tree-Sitter or alternative parsing support for that language.