Indexing bugs

[40Think]

• Disable all other plugins besides Copilot (required)
• Screenshot of note + Copilot chat pane + dev console added (required)

Logs attached
Copilot version: 3.1.0

Bug Report – Copilot Plus & Smart Connections (Obsidian plugins)
Submitted by a user who has already sent the relevant log files.

---

1. Summary

When indexing a large Obsidian vault with Copilot Plus (and, to a lesser extent, Smart Connections) the application frequently hangs or crashes. The problem appears tied to:

• high‑frequency LLM calls for embedding generation,
• batch size and context window settings,
• long absolute file paths on Ubuntu,
• complex folder hierarchies / non‑alphanumeric characters, and
• interaction with other plugins or the Obsidian developer console.

Smart Connections is comparatively stable, but both plugins suffer from index corruption after a freeze.

---

2. Environment

Item	Value (as reported)
OS	Ubuntu 22.04 (64‑bit)
Obsidian version	(user did not specify – logs contain the exact build)
Copilot Plus version	(see attached logs)
Smart Connections version	(see attached logs)
GPU	High‑end card with ample VRAM (able to process thousands of embedding batches)
Embedding model	quern-3-8b-embedding-16f (context window 40 k tokens, batch size set to 100/128)
Vault size	Tens of thousands of notes (some very small, many larger)
File‑system	Ext4, absolute paths used by the plugins

---

3. Steps to Reproduce

1. Prepare a fresh environment

   • Close all other applications.
   • Disable every Obsidian plugin except Copilot Plus (or Smart Connections).
   • Do not open the developer console – it dramatically raises the chance of a freeze.

2. Configure the vault

   • Use only alphanumeric characters (letters & digits) in folder and file names.
   • Keep paths as short as possible; avoid deep hierarchies.
   • Ensure most notes are small (< few KB).

3. Set indexing parameters

   • In Copilot Plus: batch size = 128 embeddings, embedding‑model context window = small (e.g., < 10 k tokens).
   • In Smart Connections: default settings (no special changes needed).

4. Start the indexing process while the computer is idle – no mouse/keyboard activity, no other software running, and ideally after a full system reboot.

5. Observe the UI for hangs, freezes, or crashes. When they occur, note whether the plugin shows an error message or simply becomes unresponsive (white/black screen).

6. Repeat with different batch sizes (e.g., 100 vs 128) and with single‑folder indexing (index one folder at a time, restart Obsidian between folders).

---

4. Expected Behaviour

• The plugin should process all notes, build the embedding index, and finish without hanging or corrupting the index files, regardless of vault size or batch configuration.
• Opening the developer console must not interfere with indexing.
• Errors (if any) should be reported in the UI or console rather than causing a complete freeze.

---

5. Actual Behaviour

Observation	Details
Frequent hangs / freezes	Copilot Plus becomes completely unresponsive (white/black screen) during indexing, especially when the developer console is open.
Index corruption	After a hang, the generated index files are sometimes left in an inconsistent state; subsequent attempts to re‑index fail until the entire index is deleted and rebuilt from scratch.
Batch size impact	Larger batch sizes (128) increase the probability of hangs, despite the GPU having sufficient VRAM. Small batches appear to overload internal logging mechanisms, leading to similar failures.
Path length issue	On Ubuntu the plugins resolve absolute paths; extremely long paths cause failures even when the relative path inside the vault is short.
Folder hierarchy sensitivity	Deep or complex hierarchies (many nested folders) and non‑alphanumeric characters in names dramatically raise failure rates.
Plugin interaction	Even with all other plugins disabled, Copilot Plus still crashes; however, having any additional plugin active raises the overall instability of both plugins.
Smart Connections stability	Generally stable – crashes are rare, but if the index is corrupted by Copilot Plus it can inherit the same problem.

---

6. Workarounds Attempted (by the user)

1. Full system reboot before each indexing run.
2. Index one folder at a time (hundreds of files max) and restart Obsidian between folders.
3. Keep all notes small and use very short, alphanumeric file/folder names.
4. Delete the whole index and start from zero whenever a hang occurs.
5. Avoid any mouse/keyboard activity during indexing; keep the cursor out of the Obsidian window.

These mitigations improve success probability but do not eliminate the underlying issue.

---

7. Logs & Attachments

The user has already sent the complete log files (see attached copilot_plus.log and smart_connections.log).
Key excerpts show repeated “Unhandled exception” messages followed by a silent UI freeze when the developer console is open.

---

8. Suggested Improvements for the Developers

Area	Recommendation
Graceful error handling	Catch and surface any embedding‑generation failures instead of allowing them to block the main thread (especially when the dev console is active).
Batch‑size safety checks	Dynamically limit batch size based on available GPU memory and internal logging load; provide a “safe mode” that auto‑tunes these values.
Path handling	Normalize paths to relative vault roots before persisting index entries, or enforce a maximum absolute path length on Ubuntu.
File‑name validation	Reject or automatically sanitize non‑alphanumeric characters during indexing to avoid hidden parsing errors.
Index recovery	Implement an atomic write/commit strategy for the embedding index so that partial writes do not corrupt the whole database; add a “repair” command.
Developer console isolation	Ensure that opening the dev console does not block the UI thread of Copilot Plus; run indexing in a WebWorker or separate process.
Documentation of known limits	Clearly state in the README the recommended vault structure (short paths, small files, limited hierarchy) and the impact of batch size on stability.
Testing matrix expansion	Add automated tests that combine Copilot Plus with a handful of popular third‑party plugins to detect cross‑plugin deadlocks early.

---

9. Conclusion

The current implementation of Copilot Plus (and, indirectly, Smart Connections) is highly sensitive to vault size, file‑system layout, batch configuration, and interaction with the Obsidian developer console. The user’s extensive manual testing demonstrates reproducible failure modes that lead to UI freezes and index corruption. Implementing the suggested safeguards would greatly increase robustness for power users who need to index large knowledge bases.

obsidian.md-1760869775478.log
obsidian.md-1760869753350.log
obsidian.md-1760869737365.log
obsidian.md-1760869673450.log
copilot-log.md

[ichts]

I think it's JSON's size limit issue. cc @logancyang

[logancyang]

We have moved over to index-free search in v3. Our "semantic search" option is a backward-compatible mode. Having the vector db inside the Electron/browser environment has its limitations. The right way for larger vaults is to use agentic search and/or with a dedicated standalone local vector db.

With the upcoming "self-host" mode we will have a much more scalable local vector db that is cross-platform but separate from the Obsidian environment.

FYI this is on our website:

[40Think]

@logancyang At this moment, I have found two ways to index a large number of files using your system. The first variant. Instead of running global indexing for the entire vault, I go into each note separately and press the button on the top right that indexes only that one. In such a case, so far there were no problems — never once has anything frozen in this setup with 1500 notes indexed this way. The second variant I found is: if files are very small, it was possible to index 18 thousand files. But still, sometimes there are failures which appear to be initiated not just by a single file; rather, some part of the system — perhaps access control, logging, or buffer overflows — causes these failures and freezes. It's my assumption that some queues for user action control, resource allocation, etc., cause these errors and hangs, but your system can definitely index many files. I cannot say though whether this will continue for weeks or months; indexing one file at a time as they appear might lead to another failure tomorrow or tonight — and everything could collapse again. For now it works.

If I were in your place, I would analyze what the difference is between indexing a large number of small files versus big ones. Also, what’s the difference when you index large files one by one vs. running batches of 50–100 at once?

Perhaps it would be good to conduct a small experiment — run a version of the indexing system that emulates the loop which is triggered when each file gets indexed individually and simply feed files into it one by one. Possibly if we change the architecture of mass indexing, these errors will disappear. Naturally, in the settings we'd need to hardcode batch size to exactly one file per batch. And so-called, let’s name this mode “BigVault Index Mode” or “Safety Mode”.

[40Think]

Manual index 1700 files, i think this method save index to RAM? I dont see file modification. But index dont save after app crash.

[logancyang]

No index doesn't save to RAM, it saves to disk by batch with checkpoints. What file are you looking at? Use "list all indexed files" command for more details.

[logancyang]

We have tested with several other users with 100GB+ vaults. They are using a similar approach you described - index a subset and add more with the button one by one. It's not worth the engineering hours since this approach has clear limitations. We are moving to real vector dbs in self-host mode as a proper solution.

[40Think]

e are moving to real vector dbs in self-host mode as a proper solution.

I research about better RAG system. Current my preference - https://platform.cognee.ai/auth
https://docs.cognee.ai/core-concepts/architecture

Graph+Vector human-like memory, intellectual agentic graph generation inside this memory. Can you integrate this? Maybe in MCP mode? Or API call from you agent?

Relational store — Tracks your documents, their chunks, and provenance (i.e. where each piece of data came from and how it’s linked to the source).
Vector store — Holds embeddings for semantic similarity (i.e. numerical representations that let Cognee find conceptually related text, even if the wording is different).
Graph store — Captures entities and relationships in a knowledge graph (i.e. nodes and edges that let Cognee understand structure and navigate connections between concepts).