Improve Re-indexing and Workflow Efficiency for Better Usability

[iinkov]

The plugin has some key issues that need attention for better usability and performance:
1. Re-indexing of Excluded Files
• Once a file is excluded from the index, it cannot be added back unless it is copied or recreated. This creates unnecessary complexity.
2. Indexing Priority
• Indexing functionality is one of the most critical aspects of the plugin but seems to have a low prioritization. Addressing this should be a top priority, as it is core to the plugin’s usefulness.
3. Combining Garbage Collection and Refresh
• The plugin became more usable after garbage collection was introduced. However, combining the garbage collection and refresh functions into a single command would further simplify workflow and improve efficiency.

Additional Notes

Despite these issues, the plugin is outstanding overall. It looks professional, has an excellent design for all elements, and shows great potential.

[logancyang]

@iinkov Indexing improvement is a big initiative right now, thanks for sharing your experience!

To clarify, by point 1, do you mean you cannot add back the excluded files when you use "refresh index"? If so, that's indeed the current limitation. When you modify the exclusion or inclusion filters, it needs a complete reindex (clear and index, or force reindex) to take effect. But if even force reindex doesn't add back the prev excluded files, let me know, that shouldn't happen.

My current dev work:

• Index partition for extremely large vaults
• Better "refresh index" command with garbage collection and filter updates (avoid the need to re-index where possible). This should enable "incremental indexing", where one can change the inclusion filter and index subsets of the files without removing the previous index.
• Performance improvements

This is my top priority and should be ready soon.

[iinkov]

Full reindex fixes all the issues (though sometimes it stalls), and then I restart from the very beginning. This process is problematic because reindexing takes a lot of time (up to half an hour in my case). The timeout setting doesn’t work at all. I even tried setting it to more than 30 in the config file manually, but it stalls more frequently.

I use GPT embedding and recently migrated from Smart Connection. The plugin is very buggy, but its indexing works much better—it processes 15k symbols per second and continues after any interruption.

The issue isn’t intuitive. If finding a long-term solution takes too much time, I’d suggest a quick fix:

Just bring GC under the reindexing button.

Exclusion-related issues aren’t critical.

However, there’s a powerful technique for collecting all relevant info about a topic into one folder, which does require exclusions.

[logancyang]

Keep an eye on this next release v2.7.10, which I will publish soon.

[iinkov]

I didn't find any issues. Thx for quick reaction. Deletion/Renaming, Excluding/Including, incremental indexing are good