Feat/Fix: Fix/Improve search tool #307
Open
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
* Implement token-based, whitespace-robust search in filterTools:
* Normalize queries by trimming, lowercasing, and collapsing internal whitespace.
* Split normalized query into tokens and require every token to match at least one of: localized name, description, shortDescription, or keywords.
* Keep user-type filtering behavior the same and apply it before search filtering.
* Return all tools (post user-type filter) when the normalized query is empty or only whitespace.
* Make Hero autocomplete respect the new search behavior:* Use filterOptions={(options) => options} to disable MUI Autocomplete’s internal filtering.
* Rely solely on filterTools + filteredTools so queries like "pdf " behave the same as "pdf" and support multi-word, order-insensitive searches.
* Add unit tests for Phase 1 search behavior:
* New src/tools/index.test.ts covering:
* Empty and whitespace-only queries returning all tools.
* Word-order and extra-whitespace robustness for queries like "pdf merge" vs " merge pdf ".
* Case-insensitive matching and multi-word queries for the Base64 tool.
* Trailing spaces being ignored.
* Non-English behavior by using localized Spanish strings (e.g. "unir pdf") to confirm locale-agnostic token matching.
Co-Authored-By: Warp <[email protected]>
• Extend filterTools query normalization with alpha+digit token concatenation (e.g. base + 64 -> base64) • Document keyword-based synonym usage in filterTools • Expand filterTools tests to cover alias-style queries like pdf join and base 64 Co-Authored-By: Warp <[email protected]>
• Add weighted scoring and Levenshtein-based fuzzy matching in filterTools • Ensure title matches rank above description-only matches and keep existing token behavior
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Problems
Search was strict substring match on entire query string - word order and exact spacing mattered, any typo broke matches. Examples: "pdf merge" didn't match "Merge PDF", "base 64" didn't match "Base64 Encoder/Decoder".
Keywords, description and short description were not being included in the search (although it seems that the intention was they should be)
Changes
• Token-based matching: Query is normalised (trimmed, lowercased, split on whitespace), then each token must appear as substring in tool's searchable text (name, description, shortDescription, keywords). Order no longer matters.
• Fuzzy matching with Levenshtein distance: Tokens within edit distance 1 are considered weak matches, so small typos still find tools.
• Relevance scoring: Tools ranked by weighted score (name matches weighted higher than description/keywords), exact matches weighted higher than fuzzy.
• Token variants: Handles patterns like "base 64" → "base64" by generating merged token variants.
Examples that now work
Keyword cleanup
While testing, cleaned up keywords in various tool meta files - removed duplicates, fixed inconsistencies, added missing synonyms (e.g. "b64" for base64, "join" for merge-pdf).
Tests
Added src/tools/index.test.ts with comprehensive tests covering token matching, case insensitivity, keyword synonyms, typo tolerance, and ranking behaviour.
Manually tested performance and the search isn't noticeably slower.
Comments
Keywords are still hardcoded as english but after this change it would be trivial to extract keywords to a single i18n string of search terms.