Continue implementation of experimental Redis vector search #2430

gunthercox · 2025-12-06T19:26:25Z

No description provided.

Copilot

Pull request overview

This pull request continues the implementation of experimental Redis vector search functionality for ChatterBot, introducing semantic similarity search capabilities using vector embeddings. The PR implements a "storage-aware architecture" where storage adapters can specify their preferred tagger and search algorithm, enabling automatic optimization based on backend capabilities.

Key Changes:

Added NoOpTagger class that skips spaCy processing for vector-based storage adapters
Implemented SemanticVectorSearch algorithm that relies on storage adapter's native vector similarity
Enhanced Redis storage adapter with confidence scoring from cosine distance and improved timestamp handling
Added storage adapter methods get_preferred_tagger() and get_preferred_search_algorithm() for automatic configuration

Reviewed changes

Copilot reviewed 8 out of 8 changed files in this pull request and generated 12 comments.

Show a summary per file

File	Description
docs/storage/redis.rst	Extensive documentation updates explaining semantic vector search, comparison tables, performance considerations, and usage examples
docs/storage/index.rst	Added metadata tags and storage adapter comparison guide with recommendations for different use cases
docs/index.rst	Updated main page with modern AI capabilities including semantic vector search
chatterbot/tagging.py	Introduced `NoOpTagger` class for storage adapters that don't require POS-lemma indexing
chatterbot/storage/storage_adapter.py	Added base methods for storage adapters to specify preferred tagger and search algorithm
chatterbot/storage/redis.py	Implemented preferred tagger/search algorithm methods, enhanced confidence calculation from vector scores, improved timestamp handling, and added `search_text_contains` parameter support
chatterbot/search.py	Added `SemanticVectorSearch` class that delegates similarity calculation to storage adapter
chatterbot/chatterbot.py	Integrated storage-aware architecture with automatic tagger and search algorithm selection based on storage adapter preferences

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot · 2025-12-06T21:15:37Z

chatterbot/tagging.py

+
+    def as_nlp_pipeline(
+        self,
+        texts: Union[List[str], Tuple[str, dict]],


The type annotation Union[List[str], Tuple[str, dict]] for the texts parameter is incorrect. Based on line 36 where texts[0] is checked with isinstance(texts[0], tuple), the actual expected type should be Union[List[str], List[Tuple[str, dict]]] (a list of strings OR a list of tuples). The current annotation suggests a single tuple, but the implementation expects a list/iterable.

Copilot · 2025-12-06T21:15:37Z

docs/storage/index.rst

+
+**For Semantic AI Chatbots** (Recommended for modern conversational AI):
+
+Note that as of December 2025, the Redis Vector Storage Adapter is still an experimental betafeature.


Missing space between "beta" and "feature". Should be "beta feature" instead of "betafeature".

Suggested change

Note that as of December 2025, the Redis Vector Storage Adapter is still an experimental betafeature.

Note that as of December 2025, the Redis Vector Storage Adapter is still an experimental beta feature.

Copilot · 2025-12-06T21:15:37Z

chatterbot/storage/redis.py

+        if 'search_text' in kwargs:
+            # For now, we'll treat search_text as a filter-only parameter
+            # and fall through to the regular query_search below.
+            # This prevents the broken behavior of embedding POS tags.
+            # The proper fix requires BestMatch to pass additional context
+            # or use search_text_contains instead.
+            pass


The search_text parameter is now being ignored (lines 366-372) with just a pass statement. However, the code at line 280 now handles search_text_contains instead. This creates a potential issue where legacy code or logic adapters that pass search_text will get no results. Consider logging a deprecation warning or handling backward compatibility more explicitly to help users migrate from search_text to search_text_contains.

Copilot · 2025-12-06T21:15:38Z

chatterbot/tagging.py

+    def get_text_index_string(self, text: Union[str, List[str]]):
+        """
+        Return the text unchanged (no indexing applied).
+        """
+        return text


The type annotation for the get_text_index_string parameter includes List[str], but the function returns text unchanged. If a list is passed, it will return the list as-is rather than a string. This could cause type inconsistency issues. Consider either removing List[str] from the annotation or handling the list case explicitly (e.g., by joining list elements into a string).

Copilot · 2025-12-06T21:15:38Z

docs/storage/redis.rst

+     - ✅ Yes (similar meaning)
+   * - Confidence Scoring
+     - Levenshtein distance
+     - Cosine similarity (1 - distance/2)


[nitpick] The documentation states confidence is calculated as 1 - distance/2 for cosine similarity, but the actual implementation in the code uses 1.0 - (float(vector_score) / 2.0) where vector_score is the cosine distance. This is correct, but the documentation should be clearer that distance refers to the cosine distance (ranging from 0 to 2), not a normalized value. Consider clarifying: "Cosine similarity (1 - cosine_distance/2, where distance ranges 0-2)".

Suggested change

- Cosine similarity (1 - distance/2)

- Cosine similarity (1 - cosine_distance/2, where distance ranges 0-2)

Copilot · 2025-12-06T21:15:38Z

chatterbot/storage/redis.py

-            'created_at': kwargs.get('created_at') or int(_default_date.strftime('%y%m%d')),
+            # Store created_at as Unix timestamp with microseconds (float)
+            # This provides full datetime precision while maintaining Redis NUMERIC field compatibility
+            'created_at': kwargs.get('created_at') or _default_date.timestamp(),


The code kwargs.get('created_at') or _default_date.timestamp() has a potential bug. If kwargs.get('created_at') returns a datetime object, it won't have .timestamp() called on it, and a datetime object will be stored instead of a Unix timestamp. This should be: kwargs.get('created_at').timestamp() if kwargs.get('created_at') else _default_date.timestamp() or handle the datetime-to-timestamp conversion explicitly.

Suggested change

'created_at': kwargs.get('created_at') or _default_date.timestamp(),

'created_at': (

kwargs.get('created_at').timestamp() if isinstance(kwargs.get('created_at'), datetime)

else kwargs.get('created_at') if kwargs.get('created_at') is not None

else _default_date.timestamp()

),

Copilot · 2025-12-06T21:15:39Z

chatterbot/chatterbot.py

+        if isinstance(self.tagger, NoOpTagger):
+            # NoOpTagger returns text unchanged, so we can skip the tagging call
+            if not input_statement.search_text:
+                input_statement.search_text = input_statement.text
+            if not input_statement.search_in_response_to and input_statement.in_response_to:
+                input_statement.search_in_response_to = input_statement.in_response_to


[nitpick] The check isinstance(self.tagger, NoOpTagger) at line 215 creates a tight coupling between the ChatBot class and the NoOpTagger implementation. A more maintainable approach would be to add a method like needs_text_indexing() to the tagger interface that returns False for NoOpTagger and True for others. This would make the code more extensible if other taggers with similar behavior are added in the future.

Copilot · 2025-12-06T21:15:39Z

docs/index.rst

+**Modern AI Capabilities** (2025):
+
+- **Semantic Vector Search**: Advanced context understanding using vector embeddings and Redis vector database
+- **Large Language Model (LLM) Integration**: Direct support for Ollama and OpenAI models (in development)


The statement "Large Language Model (LLM) Integration: Direct support for Ollama and OpenAI models (in development)" mentions features that are in development. This could be misleading in documentation as it suggests features that don't yet exist. Consider removing this line or moving it to a separate "Roadmap" or "Planned Features" section to avoid confusion.

Copilot · 2025-12-06T21:15:39Z

docs/storage/redis.rst

+
+   # These inputs find similar responses despite different words:
+   response1 = chatbot.get_response("What's the weather like?")
+   response2 = chatbot.get_response("How's the climate today?")  


There's a trailing space after the period on line 167. This should be removed for consistency.

Suggested change

response2 = chatbot.get_response("How's the climate today?")

response2 = chatbot.get_response("How's the climate today?")

Copilot · 2025-12-06T21:15:39Z

chatterbot/chatterbot.py

+        from chatterbot.search import SemanticVectorSearch
+


[nitpick] There's a blank line after the import statement that creates inconsistent spacing. Consider removing this trailing blank line for cleaner code formatting.

gunthercox added 6 commits December 6, 2025 11:01

Add important note to redis adapter

64627f9

Implement search text fields

300a17e

Add noop tagger

3766967

Add dedicated search class

70e0786

Order results by redis simalarity

86319b1

Update documentation

9d74c4b

gunthercox requested a review from Copilot December 6, 2025 21:10

Copilot started reviewing on behalf of gunthercox December 6, 2025 21:10 View session

Copilot AI reviewed Dec 6, 2025

View reviewed changes

gunthercox added 2 commits December 6, 2025 16:33

Simplify tagger logic

87f8534

Update imports

7814c51

gunthercox merged commit c3b0b6c into master Dec 7, 2025
8 checks passed

gunthercox deleted the redis branch December 7, 2025 20:16


		For Semantic AI Chatbots (Recommended for modern conversational AI):

		Note that as of December 2025, the Redis Vector Storage Adapter is still an experimental betafeature.

	- Cosine similarity (1 - distance/2)
	- Cosine similarity (1 - cosine_distance/2, where distance ranges 0-2)

-            'created_at': kwargs.get('created_at') or _default_date.timestamp(),
+            'created_at': (
+                kwargs.get('created_at').timestamp() if isinstance(kwargs.get('created_at'), datetime)
+                else kwargs.get('created_at') if kwargs.get('created_at') is not None
+                else _default_date.timestamp()
+            ),

	response2 = chatbot.get_response("How's the climate today?")
	response2 = chatbot.get_response("How's the climate today?")

Uh oh!

Continue implementation of experimental Redis vector search #2430

Continue implementation of experimental Redis vector search #2430

Uh oh!

Conversation

gunthercox commented Dec 6, 2025

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Copilot AI Dec 6, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Dec 6, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Dec 6, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Dec 6, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Dec 6, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Dec 6, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Dec 6, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Dec 6, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Dec 6, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Dec 6, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants