new: decouple colbert query and document tokenizer

[joein]

decouple colbert query and document tokenizer in order to avoid problems with multithreading like

File "/usr/local/lib/python3.12/dist-packages/fastembed/late_interaction/colbert.py", line 94, 
in _tokenize_query self.tokenizer.enable_padding( File "/app/model/model.py", line 58,
in predict embeddings = list(self.text_model.query_embed(texts)) 
RuntimeError: Already borrowed

[coderabbitai]

📝 Walkthrough

Walkthrough

• Adds a dedicated query tokenizer to Colbert (new public attribute: query_tokenizer: Optional[Tokenizer]).
• During ONNX model initialization, loads and configures query_tokenizer via load_tokenizer with truncation and padding rules (max length = current_max_length - 1; pad to MIN_QUERY_LENGTH using MASK).
• _tokenize_query now uses query_tokenizer.encode_batch([query]) and removes prior ad-hoc padding logic.
• Document tokenization (_tokenize_documents) continues to use the existing tokenizer.
• Imports updated to include Tokenizer type and load_tokenizer utility.
• Changes are confined to fastembed/late_interaction/colbert.py and separate query vs. document tokenization paths.

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~20 minutes

Tip

🔌 Remote MCP (Model Context Protocol) integration is now available!

Pro plan users can now connect to remote MCP servers from the Integrations page. Connect with popular remote MCPs such as Notion and Linear to add more context to your reviews and chats.

✨ Finishing Touches

• 📝 Generate Docstrings

🧪 Generate unit tests

• Create PR with unit tests
• Post copyable unit tests in a comment
• Commit unit tests in branch colbert-query-tokenizer

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

🪧 Tips

Chat

There are 3 ways to chat with CodeRabbit:

• Review comments: Directly reply to a review comment made by CodeRabbit. Example:
  • I pushed a fix in commit <commit_id>, please review it.
  • Open a follow-up GitHub issue for this discussion.
• Files and specific lines of code (under the "Files changed" tab): Tag @coderabbitai in a new review comment at the desired location with your query.
• PR comments: Tag @coderabbitai in a new PR comment to ask questions about the PR branch. For the best results, please provide a very specific query, as very limited context is provided in this mode. Examples:
  • @coderabbitai gather interesting stats about this repository and render them as a table. Additionally, render a pie chart showing the language distribution in the codebase.
  • @coderabbitai read the files in the src/scheduler package and generate a class diagram using mermaid and a README in the markdown format.

Support

Need help? Create a ticket on our support page for assistance with any issues or questions.

CodeRabbit Commands (Invoked using PR/Issue comments)

Type @coderabbitai help to get the list of available commands.

Other keywords and placeholders

• Add @coderabbitai ignore anywhere in the PR description to prevent this PR from being reviewed.
• Add @coderabbitai summary to generate the high-level summary at a specific location in the PR description.
• Add @coderabbitai anywhere in the PR title to generate the title automatically.

CodeRabbit Configuration File (.coderabbit.yaml)

• You can programmatically configure CodeRabbit by adding a .coderabbit.yaml file to the root of your repository.
• Please see the configuration documentation for more information.
• If your editor has YAML language server enabled, you can add the path at the top of this file to enable auto-completion and validation: # yaml-language-server: $schema=https://coderabbit.ai/integrations/schema.v2.json

Status, Documentation and Community

• Visit our Status Page to check the current availability of CodeRabbit.
• Visit our Documentation for detailed information on how to use CodeRabbit.
• Join our Discord Community to get help, request features, and share feedback.
• Follow us on X/Twitter for updates and announcements.

[coderabbitai]

Actionable comments posted: 0

🧹 Nitpick comments (2)

fastembed/late_interaction/colbert.py (2)

91-93: Prefer an explicit runtime check over assert for library code.

assert can be stripped with Python’s -O flag; raise a clear error instead so misuse is obvious in all environments.

-    assert self.query_tokenizer is not None
-    encoded = self.query_tokenizer.encode_batch([query])
+    if self.query_tokenizer is None:
+        raise RuntimeError("Query tokenizer is not initialized. Call load_onnx_model() first.")
+    encoded = self.query_tokenizer.encode_batch([query])

---

186-204: Use the query tokenizer’s own special-token map to derive its MASK id (robustness).

You’re configuring query_tokenizer padding with pad_id=self.mask_token_id, which was derived from the document tokenizer’s map. These will typically match (same files), but coupling to the doc tokenizer is unnecessary and could break if token additions diverge. Derive the MASK id from the query tokenizer that you just loaded.

-        self.query_tokenizer, _ = load_tokenizer(model_dir=self._model_dir)
+        self.query_tokenizer, query_special_token_to_id = load_tokenizer(model_dir=self._model_dir)

         assert self.tokenizer is not None
         self.mask_token_id = self.special_token_to_id[self.MASK_TOKEN]
         self.pad_token_id = self.tokenizer.padding["pad_id"]
         self.skip_list = {
             self.tokenizer.encode(symbol, add_special_tokens=False).ids[0]
             for symbol in string.punctuation
         }
         current_max_length = self.tokenizer.truncation["max_length"]
         # ensure not to overflow after adding document-marker
         self.tokenizer.enable_truncation(max_length=current_max_length - 1)
-        self.query_tokenizer.enable_truncation(max_length=current_max_length - 1)
-        self.query_tokenizer.enable_padding(
-            pad_token=self.MASK_TOKEN,
-            pad_id=self.mask_token_id,
-            length=self.MIN_QUERY_LENGTH,
-        )
+        self.query_tokenizer.enable_truncation(max_length=current_max_length - 1)
+        # Derive MASK id from the query tokenizer’s own map (fallback to token_to_id for safety)
+        query_mask_token_id = query_special_token_to_id.get(self.MASK_TOKEN)
+        if query_mask_token_id is None:
+            query_mask_token_id = self.query_tokenizer.token_to_id(self.MASK_TOKEN)  # type: ignore[union-attr]
+        self.query_tokenizer.enable_padding(
+            pad_token=self.MASK_TOKEN,
+            pad_id=query_mask_token_id,
+            length=self.MIN_QUERY_LENGTH,
+        )

Notes:

• Keeping self.mask_token_id as-is preserves the document path behavior; we only de-couple the query path.
• This maintains the invariant “query length before marker = MIN_QUERY_LENGTH” and “after marker = MIN_QUERY_LENGTH + 1”.

📜 Review details

Configuration used: CodeRabbit UI

Review profile: CHILL

Plan: Pro

💡 Knowledge Base configuration:

• MCP integration is disabled by default for public repositories
• Jira integration is disabled by default for public repositories
• Linear integration is disabled by default for public repositories

You can enable these sources in your CodeRabbit configuration.

📥 Commits

Reviewing files that changed from the base of the PR and between 8872392 and bda484b.

📒 Files selected for processing (1)

• fastembed/late_interaction/colbert.py (5 hunks)

🧰 Additional context used

🧬 Code graph analysis (1)

fastembed/late_interaction/colbert.py (1)

fastembed/common/preprocessor_utils.py (1)

• load_tokenizer (21-72)

⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (15)

• GitHub Check: Python 3.13.x on windows-latest test
• GitHub Check: Python 3.12.x on macos-latest test
• GitHub Check: Python 3.9.x on macos-latest test
• GitHub Check: Python 3.11.x on ubuntu-latest test
• GitHub Check: Python 3.13.x on ubuntu-latest test
• GitHub Check: Python 3.12.x on windows-latest test
• GitHub Check: Python 3.11.x on macos-latest test
• GitHub Check: Python 3.12.x on ubuntu-latest test
• GitHub Check: Python 3.10.x on windows-latest test
• GitHub Check: Python 3.10.x on ubuntu-latest test
• GitHub Check: Python 3.9.x on windows-latest test
• GitHub Check: Python 3.11.x on windows-latest test
• GitHub Check: Python 3.13.x on macos-latest test
• GitHub Check: Python 3.10.x on macos-latest test
• GitHub Check: Python 3.9.x on ubuntu-latest test

🔇 Additional comments (2)

fastembed/late_interaction/colbert.py (2)

5-7: LGTM: imports reflect the new design (separate query tokenizer).

Importing Tokenizer and load_tokenizer is appropriate for the decoupled query/document tokenization paths.

---

172-173: LGTM: explicit query_tokenizer attribute improves clarity and thread-safety of configuration.

Initializing it to None and configuring once during model load avoids the previous “Already borrowed” issue from per-call reconfiguration.