feat: Tokenizer and tool tracking [WIP]

[monoxgas]

Notes

• Working on a better structure for tooling and translations to text-based forms
• Still in progress on tokenizer integrations that play off the new transform mechanic

---

Generated Summary

• Major restructuring of tool modules:
  • Legacy modules under “rigging/tool” (API and native tool handling) have been removed and replaced by a unified “rigging/tools” package.
  • Imports throughout the codebase and documentation now reference the new “tools” namespace.
• Introduction of a new transform framework for tool calls:
  • Added the “rigging/transform” module with transforms to convert tool calls and responses into various JSON and XML formats (“json”, “json-in-xml”, “json-with-tag” and native XML).
  • ChatPipeline now supports transform callbacks (both pre- and post-) for flexible processing of tool calls.
• Enhanced tokenization support:
  • A new “rigging/tokenize” module has been introduced with utilities such as get_tokenizer, find_in_tokens, and structured token slicing (via TokenSlice and TokenizedChat).
  • Added Chat.to_tokens() method to provide tokenized chat output.
• Message model improvements:
  • Replaced the legacy “parts” attribute with “slices” for tracking parsed sections of a message.
  • Added new methods (add_slice, remove_slice, meta) and updated cloning to include metadata and compatibility flags.
• Improvements in generation parameters:
  • Added a clone() method to GenerateParams for deep copying.
  • Updated identifier generation to encode/decode complex parameter values using base64.
• Documentation updates:
  • API docs now include new sections for tools, transforms, and tokenize.
  • Topic pages have been updated to reflect the new tool calling modes and formats.
• Deprecation and backward compatibility:
  • Deprecated legacy methods such as .to_openai_spec(), .models, and .parts in favor of their updated counterparts.
  • Added detailed deprecation warnings to guide migration to the new API.

This summary was generated with ❤️ by rigging

Generated Summary

• Removed the old "tool" API – all functionality has been migrated to the new "tools" module (and related "transform" module) with updated types (e.g. ToolDefinition, ToolCall, FunctionCall) and naming for API and native tools.
• Replaced deprecated API types (ApiToolChoice, ApiToolDefinition, ApiToolCall) with new ones (ToolChoice, ToolDefinition, ToolCall) throughout the codebase.
• Changed message parsing: parsed message parts (“parts”) are now managed via “slices” to track structured content; corresponding getters, setters, add_slice, and strip functions have been updated.
• Updated chat serialization methods – .to_openai() now replaces the deprecated .to_openai_spec() and properly returns a list of dicts.
• Introduced new tokenization support – added asynchronous to_tokens() methods in Chat and ChatList which utilize the new get_tokenizer() function and TokenizedChat/TokenSlice types.
• Added new transform modules (json_tools and xml_tools) that provide transforms for converting tool calls/responses into JSON, JSON-in-XML, JSON-with-tag or native XML formats, complete with system prompt injection and post-transform callbacks.
• Updated error handling by adding new warning classes (PipelineWarning, ToolWarning, MessageWarning, TokenizeWarning) to signal non-critical issues during processing.
• Updated documentation and examples (e.g. examples/robopages.py and tokenize.ipynb) to reflect the migration and new APIs.

This summary was generated with ❤️ by rigging

Generated Summary

• Introduce new APIs in Chat and ChatList for converting chat messages to the OpenAI-compatible JSON format and for tokenizing chats (methods “to_openai” and “to_tokens”), replacing the older “to_openai_spec” approach.
• Replace the deprecated “parts” field with a new “slices” property in the Message model to track parsed content, along with updated getters/setters and deprecation warnings.
• Add comprehensive transform functionality with new modules in rigging/transform to support multiple tool call formats (e.g. “json”, “json-in-xml”, “json-with-tag”, and native XML), including helper functions and post-transform callbacks.
• Migrate and consolidate tool-related functionality from the old rigging/tool module into the new rigging/tools module with updated definitions, API interfaces, and error handling.
• Enhance XML handling in models by introducing CDATA processing (with methods such as preprocess_with_cdata and _postprocess_with_cdata) and updating both to_pretty_xml and to_xml for unescaping CDATA tags.
• Improve tokenization utilities by adding a get_tokenizer function and a new protocol for Tokenizer, along with functions (e.g. find_in_tokens and extract_json_objects) to support structured token slicing.
• Update documentation and examples to reflect the new module structure and API changes, including adjustments in docs topics and examples (e.g. use of “rg.robopages” and revised tool modes).

This summary was generated with ❤️ by rigging

Generated Summary

• Refactored message parsing by replacing “parts” with “slices” in the Message model for tracking structured content (e.g. tool calls, models), updating getters, setters, cloning, and serialization methods (to_openai, etc.).
• Deprecated the legacy to_openai_spec() method in favor of the new to_openai() method, ensuring improved backwards compatibility and clearer conversion logic.
• Removed legacy “rigging/tool” modules and consolidated tool functionality under the new “rigging/tools” package, migrating definitions, tool calls, and decorator methods (tool, tool_method).
• Added new transform modules (rigging/transform) with support for converting tool calls/responses between native XML, JSON, JSON-in-XML, and JSON-with-tag formats; updated the ChatPipeline to integrate these transforms.
• Introduced a new tokenize module (rigging/tokenize) that provides a get_tokenizer() function, tokenized chat representation (TokenizedChat, TokenSlice), and helper functions such as find_in_tokens; generator methods now support converting chats to tokens.
• Updated GenerateParams to include a clone() method and adjusted tool-related parameters to use the new ToolDefinition and ToolChoice types.
• Enhanced error handling by introducing new warning classes (PipelineWarning, ToolWarning, MessageWarning, TokenizeWarning) to signal non-critical issues in pipeline, tool execution, message processing, and tokenization.
• Updated documentation files and examples (including a new tokenize.ipynb notebook and modifications to robopages examples) to reflect these architecture changes and API updates.

This summary was generated with ❤️ by rigging

Generated Summary

• Reworked the message parsing by replacing “ParsedMessagePart” with “MessageSlice” and updating related APIs (e.g. add_slice, remove_slice, and content setter behavior) to better track structured content.
• Deprecated the legacy .to_openai_spec() method in favor of .to_openai() for message serialization, with appropriate deprecation warnings.
• Consolidated tool functionality by removing the old rigging/tool module and moving tool types and functions into rigging/tools (e.g. ToolDefinition, ToolCall, tool, tool_method) with updated APIs and type hints.
• Introduced new modules for tokenization (rigging/tokenize) and transformation (rigging/transform) to support conversion between chat messages and tokenized representations and to enable flexible tool call transforms (JSON, JSON-in-XML, JSON-with-tag, and XML native formats).
• Updated the generator and chat pipeline logic to use the new message serialization and transform callbacks, including dynamic injection and stripping of system content.
• Added several new warning classes (PipelineWarning, ToolWarning, MessageWarning, TokenizeWarning) to signal non-critical issues during pipeline execution, tool processing, message handling, and tokenization.
• Modified get_identifier and get_generator in the generator module to encode/decode extra parameters (including base64 encoding for complex values) for better identifier consistency.
• Updated documentation topics and examples to reflect these API changes, including revised tool modes and new examples for using the tokenizer and transform features.

This summary was generated with ❤️ by rigging

Generated Summary

• Refactored tool handling by replacing the old “rigging/tool” modules with new “rigging/tools” interfaces; this includes updated definitions for Tool, tool, tool_method, mcp, and robopages.
• Replaced “parsed parts” with “slices” in the Message model, deprecating the parts/models properties to improve tracking and editing of structured message content.
• Introduced a new tokenize module (with ChatFormatter, Tokenizer, TokenizedChat, etc.) and tokens utilities (find_in_tokens, get_tokenizer) for serializing chats into tokenized representations.
• Added a new transform module for converting tool calls/responses between various formats (JSON, JSON-in-XML, JSON-with-tag, and native XML) with corresponding utility transforms.
• Updated generator methods to use .to_openai() instead of the deprecated .to_openai_spec(), and improved identifier generation in GenerateParams with base64 encoding for complex values.
• Enhanced error handling by introducing new warning classes (PipelineWarning, ToolWarning, MessageWarning, TokenizeWarning) to signal non-critical issues during pipeline execution, tool handling, message processing, and tokenization.
• Adjusted several internal APIs and conversions (e.g. system prompt injection, stripping system content) to better align with new tooling and message slice structures.

This summary was generated with ❤️ by rigging

Generated Summary

• Migrated and consolidated tool handling by moving functionality from the old “rigging/tool” package to the new “rigging/tools” module, including changes to tool definitions, calls, and related decorators.
• Introduced new modules for tokenization (“rigging/tokenize”) and transformation (“rigging/transform”), with comprehensive API documentation and examples.
• Deprecated and replaced the old “to_openai_spec” method in favor of a new “to_openai” method, ensuring backward compatibility with added deprecation warnings.
• Enhanced XML serialization: added CDATA wrapping support during XML generation and postprocessing to better handle unescaped entities.
• Updated GenerateParams to use the new ToolDefinition and ToolChoice types instead of the old API-based equivalents.
• Added support for low-level transform callbacks in ChatPipeline, allowing pre- and post-transformation modifications and enabling different JSON and XML tool call formats.
• Improved error and warning handling by adding new warning classes (PipelineWarning, ToolWarning, MessageWarning, and TokenizeWarning) to better flag non-critical issues.

This summary was generated with ❤️ by rigging

Generated Summary

• Removed legacy modules (rigging/tool, rigging/tool/api, rigging/tool/native) and migrated all tool‐related functionality to the new rigging/tools namespace, unifying tool definitions, calls, and decorators.
• Updated Chat methods to use a new utility (inject_system_content_into_messages) for injecting system content and replaced deprecated “to_openai_spec” with the “to_openai” method for serializing messages.
• Added comprehensive tokenization support via a new rigging/tokenize module exposing get_tokenizer, find_in_tokens, ChatFormatter, and related types.
• Introduced a new transform system with protocols for pre‐ and post‐transform callbacks; new transforms for tool calls now support multiple modes (“xml”, “json”, “json-in-xml”, “json-with-tag”) are provided in rigging/transform.
• Revised model serialization: added methods to preprocess content with CDATA wrapping, updated pretty-printing to unescape CDATA tags, and introduced ErrorModel.from_exception.
• Adjusted generator and pipeline logic to use updated message serialization and transformation callbacks, including removal of legacy _pre_run and integration of transform callbacks.
• Updated various method signatures and type hints throughout for consistency (for example, renaming ApiToolChoice to ToolChoice, and adjustments in tool API definitions).
• Documentation and examples have been revised to reference the new tool modes (including “json-with-tag”) and updated module paths.

This summary was generated with ❤️ by rigging

Generated Summary

• Migrated and refactored tool-related code from the old rigging/tool module into a new rigging/tools package, consolidating definitions (e.g. Tool, ToolCall, ToolDefinition, FunctionCall, tool, tool_method) and integrations (mcp, robopages) under a unified namespace.
• Replaced legacy “parts” in Message with “slices” to track parsed content, deprecating properties like .parts and .models; updated content getter/setter logic to use slices.
• Introduced new modules for tokenization (rigging/tokenize) and transformation (rigging/transform) with corresponding interfaces (e.g. get_tokenizer, find_in_tokens, Transform, PostTransform) to support advanced chat formatting and tool call conversion.
• Updated Chat API methods to use .to_openai() instead of the deprecated .to_openai_spec(), and added async .to_tokens() and .transform() methods.
• Revised GenerateParams to use ToolDefinition and ToolChoice instead of the previous API tool types, and improved identifier encoding/decoding methods.
• Added new warning types (PipelineWarning, ToolWarning, MessageWarning, TokenizeWarning) to signal non-critical issues during pipeline, tool, message, and tokenization operations.
• Updated documentation across API, topics, examples, and notebooks (e.g. tokenize.ipynb, robopages.py) to reflect these structural and naming changes.

This summary was generated with ❤️ by rigging

Generated Summary

• Migrated and refactored tool-related functionality from the old “rigging.tool” package to the new “rigging.tools” package, consolidating tool definitions, API interactions, and MCP/Robopages integrations.
• Introduced new modules for tokenization under “rigging.tokenize” with types (TokenSlice, TokenizedChat, ChatFormatter, Encoder, Decoder) and helper functions (find_in_tokens, get_tokenizer).
• Added a new “rigging.transform” package providing Transform interfaces and implementations for converting tool calls/responses into various JSON/XML formats (including “json”, “json-in-xml”, and “json-with-tag” modes) with associated prompt generators.
• Updated the Chat class methods to use a helper “inject_system_content” function and to serialize messages using the new “to_openai()” method instead of the deprecated “to_openai_spec()”.
• Replaced references to ApiToolChoice and ApiToolDefinition with the more general ToolChoice and ToolDefinition throughout generator and parameter merging logic.
• Enhanced message handling by replacing the legacy ParsedMessagePart with MessageSlice, adding functionality for adding, removing, and synchronizing slices while preserving compatibility flags and metadata.
• Improved identifier generation in generators by encoding/decoding complex parameter values using base64 and TypeAdapter.
• Adjusted utility functions (e.g. extract_json_objects, escape_xml, unescape_cdata_tags, to_snake, to_xml_tag) to better support the new processing flows and XML formatting.
• Updated examples and documentation to reflect the new module structure, integration endpoints, and API changes.

This summary was generated with ❤️ by rigging

Generated Summary

• Migrated all tool-related functionality from the “rigging.tool” module to the new “rigging.tools” module, consolidating definitions, decorators, and integrations (including MCP and Robopages).
• Introduced new API documentation pages for tools, transforms, and tokenization with comprehensive examples and usage guides.
• Updated message handling to use “slices” (via MessageSlice) instead of legacy “parts” for parsed content; added new methods for adding and removing slices, plus a meta() helper.
• Implemented new tokenization support by adding the tokenize package with a TokenizedChat type, get_tokenizer(), and a find_in_tokens() helper, allowing conversion of chats to token sequences.
• Added transform support in ChatPipeline; new transform() method registers low-level callbacks (including JSON and XML tool transforms) that modify messages and parameters before generation.
• Enhanced generator parameter encoding: the get_identifier() function now encodes complex parameter values in base64, with corresponding decoding in get_generator().
• Introduced new warning classes (PipelineWarning, ToolWarning, MessageWarning, TokenizeWarning) to signal non-fatal issues in pipeline, tool execution, message processing, and tokenization.
• Revised several API methods to use the updated to_openai() serialization (replacing the deprecated to_openai_spec() method) for better compatibility with external providers.
• Updated utility functions to extract JSON objects from text and to properly escape/unescape XML and CDATA tags for secure and correct XML handling.

This summary was generated with ❤️ by rigging

Generated Summary

• Removed the old tool and API definitions and refactored tool handling under a new rigging.tools module.
• Introduced Tool, ToolCall, and ToolDefinition classes for better organization and management of tool interactions.
• Updated the Chat and ChatPipeline classes to support the new tool structure, including integration with new transform methods.
• Implemented slice functionality in the Message class to replace the previous parts implementation.
• Updated various API calls, message conversions, and tool handling to use the new tool structures.
• Introduced new warnings for mismatched tool names and other inconsistencies during tool calls.
• Renamed and restructured several functions and classes to improve clarity and maintainability.
• Added functionality to support JSON tool calls in native XML format.
• Enhanced message parsing and slice handling in the message content.
• Deprecated previous tool and API structure while ensuring backward compatibility in some areas.
• Added functionality for tokenization and transforms, including the ability to convert tool calls and responses directly to JSON or XML formats.
• Add a new transform function in the ChatPipeline class, allowing the application of transformations to messages and parameters as part of the pipeline processing.
• Implemented a method for injecting system content into messages directly.

This summary was generated with ❤️ by rigging

Generated Summary

Summary of changes:

• Refactored message and tool interaction logic to improve maintainability and clarity.
• Introduced MessageSlice and TokenSlice to represent portions of messages and tokens.
• Replaced instances of ApiToolCall, ApiToolDefinition, and related classes with a more modular structure using Tool, ToolCall, and ToolDefinition.
• Updated Chat, ChatPipeline, and related classes to utilize the new slice logic for better content tracking.
• Enhanced error handling with new warning classes such as PipelineWarning, MessageWarning, and TokenizeWarning.
• Improved the transformer functionality, allowing for better manipulation of tool calls and responses in various formats (JSON, XML).
• Added support for transform callbacks to be registered in the ChatPipeline.
• Simplified the generation of tool definitions and improved the handling of their parameters.

Key changes include:

• Migration from handling multi-part content in messages to utilizing slices for more accuracy.
• Consolidation of tool-related classes to streamline tool integration and response handling.
• Implementation of new utility functions for better JSON and XML management.
• Enhanced testing to accommodate new structure and logic, ensuring thorough coverage across new features implemented.

This refactor aims to make the codebase more modular, maintainable, and easier to enhance in the future while also improving the developer experience when using the framework for chat and tool interactions.

This summary was generated with ❤️ by rigging

Generated Summary

• Introduced a new PipelineWarning class to handle non-critical warnings during pipeline execution.
• Replaced usages of ApiToolCall, ApiToolChoice, and ApiToolDefinition with generic ToolCall, ToolChoice, and ToolDefinition, facilitating a more unified tool interface.
• Added support for transform callbacks to manipulate messages and parameters before and after generation.
• Simplified the inject_system_content logic into a new function to streamline injection of system messages in the chat.
• Deprecated to_openai_spec() in favor of to_openai(), providing a more intuitive method for converting messages to OpenAI-compatible format.
• Enhanced ChatPipeline functionality with new transform methods and improved handling of tool calls and responses.
• Updated XML and JSON handling in tools to use the new formats consistently, improving integration and usability.
• Multiple refactorings and optimizations throughout the rigging package for better clarity and performance.
• Removed the now-obsolete api.py module, consolidating functionality into the base tool definitions.
• Added comprehensive tests to validate the new transformation and serialization features, ensuring reliability.

This summary was generated with ❤️ by rigging

[moohax]

Some left over typing stuff from Transformers.
get_tokenizer could return none, which will cause chats_to_tokens to fail.