feat: Tokenizer and tool tracking [WIP]#129

Merged

moohax merged 28 commits intomainfrom

feat/tokenizer-and-tool-tracking

Jun 16, 2025

Commits on May 23, 2025

Add base64 encoding for complex parameters in identifier strings
monoxgas
committed
Associate parsed native tool calls as tool_call objects on the Message class
monoxgas
committed

Commits on May 27, 2025

Add clarity to tool_method docstring
monoxgas
committed

Commits on May 28, 2025

Add pipeline transforms. Refactor native tool calling.
monoxgas
committed

Commits on May 30, 2025

Start porting transformers
monoxgas
committed

Commits on Jun 6, 2025

More working changes
monoxgas
committed

Commits on Jun 8, 2025

Merge remote-tracking branch 'origin/main' into feat/tokenizer-and-tool-tracking
monoxgas
committed

Commits on Jun 10, 2025

Commits on Jun 15, 2025

tokenize chats
moo
committed
clean yellow squiggle for empty dict as default arg
moo
committed
update docs
moo
committed
update docs
moo
committed
typing fixes
moo
committed
typing fixes #B008
moo
committed
default tokenizer None
moo
committed
typing
moo
committed
return tokenized chat
moo
committed
clean up arguments. Test with tokenize.ipynb/gsm8k
moo
committed
return AutoTokenizer or None
moo
committed
rigging logger -> loguru
moo
committed

Commits on Jun 16, 2025

get_tokenizer return None
moo
committed