Use AutoTokenizer.from() for faster tokenizer loading

[DePasqualeOrg]

swift-transformers PR #303 offers significantly faster tokenizer loading when using AutoTokenizer.from(). It also covers the tokenizer remapping and registration that is currently done in mlx-swift-lm, so we can remove that and use the fast path here after that PR is merged.

Changes

• MLXLMCommon/Tokenizer.swift: loadTokenizer now uses AutoTokenizer.from() directly instead of manually loading configs and calling PreTrainedTokenizer init
• Embedders/Tokenizer.swift: Same change, now passes revision to AutoTokenizer.from()
• Embedders/Models.swift: Added revision parameter to ModelConfiguration for consistency with MLXLMCommon
• Embedders/Load.swift: Now passes revision to hub.snapshot()
• Embedders/EmbeddingModel.swift: Uses loadTokenizer instead of inline config loading

API Changes

Deprecated:

• loadTokenizerConfig: Use LanguageModelConfigurationFromHub from swift-transformers directly, which allows users to opt in to the fast path with stripVocabForPerformance: true.

Unavailable (breaking change):

• TokenizerReplacementRegistry / replacementTokenizers: Use AutoTokenizer.register(_:for:) from swift-transformers instead. These no longer function with the new AutoTokenizer.from() code path.

Offline Mode

The offline fallback logic has been removed, as it's handled automatically by the swift-transformers Hub API. When offline, HubApi.snapshot() detects the network state via NWPathMonitor and falls back to cached files if available.

[davidkoski]

This looks good -- is is ready to merge?

[DePasqualeOrg]

I think we should wait for huggingface/swift-transformers#303 to be merged. I'll mark this as ready for review at that time.

[davidkoski]

I think we should wait for huggingface/swift-transformers#303 to be merged. I'll mark this as ready for review at that time.

Awesome, looking at that one now! Ooops, that is one the swift-transformers side :-)