KV_cache offload

[yuzhenmao]

Hi, I am using the latest huggingface transformers (version==4.48.0.dev0). When I tried to run the demo from here, I have this error: AttributeError: 'LlamaForCausalLM' object has no attribute 'set_kv_cache_offload'.

Does anyone know the solution to solve this issue? Thanks!

[kdw4537]

I'm having the same problem (OPT as well)
Did you build a tokenizer in fork of transformers? In my case, failed.

[tjruwase]

@yuzhenmao, sorry that we missed this issue. The kv cache offload was prototyped in a transformer's fork, but never upstreamted.
https://github.com/deepspeedai/DeepSpeedExamples/blob/master/inference/huggingface/zero_inference/README.md#install

@kdw4537, are you interested in this feature?

[Freed-Wu]

It cannot be installed. @tjruwase

Collecting git+https://github.com/tjruwase/transformers@kvcache-offload-cpu
  Cloning https://github.com/tjruwase/transformers (to revision kvcache-offload-cpu) to /tmp/pip-req-build-em4nbi06
  Running command git clone --filter=blob:none --quiet https://github.com/tjruwase/transformers /tmp/pip-req-build-em4nbi06
  Running command git checkout -b kvcache-offload-cpu --track origin/kvcache-offload-cpu
  Switched to a new branch 'kvcache-offload-cpu'
  branch 'kvcache-offload-cpu' set up to track 'origin/kvcache-offload-cpu'.
  Resolved https://github.com/tjruwase/transformers to commit 35515d7deb03200e0f835120ed40e817a9f6dd06
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Preparing metadata (pyproject.toml) ... done
Requirement already satisfied: filelock in /usr/lib/python3.13/site-packages (from transformers==4.33.0.dev0) (3.18.0)
Requirement already satisfied: huggingface-hub<1.0,>=0.15.1 in /usr/lib/python3.13/site-packages (from transformers==4.33.0.dev0) (0.32.5)
Requirement already satisfied: numpy>=1.17 in /usr/lib/python3.13/site-packages (from transformers==4.33.0.dev0) (2.3.0)
Requirement already satisfied: packaging>=20.0 in /usr/lib/python3.13/site-packages (from transformers==4.33.0.dev0) (25.0)
Requirement already satisfied: pyyaml>=5.1 in /usr/lib/python3.13/site-packages (from transformers==4.33.0.dev0) (6.0.2)
Requirement already satisfied: regex!=2019.12.17 in /usr/lib/python3.13/site-packages (from transformers==4.33.0.dev0) (2024.11.6)
Requirement already satisfied: requests in /usr/lib/python3.13/site-packages (from transformers==4.33.0.dev0) (2.32.4)
Collecting tokenizers!=0.11.3,<0.14,>=0.11.1 (from transformers==4.33.0.dev0)
  Using cached https://mirrors.bfsu.edu.cn/pypi/web/packages/29/9c/936ebad6dd963616189d6362f4c2c03a0314cf2a221ba15e48dd714d29cf/tokenizers-0.13.3.tar.gz (314 kB)
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Preparing metadata (pyproject.toml) ... done
Requirement already satisfied: safetensors>=0.3.1 in /usr/lib/python3.13/site-packages (from transformers==4.33.0.dev0) (0.5.3)
Requirement already satisfied: tqdm>=4.27 in /usr/lib/python3.13/site-packages (from transformers==4.33.0.dev0) (4.67.1)
Requirement already satisfied: fsspec>=2023.5.0 in /usr/lib/python3.13/site-packages (from huggingface-hub<1.0,>=0.15.1->transformers==4.33.0.dev0) (2025.5.1)
Collecting hf-xet<2.0.0,>=1.1.2 (from huggingface-hub<1.0,>=0.15.1->transformers==4.33.0.dev0)
  Using cached https://mirrors.bfsu.edu.cn/pypi/web/packages/52/48/e929e6e3db6e4758c2adf0f2ca2c59287f1b76229d8bdc1a4c9cfc05212e/hf_xet-1.1.3-cp37-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (4.8 MB)
Requirement already satisfied: typing-extensions>=3.7.4.3 in /usr/lib/python3.13/site-packages (from huggingface-hub<1.0,>=0.15.1->transformers==4.33.0.dev0) (4.13.2)
Requirement already satisfied: charset_normalizer<4,>=2 in /usr/lib/python3.13/site-packages (from requests->transformers==4.33.0.dev0) (3.4.2)
Requirement already satisfied: idna<4,>=2.5 in /usr/lib/python3.13/site-packages (from requests->transformers==4.33.0.dev0) (3.10)
Requirement already satisfied: urllib3<3,>=1.21.1 in /usr/lib/python3.13/site-packages (from requests->transformers==4.33.0.dev0) (2.4.0)
Building wheels for collected packages: transformers, tokenizers
  Building wheel for transformers (pyproject.toml) ... done
  Created wheel for transformers: filename=transformers-4.33.0.dev0-py3-none-any.whl size=7575341 sha256=eb695cf3411bccd53d24027237a4250556f9534fb8fd777fb4bb184d5f1563e1
  Stored in directory: /tmp/pip-ephem-wheel-cache-qykwx323/wheels/da/da/01/50a4b3b541a7de5557d73e3d11f3a432b6107eee12cb29cd1e
  Building wheel for tokenizers (pyproject.toml) ... error
  error: subprocess-exited-with-error

  × Building wheel for tokenizers (pyproject.toml) did not run successfully.
  │ exit code: 1
  ╰─> [659 lines of output]
      /tmp/pip-build-env-dizg4cry/overlay/lib/python3.13/site-packages/setuptools/dist.py:759: SetuptoolsDeprecationWarning: License classifiers are deprecated.
      !!

              ********************************************************************************
              Please consider removing the following classifiers in favor of a SPDX license expression:

              License :: OSI Approved :: Apache Software License

              See https://packaging.python.org/en/latest/guides/writing-pyproject-toml/#license for details.
              ********************************************************************************

      !!
        self._finalize_license_expression()
      running bdist_wheel
      running build
      running build_py
      creating build/lib.linux-x86_64-cpython-313/tokenizers
      copying py_src/tokenizers/__init__.py -> build/lib.linux-x86_64-cpython-313/tokenizers
      creating build/lib.linux-x86_64-cpython-313/tokenizers/models
      copying py_src/tokenizers/models/__init__.py -> build/lib.linux-x86_64-cpython-313/tokenizers/models
      creating build/lib.linux-x86_64-cpython-313/tokenizers/decoders
      copying py_src/tokenizers/decoders/__init__.py -> build/lib.linux-x86_64-cpython-313/tokenizers/decoders
      creating build/lib.linux-x86_64-cpython-313/tokenizers/normalizers
      copying py_src/tokenizers/normalizers/__init__.py -> build/lib.linux-x86_64-cpython-313/tokenizers/normalizers
      creating build/lib.linux-x86_64-cpython-313/tokenizers/pre_tokenizers
      copying py_src/tokenizers/pre_tokenizers/__init__.py -> build/lib.linux-x86_64-cpython-313/tokenizers/pre_tokenizers
      creating build/lib.linux-x86_64-cpython-313/tokenizers/processors
      copying py_src/tokenizers/processors/__init__.py -> build/lib.linux-x86_64-cpython-313/tokenizers/processors
      creating build/lib.linux-x86_64-cpython-313/tokenizers/trainers
      copying py_src/tokenizers/trainers/__init__.py -> build/lib.linux-x86_64-cpython-313/tokenizers/trainers
      creating build/lib.linux-x86_64-cpython-313/tokenizers/implementations
      copying py_src/tokenizers/implementations/sentencepiece_unigram.py -> build/lib.linux-x86_64-cpython-313/tokenizers/implementations
      copying py_src/tokenizers/implementations/sentencepiece_bpe.py -> build/lib.linux-x86_64-cpython-313/tokenizers/implementations
      copying py_src/tokenizers/implementations/char_level_bpe.py -> build/lib.linux-x86_64-cpython-313/tokenizers/implementations
      copying py_src/tokenizers/implementations/byte_level_bpe.py -> build/lib.linux-x86_64-cpython-313/tokenizers/implementations
      copying py_src/tokenizers/implementations/bert_wordpiece.py -> build/lib.linux-x86_64-cpython-313/tokenizers/implementations
      copying py_src/tokenizers/implementations/base_tokenizer.py -> build/lib.linux-x86_64-cpython-313/tokenizers/implementations
      copying py_src/tokenizers/implementations/__init__.py -> build/lib.linux-x86_64-cpython-313/tokenizers/implementations
      creating build/lib.linux-x86_64-cpython-313/tokenizers/tools
      copying py_src/tokenizers/tools/visualizer.py -> build/lib.linux-x86_64-cpython-313/tokenizers/tools
      copying py_src/tokenizers/tools/__init__.py -> build/lib.linux-x86_64-cpython-313/tokenizers/tools
      copying py_src/tokenizers/__init__.pyi -> build/lib.linux-x86_64-cpython-313/tokenizers
      copying py_src/tokenizers/models/__init__.pyi -> build/lib.linux-x86_64-cpython-313/tokenizers/models
      copying py_src/tokenizers/decoders/__init__.pyi -> build/lib.linux-x86_64-cpython-313/tokenizers/decoders
      copying py_src/tokenizers/normalizers/__init__.pyi -> build/lib.linux-x86_64-cpython-313/tokenizers/normalizers
      copying py_src/tokenizers/pre_tokenizers/__init__.pyi -> build/lib.linux-x86_64-cpython-313/tokenizers/pre_tokenizers
      copying py_src/tokenizers/processors/__init__.pyi -> build/lib.linux-x86_64-cpython-313/tokenizers/processors
      copying py_src/tokenizers/trainers/__init__.pyi -> build/lib.linux-x86_64-cpython-313/tokenizers/trainers
      copying py_src/tokenizers/tools/visualizer-styles.css -> build/lib.linux-x86_64-cpython-313/tokenizers/tools
      running build_ext
      running build_rust
          Updating `bfsu` index
           Locking 300 packages to latest compatible versions
            Adding env_logger v0.7.1 (available: v0.11.8)
            Adding itertools v0.9.0 (available: v0.14.0)
            Adding ndarray v0.13.1 (available: v0.16.1)
            Adding numpy v0.18.0 (available: v0.25.0)
            Adding pyo3 v0.18.3 (available: v0.25.0)
      cargo rustc --lib --message-format=json-render-diagnostics --manifest-path Cargo.toml --release -v --features pyo3/extension-module --crate-type cdylib --
         Compiling libc v0.2.172
         Compiling proc-macro2 v1.0.95
         Compiling unicode-ident v1.0.18
         Compiling autocfg v1.4.0
         Compiling shlex v1.3.0
         Compiling cfg-if v1.0.1
         Compiling pkg-config v0.3.32
         Compiling memchr v2.7.5
         Compiling stable_deref_trait v1.2.0
         Compiling once_cell v1.21.3
         Compiling version_check v0.9.5
         Compiling typenum v1.18.0
         Compiling syn v1.0.109
         Compiling pin-project-lite v0.2.16
         Compiling bitflags v2.9.1
         Compiling smallvec v1.15.1
         ...
         Compiling tokenizers v0.13.3 (/tmp/pip-install-bk5z726k/tokenizers_7a7b16d37ce74c718e6bf722333a98db/tokenizers-lib)
           Running `rustc --crate-name tokenizers --edition=2018 tokenizers-lib/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts,future-incompat --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 -C embed-bitcode=no --cfg 'feature="cached-path"' --cfg 'feature="clap"' --cfg 'feature="cli"' --cfg 'feature="default"' --cfg 'feature="dirs"' --cfg 'feature="esaxx_fast"' --cfg 'feature="http"' --cfg 'feature="indicatif"' --cfg 'feature="onig"' --cfg 'feature="progressbar"' --cfg 'feature="reqwest"' --check-cfg 'cfg(docsrs,test)' --check-cfg 'cfg(feature, values("cached-path", "clap", "cli", "default", "dirs", "esaxx_fast", "fancy-regex", "http", "indicatif", "onig", "progressbar", "reqwest", "unstable_wasm"))' -C metadata=86ea2528eed6c520 -C extra-filename=-63e70014eb414976 --out-dir /tmp/pip-install-bk5z726k/tokenizers_7a7b16d37ce74c718e6bf722333a98db/target/release/deps -C strip=debuginfo -L dependency=/tmp/pip-install-bk5z726k/tokenizers_7a7b16d37ce74c718e6bf722333a98db/target/release/deps --extern aho_corasick=/tmp/pip-install-bk5z726k/tokenizers_7a7b16d37ce74c718e6bf722333a98db/target/release/deps/libaho_corasick-8be2932ba1d4f103.rmeta --extern cached_path=/tmp/pip-install-bk5z726k/tokenizers_7a7b16d37ce74c718e6bf722333a98db/target/release/deps/libcached_path-7e016c0836af2b36.rmeta --extern clap=/tmp/pip-install-bk5z726k/tokenizers_7a7b16d37ce74c718e6bf722333a98db/target/release/deps/libclap-fa9a37346145174a.rmeta --extern derive_builder=/tmp/pip-install-bk5z726k/tokenizers_7a7b16d37ce74c718e6bf722333a98db/target/release/deps/libderive_builder-a43212fd1b428fb6.rmeta --extern dirs=/tmp/pip-install-bk5z726k/tokenizers_7a7b16d37ce74c718e6bf722333a98db/target/release/deps/libdirs-7cd6ed3e2648e17a.rmeta --extern esaxx_rs=/tmp/pip-install-bk5z726k/tokenizers_7a7b16d37ce74c718e6bf722333a98db/target/release/deps/libesaxx_rs-06374b8edc4393f4.rmeta --extern getrandom=/tmp/pip-install-bk5z726k/tokenizers_7a7b16d37ce74c718e6bf722333a98db/target/release/deps/libgetrandom-c78eb40ece4a1da6.rmeta --extern indicatif=/tmp/pip-install-bk5z726k/tokenizers_7a7b16d37ce74c718e6bf722333a98db/target/release/deps/libindicatif-b32862cb28ea595a.rmeta --extern itertools=/tmp/pip-install-bk5z726k/tokenizers_7a7b16d37ce74c718e6bf722333a98db/target/release/deps/libitertools-5a65b501f315af7a.rmeta --extern lazy_static=/tmp/pip-install-bk5z726k/tokenizers_7a7b16d37ce74c718e6bf722333a98db/target/release/deps/liblazy_static-95b98c7937dd5b53.rmeta --extern log=/tmp/pip-install-bk5z726k/tokenizers_7a7b16d37ce74c718e6bf722333a98db/target/release/deps/liblog-c930a65bd2d33389.rmeta --extern macro_rules_attribute=/tmp/pip-install-bk5z726k/tokenizers_7a7b16d37ce74c718e6bf722333a98db/target/release/deps/libmacro_rules_attribute-8ccf184cad68b546.rmeta --extern monostate=/tmp/pip-install-bk5z726k/tokenizers_7a7b16d37ce74c718e6bf722333a98db/target/release/deps/libmonostate-9ecf66afef857f1e.rmeta --extern onig=/tmp/pip-install-bk5z726k/tokenizers_7a7b16d37ce74c718e6bf722333a98db/target/release/deps/libonig-4c8708f0dcae69c7.rmeta --extern paste=/tmp/pip-install-bk5z726k/tokenizers_7a7b16d37ce74c718e6bf722333a98db/target/release/deps/libpaste-25f1f0e59fe3fce4.so --extern rand=/tmp/pip-install-bk5z726k/tokenizers_7a7b16d37ce74c718e6bf722333a98db/target/release/deps/librand-585aa8890a2a7095.rmeta --extern rayon=/tmp/pip-install-bk5z726k/tokenizers_7a7b16d37ce74c718e6bf722333a98db/target/release/deps/librayon-664a725f0e3fb01b.rmeta --extern rayon_cond=/tmp/pip-install-bk5z726k/tokenizers_7a7b16d37ce74c718e6bf722333a98db/target/release/deps/librayon_cond-aaa74168f63f18ac.rmeta --extern regex=/tmp/pip-install-bk5z726k/tokenizers_7a7b16d37ce74c718e6bf722333a98db/target/release/deps/libregex-3df5e2487fe5240f.rmeta --extern regex_syntax=/tmp/pip-install-bk5z726k/tokenizers_7a7b16d37ce74c718e6bf722333a98db/target/release/deps/libregex_syntax-8cfad6fd4030a27d.rmeta --extern reqwest=/tmp/pip-install-bk5z726k/tokenizers_7a7b16d37ce74c718e6bf722333a98db/target/release/deps/libreqwest-8113c37e66f04d9c.rmeta --extern serde=/tmp/pip-install-bk5z726k/tokenizers_7a7b16d37ce74c718e6bf722333a98db/target/release/deps/libserde-3304991411b95fb4.rmeta --extern serde_json=/tmp/pip-install-bk5z726k/tokenizers_7a7b16d37ce74c718e6bf722333a98db/target/release/deps/libserde_json-5e260cc830bb35cb.rmeta --extern spm_precompiled=/tmp/pip-install-bk5z726k/tokenizers_7a7b16d37ce74c718e6bf722333a98db/target/release/deps/libspm_precompiled-4808fa2d00f97681.rmeta --extern thiserror=/tmp/pip-install-bk5z726k/tokenizers_7a7b16d37ce74c718e6bf722333a98db/target/release/deps/libthiserror-b855120315a6a169.rmeta --extern unicode_normalization_alignments=/tmp/pip-install-bk5z726k/tokenizers_7a7b16d37ce74c718e6bf722333a98db/target/release/deps/libunicode_normalization_alignments-9135a16f27f13bc2.rmeta --extern unicode_segmentation=/tmp/pip-install-bk5z726k/tokenizers_7a7b16d37ce74c718e6bf722333a98db/target/release/deps/libunicode_segmentation-2a9ca491bcb513eb.rmeta --extern unicode_categories=/tmp/pip-install-bk5z726k/tokenizers_7a7b16d37ce74c718e6bf722333a98db/target/release/deps/libunicode_categories-5f7a76af5f0a221c.rmeta -L native=/usr/lib -L native=/tmp/pip-install-bk5z726k/tokenizers_7a7b16d37ce74c718e6bf722333a98db/target/release/build/zstd-sys-fd2eb66a5411554c/out -L native=/tmp/pip-install-bk5z726k/tokenizers_7a7b16d37ce74c718e6bf722333a98db/target/release/build/esaxx-rs-6af3e3eb6afb5ee1/out -L native=/tmp/pip-install-bk5z726k/tokenizers_7a7b16d37ce74c718e6bf722333a98db/target/release/build/onig_sys-33dbd85663e96a6e/out`
      warning: variable does not need to be mutable
         --> tokenizers-lib/src/models/unigram/model.rs:265:21
          |
      265 |                 let mut target_node = &mut best_path_ends_at[key_pos];
          |                     ----^^^^^^^^^^^
          |                     |
          |                     help: remove this `mut`
          |
          = note: `#[warn(unused_mut)]` on by default

      warning: variable does not need to be mutable
         --> tokenizers-lib/src/models/unigram/model.rs:282:21
          |
      282 |                 let mut target_node = &mut best_path_ends_at[starts_at + mblen];
          |                     ----^^^^^^^^^^^
          |                     |
          |                     help: remove this `mut`

      warning: variable does not need to be mutable
         --> tokenizers-lib/src/pre_tokenizers/byte_level.rs:200:59
          |
      200 |     encoding.process_tokens_with_offsets_mut(|(i, (token, mut offsets))| {
          |                                                           ----^^^^^^^
          |                                                           |
          |                                                           help: remove this `mut`

      error: casting `&T` to `&mut T` is undefined behavior, even if the reference is unused, consider instead using an `UnsafeCell`
         --> tokenizers-lib/src/models/bpe/trainer.rs:526:47
          |
      522 |                     let w = &words[*i] as *const _ as *mut _;
          |                             -------------------------------- casting happened here
      ...
      526 |                         let word: &mut Word = &mut (*w);
          |                                               ^^^^^^^^^
          |
          = note: for more information, visit <https://doc.rust-lang.org/book/ch15-05-interior-mutability.html>
          = note: `#[deny(invalid_reference_casting)]` on by default

      warning: `tokenizers` (lib) generated 3 warnings
      error: could not compile `tokenizers` (lib) due to 1 previous error; 3 warnings emitted

      Caused by:
        process didn't exit successfully: `rustc --crate-name tokenizers --edition=2018 tokenizers-lib/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts,future-incompat --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 -C embed-bitcode=no --cfg 'feature="cached-path"' --cfg 'feature="clap"' --cfg 'feature="cli"' --cfg 'feature="default"' --cfg 'feature="dirs"' --cfg 'feature="esaxx_fast"' --cfg 'feature="http"' --cfg 'feature="indicatif"' --cfg 'feature="onig"' --cfg 'feature="progressbar"' --cfg 'feature="reqwest"' --check-cfg 'cfg(docsrs,test)' --check-cfg 'cfg(feature, values("cached-path", "clap", "cli", "default", "dirs", "esaxx_fast", "fancy-regex", "http", "indicatif", "onig", "progressbar", "reqwest", "unstable_wasm"))' -C metadata=86ea2528eed6c520 -C extra-filename=-63e70014eb414976 --out-dir /tmp/pip-install-bk5z726k/tokenizers_7a7b16d37ce74c718e6bf722333a98db/target/release/deps -C strip=debuginfo -L dependency=/tmp/pip-install-bk5z726k/tokenizers_7a7b16d37ce74c718e6bf722333a98db/target/release/deps --extern aho_corasick=/tmp/pip-install-bk5z726k/tokenizers_7a7b16d37ce74c718e6bf722333a98db/target/release/deps/libaho_corasick-8be2932ba1d4f103.rmeta --extern cached_path=/tmp/pip-install-bk5z726k/tokenizers_7a7b16d37ce74c718e6bf722333a98db/target/release/deps/libcached_path-7e016c0836af2b36.rmeta --extern clap=/tmp/pip-install-bk5z726k/tokenizers_7a7b16d37ce74c718e6bf722333a98db/target/release/deps/libclap-fa9a37346145174a.rmeta --extern derive_builder=/tmp/pip-install-bk5z726k/tokenizers_7a7b16d37ce74c718e6bf722333a98db/target/release/deps/libderive_builder-a43212fd1b428fb6.rmeta --extern dirs=/tmp/pip-install-bk5z726k/tokenizers_7a7b16d37ce74c718e6bf722333a98db/target/release/deps/libdirs-7cd6ed3e2648e17a.rmeta --extern esaxx_rs=/tmp/pip-install-bk5z726k/tokenizers_7a7b16d37ce74c718e6bf722333a98db/target/release/deps/libesaxx_rs-06374b8edc4393f4.rmeta --extern getrandom=/tmp/pip-install-bk5z726k/tokenizers_7a7b16d37ce74c718e6bf722333a98db/target/release/deps/libgetrandom-c78eb40ece4a1da6.rmeta --extern indicatif=/tmp/pip-install-bk5z726k/tokenizers_7a7b16d37ce74c718e6bf722333a98db/target/release/deps/libindicatif-b32862cb28ea595a.rmeta --extern itertools=/tmp/pip-install-bk5z726k/tokenizers_7a7b16d37ce74c718e6bf722333a98db/target/release/deps/libitertools-5a65b501f315af7a.rmeta --extern lazy_static=/tmp/pip-install-bk5z726k/tokenizers_7a7b16d37ce74c718e6bf722333a98db/target/release/deps/liblazy_static-95b98c7937dd5b53.rmeta --extern log=/tmp/pip-install-bk5z726k/tokenizers_7a7b16d37ce74c718e6bf722333a98db/target/release/deps/liblog-c930a65bd2d33389.rmeta --extern macro_rules_attribute=/tmp/pip-install-bk5z726k/tokenizers_7a7b16d37ce74c718e6bf722333a98db/target/release/deps/libmacro_rules_attribute-8ccf184cad68b546.rmeta --extern monostate=/tmp/pip-install-bk5z726k/tokenizers_7a7b16d37ce74c718e6bf722333a98db/target/release/deps/libmonostate-9ecf66afef857f1e.rmeta --extern onig=/tmp/pip-install-bk5z726k/tokenizers_7a7b16d37ce74c718e6bf722333a98db/target/release/deps/libonig-4c8708f0dcae69c7.rmeta --extern paste=/tmp/pip-install-bk5z726k/tokenizers_7a7b16d37ce74c718e6bf722333a98db/target/release/deps/libpaste-25f1f0e59fe3fce4.so --extern rand=/tmp/pip-install-bk5z726k/tokenizers_7a7b16d37ce74c718e6bf722333a98db/target/release/deps/librand-585aa8890a2a7095.rmeta --extern rayon=/tmp/pip-install-bk5z726k/tokenizers_7a7b16d37ce74c718e6bf722333a98db/target/release/deps/librayon-664a725f0e3fb01b.rmeta --extern rayon_cond=/tmp/pip-install-bk5z726k/tokenizers_7a7b16d37ce74c718e6bf722333a98db/target/release/deps/librayon_cond-aaa74168f63f18ac.rmeta --extern regex=/tmp/pip-install-bk5z726k/tokenizers_7a7b16d37ce74c718e6bf722333a98db/target/release/deps/libregex-3df5e2487fe5240f.rmeta --extern regex_syntax=/tmp/pip-install-bk5z726k/tokenizers_7a7b16d37ce74c718e6bf722333a98db/target/release/deps/libregex_syntax-8cfad6fd4030a27d.rmeta --extern reqwest=/tmp/pip-install-bk5z726k/tokenizers_7a7b16d37ce74c718e6bf722333a98db/target/release/deps/libreqwest-8113c37e66f04d9c.rmeta --extern serde=/tmp/pip-install-bk5z726k/tokenizers_7a7b16d37ce74c718e6bf722333a98db/target/release/deps/libserde-3304991411b95fb4.rmeta --extern serde_json=/tmp/pip-install-bk5z726k/tokenizers_7a7b16d37ce74c718e6bf722333a98db/target/release/deps/libserde_json-5e260cc830bb35cb.rmeta --extern spm_precompiled=/tmp/pip-install-bk5z726k/tokenizers_7a7b16d37ce74c718e6bf722333a98db/target/release/deps/libspm_precompiled-4808fa2d00f97681.rmeta --extern thiserror=/tmp/pip-install-bk5z726k/tokenizers_7a7b16d37ce74c718e6bf722333a98db/target/release/deps/libthiserror-b855120315a6a169.rmeta --extern unicode_normalization_alignments=/tmp/pip-install-bk5z726k/tokenizers_7a7b16d37ce74c718e6bf722333a98db/target/release/deps/libunicode_normalization_alignments-9135a16f27f13bc2.rmeta --extern unicode_segmentation=/tmp/pip-install-bk5z726k/tokenizers_7a7b16d37ce74c718e6bf722333a98db/target/release/deps/libunicode_segmentation-2a9ca491bcb513eb.rmeta --extern unicode_categories=/tmp/pip-install-bk5z726k/tokenizers_7a7b16d37ce74c718e6bf722333a98db/target/release/deps/libunicode_categories-5f7a76af5f0a221c.rmeta -L native=/usr/lib -L native=/tmp/pip-install-bk5z726k/tokenizers_7a7b16d37ce74c718e6bf722333a98db/target/release/build/zstd-sys-fd2eb66a5411554c/out -L native=/tmp/pip-install-bk5z726k/tokenizers_7a7b16d37ce74c718e6bf722333a98db/target/release/build/esaxx-rs-6af3e3eb6afb5ee1/out -L native=/tmp/pip-install-bk5z726k/tokenizers_7a7b16d37ce74c718e6bf722333a98db/target/release/build/onig_sys-33dbd85663e96a6e/out` (exit status: 1)
      error: `cargo rustc --lib --message-format=json-render-diagnostics --manifest-path Cargo.toml --release -v --features pyo3/extension-module --crate-type cdylib --` failed with code 101
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
  ERROR: Failed building wheel for tokenizers
Successfully built transformers
Failed to build tokenizers
ERROR: Failed to build installable wheels for some pyproject.toml based projects (tokenizers)