Release v0.3.0

Author: Philip Sampaio

.github/workflows/release.yml

@@ -19,7 +19,7 @@ jobs:
     strategy:
       fail-fast: false
       matrix:
-        nif: ["2.15", "2.16", "2.14"]
+        nif: ["2.15", "2.16"]
         job:
           - { target: aarch64-apple-darwin, os: macos-11 }
           - { target: aarch64-unknown-linux-gnu, os: ubuntu-20.04, use-cross: true }

CHANGELOG.md

@@ -5,6 +5,32 @@ All notable changes to this project will be documented in this file.
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
+## [Unreleased]
+
+## [v0.3.0] - 2023-03-04
+
+### Added
+
+- Add option to use cache when downloading pretrained files. We check the ETAG of
+  the file before trying to download it. This introduces the `:use_cache` and `:cache_dir`
+  options to the `Tokenizers.from_pretrained/2` function.
+
+- Support adding special tokens when creating a tokenizer. This allows a pretrained
+  tokenizer to be loaded with additional special tokens.
+
+  This change adds the `:additional_special_tokens` option to the `Tokenizers.from_pretrained/2`
+  function.
+
+- Add support for the `riscv64gc-unknown-linux-gnu` target, which is useful for Nerves
+  projects running on 64 bits RISC-V computers.
+  This means that we are precompiling the project to run on those machines.
+
+### Changed
+
+- Change minimum required version of Rustler Precompiled to `~> 0.6`. With this, we have
+  the `aarch64-unknown-linux-musl` and `riscv64gc-unknown-linux-gnu` as default targets.
+  But we also drop support for the NIF version 2.14.
+
 ## [v0.2.0] - 2022-12-07
 
 ### Added
@@ -15,3 +41,8 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 ## [v0.1.0] - 2022-08-25
 
 First release.
+
+[Unreleased]: https://github.com/elixir-nx/tokenizers/compare/v0.3.0...HEAD
+[v0.3.0]: https://github.com/elixir-nx/tokenizers/compare/v0.2.0...v0.3.0
+[v0.2.0]: https://github.com/elixir-nx/tokenizers/compare/v0.1.0...v0.2.0
+[v0.1.0]: https://github.com/elixir-nx/tokenizers/releases/tag/v0.1.0

lib/tokenizers/native.ex

@@ -8,10 +8,7 @@ defmodule Tokenizers.Native do
     crate: "ex_tokenizers",
     version: version,
     base_url: "#{github_url}/releases/download/v#{version}",
-    force_build: System.get_env("TOKENIZERS_BUILD") in ["1", "true"],
-    targets:
-      RustlerPrecompiled.Config.default_targets() ++
-        ["aarch64-unknown-linux-musl", "riscv64gc-unknown-linux-gnu"]
+    force_build: System.get_env("TOKENIZERS_BUILD") in ["1", "true"]
 
   def decode(_tokenizer, _ids, _skip_special_tokens), do: err()
   def decode_batch(_tokenizer, _ids, _skip_special_tokens), do: err()