v0.0.8

Release v0.0.8

Installation

Homebrew (macOS and Linux)

brew install agentstation/tap/tokenizer

Docker

docker run --rm -it ghcr.io/agentstation/tokenizer:0.0.8 llama3 encode "Hello, world!"

Download Binary

Download the appropriate binary for your platform from the assets below.

Using Go

go install github.com/agentstation/tokenizer/cmd/tokenizer@v0.0.8

Verify Checksums

# Download checksums.txt and the binary, then:
sha256sum -c checksums.txt

Shell Completions

Generate shell completions for your shell:

# Bash
tokenizer completion bash > /etc/bash_completion.d/tokenizer

# Zsh
tokenizer completion zsh > "${fpath[1]}/_tokenizer"

# Fish
tokenizer completion fish > ~/.config/fish/completions/tokenizer.fish

Changelog

Bug fixes

• 23b9593: fix: remove deprecated conflicts_with from goreleaser config (@jackspirou)

Others

• 0dd15a6: add svg (@jackspirou)
• 501e035: refactor: consolidate to single memory-efficient scanner implementation (@jackspirou)
• e0e3421: refactor: improve demo with better branding and features (@jackspirou)

Checksums

7da583461172e431e1c4e3ee9d7472d220050578e61f16bb587fb82fab16018d  tokenizer_0.0.8_darwin_arm64.tar.gz
d6d94b14a0473f911b447f60bac0f5a404fe49b886226b893b9159f038313b08  tokenizer_0.0.8_darwin_arm64.tar.gz.sbom.json
86ae838a8ed6d3c9bc9ef0ff433504d7afe747cf198e50d083d84a1c5a765696  tokenizer_0.0.8_darwin_x86_64.tar.gz
5a33d3cc52dc0f6b4a73aa8d5f37502a4e95e446afaa75fc563c799b844e061e  tokenizer_0.0.8_darwin_x86_64.tar.gz.sbom.json
b7c0e722dbb8319519f4af3d0ebb40db150c604c179e161249f751aae5ed44ac  tokenizer_0.0.8_freebsd_arm64.tar.gz
a1ffac8c1cc9de70ac8d0e24135b62c3429ff2aa67a546eebe5cedb4efacfead  tokenizer_0.0.8_freebsd_arm64.tar.gz.sbom.json
34f87a5e44f1e73e0ae29873a35b23c139e345a8b6959bcf9fc7981f87181532  tokenizer_0.0.8_freebsd_x86_64.tar.gz
409b82302d957c1babd63c98c87258297d94a8f4f40c7189fd6b4a8be814fd89  tokenizer_0.0.8_freebsd_x86_64.tar.gz.sbom.json
d39e2c4edcf39480bcc1fd27184f01755e9aa19442b140a2a80ab85c46cc0c60  tokenizer_0.0.8_linux_arm64.tar.gz
2f4477188d9ea78932b6083e832523287a35f7f69f08c333459065f16e2b83c9  tokenizer_0.0.8_linux_arm64.tar.gz.sbom.json
5bd43f98e1770c4ab695ff101837c576f4dc437fb0537de36cc9610914ddd493  tokenizer_0.0.8_linux_armv6.tar.gz
3739695c9e0e92ff92a70285ef6d7389a2efef706a2039258f374a56ee56ca10  tokenizer_0.0.8_linux_armv6.tar.gz.sbom.json
28d2f6a2fedd093d0a7da6587397f9e4a4ea0395cdca8b13aa77e2116a7a9b25  tokenizer_0.0.8_linux_armv7.tar.gz
bd52f152bb950a156275f57841fe844715a7d355528079c728e413b392ff7f56  tokenizer_0.0.8_linux_armv7.tar.gz.sbom.json
46200e1f0d23870376e3cfb4ffb194e2ab611a553c5a1657fb7800c9081da185  tokenizer_0.0.8_linux_x86_64.tar.gz
f30775b31f18b54745c7530915a1a134780d046a94e1e2fbde745fa146777743  tokenizer_0.0.8_linux_x86_64.tar.gz.sbom.json
d019605c3cf691d51b63afa026b088ec437e76f9d3ad41797580ff871426b7e1  tokenizer_0.0.8_windows_x86_64.zip
801f3a0dcb555acf9224eee03f2c9186346464bca449e76bb1a3272c5cd6be5c  tokenizer_0.0.8_windows_x86_64.zip.sbom.json

v0.0.7

Release v0.0.7

Installation

Homebrew (macOS and Linux)

brew install agentstation/tap/tokenizer

Docker

docker run --rm -it ghcr.io/agentstation/tokenizer:0.0.7 llama3 encode "Hello, world!"

Download Binary

Download the appropriate binary for your platform from the assets below.

Using Go

go install github.com/agentstation/tokenizer/cmd/tokenizer@v0.0.7

Verify Checksums

# Download checksums.txt and the binary, then:
sha256sum -c checksums.txt

Shell Completions

Generate shell completions for your shell:

# Bash
tokenizer completion bash > /etc/bash_completion.d/tokenizer

# Zsh
tokenizer completion zsh > "${fpath[1]}/_tokenizer"

# Fish
tokenizer completion fish > ~/.config/fish/completions/tokenizer.fish

Changelog

Bug fixes

• 049655a: fix: use single quotes in caveats to avoid Ruby syntax errors (@jackspirou)

Checksums

1adb332a1ae8997ee4bbe49fabd56dfa32e46ab9b5ab21fd7552f35857127d25  tokenizer_0.0.7_darwin_arm64.tar.gz
8e4c7af8104ab8791d6c186437090c323307c00dc1270bd2fac47991e3e78891  tokenizer_0.0.7_darwin_arm64.tar.gz.sbom.json
1e285ff789ac3c6613ae5699a9c972ab9a75bbb139b4ab528ed914b4b76fc182  tokenizer_0.0.7_darwin_x86_64.tar.gz
31a1b0aa305b4bfd27318a34d05e0f607d766a81339799e1f7f927a290a84853  tokenizer_0.0.7_darwin_x86_64.tar.gz.sbom.json
0b82807e78aee710fc3be01a8fc86b119d4d8d6bbe7e9df34404e74e917bceb1  tokenizer_0.0.7_freebsd_arm64.tar.gz
e509987933eef87d6ec08373c24a141bad4462e433fb08cd2d3bfb3b531b0e29  tokenizer_0.0.7_freebsd_arm64.tar.gz.sbom.json
40f29b230ca1969e77edb0218ebbe9af603b371ae00efa7f8b766c228c4ad895  tokenizer_0.0.7_freebsd_x86_64.tar.gz
3af151355a6c0f15b0aec68bd2ea68100b0cd240320cfa11467ed447faca6811  tokenizer_0.0.7_freebsd_x86_64.tar.gz.sbom.json
4b9ef055e38f2499b044ae0a4aae3fdbf425a680075520bc6ee4b90c1580d30a  tokenizer_0.0.7_linux_arm64.tar.gz
c4608022f98fafbb9a072f747c2c906d17133d126d9afc078108595bb55a7db1  tokenizer_0.0.7_linux_arm64.tar.gz.sbom.json
6716e48a25fa1a061c3c2edface28187d20c15ba4b4394258cd0edbad67560cb  tokenizer_0.0.7_linux_armv6.tar.gz
cc680631c4537c318635c707f5710e1a920d2a9050a13b7390a5ada1decb3ead  tokenizer_0.0.7_linux_armv6.tar.gz.sbom.json
3b62466c9ec3e3fccb529de4dd8e2d5d12b0137dcd8a1eb3e3cd50228c15dcad  tokenizer_0.0.7_linux_armv7.tar.gz
708747ff355138a852fe2bdb3079f07181d718c9e6e7e439026f1d6449484252  tokenizer_0.0.7_linux_armv7.tar.gz.sbom.json
6878830f8d1dee64d18eaa5595c14aa97338155a90e052bc3364d7c09b7de13d  tokenizer_0.0.7_linux_x86_64.tar.gz
45e02e4ba79cca23a661bf6e96a714163d8f60a4eeface1cee73e1743ff2d407  tokenizer_0.0.7_linux_x86_64.tar.gz.sbom.json
e5599ceab62e65f92ee463b960a2ee7e12debc12c15338ac4d4a149b101f979a  tokenizer_0.0.7_windows_x86_64.zip
ceb7b1b04c68c02c169fb3f8b7566267064163210a6a59eb465cc880ecd9b9e3  tokenizer_0.0.7_windows_x86_64.zip.sbom.json

v0.0.6

Release v0.0.6

Installation

Homebrew (macOS and Linux)

brew install agentstation/tap/tokenizer

Docker

docker run --rm -it ghcr.io/agentstation/tokenizer:0.0.6 llama3 encode "Hello, world!"

Download Binary

Download the appropriate binary for your platform from the assets below.

Using Go

go install github.com/agentstation/tokenizer/cmd/tokenizer@v0.0.6

Verify Checksums

# Download checksums.txt and the binary, then:
sha256sum -c checksums.txt

Shell Completions

Generate shell completions for your shell:

# Bash
tokenizer completion bash > /etc/bash_completion.d/tokenizer

# Zsh
tokenizer completion zsh > "${fpath[1]}/_tokenizer"

# Fish
tokenizer completion fish > ~/.config/fish/completions/tokenizer.fish

Changelog

New Features

• b79ddbd: feat: auto-generate and install shell completions via Homebrew (@jackspirou)
• c2de6fa: feat: migrate from deprecated brews to homebrew_casks (@jackspirou)

Checksums

eae191bf3b2a0b2294a2e7c82c04368691aa5a672b59b9c4f2889732f395f0a0  tokenizer_0.0.6_darwin_arm64.tar.gz
c5a6176a82520ed1b8f5f752a1ad8c6937fd13b8d013cdd76d37d8bc4f54b7dd  tokenizer_0.0.6_darwin_arm64.tar.gz.sbom.json
655bb99493ac1a24904194e6d74a7ebd9cc4e2652894dbb3ca3d3920b297d9c7  tokenizer_0.0.6_darwin_x86_64.tar.gz
a4ca1a2232bc673b3de6c0b55065856667377cbd01428d07e3e0ec897bfa1c62  tokenizer_0.0.6_darwin_x86_64.tar.gz.sbom.json
a23e99cbad6f107c77bf1ed8733af0d4946689a761ce566a4c974fd90465c29f  tokenizer_0.0.6_freebsd_arm64.tar.gz
8789e96486f4945353a136f169d64017451250c4f1d05ac9b6216126ecfbb587  tokenizer_0.0.6_freebsd_arm64.tar.gz.sbom.json
55ab8d93fb0c654eebc7f5f3c45c2f521ae24ac1349b4f413d319422ba406e87  tokenizer_0.0.6_freebsd_x86_64.tar.gz
c038af966c96179b5db839040b6654fee4ccf65b98a42a61b1b7022d1accfb5a  tokenizer_0.0.6_freebsd_x86_64.tar.gz.sbom.json
02694a751dcd70a13586e30e7a330f5bd9f4e0d234739dff32f167d9a5486ddc  tokenizer_0.0.6_linux_arm64.tar.gz
578fecca65d50bf43b1fc9285469f706c00c7656e33e8c95a8e306d82d9f6543  tokenizer_0.0.6_linux_arm64.tar.gz.sbom.json
a66356c6f2d436af25b5aea0ba04f8c6f71dc436288556f218a158318c4301e2  tokenizer_0.0.6_linux_armv6.tar.gz
9c4bb36832dc2cee83d7e57722573022425a758a23c9d23407204d9f60369dc1  tokenizer_0.0.6_linux_armv6.tar.gz.sbom.json
976d037838ce172a34083e2d1c0cf1d8703bd0a4eb38a4b7f7fb7e942da882dc  tokenizer_0.0.6_linux_armv7.tar.gz
4761c9fef4f29b387875ef7684e66ec671426291a67b5cd7e4572beeb7e79bff  tokenizer_0.0.6_linux_armv7.tar.gz.sbom.json
13d7261c242783d2d6585338667353cea932cad88e4f42e6c01815a27a4ae4ef  tokenizer_0.0.6_linux_x86_64.tar.gz
227e4f42ac6374d967c7252f59c9cb37dffa40cba70a3775cfca9c1a4e2188a4  tokenizer_0.0.6_linux_x86_64.tar.gz.sbom.json
c6ee609b73d13b52c30fd3c84f68dcbddccfcec0bb9cdafcbb7cb1f6def478ec  tokenizer_0.0.6_windows_x86_64.zip
4f0bb0023efef666280d2f47ef6557b02f15005b714dbe3290c2ec9b4b78de94  tokenizer_0.0.6_windows_x86_64.zip.sbom.json

v0.0.5

Release v0.0.5

Installation

Homebrew (macOS and Linux)

brew install agentstation/tap/tokenizer

Docker

docker run --rm -it ghcr.io/agentstation/tokenizer:0.0.5 llama3 encode "Hello, world!"

Download Binary

Download the appropriate binary for your platform from the assets below.

Using Go

go install github.com/agentstation/tokenizer/cmd/tokenizer@v0.0.5

Verify Checksums

# Download checksums.txt and the binary, then:
sha256sum -c checksums.txt

Shell Completions

Generate shell completions for your shell:

# Bash
tokenizer completion bash > /etc/bash_completion.d/tokenizer

# Zsh
tokenizer completion zsh > "${fpath[1]}/_tokenizer"

# Fish
tokenizer completion fish > ~/.config/fish/completions/tokenizer.fish

Changelog

New Features

• ba43fb1: feat: enhance GoReleaser configuration with best practices (@jackspirou)

Bug fixes

• 23661c4: fix: add GO_VERSION back to release workflow (@jackspirou)
• c9228ed: fix: configure Docker buildx for multi-arch images (@jackspirou)
• da15e8b: fix: install syft for SBOM generation in release workflow (@jackspirou)
• 4b358b3: fix: revert to GO_VERSION env var for Go version (@jackspirou)

Others

• 5762c1a: refactor: use Runtime.Goversion for dynamic Go version (@jackspirou)

Checksums

d9cd495dd8b7b2cb596026680572cc3a9e00016a7c3dba3f4146684c577e9ebf  tokenizer_0.0.5_darwin_arm64.tar.gz
761ebb84082cfde9460422c3b9059b430cad97baf9c2a43df11dca6c16f31e2c  tokenizer_0.0.5_darwin_arm64.tar.gz.sbom.json
61312bd0879f51ff93e0263d9e4f12c882455bde1ac9a877cd4a6f842ffa80ab  tokenizer_0.0.5_darwin_x86_64.tar.gz
7c30f57ecdda44437ffc826a59fe9ff98f674bf873c1cb00b7bc3b0699468f4f  tokenizer_0.0.5_darwin_x86_64.tar.gz.sbom.json
86b09cc78456ba9d75f00cbec25aa139b68f3b0be16074c5cb8fcada79bcb9cd  tokenizer_0.0.5_freebsd_arm64.tar.gz
963939c3bd4aee56e1eff8273f8f72c4616c95dfab3e37e986840837ad66ff1a  tokenizer_0.0.5_freebsd_arm64.tar.gz.sbom.json
17fc1f3a115856850efdc2c35e2f15889e3bd3aefa41a557124501df20814968  tokenizer_0.0.5_freebsd_x86_64.tar.gz
a7eda09d2e04e8d205b78e6f722b28cfbc483d21783d5b6132c9c011872fce85  tokenizer_0.0.5_freebsd_x86_64.tar.gz.sbom.json
400e940f157c9eec16d2fa8f568703e0ac9e46507eb68ee3dbc6f3503b078fd9  tokenizer_0.0.5_linux_arm64.tar.gz
1baa40681812e475cd33fba2a426a108d76ad79ceb2171794ef25db30db144f5  tokenizer_0.0.5_linux_arm64.tar.gz.sbom.json
01df13defdf7dbfc39c386c261c0da2b901b92cae6da28088c127b7cc2e8fb21  tokenizer_0.0.5_linux_armv6.tar.gz
3d43558d2d6c27b7f467ca97fc99077bbc2a00e83ae861b7261476df15b3cdcb  tokenizer_0.0.5_linux_armv6.tar.gz.sbom.json
78edcab68fa3cc51fa798552ea6b27c89c983bfea5811ad527d4b51c9f6c9ada  tokenizer_0.0.5_linux_armv7.tar.gz
51c63ac78ade9391fb49b318dc92b94178d7e041b8e17da3e891d6a0aaf8472e  tokenizer_0.0.5_linux_armv7.tar.gz.sbom.json
d97f00396e313fad82b23199465c86d8b86b0499d5a36154208e7a985e18fbc4  tokenizer_0.0.5_linux_x86_64.tar.gz
3aea4c0b4584cbc13055d0555bb2d8443556b2ad9453c1a3aa4bc8d50b8d54c6  tokenizer_0.0.5_linux_x86_64.tar.gz.sbom.json
1cbc165ad2ea2e37dc73fc880ce4d86a63bb4f5fced97b569125e1ea5ce03d21  tokenizer_0.0.5_windows_x86_64.zip
444f98b08bc2f2ba033c3d3ff57ed73de004f8eec28cbdf85f590859901e48c0  tokenizer_0.0.5_windows_x86_64.zip.sbom.json

v0.0.4

Release v0.0.4

Installation

Homebrew (macOS and Linux)

brew install agentstation/tap/tokenizer

Download Binary

Download the appropriate binary for your platform from the assets below.

Using Go

go install github.com/agentstation/tokenizer/cmd/tokenizer@v0.0.4

Verify Checksums

# Download checksums.txt and the binary, then:
sha256sum -c checksums.txt

Changelog

New Features

• 54e03c9: feat: add builtBy variable for better build tracking (@jackspirou)
• 4430491: feat: add demo GIF generation with VHS (@jackspirou)

Bug fixes

• db58bb6: fix(ci): correct linter names for golangci-lint v2.3.0 (@jackspirou)
• 5f4b809: fix(ci): move exclusions under linters configuration (@jackspirou)
• 9505d35: fix(ci): remove formatters from linters list (@jackspirou)
• 13bc858: fix(ci): remove invalid skip-dirs from golangci-lint config (@jackspirou)
• 4f02a67: fix(ci): update golangci-lint config for v2.3.0 compatibility (@jackspirou)
• 837c0aa: fix(lint): address critical linting issues (@jackspirou)
• 55079b9: fix(lint): address more linting issues (@jackspirou)
• 3654ee1: fix(lint): address multiple linting issues (@jackspirou)
• 68bd0a6: fix(lint): address remaining linting issues (@jackspirou)
• 79d5d57: fix(lint): auto-fix comment periods with godot tool (@jackspirou)
• 1c64bee: fix: add GO_VERSION environment variable for release (@jackspirou)
• 8c80aae: fix: handle missing git repo in GoReleaser Homebrew formula (@jackspirou)
• 4761a9b: fix: resolve CI workflow issues (@jackspirou)

Others

• 0ee3247: refactor: reduce scanner.Scan() cyclomatic complexity (@jackspirou)
• 5433ea8: style: auto-format code with goimports (@jackspirou)

v0.0.3

Release v0.0.3

Installation

Homebrew (macOS and Linux)

brew install agentstation/tap/tokenizer

Download Binary

Download the appropriate binary for your platform from the assets below.

Using Go

go install github.com/agentstation/tokenizer/cmd/tokenizer@v0.0.3

Verify Checksums

# Download checksums.txt and the binary, then:
sha256sum -c checksums.txt

Changelog

New Features

• 6f72418: feat: add automatic streaming detection for piped input (@jackspirou)
• 72a1350: feat: make encode the default action for llama3 command (@jackspirou)

Bug fixes

• 0fcfc76: fix: simplify release naming to just version tag (@jackspirou)

tokenizer-v0.0.2

tokenizer v0.0.2

Installation

Homebrew (macOS and Linux)

brew install agentstation/tap/tokenizer

Download Binary

Download the appropriate binary for your platform from the assets below.

Using Go

go install github.com/agentstation/tokenizer/cmd/tokenizer@v0.0.2

Verify Checksums

# Download checksums.txt and the binary, then:
sha256sum -c checksums.txt

Changelog

New Features

• fb17799: feat: enable automatic Homebrew formula updates via GoReleaser (@jackspirou)

Bug fixes

• c194a9e: fix: correct GoReleaser homebrew config and update README (@jackspirou)

tokenizer-v0.0.1

tokenizer v0.0.1

Installation

Download Binary

Download the appropriate binary for your platform from the assets below.

Using Go

go install github.com/agentstation/tokenizer/cmd/tokenizer@v0.0.1

Verify Checksums

# Download checksums.txt and the binary, then:
sha256sum -c checksums.txt

Changelog

New Features

• 4669248: feat: Add Llama3 tokenizer implementation (jackspirou jack.spirou@me.com)
• cefc8fb: feat: add comprehensive build tooling and development workflows (jackspirou jack.spirou@me.com)
• 140e062: feat: optimize tokenizer and enhance project structure (jackspirou jack.spirou@me.com)

Bug fixes

• 0d50f5f: fix: add dir field to goreleaser build config (jackspirou jack.spirou@me.com)
• fbfd55d: fix: correct .gitignore to only exclude root tokenizer binary (jackspirou jack.spirou@me.com)
• 3d7baaf: fix: correct ldflags in GoReleaser configuration (jackspirou jack.spirou@me.com)
• 9eb3006: fix: simplify goreleaser config to match working examples (jackspirou jack.spirou@me.com)
• 337ccf6: fix: update GoReleaser version in release workflow (jackspirou jack.spirou@me.com)
• 0efe928: fix: update to goreleaser-action v6 (jackspirou jack.spirou@me.com)

Others

• c4b5495: Initial commit (Jack Spirou jack@spirou.io)
• 4e17717: ci(deps): bump codecov/codecov-action from 4 to 5 (dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>)
• 293a415: ci(deps): bump golangci/golangci-lint-action from 4 to 8 (dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>)
• f368e35: refactor: improve naming conventions for vocabulary loaders (jackspirou jack.spirou@me.com)
• 9569825: refactor: modernize API and improve test organization (jackspirou jack.spirou@me.com)