tokenizer-v0.0.1
·
42 commits
to master
since this release
ba5a228
This commit was signed with the committer’s verified signature.
jackspirou
Jack Spirou
tokenizer v0.0.1
Installation
Download Binary
Download the appropriate binary for your platform from the assets below.
Using Go
go install github.com/agentstation/tokenizer/cmd/tokenizer@v0.0.1Verify Checksums
# Download checksums.txt and the binary, then:
sha256sum -c checksums.txtChangelog
New Features
- 4669248: feat: Add Llama3 tokenizer implementation (jackspirou jack.spirou@me.com)
- cefc8fb: feat: add comprehensive build tooling and development workflows (jackspirou jack.spirou@me.com)
- 140e062: feat: optimize tokenizer and enhance project structure (jackspirou jack.spirou@me.com)
Bug fixes
- 0d50f5f: fix: add dir field to goreleaser build config (jackspirou jack.spirou@me.com)
- fbfd55d: fix: correct .gitignore to only exclude root tokenizer binary (jackspirou jack.spirou@me.com)
- 3d7baaf: fix: correct ldflags in GoReleaser configuration (jackspirou jack.spirou@me.com)
- 9eb3006: fix: simplify goreleaser config to match working examples (jackspirou jack.spirou@me.com)
- 337ccf6: fix: update GoReleaser version in release workflow (jackspirou jack.spirou@me.com)
- 0efe928: fix: update to goreleaser-action v6 (jackspirou jack.spirou@me.com)
Others
- c4b5495: Initial commit (Jack Spirou jack@spirou.io)
- 4e17717: ci(deps): bump codecov/codecov-action from 4 to 5 (dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>)
- 293a415: ci(deps): bump golangci/golangci-lint-action from 4 to 8 (dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>)
- f368e35: refactor: improve naming conventions for vocabulary loaders (jackspirou jack.spirou@me.com)
- 9569825: refactor: modernize API and improve test organization (jackspirou jack.spirou@me.com)