feat(format): transition packaging from generated rust project builds to post link binary payloads

[besaleli]

Description

This PR introduces the encoderfile embedded binary format and packaging pipeline. Instead of embedding model artifacts via linker sections or include_bytes!, the runtime binary now has a self-describing payload appended at EOF, located via a fixed-size footer and protobuf manifest. Model artifacts (weights, tokenizer, config, transforms) are streamed from disk using seek-based IO and loaded lazily at runtime, with a single dynamic dispatch on model type followed by fully statically-typed execution. This design avoids platform-specific linker behavior, keeps builds fast, enables inspection and tooling, and cleanly separates compilation from packaging.

Benefits

• No more cargo-in-cargo
• Lazy loading of large assets (e.g., model weights)

Why manually specified file format instead of appended sections?

We intentionally avoid embedding artifacts via custom linker sections (.data, extern "C" symbols). While simpler initially, this approach is platform-specific, eagerly loads large blobs into memory, and provides no clear boundary for versioning or inspection. The appended encoderfile container instead offers predictable cross-platform behavior, lazy loading, and a stable, versioned format.

Testing

To test, run:

cargo build --bin encoderfile --release

Then run:

cargo run -- build -f test_config.yml --base-binary-path target/release/encoderfile

To run encoderfile:

chmod +x ./my-model-2.encoderfile
./my-model-2.encoderfile infer "hello"

Notes

• Tokenizer config is now located in TokenizerService instead of Config
• Base binaries are compiled from encoderfile-runtime
• Base binary must be explicitly specified until Pull base binaries from GH releases for encoderfile builds #204 is closed. For testing, run cargo build --bin encoderfile --release, and use target/release/encoderfile-runtime.
• Dockerfile now uses debian:bookworm-slim as a base image since we no longer need Cargo toolchain.
• we're explicitly encoding everything in little-endian, BE is theoretically possible but not a priority given our prioritized arch targets

Out of Scope

• Integration tests. See related issue: Integration tests for codec #203
• Pulling base binaries from GH releases. See related issue: Pull base binaries from GH releases for encoderfile builds #204
• Docs updates. See related issue: Docs: post-link encoderfile format #205
• Encoderfile image CLI refactor. See related issue: Move encoderfile image cli to encoderfile-runtime #206

[javiermtorres]

LGTM