JSON-aware compression for NDJSON sinks - 2x smaller than zstd on structured logs

[rushikeshmore]

Hey,

I built a JSON/NDJSON compressor called DataCortex in Rust. Instead of treating JSON as raw bytes like zstd/gzip, it auto-infers the schema, reorgs rows into columns, and applies type-specific encoding before compressing. The result on structured logs:

Data	Size	DataCortex	zstd-19
k8s logs (100K rows)	9.9 MB	260 KB	524 KB
nginx logs (100K rows)	9.5 MB	339 KB	549 KB
Analytics events (10K)	3.3 MB	132 KB	206 KB

Lossless, byte-exact roundtrip. 381 tests. MIT licensed, published on crates.io.

I'm wondering if this could be useful as a compression codec option in Vector for JSON/NDJSON sinks. It's pure Rust so no FFI needed. Main limitation is it only helps on JSON/NDJSON data.

Repo: https://github.com/rushikeshmore/DataCortex
Site: https://datacortex-dcx.vercel.app