Skip to content

feat(codecs): add arrow IPC stream batch encoder #24124

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
pront merged 65 commits into from
Nov 14, 2025
Merged

feat(codecs): add arrow IPC stream batch encoder #24124

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
65 commits
Select commit Hold shift + click to select a range
8c3e47f
enhancement(clickhouse sink): add support for ArrowStream format
benjamin-awd Oct 18, 2025
5852b8b
refactor: do not allow dynamic database/table for arrow
benjamin-awd Oct 18, 2025
6583753
chore: add type assertions
benjamin-awd Oct 18, 2025
3c7b4f2
refactor: use clearer function names in request builder
benjamin-awd Oct 20, 2025
ff79ca3
chore: add snafu for request builder
benjamin-awd Oct 20, 2025
631da91
chore: move imports to top package
benjamin-awd Oct 20, 2025
1d951b4
chore: handle templates for table and database
benjamin-awd Oct 20, 2025
7ae4c2a
build(deps): move arrow under external libs
benjamin-awd Oct 20, 2025
d852d5a
build(deps): bump arrow/arrow-schema to latest
benjamin-awd Oct 20, 2025
5b8f548
chore: add support for all timeunits e.g. nanoseconds
benjamin-awd Oct 20, 2025
a1e7318
chore: add handling for LowCardinality
benjamin-awd Oct 20, 2025
8e78211
chore: add support for Clickhouse decimal
benjamin-awd Oct 20, 2025
c7f05f6
chore: add capacity and string zero copy optimization
benjamin-awd Oct 20, 2025
d7e4e33
chore: use logging macros from extern crate
benjamin-awd Oct 21, 2025
609c126
chore: add debug logs for schema
benjamin-awd Oct 21, 2025
b11d2b7
chore: use simpler message for NoSchemaProvided snafu
benjamin-awd Oct 21, 2025
8c755c9
chore: DRY target scale within decimal builders
benjamin-awd Oct 21, 2025
1babc70
docs: add note about upcasting i128 -> i256
benjamin-awd Oct 21, 2025
8003c7b
chore: move i256 import statement to top
benjamin-awd Oct 21, 2025
cbd78c3
chore: remove redundant schema log
benjamin-awd Oct 21, 2025
cfe3848
chore: only log schema a single time at startup
benjamin-awd Oct 21, 2025
939f5c2
chore: use arrow payload size instead of redundant json size
benjamin-awd Oct 21, 2025
730309a
chore: add support for uint
benjamin-awd Oct 22, 2025
f169a15
chore: add support for rfc3339 timestamp
benjamin-awd Oct 22, 2025
5148128
refactor: move arrow to clickhouse sink subfolder
benjamin-awd Oct 25, 2025
69063e5
refactor: move Arrow encoder to shared util
benjamin-awd Oct 25, 2025
060a8fe
chore: remove redundant feature gates
benjamin-awd Oct 25, 2025
3396bd7
refactor: rename schema.rs -> arrow_schema.rs
benjamin-awd Oct 25, 2025
a56da91
chore: add changelog fragment
benjamin-awd Oct 25, 2025
0734f4f
chore: add helper function to handle clickhouse type parsing
benjamin-awd Oct 25, 2025
c27b09b
refactor: remove default coercions in arrow_schema
benjamin-awd Oct 25, 2025
3ae4ff7
refactor: add constants for decimal precision values
benjamin-awd Oct 25, 2025
66ca5f5
refactor: extract type modifier unwrapping into helper
benjamin-awd Oct 25, 2025
edc4c5f
refactor: remove unnecessary vector allocation
benjamin-awd Oct 25, 2025
d8ccf91
refactor: organize type mapping with comments
benjamin-awd Oct 25, 2025
bb93d26
refactor: use constants and functional style in decimal parsing
benjamin-awd Oct 25, 2025
9f93b97
chore: update LICENSE-3rdparty
benjamin-awd Oct 25, 2025
689e33c
chore: remove duplicated tests
benjamin-awd Oct 25, 2025
57fbfe4
chore: update generated docs
benjamin-awd Oct 25, 2025
56c4b29
chore: remove redundant docstring
benjamin-awd Oct 27, 2025
0b2e0a2
refactor: create generic arrow codec
benjamin-awd Oct 29, 2025
f658d0a
Merge branch 'master' into add-ch-arrow-codec
benjamin-awd Oct 31, 2025
5be6b9c
chore: remove duplicated changelog
benjamin-awd Oct 31, 2025
f430c02
chore: cargo fmt
benjamin-awd Nov 3, 2025
274739c
Merge branch 'master' into add-ch-arrow-codec
benjamin-awd Nov 4, 2025
a609590
chore: fix lock file
benjamin-awd Nov 5, 2025
304662b
Merge branch 'master' into add-ch-arrow-codec
benjamin-awd Nov 5, 2025
f591926
chore: clippy
benjamin-awd Nov 6, 2025
af58a40
chore: add missing int/float types
benjamin-awd Nov 7, 2025
5e35a57
Merge remote-tracking branch 'origin/master' into add-ch-arrow-codec
pront Nov 7, 2025
cbe6791
make fmt
pront Nov 7, 2025
39aa681
update licenses
pront Nov 7, 2025
3856e40
chore: remove schema provider logic
benjamin-awd Nov 8, 2025
d4d56dc
chore: add with_capacity for helper functions
benjamin-awd Nov 8, 2025
857d3da
chore: add support for null values
benjamin-awd Nov 8, 2025
219f64a
chore: clean up encode_events_to_arrow_ipc_stream
benjamin-awd Nov 8, 2025
d4523a2
chore: use macros for primitives and null constraint checks
benjamin-awd Nov 8, 2025
c01d99f
Merge branch 'master' into add-ch-arrow-codec
pront Nov 11, 2025
f0aa30b
add links to changelog
pront Nov 11, 2025
b93961e
chore: clippy
benjamin-awd Nov 12, 2025
a060845
chore: fix duplicate events
benjamin-awd Nov 12, 2025
e862e94
Merge branch 'master' into add-ch-arrow-codec
pront Nov 12, 2025
bf94ab0
Merge branch 'master' into add-ch-arrow-codec
benjamin-awd Nov 13, 2025
9654ddb
chore: remove duplicated import
benjamin-awd Nov 13, 2025
2e5ac56
chore: clippy
benjamin-awd Nov 13, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
304 changes: 302 additions & 2 deletions Cargo.lock
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Loading
Loading