Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
61 commits
Select commit Hold shift + click to select a range
16d2557
fix: Ensure properly nested null masks for parquet reads (#692)
scovich Feb 14, 2025
2e4f555
fix: Handle predicates on non-nullable columns without stats (#700)
adamreeve Feb 19, 2025
72b585d
feat!(ffi): new visit_schema FFI and rename old visit_schema to visit…
zachschuermann Feb 19, 2025
484eb33
feat: introduce feature flags to select major arrow versions (#654)
rtyler Feb 20, 2025
e2bcd0b
feat: Support writing to not only 3/7 protocol (#693)
nicklan Feb 20, 2025
baa3fc3
Part 4: read_table.c uses transform in ffi (#614)
nicklan Feb 21, 2025
ca18e7f
feat!(ffi): remove `visit_snapshot_schema`, add `logical_schema` (#709)
zachschuermann Feb 21, 2025
eb0f725
Support --all-features again (#708)
nicklan Feb 24, 2025
363256b
fix: Make having `need_arrow` + no arrow a compiler error (#717)
nicklan Feb 25, 2025
8b5e06d
Release 0.7.0 (#716)
zachschuermann Feb 25, 2025
4c00de4
pin chrono version to fix arrow compilation failure (#719)
hntd187 Feb 28, 2025
6110f76
fix!(ffi): Visit decimals as signed values (#724)
scovich Mar 3, 2025
ac91087
feat!(ffi): Make get_partition_column* work on a snapshot. (#697)
nicklan Mar 3, 2025
bbe8929
fix!: bump MSRV to 1.81 (#725)
zachschuermann Mar 3, 2025
76271c7
perf!: replace default engine JSON reader's `FileStream` with concurr…
zachschuermann Mar 3, 2025
aef2c9b
release 0.8.0 (#726)
zachschuermann Mar 4, 2025
15345d8
feat: extract & insert sidecar batches in `replay`'s action iterator …
sebastiantia Mar 5, 2025
95c06f1
feat: support the `v2Checkpoint` reader/writer feature (#685)
sebastiantia Mar 6, 2025
b663850
Update HDFS dependencies (#689)
rzepinskip Mar 7, 2025
bdbb2f7
feat: Add check for whether appendOnly table feature is supported or …
OussamaSaoudi Mar 11, 2025
cd5ea20
tests: add V2 checkpoint read support integration tests (#690)
sebastiantia Mar 12, 2025
db86d97
feat: Add basic partition pruning support (#713)
scovich Mar 12, 2025
ce31e97
feat: add `DeletionVectors` to supported writer features (#735)
zachschuermann Mar 13, 2025
341eb08
feat: add writer version 2/invariant table feature support (#734)
zachschuermann Mar 17, 2025
5f23fad
fix!: change metadata values for column metadata to i64 (#733)
hntd187 Mar 19, 2025
34b9d1a
ci: use maintained action to setup rust toolchain (#585)
roeap Mar 19, 2025
72bab02
refactor: split up `arrow_expression` module (#750)
maruschin Mar 20, 2025
74aab72
refactor: flatten deeply nested match statement (#756)
scovich Mar 20, 2025
3065a76
fix: return `Error::unsupported` instead of panic in `Scalar::to_arra…
zachschuermann Mar 20, 2025
ec47584
fix!: get prefix from offset path (#699)
roeap Mar 21, 2025
2573424
fix: remove 'default-members' in workspace, default to all crates (#752)
zachschuermann Mar 21, 2025
3dcad08
refactor!: make `snapshot.schema()` return a `SchemaRef` (#751)
maruschin Mar 21, 2025
79b9f24
refactor: Simplify predicate evaluation by supporting inversion (#761)
scovich Mar 22, 2025
512a31a
feat: robustify pre-signed URL checks (#760)
roeap Mar 24, 2025
7e5476a
refactor: rename `LogSegment::replay` to `LogSegment::read_actions` (…
zachschuermann Mar 25, 2025
fff0052
refactor!: make visit_expression_internal private, and unwrap_kernel…
nicklan Mar 26, 2025
cb70e58
fix!: make actions types pub(crate) instead of pub (#405)
zachschuermann Mar 26, 2025
5812eff
misc: add .cargo/config.toml with native instruction codegen (#772)
zachschuermann Mar 27, 2025
9290930
feat!: new `null_row` ExpressionHandler API (#662)
zachschuermann Mar 28, 2025
06fb373
refactor: extract deduplication logic from `AddRemoveDedupVisitor` in…
sebastiantia Mar 31, 2025
f062e87
refactor: Move testing helper function to `test_utils` mod (#794)
sebastiantia Apr 2, 2025
a244dd1
feat: add `CheckpointMetadata` action (#781)
sebastiantia Apr 2, 2025
7d99023
refactor: Rename `_last_checkpoint` from `CheckpointMetadata` to `Las…
sebastiantia Apr 2, 2025
eb91e51
fix: update compilation error and clippy lints for rustc 1.86 (#800)
zachschuermann Apr 3, 2025
bd6034e
refactor!: Rename enums of ReaderFeatures and WriterFeatures (#802)
hntd187 Apr 3, 2025
72e369f
refactor: Use ExpressionTransform instead of adhoc expression travers…
scovich Apr 4, 2025
2287455
refactor: extract log replay processing structure into `LogReplayProc…
sebastiantia Apr 7, 2025
4ad2bc6
refactor!: Remove get_ prefix from engine getters (#804)
scovich Apr 7, 2025
c84b079
feat: add classic and uuid parquet checkpoint path generation (#782)
sebastiantia Apr 7, 2025
9d36e25
refactor!: Rename FileSystemClient to StorageHandler (#805)
scovich Apr 7, 2025
0186dc4
feat: `Snapshot::try_new_from()` API (#549)
zachschuermann Apr 7, 2025
8961e97
refactor!: adopt types for table features (#684)
hntd187 Apr 8, 2025
10bdee7
refactor!: rename `ScanData` to `ScanMetadata` (#817)
zachschuermann Apr 9, 2025
b38bc5d
refactor!: update `ScanMetadata` to struct with new `FilteredEngineDa…
sebastiantia Apr 10, 2025
7e62d12
release 0.9.0 (#815)
zachschuermann Apr 10, 2025
929ac08
feat: add `CheckpointVisitor` in new `checkpoint` mod (#738)
sebastiantia Apr 10, 2025
e74d18b
docs: correct docstring for DefaultEngine::new (#821)
gotocoding-DB Apr 10, 2025
b8ae33f
refactor: Rename predicates mod to kernel_predicates (#822)
scovich Apr 11, 2025
e98aeae
Merge remote-tracking branch 'delta-io/main' into update-delta
kssenii Apr 11, 2025
2edeb8a
Update cargo.lock
kssenii Apr 14, 2025
1ee29c8
Update lockfile
May 19, 2025
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions .cargo/config.toml
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
[build]
rustflags = ["-C", "target-cpu=native"]
94 changes: 44 additions & 50 deletions .github/workflows/build.yml
Original file line number Diff line number Diff line change
Expand Up @@ -10,27 +10,20 @@ jobs:
format:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v3
- name: Install minimal stable with clippy and rustfmt
uses: actions-rs/toolchain@v1
- uses: actions/checkout@v4
- name: Install minimal stable with rustfmt
uses: actions-rust-lang/setup-rust-toolchain@v1
with:
profile: default
toolchain: stable
override: true
- uses: Swatinem/rust-cache@v2
components: rustfmt
- name: format
run: cargo fmt -- --check

msrv:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v3
- uses: actions/checkout@v4
- name: Install minimal stable and cargo msrv
uses: actions-rs/toolchain@v1
with:
profile: default
toolchain: stable
override: true
- uses: Swatinem/rust-cache@v2
uses: actions-rust-lang/setup-rust-toolchain@v1
- name: Install cargo-msrv
shell: bash
run: |
Expand All @@ -46,11 +39,7 @@ jobs:
steps:
- uses: actions/checkout@v4
- name: Install minimal stable and cargo msrv
uses: actions-rs/toolchain@v1
with:
profile: default
toolchain: stable
override: true
uses: actions-rust-lang/setup-rust-toolchain@v1
- uses: Swatinem/rust-cache@v2
- name: Install cargo-msrv
shell: bash
Expand All @@ -74,16 +63,34 @@ jobs:
env:
RUSTDOCFLAGS: -D warnings
steps:
- uses: actions/checkout@v3
- uses: actions/checkout@v4
- name: Install minimal stable
uses: actions-rs/toolchain@v1
with:
profile: default
toolchain: stable
override: true
- uses: Swatinem/rust-cache@v2
uses: actions-rust-lang/setup-rust-toolchain@v1
- name: build docs
run: cargo doc


# When we run cargo { build, clippy } --no-default-features, we want to build/lint the kernel to
# ensure that we can build the kernel without any features enabled. Unfortunately, due to how
# cargo resolves features, if we have a workspace member that depends on the kernel with features
# enabled, the kernel will be compiled with those features (even if we specify
# --no-default-features).
#
# To cope with this, we split build/clippy --no-default-features into two runs:
# 1. build/clippy all packages that depend on the kernel with some features enabled:
# - acceptance
# - test_utils
# - feature_tests
# (and examples)
# - inspect-table
# - read-table-changes
# - read-table-multi-threaded
# - read-table-single-threaded
# 2. build/clippy all packages that only have no-feature kernel dependency
# - delta_kernel
# - delta_kernel_derive
# - delta_kernel_ffi
# - delta_kernel_ffi_macros
build:
runs-on: ${{ matrix.os }}
strategy:
Expand All @@ -93,20 +100,17 @@ jobs:
- ubuntu-latest
- windows-latest
steps:
- uses: actions/checkout@v3
- name: Install minimal stable with clippy and rustfmt
uses: actions-rs/toolchain@v1
- uses: actions/checkout@v4
- name: Install minimal stable with clippy
uses: actions-rust-lang/setup-rust-toolchain@v1
with:
profile: default
toolchain: stable
override: true
- uses: Swatinem/rust-cache@v2
- name: check kernel builds with no-default-features
run: cargo build -p delta_kernel --no-default-features
components: clippy
- name: build and lint with clippy
run: cargo clippy --benches --tests --all-features -- -D warnings
- name: lint without default features
run: cargo clippy --no-default-features -- -D warnings
- name: lint without default features - packages which depend on kernel with features enabled
run: cargo clippy --workspace --no-default-features --exclude delta_kernel --exclude delta_kernel_ffi --exclude delta_kernel_derive --exclude delta_kernel_ffi_macros -- -D warnings
- name: lint without default features - packages which don't depend on kernel with features enabled
run: cargo clippy --no-default-features --package delta_kernel --package delta_kernel_ffi --package delta_kernel_derive --package delta_kernel_ffi_macros -- -D warnings
- name: check kernel builds with default-engine
run: cargo build -p feature_tests --features default-engine
- name: check kernel builds with default-engine-rustls
Expand All @@ -120,14 +124,9 @@ jobs:
- ubuntu-latest
- windows-latest
steps:
- uses: actions/checkout@v3
- uses: actions/checkout@v4
- name: Install minimal stable with clippy and rustfmt
uses: actions-rs/toolchain@v1
with:
profile: default
toolchain: stable
override: true
- uses: Swatinem/rust-cache@v2
uses: actions-rust-lang/setup-rust-toolchain@v1
- name: test
run: cargo test --workspace --verbose --all-features -- --skip read_table_version_hdfs

Expand Down Expand Up @@ -220,14 +219,9 @@ jobs:
steps:
- uses: actions/checkout@v4
- name: Install rust
uses: actions-rs/toolchain@v1
with:
profile: default
toolchain: stable
override: true
uses: actions-rust-lang/setup-rust-toolchain@v1
- name: Install cargo-llvm-cov
uses: taiki-e/install-action@cargo-llvm-cov
- uses: Swatinem/rust-cache@v2
- name: Generate code coverage
run: cargo llvm-cov --all-features --workspace --codecov --output-path codecov.json -- --skip read_table_version_hdfs
- name: Upload coverage to Codecov
Expand Down
12 changes: 3 additions & 9 deletions .github/workflows/run_integration_test.yml
Original file line number Diff line number Diff line change
Expand Up @@ -18,17 +18,11 @@ jobs:
- name: Skip job for pull requests on Windows
if: ${{ matrix.skip }}
run: echo "Skipping job for pull requests on Windows."
- uses: actions/checkout@v3
- uses: actions/checkout@v4
if: ${{ !matrix.skip }}
- name: Install minimal stable rust
if: ${{ !matrix.skip }}
uses: actions-rs/toolchain@v1
with:
profile: default
toolchain: stable
override: true
- uses: Swatinem/rust-cache@v2
- name: Setup rust toolchain
if: ${{ !matrix.skip }}
uses: actions-rust-lang/setup-rust-toolchain@v1
- name: Run integration tests
if: ${{ !matrix.skip }}
shell: bash
Expand Down
7 changes: 1 addition & 6 deletions .github/workflows/semver-checks.yml
Original file line number Diff line number Diff line change
Expand Up @@ -25,12 +25,7 @@ jobs:
fetch-depth: 0
ref: ${{ github.event.pull_request.head.sha }}
- name: Install minimal stable
uses: actions-rs/toolchain@v1
with:
profile: default
toolchain: stable
override: true
- uses: Swatinem/rust-cache@v2
uses: actions-rust-lang/setup-rust-toolchain@v1
- name: Install cargo-semver-checks
shell: bash
run: |
Expand Down
1 change: 0 additions & 1 deletion .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,6 @@
.zed

# Rust
.cargo/
target/
integration-tests/Cargo.lock

Expand Down
Loading
Loading