test(spill): add unit tests and integration tests for spill-to-disk by dalingmeng · Pull Request #272 · alibaba/paimon-cpp

dalingmeng · 2026-05-11T11:33:02Z

Purpose

Linked issue: #149
add unit tests and integration tests for spill-to-disk

Tests

WriteInteTest.TestPkSpillable.*
KeyValueFileStoreWriteTest.TestSpill.*
MergeTreeWriterTest.TestSpill.*
WriteBufferTest.TestSpill.*
SpillReaderWriterTest

API and Format

Documentation

Generative AI tooling

Claude opus 4.6

Copilot

Pull request overview

Note

Copilot was unable to run its full agentic suite in this review.

Adds comprehensive unit/integration coverage for spill-to-disk behavior (incl. quota exhaustion, cleanup, dedup, intermediate merge, and IO error handling), linked to issue #149.

Changes:

Adds new spill-focused integration tests to validate correctness, cleanup, and failure fallbacks.
Adds unit tests around spill file quotas, reader ordering, merge behavior, and resource cleanup.
Refactors repeated commit/scan logic in integration tests into helper methods.

Reviewed changes

Copilot reviewed 5 out of 5 changed files in this pull request and generated 6 comments.

Show a summary per file

File	Description
test/inte/write_inte_test.cpp	Adds PK spill integration tests + introduces Commit/Scan helper utilities to reduce duplication.
src/paimon/core/operation/key_value_file_store_write_test.cpp	Adds spill tests for disk-quota fallback and multi-round dedup semantics.
src/paimon/core/mergetree/write_buffer_test.cpp	Adds extensive WriteBuffer spill/quota/reader/cleanup test coverage.
src/paimon/core/mergetree/spill_reader_writer_test.cpp	Adds tests for writer Close idempotency and schema mismatch error cases.
src/paimon/core/mergetree/merge_tree_writer_test.cpp	Adds MergeTreeWriter spill tests: dedup, intermediate merge bounds, quota fallback, cleanup.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

+    Status CommitMessages(const std::string& table_path,
+                          const std::vector<std::shared_ptr<CommitMessage>>& commit_messages,
+                          const std::map<std::string, std::string>& commit_options,
+                          bool ignore_empty_commit = true,
+                          int64_t commit_identifier = BATCH_WRITE_COMMIT_IDENTIFIER) const {
+        CommitContextBuilder commit_builder(table_path, "commit_user_1");
+        commit_builder.SetOptions(commit_options);
+        commit_builder.IgnoreEmptyCommit(ignore_empty_commit);
+        PAIMON_ASSIGN_OR_RAISE(auto commit_context, commit_builder.Finish());
+        PAIMON_ASSIGN_OR_RAISE(auto file_store_commit,
+                               FileStoreCommit::Create(std::move(commit_context)));
+        return file_store_commit->Commit(commit_messages, commit_identifier);
+    }


dalingmeng · 2026-05-13T09:33:45Z

+    ASSERT_TRUE(success_b1);
+}
+
+TEST_P(WriteInteTest, TestPkSpillableWithIOException) {


unnecessary

+                         schema_manger.ReadSchema(/*schema_id=*/0));
+    DataGenerator gen(table_schema, pool_);
+
+    for (size_t i = 0; i < 2000; i++) {


dalingmeng · 2026-05-13T09:33:19Z

+    for (const auto& split : data_splits) {
+        auto split_impl = dynamic_cast<DataSplitImpl*>(split.get());
+        ASSERT_OK_AND_ASSIGN(std::string partition_str,
+                             helper->PartitionStr(split_impl->Partition()));
+        splits_by_partition[partition_str].push_back(split);
+    }


unnecessary

dalingmeng force-pushed the feat/spill-test branch from 9eb5c57 to f441ff9 Compare May 11, 2026 11:39