Skip to content

[fix](test) Stabilize export parallelism CSV reload#65395

Open
shuke987 wants to merge 1 commit into
apache:masterfrom
shuke987:fix-export-parallelism-csv-schema
Open

[fix](test) Stabilize export parallelism CSV reload#65395
shuke987 wants to merge 1 commit into
apache:masterfrom
shuke987:fix-export-parallelism-csv-schema

Conversation

@shuke987

@shuke987 shuke987 commented Jul 8, 2026

Copy link
Copy Markdown
Collaborator

Summary

  • add an explicit CSV schema when reloading CSV-family export fragments through S3 TVF
  • use a unique S3 export prefix per export invocation to avoid cross-round/object namespace interference

Testing

  • git diff --check
  • groovyc -d /tmp/doris_groovy_check regression-test/suites/export_p2/test_export_with_parallelism.groovy
  • not run against a Doris cluster

@hello-stephen

Copy link
Copy Markdown
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@shuke987

shuke987 commented Jul 8, 2026

Copy link
Copy Markdown
Collaborator Author

run buildall

@shuke987

shuke987 commented Jul 8, 2026

Copy link
Copy Markdown
Collaborator Author

/review

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review opinion: no blocking issue found.

I reviewed PR 65395 at head 22eb45f. The PR changes only regression-test/suites/export_p2/test_export_with_parallelism.groovy.

Critical checkpoint conclusions:

  • Goal and proof: the change is a test stabilization for export parallelism reloads. The per-export UUID S3 prefix prevents stale files from previous formats/runs from being matched, and the CSV-family reloads now pass the TVF schema needed for stable typed reads. The existing generated .out file still has result blocks matching the suite's invocations.
  • Scope and clarity: the patch is small and localized to the affected regression suite.
  • Concurrency: no production concurrency path is changed. For the test, using a UUID-specific object prefix reduces cross-run/object collision risk.
  • Lifecycle/static initialization: not applicable; no lifecycle or static initialization code changes.
  • Configuration/session propagation: no new config is added. The existing Nereids session setup is unchanged. The added CSV TVF properties are local to the reload query and only applied to CSV-family formats.
  • Compatibility/storage/protocol: not applicable; no production serialization, storage, or FE-BE protocol change.
  • Parallel code paths: csv, parquet, orc, csv_with_names, and csv_with_names_and_types paths were checked. The CSV header formats still rely on their existing header-skip behavior, with csv_schema only stabilizing TVF column typing.
  • Special conditions: the format in ["csv", "csv_with_names", "csv_with_names_and_types"] condition matches the formats that need CSV TVF schema and separator properties.
  • Test coverage/results: this is itself a regression-test change. The expected output file is not changed, and the output block count remains consistent with the 21 reload checks plus the initial source-table check.
  • Observability: no additional observability is needed for this test-only change.
  • Transaction/persistence/data writes: no production transaction or persistence path is changed.
  • FE-BE variables/performance: no new transmitted variable or performance-sensitive production path is changed.

User focus points: none were provided beyond the full PR review.

Subagent conclusions:

  • optimizer-rewrite: no optimizer/rewrite, semantic-equivalence, parallel join, or aggregate-path issue found.
  • tests-session-config: no regression/session/config/CI issue found.
  • Final convergence round 1 ended with both live subagents replying NO_NEW_VALUABLE_FINDINGS for the same current ledger and empty proposed final comment set.

Validation limits: static review only. This checkout is missing .worktree_initialized, thirdparty/installed, and thirdparty/installed/bin/protoc, so I did not run the S3 regression suite locally.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants