Skip to content

SNOW-1887892: improve process pool + thread pool logic, add retry logic, add timeout logic#2924

Merged
sfc-gh-yuwang merged 6 commits intodev/data-sourcefrom
SNOW-1887892
Jan 30, 2025
Merged

SNOW-1887892: improve process pool + thread pool logic, add retry logic, add timeout logic#2924
sfc-gh-yuwang merged 6 commits intodev/data-sourcefrom
SNOW-1887892

Conversation

@sfc-gh-yuwang
Copy link
Collaborator

@sfc-gh-yuwang sfc-gh-yuwang commented Jan 23, 2025

…not wait, add retry logic

  1. Which Jira issue is this PR addressing? Make sure that there is an accompanying issue to your PR.

    Fixes SNOW-1887892

  2. Fill out the following pre-review checklist:

    • I am adding a new automated test(s) to verify correctness of my new code
      • If this test skips Local Testing mode, I'm requesting review from @snowflakedb/local-testing
    • I am adding new logging messages
    • I am adding a new telemetry message
    • I am adding new credentials
    • I am adding a new dependency
    • If this is a new feature/behavior, I'm adding the Local Testing parity changes.
    • I acknowledge that I have ensured my changes to be thread-safe. Follow the link for more information: Thread-safe Developer Guidelines
  3. Please describe how your code solves the related issue.

    this PR meant to do two things:
    i. decouple process pool download and thread pool upload/copy into table
    ii. add retry logic for both process pool and thread pool
    test for partition download will be added in other PR

@github-actions
Copy link

Seems like your changes contain some Local Testing changes, please request review from @snowflakedb/local-testing

@github-actions
Copy link

Seems like your changes contain some Local Testing changes, please request review from @snowflakedb/local-testing

@sfc-gh-yuwang sfc-gh-yuwang marked this pull request as ready for review January 24, 2025 18:55
@sfc-gh-yuwang sfc-gh-yuwang requested review from a team as code owners January 24, 2025 18:55
@sfc-gh-yuwang sfc-gh-yuwang requested review from sfc-gh-aalam and sfc-gh-jrose and removed request for a team January 24, 2025 18:55
@github-actions
Copy link

Seems like your changes contain some Local Testing changes, please request review from @snowflakedb/local-testing

@sfc-gh-yuwang sfc-gh-yuwang changed the title SNOW-1887892: improve process pool + thread pool logic, add retry logic SNOW-1887892: improve process pool + thread pool logic, add retry logic, add timeout logic Jan 27, 2025
@github-actions
Copy link

Seems like your changes contain some Local Testing changes, please request review from @snowflakedb/local-testing

@github-actions
Copy link

Seems like your changes contain some Local Testing changes, please request review from @snowflakedb/local-testing

@sfc-gh-yuwang sfc-gh-yuwang merged commit 93b5a10 into dev/data-source Jan 30, 2025
20 of 38 checks passed
@sfc-gh-yuwang sfc-gh-yuwang deleted the SNOW-1887892 branch January 30, 2025 21:14
@github-actions github-actions bot locked and limited conversation to collaborators Jan 30, 2025
@sfc-gh-yuwang sfc-gh-yuwang restored the SNOW-1887892 branch February 13, 2025 22:51
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants