Skip to content

DOG-6369 : Add Dataset support and upload queue size#487

Merged
Hmnt39 merged 3 commits intomasterfrom
DOG-6369-dataset-configuration-support
Nov 5, 2025
Merged

DOG-6369 : Add Dataset support and upload queue size#487
Hmnt39 merged 3 commits intomasterfrom
DOG-6369-dataset-configuration-support

Conversation

@Hmnt39
Copy link
Copy Markdown
Contributor

@Hmnt39 Hmnt39 commented Oct 28, 2025

This PR adds DataSet configuration support to ConnectionConfig (with get_data_set method) and makes upload queue size configurable in ExtractorConfig.

Changes:

  1. For DataSet support: Add data-set-external-id or data-set configuration and a method to get the data set details.
  2. For custom queue size: Add upload-queue-size configuration

@Hmnt39 Hmnt39 requested a review from a team as a code owner October 28, 2025 11:07
@Hmnt39 Hmnt39 changed the title Add Dataset support and upload queue size DOG-6369 : Add Dataset support and upload queue size Oct 28, 2025
@codecov
Copy link
Copy Markdown

codecov bot commented Oct 28, 2025

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 80.65%. Comparing base (981425d) to head (061d3b8).
⚠️ Report is 1 commits behind head on master.

Additional details and impacted files
@@            Coverage Diff             @@
##           master     #487      +/-   ##
==========================================
+ Coverage   80.63%   80.65%   +0.01%     
==========================================
  Files          43       43              
  Lines        4194     4203       +9     
==========================================
+ Hits         3382     3390       +8     
- Misses        812      813       +1     
Files with missing lines Coverage Δ
...te/extractorutils/unstable/configuration/models.py 84.46% <100.00%> (+0.69%) ⬆️

... and 1 file with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@Hmnt39
Copy link
Copy Markdown
Contributor Author

Hmnt39 commented Oct 29, 2025

/gemini review

Copy link
Copy Markdown

@gemini-code-assist gemini-code-assist bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces DataSet configuration support to ConnectionConfig and allows configuring the upload queue size in ExtractorConfig. The changes involve adding data-set-external-id or data-set configurations, a method to retrieve data set details, and an upload-queue-size configuration. The code changes also include adding tests for the new functionality. The review focuses on correctness and potential issues related to the implementation of the get_data_set method in ExtractorConfig.

@Hmnt39 Hmnt39 requested a review from toondaey October 31, 2025 00:24
Copy link
Copy Markdown

@121yaseen 121yaseen left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks Good

@vikramlc-cognite
Copy link
Copy Markdown

First round of sanity done. LGTM. Need to be looked further by another team member.

@Hmnt39 Hmnt39 added the waiting-for-risk-review Waiting for a member of the risk review team to take an action label Nov 4, 2025
@rsjr rsjr self-assigned this Nov 5, 2025
@rsjr rsjr added the risk-review-ongoing Risk review is in progress label Nov 5, 2025
Copy link
Copy Markdown
Contributor

@rsjr rsjr left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

risk review ok

@rsjr rsjr added waiting-for-team Waiting for the submitter or reviewer of the PR to take an action and removed waiting-for-risk-review Waiting for a member of the risk review team to take an action labels Nov 5, 2025
@Hmnt39 Hmnt39 merged commit ba403a8 into master Nov 5, 2025
6 checks passed
@Hmnt39 Hmnt39 deleted the DOG-6369-dataset-configuration-support branch November 5, 2025 12:20
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

risk-review-ongoing Risk review is in progress waiting-for-team Waiting for the submitter or reviewer of the PR to take an action

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants