DOG-6369 : Add Dataset support and upload queue size#487
Conversation
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## master #487 +/- ##
==========================================
+ Coverage 80.63% 80.65% +0.01%
==========================================
Files 43 43
Lines 4194 4203 +9
==========================================
+ Hits 3382 3390 +8
- Misses 812 813 +1
🚀 New features to boost your workflow:
|
|
/gemini review |
There was a problem hiding this comment.
Code Review
This pull request introduces DataSet configuration support to ConnectionConfig and allows configuring the upload queue size in ExtractorConfig. The changes involve adding data-set-external-id or data-set configurations, a method to retrieve data set details, and an upload-queue-size configuration. The code changes also include adding tests for the new functionality. The review focuses on correctness and potential issues related to the implementation of the get_data_set method in ExtractorConfig.
|
First round of sanity done. LGTM. Need to be looked further by another team member. |
This PR adds DataSet configuration support to
ConnectionConfig(withget_data_setmethod) and makesupload queue sizeconfigurable in ExtractorConfig.Changes:
data-set-external-idordata-setconfiguration and a method to get the data set details.upload-queue-sizeconfiguration