Custom query and preprocess missing columns handling #19
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…b_connection_analysis
…tialize test for function, make change to how custom_query is used in function
…tering by status, GPU, and elapsed time for load_jobs_dataframe_from_duckdb, add associated test
…lysis Merge with origin/main.
… for querying columns
…logic in preprocess code
…o handle missing columns
… TODO and temporarily finalize handling of missing columns in preprocess
…_back filtering options
…r allocated_vram vram_constraints calculation when deal with empty dataframe in preprocess
…jobs_dataframe_from_duckdb, finalize test for load_jobs_duck_db
…Error and warnings for preprocess
…nd add comments explaining missing columns logic
…and test for this change
…ry and the other with custom query options, changed tests accordingly
Espiobest
reviewed
Aug 12, 2025
Espiobest
reviewed
Aug 12, 2025
Espiobest
reviewed
Aug 12, 2025
Collaborator
|
Changes look good to me |
…ing test_loads_and_preprocessed_jobs
…ue_counts in test_preprocess, adjust mock data to have more recent date
Espiobest
reviewed
Aug 17, 2025
Collaborator
Espiobest
left a comment
Espiobest
left a comment
There was a problem hiding this comment.
I just had 2 small comments with the tests. Otherwise the implementation looks good to me!
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR includes change made to preprocess to include missing columns handling logic and
load_preprocessed_jobs_dataframe_from_duckdbfor checking a date_back options in custom query.For all columns that are necessary for calculation in preprocess and EfficiencyAnalysis's jobs metrics calculation (GPU-related metrics), this will enforce the dataframe to have those columns. Otherwise, it will raise the warning if the dataframe is missing some columns.
For
load_preprocessed_jobs_dataframe_from_duckdb, it has some new parameters to include the dates_back options for SQL query and also support parameters of preprocess so user can use this as a complete wrapper aroundDatabaseConnectionandpreprocess_data.