Skip to content

Comments

Add workflow to run SDGym multi-table benchmark monthly and publish results#518

Merged
R-Palazzo merged 26 commits intofeature_branch/mutli_table_benchmarkfrom
issue-516-add-workflows
Jan 8, 2026
Merged

Add workflow to run SDGym multi-table benchmark monthly and publish results#518
R-Palazzo merged 26 commits intofeature_branch/mutli_table_benchmarkfrom
issue-516-add-workflows

Conversation

@R-Palazzo
Copy link
Collaborator

@R-Palazzo R-Palazzo commented Dec 22, 2025

Resolve #516
CU-86b7w17ze

@amontanez24 From what I've experienced, we can currently run 4-gpu machines at the same location:

RuntimeError: GCP instance creation failed: Quota 'NVIDIA_T4_GPUS' exceeded.  Limit: 4.0 in region us-central1.

The initial plan is to have 6 instances running on January 1st (4 for single-table, 2 for multi-table), so I wanted to discuss the options with you:

  • Run only 4 machines (start with single-table on January 1st and multi-table on January 5th, so machines are available again)
  • Run 6 machines in a different location, if that's allowed.
  • Increase the number of machines, if possible.

Maybe there is a better solution that I'm happy to discuss.

@R-Palazzo R-Palazzo self-assigned this Dec 22, 2025
@R-Palazzo R-Palazzo requested a review from a team as a code owner December 22, 2025 17:48
@sdv-team
Copy link
Contributor

@R-Palazzo R-Palazzo removed the request for review from a team December 22, 2025 17:50
@codecov
Copy link

codecov bot commented Dec 22, 2025

Codecov Report

❌ Patch coverage is 87.69231% with 8 lines in your changes missing coverage. Please review.
✅ Project coverage is 78.87%. Comparing base (d52e10b) to head (84b562d).
⚠️ Report is 1 commits behind head on feature_branch/mutli_table_benchmark.

Files with missing lines Patch % Lines
sdgym/run_benchmark/upload_benchmark_results.py 83.33% 3 Missing ⚠️
sdgym/run_benchmark/utils.py 88.00% 3 Missing ⚠️
sdgym/benchmark.py 0.00% 2 Missing ⚠️
Additional details and impacted files
@@                           Coverage Diff                            @@
##           feature_branch/mutli_table_benchmark     #518      +/-   ##
========================================================================
+ Coverage                                 78.78%   78.87%   +0.08%     
========================================================================
  Files                                        33       33              
  Lines                                      2791     2826      +35     
========================================================================
+ Hits                                       2199     2229      +30     
- Misses                                      592      597       +5     
Flag Coverage Δ
integration 54.67% <0.00%> (-0.62%) ⬇️
unit 73.38% <87.69%> (+0.08%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@R-Palazzo R-Palazzo force-pushed the issue-516-add-workflows branch from 223d2b2 to 9f10efc Compare December 23, 2025 11:48
Comment on lines 52 to 55
MODALITY_TO_GDRIVE_LINK = {
'single_table': 'https://docs.google.com/spreadsheets/d/1W3tsGOOtbtTw3g0EVE0irLgY_TN_cy2W4ONiZQ57OPo/edit?usp=sharing',
'multi_table': 'https://docs.google.com/spreadsheets/d/1R13RktVvKnxRecYIge07OBpbX1vbEkE2D1_2idNAKSY/edit?usp=sharing',
}
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I know this won't change probably, but could we use the id from sdgym/run_benchmark/upload_benchmark_results.py if it doesn't cause some circular dependencies ? Or maybe define them in a constants.py file. That way we should just change the id in one file and not have to worry about it.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

if we do this, let's move to constants file

Copy link
Contributor

@amontanez24 amontanez24 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

@R-Palazzo R-Palazzo force-pushed the issue-516-add-workflows branch from 7f02321 to f74bb9a Compare January 5, 2026 16:34
Base automatically changed from issue-515-_benchmark_multi_table_compute_gcp to feature_branch/mutli_table_benchmark January 7, 2026 10:08
@R-Palazzo R-Palazzo force-pushed the issue-516-add-workflows branch from f74bb9a to 5842fcd Compare January 7, 2026 11:02
@R-Palazzo R-Palazzo merged commit 341f476 into feature_branch/mutli_table_benchmark Jan 8, 2026
43 of 44 checks passed
@R-Palazzo R-Palazzo deleted the issue-516-add-workflows branch January 8, 2026 10:11
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants