Add workflow to run SDGym multi-table benchmark monthly and publish results#518
Conversation
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## feature_branch/mutli_table_benchmark #518 +/- ##
========================================================================
+ Coverage 78.78% 78.87% +0.08%
========================================================================
Files 33 33
Lines 2791 2826 +35
========================================================================
+ Hits 2199 2229 +30
- Misses 592 597 +5
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
223d2b2 to
9f10efc
Compare
| MODALITY_TO_GDRIVE_LINK = { | ||
| 'single_table': 'https://docs.google.com/spreadsheets/d/1W3tsGOOtbtTw3g0EVE0irLgY_TN_cy2W4ONiZQ57OPo/edit?usp=sharing', | ||
| 'multi_table': 'https://docs.google.com/spreadsheets/d/1R13RktVvKnxRecYIge07OBpbX1vbEkE2D1_2idNAKSY/edit?usp=sharing', | ||
| } |
There was a problem hiding this comment.
I know this won't change probably, but could we use the id from sdgym/run_benchmark/upload_benchmark_results.py if it doesn't cause some circular dependencies ? Or maybe define them in a constants.py file. That way we should just change the id in one file and not have to worry about it.
There was a problem hiding this comment.
if we do this, let's move to constants file
7f02321 to
f74bb9a
Compare
f74bb9a to
5842fcd
Compare
341f476
into
feature_branch/mutli_table_benchmark
Resolve #516
CU-86b7w17ze
@amontanez24 From what I've experienced, we can currently run 4-gpu machines at the same location:
The initial plan is to have 6 instances running on January 1st (4 for single-table, 2 for multi-table), so I wanted to discuss the options with you:
Maybe there is a better solution that I'm happy to discuss.