Skip to content

Use linux.4xlarge.memory instead of linux.12xlarge #6896

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 11 commits into from
Nov 26, 2024
Merged

Use linux.4xlarge.memory instead of linux.12xlarge #6896

Changes from 6 commits
Commits
Show all changes
11 commits
Select commit Hold shift + click to select a range
3d3c8a6
Try linux.4xlarge.memory
huydhn Nov 15, 2024
95a912e
Testing
huydhn Nov 15, 2024
6cc16d9
Ready to land
huydhn Nov 18, 2024
71ff610
Merge branch 'main' into try-r5-instances
huydhn Nov 18, 2024
a41f4b4
Merge branch 'main' into try-r5-instances
huydhn Nov 26, 2024
2d84696
More tests
huydhn Nov 26, 2024
245b1fe
Forget one line change
huydhn Nov 26, 2024
34ee8d6
Use linux.8xlarge.memory for llama3_2_text_decoder
huydhn Nov 26, 2024
1df29d1
Increase timeout value
huydhn Nov 26, 2024
038265e
Skip llama3_2_text_decoder because it takes too long to export
huydhn Nov 26, 2024
8204737
Ready to land
huydhn Nov 26, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
20 changes: 10 additions & 10 deletions .ci/scripts/gather_test_models.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -20,16 +20,16 @@
CUSTOM_RUNNERS = {
"linux": {
# This one runs OOM on smaller runner, the root cause is unclear (T163016365)
"w2l": "linux.12xlarge",
"ic4": "linux.12xlarge",
"resnet50": "linux.12xlarge",
"llava": "linux.12xlarge",
"llama3_2_vision_encoder": "linux.12xlarge",
# "llama3_2_text_decoder": "linux.12xlarge", # TODO: re-enable test when Huy's change is in / model gets smaller.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you attach the job link to this model since we re-enable it ?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good catch, I think I will comment it out and leave it for latter. It doesn't OOM but take forever to export (close to 6 hours so far). I don't have much context, so probably need help from @dvorjackz to figure this one out

"w2l": "linux.4xlarge.memory",
"ic4": "linux.4xlarge.memory",
"resnet50": "linux.4xlarge.memory",
"llava": "linux.4xlarge.memory",
"llama3_2_vision_encoder": "linux.4xlarge.memory",
"llama3_2_text_decoder": "linux.4xlarge.memory",
# This one causes timeout on smaller runner, the root cause is unclear (T161064121)
"dl3": "linux.12xlarge",
"emformer_join": "linux.12xlarge",
"emformer_predict": "linux.12xlarge",
"dl3": "linux.4xlarge.memory",
"emformer_join": "linux.4xlarge.memory",
"emformer_predict": "linux.4xlarge.memory",
}
}

Expand Down Expand Up @@ -88,7 +88,7 @@ def model_should_run_on_event(model: str, event: str) -> bool:
We put higher priority and fast models to pull request and rest to push.
"""
if event == "pull_request":
return model in ["mv3", "vit"]
return model in ["mv3", "vit", "llama3_2_text_decoder"]
elif event == "push":
# These are super slow. Only run it periodically
return model not in ["dl3", "edsr", "emformer_predict"]
Expand Down
Loading