Skip to content

Add warning about not supporting torch.nn.SyncBatchNorm #5046

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 3 commits into from
Feb 18, 2025
Merged

Add warning about not supporting torch.nn.SyncBatchNorm #5046

merged 3 commits into from
Feb 18, 2025
+3 −0

Conversation

pintaoz-aws
Copy link
Contributor

Issue #, if available:
#2571
Description of changes:
Add warning about not supporting torch.nn.SyncBatchNorm
Testing done:

Merge Checklist

Put an x in the boxes that apply. You can also fill these out after creating the PR. If you're unsure about any of them, don't hesitate to ask. We're here to help! This is simply a reminder of what we are going to look for before merging your pull request.

General

  • I have read the CONTRIBUTING doc
  • I certify that the changes I am introducing will be backward compatible, and I have discussed concerns about this, if any, with the Python SDK team
  • I used the commit message format described in CONTRIBUTING
  • I have passed the region in to all S3 and STS clients that I've initialized as part of this change.
  • I have updated any necessary documentation, including READMEs and API docs (if appropriate)

Tests

  • I have added tests that prove my fix is effective or that my feature works (if appropriate)
  • I have added unit and/or integration tests as appropriate to ensure backward compatibility of the changes
  • I have checked that my tests are not configured for a specific region or account (if appropriate)
  • I have used unique_name_from_base to create resource names in integ tests (if appropriate)
  • If adding any dependency in requirements.txt files, I have spell checked and ensured they exist in PyPi

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

@pintaoz-aws pintaoz-aws requested a review from a team as a code owner February 18, 2025 01:03
nargokul
nargokul reviewed Feb 18, 2025
View reviewed changes
doc/frameworks/pytorch/using_pytorch.rst Outdated
Comment on lines 378 to 379
Warning: ``torch.nn.SyncBatchNorm`` is not supported and its existence in
``init_process_group`` will cause an exception during distributed training.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we make the wording more generic instead of just this use case ?

smdistributed with init_process_group will not allow use of certain other torch features, such as (and likely not limited to) torch.nn.SyncBatchNorm.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

updated

@pintaoz-aws pintaoz-aws requested a review from nargokul February 18, 2025 19:38
@chad119 chad119 merged commit 4f19de5 into aws:master Feb 18, 2025
12 of 14 checks passed
@pintaoz-aws pintaoz-aws mentioned this pull request Feb 18, 2025
Closed
evakravi pushed a commit to evakravi/sagemaker-python-sdk that referenced this pull request Mar 20, 2025
@pintaoz-aws @sagemaker-bot
Add warning about not supporting torch.nn.SyncBatchNorm (aws#5046)
379e8ee 
* Add warning about not supporting

* update wording

---------

Co-authored-by: pintaoz <[email protected]>
pravali96 pushed a commit to pravali96/sagemaker-python-sdk that referenced this pull request Apr 21, 2025
@pintaoz-aws
Add warning about not supporting torch.nn.SyncBatchNorm (aws#5046)
35acb3a 
* Add warning about not supporting

* update wording

---------

Co-authored-by: pintaoz <[email protected]>
uyoldas pushed a commit to uyoldas/sagemaker-python-sdk that referenced this pull request May 23, 2025
@pintaoz-aws
Add warning about not supporting torch.nn.SyncBatchNorm (aws#5046)
29271ac 
* Add warning about not supporting

* update wording

---------

Co-authored-by: pintaoz <[email protected]>
sage-maker added a commit that referenced this pull request Jul 1, 2025
@uyoldas @pintaoz-aws
@sage-maker @timkuo-amazon @zhaoqizqwang @sagemaker-bot
feature: integrate amtviz for visualization of tuning jobs (#5044)
5164316 
* feature: integrate amtviz for visualization of tuning jobs

* Move RecordSerializer and RecordDeserializer to sagemaker.serializers and sagemaker.deserialzers (#5037)

* Move RecordSerializer and RecordDeserializer to sagemaker.serializers and sagemaker.deserializers

* fix codestyle

* fix test

---------

Co-authored-by: pintaoz <[email protected]>

* Add framework_version to all TensorFlowModel examples (#5038)

* Add framework_version to all TensorFlowModel examples

* update framework_version to x.x.x

---------

Co-authored-by: pintaoz <[email protected]>

* Fix hyperparameter strategy docs (#5045)

* fix: pass in inference_ami_version to model_based endpoint type (#5043)

* fix: pass in inference_ami_version to model_based endpoint type

* documentation: update contributing.md w/ venv instructions and pip install fixes

---------

Co-authored-by: Zhaoqi <[email protected]>

* Add warning about not supporting torch.nn.SyncBatchNorm (#5046)

* Add warning about not supporting

* update wording

---------

Co-authored-by: pintaoz <[email protected]>

* prepare release v2.239.2

* update development version to v2.239.3.dev0

* change: update image_uri_configs  02-19-2025 06:18:15 PST

* fix: codestyle, type hints, license, and docstrings

* documentation: add docstring for amtviz module

* fix: fix docstyle and flake8 errors

* fix: code reformat using black

---------

Co-authored-by: Uemit Yoldas <[email protected]>
Co-authored-by: pintaoz-aws <[email protected]>
Co-authored-by: pintaoz <[email protected]>
Co-authored-by: parknate@ <[email protected]>
Co-authored-by: timkuo-amazon <[email protected]>
Co-authored-by: Zhaoqi <[email protected]>
Co-authored-by: ci <ci>
Co-authored-by: sagemaker-bot <[email protected]>
HollowTube pushed a commit to HollowTube/sagemaker-python-sdk that referenced this pull request Jul 15, 2025
@uyoldas @pintaoz-aws
@sage-maker @timkuo-amazon @zhaoqizqwang @sagemaker-bot
feature: integrate amtviz for visualization of tuning jobs (aws#5044)
abf7367 
* feature: integrate amtviz for visualization of tuning jobs

* Move RecordSerializer and RecordDeserializer to sagemaker.serializers and sagemaker.deserialzers (aws#5037)

* Move RecordSerializer and RecordDeserializer to sagemaker.serializers and sagemaker.deserializers

* fix codestyle

* fix test

---------

Co-authored-by: pintaoz <[email protected]>

* Add framework_version to all TensorFlowModel examples (aws#5038)

* Add framework_version to all TensorFlowModel examples

* update framework_version to x.x.x

---------

Co-authored-by: pintaoz <[email protected]>

* Fix hyperparameter strategy docs (aws#5045)

* fix: pass in inference_ami_version to model_based endpoint type (aws#5043)

* fix: pass in inference_ami_version to model_based endpoint type

* documentation: update contributing.md w/ venv instructions and pip install fixes

---------

Co-authored-by: Zhaoqi <[email protected]>

* Add warning about not supporting torch.nn.SyncBatchNorm (aws#5046)

* Add warning about not supporting

* update wording

---------

Co-authored-by: pintaoz <[email protected]>

* prepare release v2.239.2

* update development version to v2.239.3.dev0

* change: update image_uri_configs  02-19-2025 06:18:15 PST

* fix: codestyle, type hints, license, and docstrings

* documentation: add docstring for amtviz module

* fix: fix docstyle and flake8 errors

* fix: code reformat using black

---------

Co-authored-by: Uemit Yoldas <[email protected]>
Co-authored-by: pintaoz-aws <[email protected]>
Co-authored-by: pintaoz <[email protected]>
Co-authored-by: parknate@ <[email protected]>
Co-authored-by: timkuo-amazon <[email protected]>
Co-authored-by: Zhaoqi <[email protected]>
Co-authored-by: ci <ci>
Co-authored-by: sagemaker-bot <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@chad119 chad119 chad119 approved these changes

@akrishna1995 akrishna1995 Awaiting requested review from akrishna1995 akrishna1995 is a code owner automatically assigned from aws/sagemaker-ml-frameworks

@nargokul nargokul Awaiting requested review from nargokul

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@pintaoz-aws @chad119 @nargokul