Skip to content

Call spec.loader.exec_module in _load_pybind11_module utility #835

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 1 commit into from
Aug 22, 2025
Merged

Call spec.loader.exec_module in _load_pybind11_module utility #835

merged 1 commit into from
Aug 22, 2025

Conversation

@traversaro
Copy link
Collaborator

@traversaro traversaro commented Aug 22, 2025

I am not 100% sure about this change, but without this change the torchcodec build of conda-forge fails after the upgrade from pybind11 2.* to pybind11 3.* during tests with error like:

 │ =========================== short test summary info ============================
 │ FAILED test/test_decoders.py::TestDecoder::test_create[file_like_rawio-VideoDecoder-asset0] - AttributeError: module 'core_pybind_ops' has no attribute 'create_from_file_like'
 │ FAILED test/test_decoders.py::TestDecoder::test_create[file_like_rawio-AudioDecoder-asset1] - AttributeError: module 'core_pybind_ops' has no attribute 'create_from_file_like'
 │ FAILED test/test_decoders.py::TestDecoder::test_create[file_like_rawio-AudioDecoder-asset2] - AttributeError: module 'core_pybind_ops' has no attribute 'create_from_file_like'
 │ FAILED test/test_decoders.py::TestDecoder::test_create[file_like_bufferedio-VideoDecoder-asset0] - AttributeError: module 'core_pybind_ops' has no attribute 'create_from_file_like'
 │ FAILED test/test_decoders.py::TestDecoder::test_create[file_like_bufferedio-AudioDecoder-asset1] - AttributeError: module 'core_pybind_ops' has no attribute 'create_from_file_like'
 │ FAILED test/test_decoders.py::TestDecoder::test_create[file_like_bufferedio-AudioDecoder-asset2] - AttributeError: module 'core_pybind_ops' has no attribute 'create_from_file_like'
 │ FAILED test/test_decoders.py::TestDecoder::test_create[file_like_custom-VideoDecoder-asset0] - AttributeError: module 'core_pybind_ops' has no attribute 'create_from_file_like'
 │ FAILED test/test_decoders.py::TestDecoder::test_create[file_like_custom-AudioDecoder-asset1] - AttributeError: module 'core_pybind_ops' has no attribute 'create_from_file_like'
 │ FAILED test/test_decoders.py::TestDecoder::test_create[file_like_custom-AudioDecoder-asset2] - AttributeError: module 'core_pybind_ops' has no attribute 'create_from_file_like'
 │ FAILED test/test_ops.py::TestVideoDecoderOps::test_create_decoder[file_like_rawio-cpu] - AttributeError: module 'core_pybind_ops' has no attribute 'create_from_file_like'
 │ FAILED test/test_ops.py::TestVideoDecoderOps::test_create_decoder[file_like_bufferedio-cpu] - AttributeError: module 'core_pybind_ops' has no attribute 'create_from_file_like'
 │ FAILED test/test_ops.py::TestAudioDecoderOps::test_file_like_decoding[cpu-0] - AttributeError: module 'core_pybind_ops' has no attribute 'create_from_file_like'
 │ FAILED test/test_ops.py::TestAudioDecoderOps::test_file_like_decoding[cpu-1024] - AttributeError: module 'core_pybind_ops' has no attribute 'create_from_file_like'
 │ FAILED test/test_ops.py::TestAudioDecoderOps::test_file_like_method_check_fails - AttributeError: module 'core_pybind_ops' has no attribute 'create_from_file_like'
 │ FAILED test/test_ops.py::TestAudioDecoderOps::test_file_like_read_fails - AttributeError: module 'core_pybind_ops' has no attribute 'create_from_file_like'
 │ FAILED test/test_ops.py::TestAudioDecoderOps::test_file_like_read_less_than_requested[half] - AttributeError: module 'core_pybind_ops' has no attribute 'create_from_file_like'
 │ FAILED test/test_ops.py::TestAudioDecoderOps::test_file_like_read_less_than_requested[minus_10] - AttributeError: module 'core_pybind_ops' has no attribute 'create_from_file_like'
 │ ============ 17 failed, 702 passed, 98 skipped in 109.81s (0:01:49) ============

to be honest, I do not know enough about Python module loading to understand why this happens in conda-forge CI, and not in torchcodec CI that also uses pybind11 3.* . However, the problem goes away if after loading the module, we also execute spec.loader.exec_module . Looking around it seems that this may be related to multi-phase init (for example pybind/pybind11#5574), but to be honest I do not know enough about that to understand if that is the case.

See conda-forge/torchcodec-feedstock#27 for the downstream related PR.

@meta-cla meta-cla bot added the CLA Signed This label is managed by the Meta Open Source bot. label Aug 22, 2025
@traversaro
Copy link
Collaborator Author

traversaro commented Aug 22, 2025

Sorry for the noise in #833 and #834, I did not realized that GitHub silently switched to let me created branches in torchcodec itself.

@traversaro traversaro mentioned this pull request Aug 22, 2025
Merged
5 tasks
@traversaro traversaro marked this pull request as draft August 22, 2025 10:23
@traversaro
Copy link
Collaborator Author

traversaro commented Aug 22, 2025

There are some test failures, but they do not seem related to the PR (see https://github.com/pytorch/torchcodec/actions/runs/17136952598/job/48617188664 for another PR with similar failures).

@traversaro traversaro marked this pull request as ready for review August 22, 2025 11:22
NicolasHug
NicolasHug approved these changes Aug 22, 2025
View reviewed changes
Copy link
Contributor

@NicolasHug NicolasHug left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the PR @traversaro , this LGTM. I... don't understand why this may or may not be needed either, but if it fixes something on conda-forge then sure.

Waiting on @scotts to take a look

scotts
scotts approved these changes Aug 22, 2025
View reviewed changes
Copy link
Contributor

@scotts scotts left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@traversaro, great catch! It looks like we definitely need to do this.

@scotts scotts merged commit b141814 into meta-pytorch:main Aug 22, 2025
60 of 68 checks passed
@traversaro traversaro deleted the fixpybind113loadconda branch August 22, 2025 15:46
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@scotts scotts scotts approved these changes

@NicolasHug NicolasHug NicolasHug approved these changes

Assignees

No one assigned

Labels

CLA Signed This label is managed by the Meta Open Source bot.

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@traversaro @scotts @NicolasHug