Skip to content

lib: fix Already borrowed#78

Merged
d4l3k merged 1 commit intomainfrom
d4l3k/already_borrowed
Jan 23, 2025
Merged

lib: fix Already borrowed#78
d4l3k merged 1 commit intomainfrom
d4l3k/already_borrowed

Conversation

@d4l3k
Copy link
Member

@d4l3k d4l3k commented Jan 23, 2025

This fixes errors from Rust when calling the manager from multiple threads.

[rank1]: Traceback (most recent call last):
[rank1]:   File "/home/mreso/torchft/train_fsdp.py", line 176, in <module>
[rank1]:     main()
[rank1]:   File "/home/mreso/torchft/train_fsdp.py", line 169, in main
[rank1]:     optimizer.step()
[rank1]:   File "/home/mreso/torchft/torchft/optim.py", line 53, in step
[rank1]:     if self.manager.should_commit():
[rank1]:   File "/home/mreso/torchft/torchft/manager.py", line 556, in should_commit
[rank1]:     should_commit = self._client.should_commit(
[rank1]: RuntimeError: Already borrowed

We have the Manager receiver type set as &mut self when it really doesn't need to be. We instead do a lightweight clone of the client when making network calls.

See PyO3/pyo3#2465 for more details

Test plan:

pytest
cargo test

@d4l3k d4l3k requested review from H-Huang, fegin and mreso January 23, 2025 00:13
@facebook-github-bot facebook-github-bot added the CLA Signed This label is managed by the Meta Open Source bot. label Jan 23, 2025
Copy link
Contributor

@mreso mreso left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@d4l3k d4l3k merged commit bed29d2 into main Jan 23, 2025
6 checks passed
@d4l3k d4l3k deleted the d4l3k/already_borrowed branch January 23, 2025 00:31
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

CLA Signed This label is managed by the Meta Open Source bot.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants