Python version inconsistencies in Forge dependencies

[joecummings]

Problem

With this PR (pytorch/torchtitan#1778), TorchTitan upgraded their Python version to be only compatible with Python >= 3.12. It would appear this was an attempt to solve an issue in CI. We are now unable to install TorchTitan on Python < 3.12; we need TorchTitan for our model implementations / parallelisms.

Option 1: Upgrade everything to Python 3.12

Pros:

• More modern version of Python
• Less versions we have to test in CI / make guarantees about working

Cons:

• Diverges with PyTorch's schedule of supported Python versions (they are set to drop 3.9 officially in 2.9 on Oct 15)
• Forces all of our other dependencies to be upgraded to Python 3.12 (Monarch and TorchStore are mostly tested on Python 3.10), this could take a while to ensure these other dependencies work well

Option 2: Allow Python 3.10 in Monarch

Pros:

• Operate as we did before, no additional changes needed
• Titan stays in Python version schedule with PyTorch core

Cons:

• CI might fail in Titan - do not fully understand the problem there and could use input from their team

---

I have a slight preference for Option 2, but very open to hearing thoughts from everyone.

cc @tianyu-l @wwwjn @allenwang28 @ebsmothers

[allenwang28]

wondering what the CI issue is in Titan? We can probably try an upgrade on our end - with @ebsmothers PR #288 landing soon it will make it easier for us to do so

but my fear is that we're really close to PTC and this is yet another thing we will have to test

[wwwjn]

CI might fail in Titan - do not fully understand the problem there and could use input from their team

We found torchtitan CI failed on torch.compile related jobs. Per @ezyang @xmfan , some PR in PyTorch has been reverted to fix the torch.compile failure, but also upgrade python version solves the issue as well. Notably, it's a python 3.11 specific issue.

Monarch and TorchStore are mostly tested on Python 3.10), this could take a while to ensure these other dependencies work well

If we are lucky enough to get Monarch and TorchStore works with 3.12, it would be a happy ending. But if not, IIUC torchtitan can switch back to 3.11 because the torch.compile PR has been reverted @tianyu-l

[joecummings]

If we are lucky enough to get Monarch and TorchStore works with 3.12, it would be a happy ending. But if not, IIUC torchtitan can switch back to 3.11 because the torch.compile PR has been reverted @tianyu-l

I completely agree that this is the ideal end state and we will work towards it ASAP! In the meantime; however, it would make our lives much less stressful to be able to work with Titan on Python 3.10.

[joecummings]

Resolved w/ Option 2 through pytorch/torchtitan#1787