feat: better handling of Python packages and environments

[gnikit]

Is there an existing request for this?

• I have searched the existing issues

I have tried the Pre-Release of the extension.

• Feature is not present in the latest Pre-Release

Feature Request

The extension uses pip in order to install third party tools like fortls or fprettify. The problem is that certain user setups are not properly configured hence the install location for pip is not in the PATH, hence the initialisation of the tools fails. Another issue is that users often rely on virtualisation venv, conda, etc. in order to install these tools, and they would prefer if the extension used the virtual environment over the user's global env.

The majority of these user pains have solutions and workarounds already in place, i.e. one can specify explicitly the location of fortls or findent. Or one can spin up VS Code from inside the venv/conda environment thus the installation infrastructure of Modern Fortran will be aware of the venv variables.

The Issue here is meant to try and ease some of those pains if possible without over-complicating the extension.

There are multiple ways forward to this problem.

1) pipx

Using pipx to isolate and run these binaries would simplify the process. One would only have to detect/install pipx in the corresponding system. There would be a small performance slow-down given that launching fortls or findent with pipx is noticeably slower.

2) Python Environment Manager

Create a Python Manager class inside the Modern Fortran extension that is aware of installation locations, potential virtualisation. Shares the terminal env variables, etc.

The main problem is the complexity of this solution. This is a Fortran extension that would end up implementing a full Python environment manager (clearly out of scope), for user pains that can already be resolved with the existing settings.

3) Ship binaries with Modern Fortran

Use some tool (there are a few out there) to either ship standalone binaries of the python packages or a lightweight Python virtual environment inside the Modern Fortran vsix.

This would raise some security concerns if done poorly, but I am relatively sure there are well-established solutions out there.

4) Integrate with the ms-python extension

If the extension is available, and if this is techinically possible we could integrate with the Python extension for VS Code and potentially avoid writing a Python Environment Manager completely.

Related Issues are #957, #773, #771

[aviralgarg05]

I would like to solve the issue. Please assign me the issue

[gnikit]

I would like to solve the issue. Please assign me the issue

I am happy for the enthusiasm, but there are 4 suggested solutions each with their pros and cons, not all of which are listed in the Issue (security, user-frustation, non-standard builds, etc.). Some solutions could be implemented in parallel for increased robustness in the extension.

Assigning the Issue to you will not do much on its own. How do you plan on tackling this Issue and evaluate all the proposed solutions?

[aviralgarg05]

I would like to solve the issue. Please assign me the issue

I am happy for the enthusiasm, but there are 4 suggested solutions each with their pros and cons, not all of which are listed in the Issue (security, user-frustation, non-standard builds, etc.). Some solutions could be implemented in parallel for increased robustness in the extension.

Assigning the Issue to you will not do much on its own. How do you plan on tackling this Issue and evaluate all the proposed solutions?

I will send a proposal for it

[aviralgarg05]

Hi @gnikit , here is my proposal to move this forward.

1. Evaluation of Options
   Custom Env Manager: Discard. Too much complexity/maintenance for this extension.
   Ship Binaries: Discard. excessive maintenance burden and security risks.
   pipx: Fallback. Good for isolation, but adds a tool dependency.
   ms-python Integration: Primary. Robust and seamless for most users.

2. Proposed Solution: Tiered Resolution
   I propose a Hybrid Approach that degrades gracefully:

Tier 1 (Priority): Use the @vscode/python-extension API.
If the user has the Python extension, we use their selected environment (venv/conda) automatically.
This solves the "venv aware" requirement without extra config.
Tier 2 (Fallback): Native pipx detection or Global PATH.
If no Python extension, check pipx run ... or standard global PATH.
3. Implementation Plan
I will split this into two phases:

Phase 1: Add @vscode/python-extension dependency and implement the logic to resolve the python path from their API.
Phase 2: Add the pipx fallback and improved "tool not found" diagnostics.
I can start on Phase 1 immediately. Does this align with your view?

[gnikit]

1. Evaluation of Options
   Custom Env Manager: Discard. Too much complexity/maintenance for this extension.

Agree, which is why I abandoned the idea too after a few failed PRs, but you'll see the other alternatives are not much better.

Ship Binaries: Discard. excessive maintenance burden and security risks.

This is not that cut and dry. From a security point it's the same. Installing packages with pip and shipping their binaries into the extension are pretty much the same. From a maintenance point, using pex or shiv to generate a signed compiled file that is downloaded is probably not too much work, but you still need a Python interpreter (which is not a big deal).

Other vscode extensions for example the C/C++ do similar stuff were they embed their proprietary binaries during packaging.

We also control all of these binaries fortls, fprettify, and findent (its wheel creation not the source code).

pipx: Fallback. Good for isolation, but adds a tool dependency.

pipx would solve the cases where the user has fortls for example but has not included the user install location to PATH.
You still need to install pipx which varies depending on the OS. The recommended installation for MacOS +14 is usually through brew or some other packaging mechanism. For linux it's usually through the various package managers or pip and on Windows scoop. I've got no idea how windows behaves when multiple python interpreters are in the PATH and pipx is installed via pip.

Aside from this additional Python dependency, which many users won't have, you have to modify the users PATH/ shell config files to ensure pipx binaries can be discovered. A Fortran VS Code extension modifying global state and PATHs is an overreach and some users will rightfully be upset.

ms-python Integration: Primary. Robust and seamless for most users.

This adds an extension dependency to ms-python.python which in turn installs a bunch of other extensions.
I've had users rightfully complaint about the dependency to C/C++ which brings in Debugging among other very useful things.
I expect similar backlash from users "why do I need this if I only code in Fortran". The dependency can be made soft but that would make the testing and user experience brittle. Also hard to diagnose where errors are stemming if a tool's installation fails, since there will be multiple methods for installing.

Moreover, Microsoft is reworking their entire Python environment handling with Python Environments, but the extension is in Preview and has horrible ratings. Microsoft has vouched that this extension will ultimately supersede how virtual envs are handled in Python. I would hate to have to rework this feature in agony when Microsoft makes the switch. Their development pace is simply too fast for single author open-source extensions to match.

That being said I have a branch locally where I implemented a soft-dependency in ms-python.python. Integration testing was an issue, so it's still WIP. I can rebase and push if you want to have a look.

2. Proposed Solution: Tiered Resolution
   I propose a Hybrid Approach that degrades gracefully:

Tier 1 (Priority): Use the @vscode/python-extension API. If the user has the Python extension, we use their selected environment (venv/conda) automatically. This solves the "venv aware" requirement without extra config. Tier 2 (Fallback): Native pipx detection or Global PATH. If no Python extension, check pipx run ... or standard global PATH. 3. Implementation

So in summary, I would say the extension should try ms-python.python if present and then just revert to the existing mechanism. The ideal scenario for me would be completely independent of Python package managers but that's a lot more work.

I'll see if I can locate my branch for ms-python.python

[aviralgarg05]

I see you've just opened PR #1290, Instead of a separate branch, I can contribute directly to this PR (or a branch off it) to:

Implement the Execution Logic:

• Update fortls / fprettify spawning to use the interpreter found by getPythonExecutable.
• Add Event Watchers: Hook into the Python extension's environments.onDidChange event to trigger re-checks.
• Add the Fallback: Implement the Tier 2 fallback (check pipx or global PATH) if the Python extension API returns nothing.

Would you prefer I push to your branch or open a PR against yours?