Provide an option to turn off the rebuild-on-import feature of editable installs.

[inklesspen]

I understand the rationale for this feature ("meson-python extends editable installs to package components that require a compilation step such as extension modules"), but at the moment it is proving inconvenient for me.

Assuming I'm willing to take responsibility for rebuilding/reinstalling the package when compilation is required, is there some way I can disable the rebuild-on-import feature for my package, but still use Python's editable installs otherwise?

[rgommers]

Thanks for suggesting this @inklesspen. I think it's time to implement this option indeed. It has come up several times now. We should also be able to avoid failure in the case build isolation is used - that happens both for users who don't read the docs, and when tools like uv and pixi implement generic support for editable installs across build backends with a default to build isolation. xref astral-sh/uv#10214

I think the current behavior is more powerful and hence a nice default, but it doesn't work smoothly in all cases, so providing an opt-out is reasonable. At the moment meson-python and scikit-build-core to my knowledge are the only build backends which implement editable behavior also for compiled extension modules, and scikit-build-core defaults to no rebuilding (see the editable.rebuild setting in https://scikit-build-core.readthedocs.io/en/latest/#configuration).

We should figure out what form the option should take exactly; probably similar to how we treat verbose mode.

[virtuald]

I don't really like this feature at all, it just feels too magical and I'd rather be in control of compilation (particularly for large projects, even with ninja's excellent incremental build) -- I'd vote for it being off by default, but I suspect that it's too late for that here. I didn't implement it at all in hatch-meson.

[dnicolodi]

@virtuald The whole point of editable installs is to have edits to the source tree to be immediately reflected into the build tree. If you do not like this feature, why do you use editable installs at all? If you want to control the rebuild explicitly, just use a regular installation (using an explicit build directory to enable incremental builds).

[dnicolodi]

I understand the rationale for this feature ("meson-python extends editable installs to package components that require a compilation step such as extension modules"), but at the moment it is proving inconvenient for me.

In which way is this feature being inconvenient?

Assuming I'm willing to take responsibility for rebuilding/reinstalling the package when compilation is required, is there some way I can disable the rebuild-on-import feature for my package, but still use Python's editable installs otherwise?

Same comment here. The whole point of editable installs is to have edits to the source tree be immediately effective. If you want to explicitly control rebuild and reinstall, just use a regular wheel installation (using an explicit build directory at installation time, to enable incremental builds).

[dnicolodi]

It exist a way to disable automatic rebuilds of editable install (and anyone reading the source code can easily find it) but I am hesitant to document it because so far it has been an implementation detail and it does not have very nice ergonomics and I am quite sure it does not do what users expect: if build dependencies are not installed in the environment or build isolation is enabled, any attempt at rebuilding the package will fail, forcing the user to do a re-installation from source. For large packages this is really inconvenient.

I think the issues with uv and pixi handling of editable installs when building project environments are design issues at the intersection of the concepts of editable wheels and build isolation that cannot be solved disabling editable installations automatic rebuilds, not in a satisfactory way, at least. IMO editable wheels do not make sense in combination with build isolation.

[rgommers]

IMO editable wheels do not make sense in combination with build isolation.

I agree with you there, but unfortunately that ship has sailed. Python packaging design / PEP 660 does not take compiled code into account properly, but that is what it is.

Our default is the right one I think, since it's strictly more powerful and doesn't provide an artificial split between pure Python files and everything else, but it will never play well with build isolation. I think at this point we have a choice between providing users who want to work on pure Python files in their package a choice an extra opt-in mode that addresses their particular use case and plays well with uv et al. (at least to the same extent as setuptools and scikit-build-core do), or not.

are design issues at the intersection of the concepts of editable wheels and build isolation that cannot be solved disabling editable installations automatic rebuilds, not in a satisfactory way, at least

Isn't that in the eye of the beholder? If you're used to the limitations coming from setuptools, it may very well be satisfactory.

[dnicolodi]

are design issues at the intersection of the concepts of editable wheels and build isolation that cannot be solved disabling editable installations automatic rebuilds, not in a satisfactory way, at least

Isn't that in the eye of the beholder? If you're used to the limitations coming from setuptools, it may very well be satisfactory.

Maybe. What would not work is something like this:

> git clone https://github.com/numpy/numpy.git
> cd numpy
> uv venv
> . .venv/bin/activate
> up pip install -e . -Ceditable-rebuild=false  # or whatever the option will look like
> ninja -C build/py313

The last command would be the natural way to rebuild the package incrementally, and it would fail because the build dependencies are not installed. Also the -Crebuild=false argument would not be added when initializing a uv or pixi project, and I am not sure whether there is a way to add it in the project definition. Rebuilding the compiled part of the project would require a re-installation:

> up pip install -e . -Ceditable-rebuild=false

but I have no idea how you would trigger this from a uv or pixi project (I haven't used either tool project management features much).

[rgommers]

The last command would be the natural way to rebuild the package incrementally,

You just can't do it like that, I agree. Very few people would do it like that though. Like with setuptools, the rule will be that if you touch compiled code, you need to re-run the pip install -e command.

[inklesspen]

I understand the rationale for this feature ("meson-python extends editable installs to package components that require a compilation step such as extension modules"), but at the moment it is proving inconvenient for me.

In which way is this feature being inconvenient?

It doesn't play nicely with uv sync, as rgommers noted. Of course, uv sync is merely a nice thing to have (if you like how it works), not a necessity. But I was never a huge fan of a Python import triggering a ninja build step anyway; it seemed against the Zen of Python.

[dnicolodi]

It doesn't play nicely with uv sync, as rgommers noted. Of course, uv sync is merely a nice thing to have (if you like how it works), not a necessity

uv sync applies the version specifications from a lock file to the current environment. What does have it to do with editable installs?

But I was never a huge fan of a Python import triggering a ninja build step anyway; it seemed against the Zen of Python.

Which part exactly? I don't think having Python source edits being reflected immediately in the execution environment but edit to source files needing a compilation step requiring an explicit compilation is in any way less surprising that having all edits being immediately reflected in the execution environment.

[inklesspen]

Well, if you have a uv workspace configured, and one of the non-root packages in the workspace uses meson-python, uv will automatically build and install the package, but doesn't allow turning off build isolation for that package.

I'll put together an example repo for you to play with.

In any event, I'm asking for an override for the default behavior; I'm not asking for the default behavior to be changed.

[dnicolodi]

Thanks for the clarification. However, I still don't understand how you envision this to work.

How are you teaching uv to disable automatic rebuilds for the interested packages? The ways in which I can imagine this to be done is either through a build config setting or via an environment variable. How do you teach uv to pass a build config setting or to set an environment variable?

Assuming you turn off automatic rebuilds of the non-root packages, how do you trigger a rebuild of these packages when the compiled parts change?

If the solution requires adding some facility in uv to play nicely with this, maybe the effort can be redirected into adding to uv the option to turn off build isolation for specific packages.

[inklesspen]

I would expect it to be a setting in the pyproject.toml, the way meson-python's other settings are. This setting would change how Meson installs things when doing an editable install.

[dnicolodi]

meson-python has project settings stored in pyproject.toml and ephemeral build config settings. Build config settings are build specific and are passed by the user via the the build front-end. These most commonly take the form of command line arguments specified to the build front-end. Some settings can be specified as project settings, but can be overridden as build settings. Others, like the build directory and the verbosity of editable builds are build config settings only as they are relevant for a specific deployment only. Something like the editable-rebuild setting proposed here is definitely in the second category because not all users deploying the package are expected to do the same choice regarding this.

Summarizing: if we add a way to disable the automatic rebuild of editable installs, it is not going to be specified in pyproject.toml.

Anyhow, setting this property solved only half of the problem: once you have a package installed in editable mode with automatic rebuilds disabled, how do you trigger a rebuild when needed?