Editable dependencies link/unlink - correct methodology to work

[liran-cohen-hs]

Details:
I will start by describing the scenario we are trying to solve
We are working extensively on Python with multiple projects across the company and numerous internal libraries
when a developer wants to work on a service (let's call it A) let's say he clones it from github and runs pdm sync, now I expect the service dependencies to be installed as they are in production - specifically internal libraries should be retrieved with the relevant github tag
Up until here, everything is covered and works as expected

The problem:
The problem starts when the same developer wants to change something in an internal library, let's say service A is dependent on library P, now I want to keep short loops, the developer has P cloned on his working station and once he changes it I want it to affect service A (locally) immediately - or in simple words - I want it to be an editable dependency
The thing is - I couldn't get a way to make it work with any configuration, I tried multiple lock files but it doesn't fully answer it or add a lot of overhead

Wanted behavior:
If I try to imagine the dream scenario I would say that in pyproject.toml I should have dependency P, described maybe in 2 locations - one is github/pypi repository and the other one is a local relative path and by a simple command, transition from one to another
I looked in other tools for the same functionality and I guess that yarn link does something similar

[frostming]

It's doable with PDM.

pyproject.toml of project A:

[project]
# other fields
dependencies = [
  "P"  # from pypi
]

[tool.pdm.dev-dependencies]
dev = [
  "-e file:///${PROJECT_ROOT}/../P"  # editable P from P directory next to project A
]

And you create two lockfiles:

pdm lock --prod
pdm lock --dev --lockfile pdm.dev.lock

The default pdm.lock created by the first command contains the pinned version of P from PyPI, and the pdm.dev.lock created by the second command contains a path to the editable local package. In subsequent install or sync commands remember to pass --lockfile pdm.dev.lock to use the second lockfile.

Documentation: https://pdm.fming.dev/latest/usage/dependency/#specify-the-lockfile-to-use

[liran-cohen-hs]

on a clean pdm project it fails:

pdm add git+ssh://[email protected]/HiredScore/P.git#egg=P
Adding group default to lockfile
Adding packages to default dependencies: config-resolver @ git+ssh://[email protected]/P.git
🔒 Lock successful
Changes are written to pyproject.toml.
Synchronizing working set with lock file: 1 to add, 0 to update, 0 to remove

  ✔ Install config-resolver 0.0.1 successful

🎉 All complete!



 pdm add -e ../../../P --dev -L pdm.dev.lock
Adding group dev to lockfile
Adding packages to dev dev-dependencies: -e file:///${PROJECT_ROOT}/../../../P#egg=P
🔒 Lock successful
Changes are written to pyproject.toml.
Synchronizing working set with lock file: 0 to add, 1 to update, 0 to remove

  ✔ Update config-resolver 0.0.1 -> 0.0.1 successful

🎉 All complete!



 pdm lock --prod
🔒 Lock successful
Changes are written to pdm.lock.



pdm lock --dev -L pdm.dev.lock
🔒 Lock failed
Unable to find a resolution for config-resolver
because of the following conflicts:
  -e file:///${PROJECT_ROOT}/../../../P#egg=P (from project)
  P @ git+ssh://[email protected]/HiredScore/P.git (from project)
To fix this, you could loosen the dependency version constraints in pyproject.toml. See https://pdm.fming.dev/latest/usage/dependency/#solve-the-locking-failure for more details.
See /var/folders/nr/92733x4j3d54n5802mg7094h0000gp/T/pdm-lock-bmmtx7k_.log for detailed debug log.
[ResolutionImpossible]: Unable to find a resolution
Add '-v' to see the detailed traceback



pdm -V                        
PDM, version 2.7.4

[frostming]

In the second command, exclude the default group to avoid conflict:

pdm lock --no-default --dev -L pdm.dev.lock

If there are common dependencies in the default group that need to be installed for both, move the config-resolver dependency to its own group so it can be picked and unpicked by its own.

[liran-cohen-hs]

I understand it better now :)
thanks, will implement it in our projects
@frostming another question in the same topic:
I've setup a pyproject.toml as discussed, now for example I run the following:

pdm sync  # installs all dependencies, including the group `inner`
pdm sync -L pdm.dev.lock  # replaces inner dependency with the one from dev which is editable
pdm sync  # Does nothing, I expected it to return the inner dependency to be from github rather than locally + editable

two solutions I found:
pdm sync --no-editable
pdm sync -r

the latter can be costly, the first one is a nice option for a developer to get back to the service "regular"/"prod" state, my question is- is it a bug that pdm sync by itself does not identify the change and return the dependency to it's state as it described in the "inner" dependency group?

Thanks !

[frostming]

is it a bug that pdm sync by itself does not identify the change and return the dependency to it's state as it described in the "inner" dependency group?

It's intentional, editable dependencies are skipped for comparison unless --no-editable is passed. This policy is more convenient back when the selected lock groups are not implemented.