Support Direct URL editable requirements

This is crucial for the deprecation of non-bare egg URL fragments as they
used to be the sole way to request an extra for a VCS editable install.

I've also removed all references to egg fragments from the user docs and
added further explanation of the Direct URL form in the VCS support
topic.

Getting the error handling right was a pain. That's why I extracted the
two parsing flows into their own functions.

Co-authored-by: Tzu-ping Chung <[email protected]>

Author: ichard26

docs/html/cli/pip_install.rst

@@ -66,8 +66,8 @@ for the name and project version (this is in theory slightly less reliable
 than using the ``egg_info`` command, but avoids downloading and processing
 unnecessary numbers of files).
 
-Any URL may use the ``#egg=name`` syntax (see :doc:`../topics/vcs-support`) to
-explicitly state the project name.
+The :pep:`508` requirement syntax can be used to explicitly state the project
+name (see :doc:`../topics/vcs-support`).
 
 Satisfying Requirements
 -----------------------
@@ -367,21 +367,21 @@ Examples
 
       .. code-block:: shell
 
-         python -m pip install -e 'git+https://git.repo/some_pkg.git#egg=SomePackage'          # from git
-         python -m pip install -e 'hg+https://hg.repo/some_pkg.git#egg=SomePackage'            # from mercurial
-         python -m pip install -e 'svn+svn://svn.repo/some_pkg/trunk/#egg=SomePackage'         # from svn
-         python -m pip install -e 'git+https://git.repo/some_pkg.git@feature#egg=SomePackage'  # from 'feature' branch
-         python -m pip install -e 'git+https://git.repo/some_repo.git#egg=subdir&subdirectory=subdir_path' # install a python package from a repo subdirectory
+         python -m pip install -e 'SomePackage @ git+https://git.repo/some_pkg.git'          # from git
+         python -m pip install -e 'SomePackage @ hg+https://hg.repo/some_pkg.git'            # from mercurial
+         python -m pip install -e 'SomePakcage @ svn+svn://svn.repo/some_pkg/trunk/'         # from svn
+         python -m pip install -e 'SomePackage @ git+https://git.repo/some_pkg.git@feature'  # from 'feature' branch
+         python -m pip install -e 'subdir @ git+https://git.repo/some_repo.git#subdirectory=subdir_path' # install a python package from a repo subdirectory
 
    .. tab:: Windows
 
       .. code-block:: shell
 
-         py -m pip install -e "git+https://git.repo/some_pkg.git#egg=SomePackage"          # from git
-         py -m pip install -e "hg+https://hg.repo/some_pkg.git#egg=SomePackage"            # from mercurial
-         py -m pip install -e "svn+svn://svn.repo/some_pkg/trunk/#egg=SomePackage"         # from svn
-         py -m pip install -e "git+https://git.repo/some_pkg.git@feature#egg=SomePackage"  # from 'feature' branch
-         py -m pip install -e "git+https://git.repo/some_repo.git#egg=subdir&subdirectory=subdir_path" # install a python package from a repo subdirectory
+         py -m pip install -e "SomePackage @ git+https://git.repo/some_pkg.git"          # from git
+         py -m pip install -e "SomePackage @ hg+https://hg.repo/some_pkg.git"            # from mercurial
+         py -m pip install -e "SomePackage @ svn+svn://svn.repo/some_pkg/trunk/"         # from svn
+         py -m pip install -e "SomePackage @ git+https://git.repo/some_pkg.git@feature"  # from 'feature' branch
+         py -m pip install -e "subdir @ git+https://git.repo/some_repo.git#subdirectory=subdir_path" # install a python package from a repo subdirectory
 
 #. Install a package with extras, i.e., optional dependencies
    (:ref:`specification <pypug:dependency-specifiers>`).

docs/html/topics/vcs-support.md

@@ -10,6 +10,25 @@ control system being used). It is used through URL prefixes:
 - Subversion -- `svn+`
 - Bazaar -- `bzr+`
 
+The general form of a VCS requirement is `ProjectName @ VCS_URL`, e.g.
+
+```none
+MyProject @ git+https://git.example.com/MyProject
+MyProject[extra] @ git+https:/git.example.com/MyProject
+```
+
+This is the Direct URL ({pep}`508`) requirement syntax. It is also permissible
+to remove `MyProject @` portion is removed and provide a bare VCS URL.
+
+```none
+git+https://git.example.com/MyProject
+```
+
+This is a pip specific extension. This form can be used as long as pip does
+not need to know the project name in advance. pip is generally able to infer
+the project name except in the case of {ref}`editable-vcs-installs`. In
+addition, extras cannot be requested using a bare VCS URL.
+
 ## Supported VCS
 
 ### Git
@@ -81,8 +100,8 @@ MyProject @ svn+ssh://[email protected]/MyProject
 You can also give specific revisions to an SVN URL, like so:
 
 ```none
--e svn+http://svn.example.com/svn/MyProject/trunk@2019#egg=MyProject
--e svn+http://svn.example.com/svn/MyProject/trunk@{20080101}#egg=MyProject
+-e MyProject @ svn+http://svn.example.com/svn/MyProject/trunk@2019
+-e MyProject @ svn+http://svn.example.com/svn/MyProject/trunk@{20080101}
 ```
 
 Note that you need to use [Editable VCS installs](#editable-vcs-installs) for
@@ -115,6 +134,9 @@ MyProject @ bzr+http://bzr.example.com/MyProject/[email protected]
 VCS projects can be installed in {ref}`editable mode <editable-installs>` (using
 the {ref}`--editable <install_--editable>` option) or not.
 
+In editable mode, the project name must be provided upfront using the Direct URL
+(`MyProject @ URL`) form so pip can determine the VCS clone location.
+
 - The default clone location (for editable installs) is:
 
   - `<venv path>/src/SomeProject` in virtual environments
@@ -133,15 +155,15 @@ take on the VCS requirement (not the commit itself).
 ## URL fragments
 
 pip looks at the `subdirectory` fragments of VCS URLs for specifying the path to the
-Python package, when it is not in the root of the VCS directory. eg: `pkg_dir`.
+Python package, when it is not in the root of the VCS directory.
 
-pip also looks at the `egg` fragment specifying the "project name". In practice the
-`egg` fragment is only required to help pip determine the VCS clone location in editable
-mode. In all other circumstances, the `egg` fragment is not necessary and its use is
-discouraged.
+```{note}
+pip also supports an `egg` fragment to specify the "project name". This is a legacy
+feature and its use is discouraged in favour of the Direct URL ({pep}`508`) form.
 
 The `egg` fragment **should** be a bare {ref}`project name <pypug:name-normalization>`.
 Anything else is not guaranteed to work.
+```
 
 ````{admonition} Example
 If your repository layout is:
@@ -164,6 +186,6 @@ $ pip install "pkg @ vcs+protocol://repo_url/#subdirectory=pkg_dir"
 or:
 
 ```{pip-cli}
-$ pip install -e "vcs+protocol://repo_url/#egg=pkg&subdirectory=pkg_dir"
+$ pip install -e "pkg @ vcs+protocol://repo_url/#subdirectory=pkg_dir"
 ```
 ````

docs/html/user_guide.rst

@@ -183,7 +183,7 @@ In practice, there are 4 common uses of Requirements files:
    ``sometag``.  You'd reference it in your requirements file with a line like
    so::
 
-     git+https://myvcs.com/some_dependency@sometag#egg=SomeDependency
+     SomeDependency @ git+https://myvcs.com/some_dependency@sometag
 
    If ``SomeDependency`` was previously a top-level requirement in your
    requirements file, then **replace** that line with the new line. If

news/13495.feature.rst

@@ -0,0 +1 @@
+Add support installing an editable requirement written as a Direct URL.

src/pip/_internal/req/constructors.py

@@ -86,17 +86,25 @@ def _set_requirement_extras(req: Requirement, new_extras: set[str]) -> Requireme
     return get_requirement(f"{pre}{extras}{post}")
 
 
-def parse_editable(editable_req: str) -> tuple[str | None, str, set[str]]:
-    """Parses an editable requirement into:
-        - a requirement name
-        - an URL
-        - extras
-        - editable options
-    Accepted requirements:
-        svn+http://blahblah@rev#egg=Foobar[baz]&subdirectory=version_subdir
-        .[some_extra]
-    """
+def _parse_direct_url_editable(editable_req: str) -> tuple[str | None, str, set[str]]:
+    try:
+        req = Requirement(editable_req)
+    except InvalidRequirement:
+        pass
+    else:
+        if req.url and "://" in req.url:
+            # Join the marker back into the name part. This will be parsed out
+            # later into a Requirement again.
+            if req.marker:
+                name = f"{req.name} ; {req.marker}"
+            else:
+                name = req.name
+            return (name, req.url, req.extras)
+
+    raise ValueError
 
+
+def _parse_pip_syntax_editable(editable_req: str) -> tuple[str | None, str, set[str]]:
     url = editable_req
 
     # If a file path is specified with extras, strip off the extras.
@@ -122,23 +130,41 @@ def parse_editable(editable_req: str) -> tuple[str | None, str, set[str]]:
             url = f"{version_control}+{url}"
             break
 
+    return Link(url).egg_fragment, url, set()
+
+
+def parse_editable(editable_req: str) -> tuple[str | None, str, set[str]]:
+    """Parses an editable requirement into:
+        - a requirement name
+        - an URL
+        - extras
+    Accepted requirements:
+        - svn+http://blahblah@rev#egg=Foobar[baz]&subdirectory=version_subdir
+        - local_path[some_extra]
+        - Foobar[extra] @ svn+http://blahblah@rev#subdirectory=subdir ; markers
+    """
+    try:
+        package_name, url, extras = _parse_direct_url_editable(editable_req)
+    except ValueError:
+        package_name, url, extras = _parse_pip_syntax_editable(editable_req)
+
     link = Link(url)
 
-    if not link.is_vcs:
+    if not link.is_vcs and not link.url.startswith("file:"):
         backends = ", ".join(vcs.all_schemes)
         raise InstallationError(
             f"{editable_req} is not a valid editable requirement. "
             f"It should either be a path to a local project or a VCS URL "
             f"(beginning with {backends})."
         )
 
-    package_name = link.egg_fragment
-    if not package_name:
+    # The project name can be inferred from local file URIs easily.
+    if not package_name and not link.url.startswith("file:"):
         raise InstallationError(
             f"Could not detect requirement name for '{editable_req}', "
-            "please specify one with #egg=your_package_name"
+            "please specify one with your_package_name @ URL"
         )
-    return package_name, url, set()
+    return package_name, url, extras
 
 
 def check_first_requirement_in_file(filename: str) -> None:

src/pip/_internal/resolution/resolvelib/candidates.py

@@ -85,8 +85,12 @@ def make_install_req_from_editable(
     link: Link, template: InstallRequirement
 ) -> InstallRequirement:
     assert template.editable, "template not editable"
+    if template.name:
+        req_string = f"{template.name} @ {link.url}"
+    else:
+        req_string = link.url
     ireq = install_req_from_editable(
-        link.url,
+        req_string,
         user_supplied=template.user_supplied,
         comes_from=template.comes_from,
         use_pep517=template.use_pep517,

tests/functional/test_install_reqs.py

@@ -10,6 +10,7 @@
     PipTestEnvironment,
     ResolverVariant,
     TestData,
+    _create_test_package,
     _create_test_package_with_subdirectory,
     create_basic_sdist_for_package,
     create_basic_wheel_for_package,
@@ -941,3 +942,52 @@ def test_nonpep517_setuptools_import_failure(script: PipTestEnvironment) -> None
     exc_message = "ImportError: this 'setuptools' was intentionally poisoned"
     assert nice_message in result.stderr
     assert exc_message in result.stderr
+
+
+class TestEditableDirectURL:
+    def test_install_local_project(
+        self, script: PipTestEnvironment, data: TestData, common_wheels: Path
+    ) -> None:
+        uri = (data.src / "simplewheel-2.0").as_uri()
+        script.pip(
+            "install", "--no-index", "-e", f"simplewheel @ {uri}", "-f", common_wheels
+        )
+        script.assert_installed(simplewheel="2.0")
+
+    def test_install_local_project_with_extra(
+        self, script: PipTestEnvironment, data: TestData, common_wheels: Path
+    ) -> None:
+        uri = (data.src / "requires_simple_extra").as_uri()
+        script.pip(
+            "install",
+            "--no-index",
+            "-e",
+            f"requires-simple-extra[extra] @ {uri}",
+            "-f",
+            common_wheels,
+            "-f",
+            data.packages,
+        )
+        script.assert_installed(requires_simple_extra="0.1")
+        script.assert_installed(simple="1.0")
+
+    def test_install_local_git_repo(
+        self, script: PipTestEnvironment, common_wheels: Path
+    ) -> None:
+        repo_path = _create_test_package(script.scratch_path, "simple")
+        url = "git+" + repo_path.as_uri()
+        script.pip(
+            "install", "--no-index", "-e", f"simple @ {url}", "-f", common_wheels
+        )
+        script.assert_installed(simple="0.1")
+
+    @pytest.mark.network
+    def test_install_remote_git_repo_with_extra(
+        self, script: PipTestEnvironment, data: TestData, common_wheels: Path
+    ) -> None:
+        req = "pip-test-package[extra] @ git+https://github.com/pypa/pip-test-package"
+        script.pip(
+            "install", "--no-index", "-e", req, "-f", common_wheels, "-f", data.packages
+        )
+        script.assert_installed(pip_test_package="0.1.1")
+        script.assert_installed(simple="3.0")

tests/unit/test_req.py

@@ -853,6 +853,25 @@ def test_install_req_extend_extras(
         assert extended.permit_editable_wheels == req.permit_editable_wheels
 
 
+@pytest.mark.parametrize(
+    "req_str, expected",
+    [
+        (
+            'foo[extra] @ svn+http://foo ; os_name == "nt"',
+            ('foo ; os_name == "nt"', "svn+http://foo", {"extra"}),
+        ),
+        (
+            "foo @ svn+http://foo",
+            ("foo", "svn+http://foo", set()),
+        ),
+    ],
+)
+def test_parse_editable_pep508(
+    req_str: str, expected: tuple[str, str, set[str]]
+) -> None:
+    assert parse_editable(req_str) == expected
+
+
 @mock.patch("pip._internal.req.req_install.os.path.abspath")
 @mock.patch("pip._internal.req.req_install.os.path.exists")
 @mock.patch("pip._internal.req.req_install.os.path.isdir")