Support Direct URL editable requirements

This is crucial for the deprecation of non-bare egg URL fragments as they
used to be the sole way to request an extra for a VCS editable install.

I've also removed all references to egg fragments from the user docs and
added further explanation of the Direct URL form in the VCS support
topic.

Author: ichard26

docs/html/cli/pip_install.rst

@@ -66,8 +66,8 @@ for the name and project version (this is in theory slightly less reliable
 than using the ``egg_info`` command, but avoids downloading and processing
 unnecessary numbers of files).
 
-Any URL may use the ``#egg=name`` syntax (see :doc:`../topics/vcs-support`) to
-explicitly state the project name.
+The :pep:`508` requirement syntax can be used to explicitly state the project
+name (see :doc:`../topics/vcs-support`).
 
 Satisfying Requirements
 -----------------------
@@ -367,21 +367,21 @@ Examples
 
       .. code-block:: shell
 
-         python -m pip install -e 'git+https://git.repo/some_pkg.git#egg=SomePackage'          # from git
-         python -m pip install -e 'hg+https://hg.repo/some_pkg.git#egg=SomePackage'            # from mercurial
-         python -m pip install -e 'svn+svn://svn.repo/some_pkg/trunk/#egg=SomePackage'         # from svn
-         python -m pip install -e 'git+https://git.repo/some_pkg.git@feature#egg=SomePackage'  # from 'feature' branch
-         python -m pip install -e 'git+https://git.repo/some_repo.git#egg=subdir&subdirectory=subdir_path' # install a python package from a repo subdirectory
+         python -m pip install -e 'SomePackage @ git+https://git.repo/some_pkg.git'          # from git
+         python -m pip install -e 'SomePackage @ hg+https://hg.repo/some_pkg.git'            # from mercurial
+         python -m pip install -e 'SomePakcage @ svn+svn://svn.repo/some_pkg/trunk/'         # from svn
+         python -m pip install -e 'SomePackage @ git+https://git.repo/some_pkg.git@feature'  # from 'feature' branch
+         python -m pip install -e 'subdir @ git+https://git.repo/some_repo.git#subdirectory=subdir_path' # install a python package from a repo subdirectory
 
    .. tab:: Windows
 
       .. code-block:: shell
 
-         py -m pip install -e "git+https://git.repo/some_pkg.git#egg=SomePackage"          # from git
-         py -m pip install -e "hg+https://hg.repo/some_pkg.git#egg=SomePackage"            # from mercurial
-         py -m pip install -e "svn+svn://svn.repo/some_pkg/trunk/#egg=SomePackage"         # from svn
-         py -m pip install -e "git+https://git.repo/some_pkg.git@feature#egg=SomePackage"  # from 'feature' branch
-         py -m pip install -e "git+https://git.repo/some_repo.git#egg=subdir&subdirectory=subdir_path" # install a python package from a repo subdirectory
+         py -m pip install -e "SomePackage @ git+https://git.repo/some_pkg.git"          # from git
+         py -m pip install -e "SomePackage @ hg+https://hg.repo/some_pkg.git"            # from mercurial
+         py -m pip install -e "SomePackage @ svn+svn://svn.repo/some_pkg/trunk/"         # from svn
+         py -m pip install -e "SomePackage @ git+https://git.repo/some_pkg.git@feature"  # from 'feature' branch
+         py -m pip install -e "subdir @ git+https://git.repo/some_repo.git#subdirectory=subdir_path" # install a python package from a repo subdirectory
 
 #. Install a package with extras, i.e., optional dependencies
    (:ref:`specification <pypug:dependency-specifiers>`).

docs/html/topics/vcs-support.md

@@ -10,6 +10,25 @@ control system being used). It is used through URL prefixes:
 - Subversion -- `svn+`
 - Bazaar -- `bzr+`
 
+The general form of a VCS requirement is `ProjectName @ VCS_URL`, e.g.
+
+```none
+MyProject @ git+https://git.example.com/MyProject
+MyProject[extra] @ git+https:/git.example.com/MyProject
+```
+
+This is the Direct URL ({pep}`508`) requirement syntax. It is also permissible
+to remove `MyProject @` portion is removed and provide a bare VCS URL.
+
+```none
+git+https://git.example.com/MyProject
+```
+
+This is a pip specific extension. This form can be used as long as pip does
+not need to know the project name in advance. pip is generally able to infer
+the project name except in the case of {ref}`editable-vcs-installs`. In
+addition, extras cannot be requested using a bare VCS URL.
+
 ## Supported VCS
 
 ### Git
@@ -81,8 +100,8 @@ MyProject @ svn+ssh://[email protected]/MyProject
 You can also give specific revisions to an SVN URL, like so:
 
 ```none
--e svn+http://svn.example.com/svn/MyProject/trunk@2019#egg=MyProject
--e svn+http://svn.example.com/svn/MyProject/trunk@{20080101}#egg=MyProject
+-e MyProject @ svn+http://svn.example.com/svn/MyProject/trunk@2019
+-e MyProject @ svn+http://svn.example.com/svn/MyProject/trunk@{20080101}
 ```
 
 Note that you need to use [Editable VCS installs](#editable-vcs-installs) for
@@ -115,6 +134,9 @@ MyProject @ bzr+http://bzr.example.com/MyProject/[email protected]
 VCS projects can be installed in {ref}`editable mode <editable-installs>` (using
 the {ref}`--editable <install_--editable>` option) or not.
 
+In editable mode, the project name must be provided upfront using the Direct URL
+(`MyProject @ URL`) form so pip can determine the VCS clone location.
+
 - The default clone location (for editable installs) is:
 
   - `<venv path>/src/SomeProject` in virtual environments
@@ -133,15 +155,15 @@ take on the VCS requirement (not the commit itself).
 ## URL fragments
 
 pip looks at the `subdirectory` fragments of VCS URLs for specifying the path to the
-Python package, when it is not in the root of the VCS directory. eg: `pkg_dir`.
+Python package, when it is not in the root of the VCS directory.
 
-pip also looks at the `egg` fragment specifying the "project name". In practice the
-`egg` fragment is only required to help pip determine the VCS clone location in editable
-mode. In all other circumstances, the `egg` fragment is not necessary and its use is
-discouraged.
+```{note}
+pip also supports an `egg` fragment to specify the "project name". This is a legacy
+feature and its use is discouraged in favour of the Direct URL ({pep}`508`) form.
 
 The `egg` fragment **should** be a bare {ref}`project name <pypug:name-normalization>`.
 Anything else is not guaranteed to work.
+```
 
 ````{admonition} Example
 If your repository layout is:
@@ -164,6 +186,6 @@ $ pip install "pkg @ vcs+protocol://repo_url/#subdirectory=pkg_dir"
 or:
 
 ```{pip-cli}
-$ pip install -e "vcs+protocol://repo_url/#egg=pkg&subdirectory=pkg_dir"
+$ pip install -e "pkg @ vcs+protocol://repo_url/#subdirectory=pkg_dir"
 ```
 ````

docs/html/user_guide.rst

@@ -183,7 +183,7 @@ In practice, there are 4 common uses of Requirements files:
    ``sometag``.  You'd reference it in your requirements file with a line like
    so::
 
-     git+https://myvcs.com/some_dependency@sometag#egg=SomeDependency
+     SomeDependency @ git+https://myvcs.com/some_dependency@sometag
 
    If ``SomeDependency`` was previously a top-level requirement in your
    requirements file, then **replace** that line with the new line. If

news/13495.feature.rst

@@ -0,0 +1 @@
+Add support installing an editable requirement written as a Direct URL.

src/pip/_internal/req/constructors.py

@@ -86,17 +86,25 @@ def _set_requirement_extras(req: Requirement, new_extras: set[str]) -> Requireme
     return get_requirement(f"{pre}{extras}{post}")
 
 
-def parse_editable(editable_req: str) -> tuple[str | None, str, set[str]]:
-    """Parses an editable requirement into:
-        - a requirement name
-        - an URL
-        - extras
-        - editable options
-    Accepted requirements:
-        svn+http://blahblah@rev#egg=Foobar[baz]&subdirectory=version_subdir
-        .[some_extra]
-    """
+def _parse_direct_url_editable(editable_req: str) -> tuple[str | None, str, set[str]]:
+    try:
+        req = Requirement(editable_req)
+    except InvalidRequirement:
+        pass
+    else:
+        if req.url and "://" in req.url:
+            # Join the marker back into the name part. This will be parsed out
+            # later into a Requirement again.
+            if req.marker:
+                name = f"{req.name} ; {req.marker}"
+            else:
+                name = req.name
+            return (name, req.url, req.extras)
+
+    raise ValueError
 
+
+def _parse_pip_syntax_editable(editable_req: str) -> tuple[str | None, str, set[str]]:
     url = editable_req
 
     # If a file path is specified with extras, strip off the extras.
@@ -122,23 +130,41 @@ def parse_editable(editable_req: str) -> tuple[str | None, str, set[str]]:
             url = f"{version_control}+{url}"
             break
 
+    return Link(url).egg_fragment, url, set()
+
+
+def parse_editable(editable_req: str) -> tuple[str | None, str, set[str]]:
+    """Parses an editable requirement into:
+        - a requirement name
+        - an URL
+        - extras
+    Accepted requirements:
+        - svn+http://blahblah@rev#egg=Foobar[baz]&subdirectory=version_subdir
+        - local_path[some_extra]
+        - Foobar[extra] @ svn+http://blahblah@rev#subdirectory=subdir ; markers
+    """
+    try:
+        package_name, url, extras = _parse_direct_url_editable(editable_req)
+    except ValueError:
+        package_name, url, extras = _parse_pip_syntax_editable(editable_req)
+
     link = Link(url)
 
-    if not link.is_vcs:
+    if not link.is_vcs and not link.is_file:
         backends = ", ".join(vcs.all_schemes)
         raise InstallationError(
             f"{editable_req} is not a valid editable requirement. "
             f"It should either be a path to a local project or a VCS URL "
             f"(beginning with {backends})."
         )
 
-    package_name = link.egg_fragment
-    if not package_name:
+    # The project name can be inferred from local file URIs easily.
+    if not package_name and not link.is_file:
         raise InstallationError(
             f"Could not detect requirement name for '{editable_req}', "
-            "please specify one with #egg=your_package_name"
+            "please specify one with your_package_name @ URL"
         )
-    return package_name, url, set()
+    return package_name, url, extras
 
 
 def check_first_requirement_in_file(filename: str) -> None:

src/pip/_internal/resolution/resolvelib/candidates.py

@@ -85,8 +85,12 @@ def make_install_req_from_editable(
     link: Link, template: InstallRequirement
 ) -> InstallRequirement:
     assert template.editable, "template not editable"
+    if template.name:
+        req_string = f"{template.name} @ {link.url}"
+    else:
+        req_string = link.url
     ireq = install_req_from_editable(
-        link.url,
+        req_string,
         user_supplied=template.user_supplied,
         comes_from=template.comes_from,
         use_pep517=template.use_pep517,

tests/functional/test_install_reqs.py

@@ -10,6 +10,7 @@
     PipTestEnvironment,
     ResolverVariant,
     TestData,
+    _create_test_package,
     _create_test_package_with_subdirectory,
     create_basic_sdist_for_package,
     create_basic_wheel_for_package,
@@ -941,3 +942,52 @@ def test_nonpep517_setuptools_import_failure(script: PipTestEnvironment) -> None
     exc_message = "ImportError: this 'setuptools' was intentionally poisoned"
     assert nice_message in result.stderr
     assert exc_message in result.stderr
+
+
+class TestEditableDirectURL:
+    def test_install_local_project(
+        self, script: PipTestEnvironment, data: TestData, common_wheels: Path
+    ) -> None:
+        uri = (data.src / "simplewheel-2.0").as_uri()
+        script.pip(
+            "install", "--no-index", "-e", f"simplewheel @ {uri}", "-f", common_wheels
+        )
+        script.assert_installed(simplewheel="2.0")
+
+    def test_install_local_project_with_extra(
+        self, script: PipTestEnvironment, data: TestData, common_wheels: Path
+    ) -> None:
+        uri = (data.src / "requires_simple_extra").as_uri()
+        script.pip(
+            "install",
+            "--no-index",
+            "-e",
+            f"requires-simple-extra[extra] @ {uri}",
+            "-f",
+            common_wheels,
+            "-f",
+            data.packages,
+        )
+        script.assert_installed(requires_simple_extra="0.1")
+        script.assert_installed(simple="1.0")
+
+    def test_install_local_git_repo(
+        self, script: PipTestEnvironment, common_wheels: Path
+    ) -> None:
+        repo_path = _create_test_package(script.scratch_path, "simple")
+        url = "git+" + repo_path.as_uri()
+        script.pip(
+            "install", "--no-index", "-e", f"simple @ {url}", "-f", common_wheels
+        )
+        script.assert_installed(simple="0.1")
+
+    @pytest.mark.network
+    def test_install_remote_git_repo_with_extra(
+        self, script: PipTestEnvironment, data: TestData, common_wheels: Path
+    ) -> None:
+        req = "pip-test-package[extra] @ git+https://github.com/pypa/pip-test-package"
+        script.pip(
+            "install", "--no-index", "-e", req, "-f", common_wheels, "-f", data.packages
+        )
+        script.assert_installed(pip_test_package="0.1.1")
+        script.assert_installed(simple="3.0")

tests/unit/test_req.py

@@ -853,6 +853,25 @@ def test_install_req_extend_extras(
         assert extended.permit_editable_wheels == req.permit_editable_wheels
 
 
+@pytest.mark.parametrize(
+    "req_str, expected",
+    [
+        (
+            'foo[extra] @ svn+http://foo ; os_name == "nt"',
+            ('foo ; os_name == "nt"', "svn+http://foo", {"extra"}),
+        ),
+        (
+            "foo @ svn+http://foo",
+            ("foo", "svn+http://foo", set()),
+        ),
+    ],
+)
+def test_parse_editable_pep508(
+    req_str: str, expected: tuple[str, str, set[str]]
+) -> None:
+    assert parse_editable(req_str) == expected
+
+
 @mock.patch("pip._internal.req.req_install.os.path.abspath")
 @mock.patch("pip._internal.req.req_install.os.path.exists")
 @mock.patch("pip._internal.req.req_install.os.path.isdir")