Backports for 1.12 (#4493)

Co-authored-by: KristofferC <[email protected]>
Co-authored-by: Ian Butterworth <[email protected]>
Co-authored-by: Jerry Ling <[email protected]>
Co-authored-by: Guillaume Dalle <[email protected]>

Author: 5 people

Project.toml

@@ -43,7 +43,7 @@ Markdown = "1.11"
 Printf = "1.11"
 Random = "1.11"
 REPL = "1.11"
-SHA = "0.7"
+SHA = "0.7, 1"
 TOML = "1"
 Tar = "1.10"
 UUIDs = "1.11"

docs/src/creating-packages.md

@@ -275,79 +275,100 @@ test-specific dependencies, are available, see below.
 
 ### Test-specific dependencies
 
-There are two ways of adding test-specific dependencies (dependencies that are not dependencies of the package but will still be available to
-load when the package is tested).
+Test-specific dependencies are dependencies that are not dependencies of the package itself but are available when the package is tested.
 
-#### `target` based test specific dependencies
+#### Recommended approach: Using workspaces with `test/Project.toml`
 
-Using this method of adding test-specific dependencies, the packages are added under an `[extras]` section and to a test target,
-e.g. to add `Markdown` and `Test` as test dependencies, add the following to the `Project.toml` file:
-
-```toml
-[extras]
-Markdown = "d6f4376e-aef5-505a-96c1-9c027394607a"
-Test = "8dfed614-e22c-5e08-85e1-65c5234f0b40"
-
-[targets]
-test = ["Markdown", "Test"]
-```
-
-Note that the only supported targets are `test` and `build`, the latter of which (not recommended) can be used
-for any `deps/build.jl` scripts.
-
-#### Alternative approach: `test/Project.toml` file test specific dependencies
+!!! compat
+    Workspaces require Julia 1.12+. For older Julia versions, see the legacy approaches below.
 
-!!! note
-    The exact interaction between `Project.toml`, `test/Project.toml` and their corresponding
-    `Manifest.toml`s are not fully worked out and may be subject to change in future versions.
-    The older method of adding test-specific dependencies, described in the previous section,
-    will therefore be supported throughout all Julia 1.X releases.
+The recommended way to add test-specific dependencies is to use workspaces. This is done by:
 
-In Julia 1.2 and later test dependencies can be declared in `test/Project.toml`. When running
-tests, Pkg will automatically merge this and the package Projects to create the test environment.
+1. Adding a `[workspace]` section to your package's `Project.toml`:
 
-!!! note
-    If no `test/Project.toml` exists Pkg will use the `target` based test specific dependencies.
+```toml
+[workspace]
+projects = ["test"]
+```
 
-To add a test-specific dependency, i.e. a dependency that is available only when testing,
-it is thus enough to add this dependency to the `test/Project.toml` project. This can be
-done from the Pkg REPL by activating this environment, and then use `add` as one normally
-does. Let's add the `Test` standard library as a test dependency:
+2. Creating a `test/Project.toml` file with your test dependencies:
 
 ```julia-repl
 (HelloWorld) pkg> activate ./test
 [ Info: activating environment at `~/HelloWorld/test/Project.toml`.
 
-(test) pkg> add Test
+(HelloWorld/test) pkg> dev .  # add current package to test dependencies using its path
+ Resolving package versions...
+  Updating `~/HelloWorld/test/Project.toml`
+  [8dfed614] + HelloWorld v0.1.0 `..`
+
+(HelloWorld/test) pkg> add Test  # add other test dependencies
  Resolving package versions...
   Updating `~/HelloWorld/test/Project.toml`
   [8dfed614] + Test
-  Updating `~/HelloWorld/test/Manifest.toml`
-  [...]
 ```
 
-We can now use `Test` in the test script and we can see that it gets installed when testing:
+When using workspaces, the package manager resolves dependencies for all projects in the workspace together, and creates a single `Manifest.toml` next to the base `Project.toml`. This provides better dependency resolution and makes it easier to manage test-specific dependencies.
+
+!!! warning
+  Unlike some earlier test dependency workflows, this one explicitly requires adding `HelloWorld` (the parent package) to your `test/Project.toml`.
+
+You can now use `Test` in the test script:
 
 ```julia-repl
 julia> write("test/runtests.jl",
              """
-             using Test
+             using HelloWorld, Test
              @test 1 == 1
              """);
 
-(test) pkg> activate .
+(HelloWorld/test) pkg> activate .
 
 (HelloWorld) pkg> test
    Testing HelloWorld
  Resolving package versions...
-  Updating `/var/folders/64/76tk_g152sg6c6t0b4nkn1vw0000gn/T/tmpPzUPPw/Project.toml`
-  [d8327f2a] + HelloWorld v0.1.0 [`~/.julia/dev/Pkg/HelloWorld`]
-  [8dfed614] + Test
-  Updating `/var/folders/64/76tk_g152sg6c6t0b4nkn1vw0000gn/T/tmpPzUPPw/Manifest.toml`
-  [d8327f2a] + HelloWorld v0.1.0 [`~/.julia/dev/Pkg/HelloWorld`]
-   Testing HelloWorld tests passed```
+   Testing HelloWorld tests passed
 ```
 
+Workspaces can also be used for other purposes, such as documentation or benchmarks, by adding additional projects to the workspace:
+
+```toml
+[workspace]
+projects = ["test", "docs", "benchmarks"]
+```
+
+See the section on [Workspaces](@ref) in the `Project.toml` documentation for more details.
+
+#### Legacy approach: `target` based test specific dependencies
+
+!!! warning
+    This approach is legacy and maintained for compatibility. New packages should use workspaces instead.
+
+Using this method, test-specific dependencies are added under an `[extras]` section and to a test target:
+
+```toml
+[extras]
+Markdown = "d6f4376e-aef5-505a-96c1-9c027394607a"
+Test = "8dfed614-e22c-5e08-85e1-65c5234f0b40"
+
+[targets]
+test = ["Markdown", "Test"]
+```
+
+Note that the only supported targets are `test` and `build`, the latter of which (not recommended) can be used for any `deps/build.jl` scripts.
+
+#### Legacy approach: `test/Project.toml` without workspace
+
+!!! warning
+    This approach is legacy and maintained for compatibility. New packages should use workspaces instead.
+
+In Julia 1.2 and later, test dependencies can be declared in `test/Project.toml` without using a workspace. When running tests, Pkg will automatically merge the package and test projects to create the test environment.
+
+!!! note
+    If no `test/Project.toml` exists, Pkg will use the `target` based test specific dependencies.
+
+This approach works similarly to the workspace approach, but without the workspace declaration in the main `Project.toml`.
+
 ## Compatibility on dependencies
 
 Every dependency should in general have a compatibility constraint on it.
@@ -450,9 +471,7 @@ Extensions can have arbitrary names (here `ContourExt`), following the format of
 In `Pkg` output, extension names are always shown together with their parent package name.
 
 !!! compat
-    Often you will put the extension dependencies into the `test` target so they are loaded when running e.g. `Pkg.test()`. On earlier Julia versions
-    this requires you to also put the package in the `[extras]` section. This is unfortunate but the project verifier on older Julia versions will
-    complain if this is not done.
+    Often you will want to load extension dependencies when testing your package. The recommended approach is to use workspaces and add the extension dependencies to your `test/Project.toml` (see [Test-specific dependencies](@ref adding-tests-to-packages)). For older Julia versions that don't support workspaces, you can put the extension dependencies into the `test` target, which requires you to also put the package in the `[extras]` section. The project verifier on older Julia versions will complain if this is not done.
 
 !!! note
     If you use a manifest generated by a Julia version that does not know about extensions with a Julia version that does
@@ -557,12 +576,14 @@ This is done by making the following changes (using the example above):
 
 In the case where one wants to use an extension (without worrying about the
 feature of the extension being available on older Julia versions) while still
-supporting older Julia versions the packages under `[weakdeps]` should be
+supporting older Julia versions without workspace support, the packages under `[weakdeps]` should be
 duplicated into `[extras]`. This is an unfortunate duplication, but without
 doing this the project verifier under older Julia versions will throw an error
 if it finds packages under `[compat]` that is not listed in `[extras]`.
 
-## Package naming rules
+For Julia 1.12+, using workspaces is recommended and this duplication is not necessary.
+
+## [Package naming guidelines](@id Package-naming-rules)
 
 Package names should be sensible to most Julia users, *even to those who are not domain experts*.
 The following rules apply to the `General` registry but may be useful for other package
@@ -623,7 +644,7 @@ may fit your package better.
 9. Packages should follow the [Stylistic Conventions](https://docs.julialang.org/en/v1/manual/variables/#Stylistic-Conventions).
      * The package name begin with a capital letter and word separation is shown with upper camel case
      * Packages that provide the functionality of a project from another language should use the Julia convention
-     * Packages that [provide pre-built libraries and executables](https://docs.binarybuilder.org/stable/jll/) can keep orignal name, but should get `_jll`as a suffix. For example `pandoc_jll` wraps pandoc. However, note that the generation and release of most JLL packages is handled by the [Yggdrasil](https://github.com/JuliaPackaging/Yggdrasil) system. 
+     * Packages that [provide pre-built libraries and executables](https://docs.binarybuilder.org/stable/jll/) can keep orignal name, but should get `_jll`as a suffix. For example `pandoc_jll` wraps pandoc. However, note that the generation and release of most JLL packages is handled by the [Yggdrasil](https://github.com/JuliaPackaging/Yggdrasil) system.
 
 ## Registering packages
 

docs/src/toml-files.md

@@ -39,7 +39,7 @@ name = "Example"
 The name must be a valid [identifier](https://docs.julialang.org/en/v1/base/base/#Base.isidentifier)
 (a sequence of Unicode characters that does not start with a number and is neither `true` nor `false`).
 For packages, it is recommended to follow the
-[package naming rules](@ref Package-naming-rules). The `name` field is mandatory
+[package naming guidelines](@ref Package-naming-rules). The `name` field is mandatory
 for packages.
 
 
@@ -138,7 +138,7 @@ constraints in detail. It is also possible to list constraints on `julia` itself
 julia = "1.1"
 ```
 
-### The `[workspace]` section
+### [The `[workspace]` section](@id Workspaces)
 
 A project file can define a workspace by giving a set of projects that is part of that workspace.
 Each project in a workspace can include their own dependencies, compatibility information, and even function as full packages.
@@ -152,7 +152,7 @@ A workspace is defined in the base project by giving a list of the projects in i
 projects = ["test", "docs", "benchmarks", "PrivatePackage"]
 ```
 
-This structure is particularly beneficial for developers using a monorepo approach, where a large number of unregistered packages may be involved. It's also useful for adding documentation or benchmarks to a package by including additional dependencies beyond those of the package itself.
+This structure is particularly beneficial for developers using a monorepo approach, where a large number of unregistered packages may be involved. It's also useful for adding test-specific dependencies to a package by including a `test` project in the workspace (see [Test-specific dependencies](@ref adding-tests-to-packages)), or for adding documentation or benchmarks with their own dependencies.
 
 Workspace can be nested: a project that itself defines a workspace can also be part of another workspace.
 In this case, the workspaces are "merged" with a single manifest being stored alongside the "root project" (the project that doesn't have another workspace including it).

ext/REPLExt/precompile.jl

@@ -36,7 +36,12 @@ let
     end
 
     if Base.generating_output()
-        pkgreplmode_precompile()
+        ccall(:jl_tag_newly_inferred_enable, Cvoid, ())
+        try
+            pkgreplmode_precompile()
+        finally
+            ccall(:jl_tag_newly_inferred_disable, Cvoid, ())
+        end
     end
 
 end # let

src/Apps/Apps.jl

@@ -7,6 +7,7 @@ using Pkg.Types: AppInfo, PackageSpec, Context, EnvCache, PackageEntry, Manifest
 using Pkg.Operations: print_single, source_path, update_package_add
 using Pkg.API: handle_package_input!
 using TOML, UUIDs
+using Dates
 import Pkg.Registry
 
 app_env_folder() = joinpath(first(DEPOT_PATH), "environments", "apps")
@@ -183,6 +184,9 @@ function add(pkg::PackageSpec)
     handle_package_input!(pkg)
 
     ctx = app_context()
+
+    Pkg.Operations.update_registries(ctx; force = false, update_cooldown = Day(1))
+
     manifest = ctx.env.manifest
     new = false
 
@@ -211,6 +215,9 @@ function add(pkg::PackageSpec)
     manifest.deps[pkg.uuid] = entry
 
     _resolve(manifest, pkg.name)
+    if new === true || (new isa Set{UUID} && pkg.uuid in new)
+        Pkg.Operations.build_versions(ctx, Set([pkg.uuid]); verbose = true)
+    end
     precompile(pkg.name)
 
     @info "For package: $(pkg.name) installed apps $(join(keys(project.apps), ","))"
@@ -247,8 +254,10 @@ function develop(pkg::PackageSpec)
     manifest = ctx.env.manifest
     manifest.deps[pkg.uuid] = entry
 
-    _resolve(manifest, pkg.name)
-    precompile(pkg.name)
+    # For dev, we don't create an app environment - just point shims directly to the dev'd project
+    write_manifest(manifest, app_manifest_file())
+    generate_shims_for_apps(pkg.name, project.apps, sourcepath, joinpath(Sys.BINDIR, "julia"))
+
     @info "For package: $(pkg.name) installed apps: $(join(keys(project.apps), ","))"
     check_apps_in_path(project.apps)
 end

src/Pkg.jl

@@ -177,7 +177,7 @@ Pkg.add(name="Example", version="0.3") # Specify version; latest release in the
 Pkg.add(name="Example", version="0.3.1") # Specify version; exact release
 Pkg.add(url="https://github.com/JuliaLang/Example.jl", rev="master") # From url to remote gitrepo
 Pkg.add(url="/remote/mycompany/juliapackages/OurPackage") # From path to local gitrepo
-Pkg.add(url="https://github.com/Company/MonoRepo", subdir="juliapkgs/Package.jl)") # With subdir
+Pkg.add(url="https://github.com/Company/MonoRepo", subdir="juliapkgs/Package.jl") # With subdir
 ```
 
 After the installation of new packages the project will be precompiled. See more at [Environment Precompilation](@ref).

src/REPLMode/argument_parsers.jl

@@ -34,9 +34,46 @@ end
 
 # Simple URL detection
 function looks_like_url(str::String)
-    return startswith(str, "http://") || startswith(str, "https://") ||
-           startswith(str, "git@") || startswith(str, "ssh://") ||
-           contains(str, ".git")
+    if startswith(str, "http://") || startswith(str, "https://") ||
+            startswith(str, "git@") || startswith(str, "ssh://") ||
+            contains(str, ".git")
+        return true
+    end
+
+    # Check for user@host:path pattern (SSH URL with user)
+    # This handles cases like: [email protected]:PackageName.jl
+    # The host part should not contain / or @ characters, and should come before the :
+    at_pos = findfirst('@', str)
+    if at_pos !== nothing
+        colon_pos = findnext(':', str, nextind(str, at_pos))
+        if colon_pos !== nothing
+            host_part = str[nextind(str, at_pos):(prevind(str, colon_pos))]
+            # Host should not contain / (which would suggest this is package@version:subdir syntax)
+            # and should look like a hostname or IP address (no spaces, etc.)
+            # Additionally, exclude things that look like version numbers (e.g., "1.0", "1.0.0")
+            # by checking if the host contains only digits and dots (which would be a version or IP)
+            # If it's all digits and dots, it must have at least 3 dots to be an IP (X.X.X.X)
+            if !contains(host_part, '/') && !contains(host_part, ' ') && !isempty(host_part)
+                # Check if this looks like a version number (e.g., "1.0", "1.0.0")
+                # vs an IP address (e.g., "10.20.30.40") or hostname (e.g., "server.com")
+                if all(c -> isdigit(c) || c == '.', host_part)
+                    # All digits and dots - could be version or IP
+                    # Count dots: version has 1-2 dots, IP has 3 dots
+                    dot_count = count(==('.'), host_part)
+                    if dot_count >= 3
+                        # Likely an IP address (X.X.X.X)
+                        return true
+                    end
+                    # else: likely a version number, not a URL
+                else
+                    # Contains letters or other chars - likely a hostname
+                    return true
+                end
+            end
+        end
+    end
+
+    return false
 end
 
 # Simple path detection
@@ -119,14 +156,13 @@ end
 function is_url_structure_colon(input::String, colon_pos::Int)
     after_colon = input[nextind(input, colon_pos):end]
 
-    # Check for git@host:path syntax
-    if startswith(input, "git@")
-        at_pos = findfirst('@', input)
-        if at_pos !== nothing
-            between_at_colon = input[nextind(input, at_pos):prevind(input, colon_pos)]
-            if !contains(between_at_colon, '/')
-                return true
-            end
+    # Check for user@host:path syntax (including git@host:path)
+    at_pos = findfirst('@', input)
+    if at_pos !== nothing && at_pos < colon_pos
+        between_at_colon = input[nextind(input, at_pos):prevind(input, colon_pos)]
+        # If there's no '/' between @ and :, this colon is part of the SSH URL structure
+        if !contains(between_at_colon, '/')
+            return true
         end
     end
 

src/Resolve/graphtype.jl

@@ -1078,6 +1078,11 @@ function propagate_constraints!(graph::Graph, sources::Set{Int} = Set{Int}(); lo
 
     seen = copy(staged)
 
+    # Pre-allocate workspace for added constraints
+    max_spp = maximum(spp, init = 0)
+    added_constr1 = BitVector(undef, max_spp)
+    old_gconstr1 = BitVector(undef, max_spp)
+
     while !isempty(staged)
         staged_next = Set{Int}()
         for p0 in staged
@@ -1091,16 +1096,28 @@ function propagate_constraints!(graph::Graph, sources::Set{Int} = Set{Int}(); lo
                 pkgs[p1] == uuid_julia && continue
 
                 msk = gmsk[p0][j1]
-                # consider the sub-mask with only allowed versions of p0
-                sub_msk = msk[:,gconstr0]
                 # if an entire row of the sub-mask is false, that version of p1
                 # is effectively forbidden
                 # (this is just like calling `any` row-wise)
-                added_constr1 = any!(BitVector(undef, spp[p1]), sub_msk)
+                # sub_msk = msk[:, gconstr0]
+                # added_constr1 = any!(BitVector(undef, spp[p1]), sub_msk)
+                # The code below is equivalent to the shorter code above, but avoids allocating
+                spp1 = spp[p1]
+                resize!(added_constr1, spp1)
+                fill!(added_constr1, false)
+                for v1 in 1:spp1
+                    for v0 in 1:spp[p0]
+                        if gconstr0[v0] && msk[v1, v0]
+                            added_constr1[v1] = true
+                            break
+                        end
+                    end
+                end
+
                 # apply the new constraints, checking for contradictions
                 # (keep the old ones for comparison)
                 gconstr1 = gconstr[p1]
-                old_gconstr1 = copy(gconstr1)
+                copy!(old_gconstr1, gconstr1)
                 gconstr1 .&= added_constr1
                 # if the new constraints are more restrictive than the
                 # previous ones, record it and propagate them next