Fix rare git treehashing bug (#2881)

If we have a symlink that contains a name that is a prefix of another
file within a directory, the ordering of treehashing changed between
v1.6.X and v1.7.0.  This is a problem, as it makes certain (rare) cases
hash to invalid treehashes.

This is because when `tree_hash()` was refactored in `v1.7`, I
simplified `git_mode(f) == mode_dir` to `isdir(f)`, but it turns out
that `isdir(f)` can return true if a symlink points to a directory.
This in turn prevents `/` from being appended to the filenames when
they are sorted, which is what causes the potential ordering change.

This only happens when the symlink has a name such as `5.28` which is a
prefix of another file, say `5.28.1`, and the next character sorts
higher than `/`, as `.` does.  Therefore, on `v1.6.4`, we would get:

```julia
julia> sort(["5.28", "5.28.1/"])
2-element Vector{String}:
 "5.28"
 "5.28.1/"
```

Whereas on `v1.7.0`, because `5.28` is a symlink pointing to a
directory, we get:

```julia
julia> sort(["5.28/", "5.28.1/"])
2-element Vector{String}:
 "5.28.1/"
 "5.28/"
```

Which changes the ordering.  Consulting my ASCII table, the number of
valid pathname characters that can cause this confusion is small, but
enough that most "rootfs image" style artifacts will run into it, thanks
to versioning schemes with symlinks such as the one that I found that
triggered this whole issue.

Author: staticfloat

src/GitTools.jl

@@ -274,7 +274,7 @@ Calculate the git tree hash of a given path.
 """
 function tree_hash(::Type{HashType}, root::AbstractString; debug_out::Union{IO,Nothing} = nothing, indent::Int=0) where HashType
     entries = Tuple{String, Vector{UInt8}, GitMode}[]
-    for f in sort(readdir(root; join=true); by = f -> isdir(f) ? f*"/" : f)
+    for f in sort(readdir(root; join=true); by = f -> gitmode(f) == mode_dir ? f*"/" : f)
         # Skip `.git` directories
         if basename(f) == ".git"
             continue

test/new.jl

@@ -2598,6 +2598,18 @@ tree_hash(root::AbstractString; kwargs...) = bytes2hex(@inferred Pkg.GitTools.tr
               tree_hash(joinpath(dir, "FooGit")) ==
               "2f42e2c1c1afd4ef8c66a2aaba5d5e1baddcab33"
     end
+
+    # Test for symlinks that are a prefix of another directory, causing sorting issues
+    if !Sys.iswindows()
+        mktempdir() do dir
+            mkdir(joinpath(dir, "5.28.1"))
+            write(joinpath(dir, "5.28.1", "foo"), "")
+            chmod(joinpath(dir, "5.28.1", "foo"), 0o644)
+            symlink("5.28.1", joinpath(dir, "5.28"))
+
+            @test tree_hash(dir) == "5e50a4254773a7c689bebca79e2954630cab9c04"
+       end
+   end
 end
 
 @testset "multiple registries overlapping version ranges for different versions" begin