add safer, more versatile version of rapidhash for any array and for any iterable

[vtjnash]

Tested that array gives the same answer on random data, and that performance is equivalent (with LLVM 20 on AArch64). Neither of these methods is currently used for anything, but the intent is to change some users (e.g. String) to the array interface and other users (e.g. large Numbers and AbstractString) to the iterator interface (I already have both written–or rather Claude does–but in a different repo right now). The iterator interface does not exist anywhere else, since it is about 8x slower (essentially operating byte by byte instead of the intended 64-bit chunks).

julia> using BenchmarkTools, Random; for n = 0:67:1000
         a = (i % UInt8 for i in 1:n)
         b = collect(a)
         ## Same results for [] and String
         # b = codeunits(randstring(n))
         # a = Base.Generator(identity, b)
         @btime Base.hash_bytes($a, Base.HASH_SEED, Base.HASH_SECRET)
         @btime Base.hash_bytes($b, Base.HASH_SEED, Base.HASH_SECRET)
         @btime Base.hash_bytes(pointer($b), length($b), Base.HASH_SEED, Base.HASH_SECRET)
         println()
       end
  5.666 ns (0 allocations: 0 bytes) # iterator
  3.625 ns (0 allocations: 0 bytes) # array
  3.625 ns (0 allocations: 0 bytes) # pointer

  20.269 ns (0 allocations: 0 bytes)
  5.375 ns (0 allocations: 0 bytes)
  5.083 ns (0 allocations: 0 bytes)

  35.624 ns (0 allocations: 0 bytes)
  6.250 ns (0 allocations: 0 bytes)
  6.208 ns (0 allocations: 0 bytes)

  50.954 ns (0 allocations: 0 bytes)
  7.625 ns (0 allocations: 0 bytes)
  7.334 ns (0 allocations: 0 bytes)

  66.496 ns (0 allocations: 0 bytes)
  9.083 ns (0 allocations: 0 bytes)
  8.875 ns (0 allocations: 0 bytes)

  82.299 ns (0 allocations: 0 bytes)
  10.719 ns (0 allocations: 0 bytes)
  10.511 ns (0 allocations: 0 bytes)

  98.346 ns (0 allocations: 0 bytes)
  12.888 ns (0 allocations: 0 bytes)
  12.513 ns (0 allocations: 0 bytes)

PR implemented mostly by Claude, though I had to fix its math in several places since it got quite confused by the logical nesting of if branches and the implications of extracting trailing bytes.

Test "proof" of correctness:

julia> @time for n = 0:10000
         b = codeunits(randstring(n))
         a = Base.Generator(identity, b)
         h1 = Base.hash_bytes(a, Base.HASH_SEED, Base.HASH_SECRET)
         h2 = Base.hash_bytes(b, Base.HASH_SEED, Base.HASH_SECRET)
         h3 = Base.hash_bytes(pointer(b), length(b), Base.HASH_SEED, Base.HASH_SECRET)
         @assert h1 == h2 == h3
       end
  0.174760 seconds (10.00 k allocations: 48.855 MiB, 3.97% gc time)

[adienes]

would it be crazy to loosen this signature?

I can imagine there would potentially be types with eltype == UInt8 and index step size 1 but not <: AbstractArray

[vtjnash]

Depends. This is a specialization of the function for array-like containers. The iterator fallback is actually usually slightly faster than this function in general (since iterate is always much cheaper to compute than index), except when the compiler can be especially clever and realize that this accesses dense memory, and can entirely skip computing indices

[adienes]

I tried this and in local benchmarks it seems to be about 1.5-3x faster depending on the length

julia> @btime Base.hash_bytes($(Base.Generator(identity, UInt8[1,2,3,4,5])), Base.HASH_SEED, Base.HASH_SECRET)
  7.458 ns (0 allocations: 0 bytes)
0xae87681e85516f3c

julia> @btime Base.hash_bytes($(Base.Generator(identity, UInt8.(collect(1:10000) .% UInt8))), Base.HASH_SEED, Base.HASH_SECRET)
  2.361 μs (0 allocations: 0 bytes)

vs

julia> @btime Base.hash_bytes($(Base.Generator(identity, UInt8[1,2,3,4,5])), Base.HASH_SEED, Base.HASH_SECRET)
  12.012 ns (0 allocations: 0 bytes)
0xae87681e85516f3c

julia> @btime Base.hash_bytes($(Base.Generator(identity, UInt8.(collect(1:10000) .% UInt8))), Base.HASH_SEED, Base.HASH_SECRET)
  7.761 μs (0 allocations: 0 bytes)

@generated function read_uint64_from_uint8_iter(_, ::Nothing)
    :(zero(UInt64), nothing, 0)
end

@generated function read_uint64_from_uint8_iter(iter)
    quote
        next_result = iterate(iter)
        next_result === nothing && return (zero(UInt64), nothing, 0)
        # next_result === nothing && return nothing
        (byte, state) = next_result
        value = UInt64(byte)

        begin
            $(Expr(:block, [
                :(next_result = iterate(iter, state);
                  next_result === nothing && return (value, state, $(i-1));
                  (byte, state) = next_result;
                  value |= (UInt64(byte) << $((i-1)*8));)
                for i = 2:8
            ]...))
        end
        return (value, state, 8)
    end
end

@generated function read_uint64_from_uint8_iter(iter, state)
    quote
        value = zero(UInt64)
        begin
            $(Expr(:block, [
                :(next_result = iterate(iter, state);
                  next_result === nothing && return (value, state, $(i-1));
                  (byte, state) = next_result;
                  value |= (UInt64(byte) << $((i-1)*8));)
                for i = 1:8
            ]...))
        end
        return (value, state, 8)
    end
end

@assume_effects :terminates_globally function hash_bytes(
        iter,
        seed::UInt64,
        secret::NTuple{4, UInt64}
    )
    ...
    l0, state, b0 = read_uint64_from_uint8_iter(iter)
    if state !== nothing
        while true
            ...

[vtjnash]

I don't see anything like that speedup, and that's also unsound, but it does seem worth unrolling this from that result so I did that too (I realized I hadn't tried before since it did so badly when I had unrolled the array version that I assumed the iterator version would do the same).

[vtjnash]

All the code user(s) examples now rebased on top of this PR (for all Integers, AbstractStrings, etc): https://github.com/JuliaLang/julia/pull/new/jn/rapidhash-all-the-things

[adienes]

I ran into similar bootstrap problems with @nexrps in #58252 hence why I put them (somewhat awkwardly) in multidimensional.jl

maybe it is worth splitting hashing.jl into hashingsimple.jl and hashingfancy.jl and moving the non-core stuff (incl. the methods currently in multidimensional.jl) to later in compilation order

[oscardssmith]

32 bit failures look real @vtjnash:

Error in testset hashing:
Error During Test at /cache/build/tester-amdci5-15/julialang/julia-master/julia-3728f0bbac/share/julia/test/hashing.jl:325
  Got exception outside of a @test
  MethodError: no method matching hash_bytes(::Vector{UInt8}, ::UInt32, ::NTuple{4, UInt64})
  The function `hash_bytes` exists, but no method is defined for this combination of argument types.
  Closest candidates are:
    hash_bytes(::AbstractArray{UInt8}, !Matched::UInt64, ::NTuple{4, UInt64})
     @ Base hashing.jl:368
    hash_bytes(::Any, !Matched::UInt64, ::NTuple{4, UInt64})
     @ Base hashing.jl:478
    hash_bytes(!Matched::Ptr{UInt8}, !Matched::Int32, !Matched::UInt64, !Matched::NTuple{4, UInt64})
     @ Base hashing.jl:272

[Seelengrab]

So we now have this ultra-specific reinterpret with a very undescriptive name?

[oscardssmith]

I would definitely accept a PR removing the method.

[adienes]

maybe relevant to that: #31305