Native Julia API for Memory Allocations

[Tokazama]

While working on #48728 I've found out a couple things that may have pretty big implications and the limitations of Julia (some of this may be obvious to core devs here, so I apologize if I'm missing the mark a bit on any of this). It's possible to get and set values to a previously allocated block of memory all in native Julia (although this is likely not intended to be a stable interface and takes some careful work to accomplish with decent performance). This does require some extra calls to C
for boxed values (jl_value_ptr and jl_gc_queue_root), but seems to work without any errors. This can be used to push Julia's performance and more generally is in line with Julia's philosophy of solving the two language problem. However, the ability to manual intervene in memory using Julia stops as soon as (re)allocating memory is involved.

It's impractical to wholly rely on a dedicated memory allocation type from Base that is essentially

struct Memory
    length::Int
    data::Ptr{Cvoid}
end

This precludes other potential layouts such as something table-like

struct MemoryTable
    nrows::Int
    ncols::Int
    data::Ptr{Ptr{Cvoid}}
end

I won't belabor the variety of layouts anymore than this, but the point stands that Julia really needs to have a method to accommodate more flexibility in allocating blocks of memory itself.

I understand that the underlying management of memory in Julia is subject to change and undergoing continual optimization at this time, making commitment to any API at this time difficult. However, even something as minimal as GC.alloc and/or GC.realloc would be extremely helpful. This wouldn't commit to any specific heap vs stack allocation schema but would allow moving a lot more code to native Julia.

I figure that ordinary pointers would be an issue since we still want the GC to track this chunk of memory. Maybe if we had something that was equivalent to mutable struct OpaqueMemoryChunk end?

[mkitti]

I really do think we need to study Java's new foreign memory API closely to both not reinvent the wheel and avoid the same mistakes. Java is a garbage collected language that is quite pointer heavy. It uses layouts similar to your MemoryTable layout.

https://docs.oracle.com/en/java/javase/19/docs/api/java.base/java/lang/foreign/MemorySession.html

[mkitti]

A MemorySegment is basically your OpaqueMemoryChunk that refers to a contiguous chunk of memory. A MemorySession is used to manage multiple MemorySegments or other resources.

Java basically has similar issues as we have in Julia. In fact, it's a bit worse in Java since Java arrays (1D) are generally not guaranteed to be contiguous in memory. Perhaps this is what your second point is getting to... an array may be backed by non-contiguous memory.

The overall problem of having a bunch of mutable objects that manage low level memory is what Java's foreign memory API is taking on.

[Tokazama]

If I understand what MemorySegment is correctly, it appears to be very close to what buffers are in #48728. It might be worth factoring out some of that PR into a separate PR that just allocates n-bytes of a certain alignment and then wrap the element typed object around that.

[PallHaraldsson]

In fact, it's a bit worse in Java since Java arrays (1D) are generally not guaranteed to be contiguous in memory.

I was so surprised by this, that I looked it up, and I think you're wrong (if not how would indexing be O(1) and fast?). And even if theoretically allowed by the spec, then in practice not done:

https://stackoverflow.com/questions/10224888/java-are-1-d-arrays-always-contiguous-in-memory

"Everything which is not forbidden is allowed" principle. [..] Thus it definitely means that array might be not continuous. [..]
As all arrays are stored in heap and heap might be not continuous, it follows that arrays also might be not continuous.

I do not agree this to be a valid conclusion, the heap might not be, but you would still want each array in the heap to be.

Another answer there:

[..] Note that using Unsafe pretty much automatically means that you're strolling outside the realm of the spec.

That being said, I'd venture that most JVM implementations do in fact use a linear layout for (one-dimensional) arrays.

I believe Julia requires (regular) arrays to be contiguous (and you can get a pointer to), but also the whole heap? And would there be a good reason to not have it so (maybe on uncommon CPUs only without MMU)?

[mkitti]

Anything but a primitive array in Java is probably an array of pointers internally, and technically an array of pointers is still O(1).

While arrays are probably mostly contiguous in memory, it's also possible that chunks of an array live in different parts of the heap, especially if the array is resized.

Also note that the heap itself is not contiguous:
https://docs.oracle.com/javase/specs/jvms/se8/html/jvms-2.html#jvms-2.5.3

The memory for the heap does not need to be contiguous.

To get a guarantee of contiguous memory in Java that can be accessed natively, you have to jump through a lot of hoops. The older version is to allocate a direct ByteBuffer:

https://stackoverflow.com/questions/5670862/bytebuffer-allocate-vs-bytebuffer-allocatedirect

[Tokazama]

It seems like we'd want to emulate something closer to ByteBuffer by default and then have pooled memory structures composed of that using the type system where possible.

[mkitti]

What does GC.alloc or GC.realloc do that Libc.malloc or Libc.realloc does not do?

[Tokazama]

The GC versions would be tracked by the garbage collector and hopefully not as slow as the the Libc versions.

[mkitti]

Why are the Libc versions slow?

[mkitti]

Do you mean GC.alloc would work something like the following?

julia> function gc_alloc(sz::Integer)
           ptr = ccall(:jl_malloc, Ptr{Nothing}, (Csize_t,), sz)
           ref = Ref{Ptr{Nothing}}(ptr)
           finalizer(ref) do r
               ccall(:jl_free, Cvoid, (Ptr{Nothing},), r[])
               old_r = r[] # diagnostic
               r[] = C_NULL
               @async @info "Freed" old_r #diagnostic
           end
       end
gc_alloc (generic function with 1 method)

julia> p = gc_alloc(1024)
Base.RefValue{Ptr{Nothing}}(Ptr{Nothing} @0x0000018685339860)

julia> p = nothing

julia> GC.gc()
┌ Info: Freed
└   old_r = Ptr{Nothing} @0x0000018685339860

[Tokazama]

I'm assuming that Libc is slower because whatever approach a given OS uses for memory management can't be improved with what is know about (de)allocations occuring throughout Julia's runtime.

That's pretty similar to what I was thinking but it would be nice if it could act more like a pointer than a ref and we had a way to root other Julia values to the GC pointer.

[Tokazama]

This is an interesting reference on how Rust is developing its allocation protocols https://github.com/rust-lang/rfcs/pull/1398/files

[mkitti]

This is similar to the approach I started at https://github.com/mkitti/ArrayAllocators.jl