Proposal: Allow short isolated computations to use private heap avoiding GC (arena allocation approach)

[savahmel]

The idea is to allow short isolated computations to allocate objects on a private thread-local heap. The heap is discarded when the computation is over, avoiding garbage collection and finalization queue overheads.

Some methods will be declared as 'isolated' and the compiler will ensure that any object newed within these methods cannot outlive the isolated heap lifetime by preventing managed heap objects from referencing isolated heap objects.

---

Additions to the language:

1. The “isolated” keyword to denote a block of code that defines an isolated context.

void Foo() { 
     // some code (allocates on managed heap)
     isolated { // private heap created
	    new Bar().IsolatedMethod(); /* Bar and any objects created in 
            IsolatedMethod are allocated on the private heap */
     } // finalizers called; private heap destroyed
     // more code (allocates on managed heap)
}

2. [Isolated] attribute applicable to a method, constructor, class, struct, parameter, return type.

method/constructor

Isolated methods allocate new objects on the isolated heap when called from an isolated context.
When called normally, they allocate on the managed heap as usual.
An isolated method cannot be abstract, virtual or async or belong to an interface.
If not specified otherwise (with [NonIsolated] attribute) parameters and return value are treated by the compiler as isolated heap objects.

class/struct

Makes all methods, constructors, finalizer, fields, properties, nested classes/structs isolated.
The isolated class cannot contain static fields (unless they are marked with [NonIsolated] attribute)

parameter/return type

For non-isolated method, tells the compiler that parameter/return value can reference an isolated heap object.

3. [NonIsolated] attribute applicable to a method, constructor, class, struct, field, property, parameter, return type.

Allows to opt-out certain elements, telling the compiler that they can belong to the managed heap only.

4. is_isolated() a static method that returns true when called in an isolated context.

Allows differentiating isolated and non-isolated scenarios.

---

To ensure memory safety:

The compiler needs to ensure that:

1. No reference is created from the managed heap to the isolated heap.
2. Every reference from the isolated heap to the managed heap is pinned for the lifetime of the isolation context.

So the compiler should prevent:

1. Assignment a = b if 'a' is a reference stored on the managed heap and 'b' can belong to the isolated heap
2. Passing isolated heap objects as non-isolated arguments (when a method/constructor or parameter is not marked with [Isolated] attribute)
3. References to the isolated heap from leaving the current thread (hence no async/awaits)

This can be relaxed for strings, and a copy on assignment can be done when a string from the isolated heap is assigned to a managed heap reference.

For example:

Static fields belong to the managed heap, so isolated objects cannot be assigned to static fields.
A reference a.b, where b is a field or property, is considered isolated if a is isolated (unless b was marked with [NonIsolated] attribute)

---

Possible implementation:

The compiler can generate two versions of each isolated method, one for isolated and another for non-isolated context.
The isolated version will have the heap object as an implicit argument and will emit pin statements for managed heap references. The heap object will keep a record of all pinned and finalizable objects to be unpinned/finalized when the isolation scope exits.

[canton7]

This is called an "Arena".

Can you clarify why you want this? "Avoiding garbage collection overheads" is very general: which specific overheads are you worried about, and how exactly will using an arena address them?

Allocating objects is basically free most of the time. However presumably a new block of memory is going to have to be allocated for your arena (perhaps several, given that it's going to have to dynamically expand), which won't be free.

Remember that the GC doesn't have to consider unreferenced objects when doing a collection. So if you're worried that allocating lots of objects and then releasing them all will put pressure on the GC, don't be.

If you're worried about the GC running during your computation, there's GC.TryStartNoGCRegion.

[DaZombieKiller]

You may be interested in this, which discusses a potential arena allocation API for .NET.
Also related: object-stack-allocation.md

[AartBluestoke]

if it was currently possible for the compiler to guarantee that

"No reference is created from the managed heap to the isolated heap.
Every reference from the isolated heap to the managed heap is pinned for the lifetime of the isolation context."

then the compiler / jit would be able to stack-allocate the objects, knowing they would be destroyed on function return. The problem is that it is "hard" to prove that no pointer escapes. ie, is it safe to call Console.Writeline("...",yourObject) in this context?, what about List.Add(yo)?, Math.Max(...)?

MyClass x=new x(y); // is y now captured past the lifetime of x in some static cache?

(my personal opinion is that that some type of intent attribute, plus a just-in-time allocation of a stack object to the heap if it escapes should be the best way to go)

[theunrepentantgeek]

There's significant work under way to do this at runtime, where it will be able to benefit all code without needing compiler tricks, new language keywords, or generating multiple copies of each method.

See https://github.com/dotnet/coreclr/blob/master/Documentation/design-docs/object-stack-allocation.md

[svick]

@AartBluestoke

if it was currently possible for the compiler to guarantee that

"No reference is created from the managed heap to the isolated heap.
Every reference from the isolated heap to the managed heap is pinned for the lifetime of the isolation context."

then the compiler / jit would be able to stack-allocate the objects, knowing they would be destroyed on function return.

Not necessarily, if the function that does the allocating is not the same as the one that guarantees that the object won't be referenced anymore.

Consider this code:

void F()
{
    var list = new List<int>(capacity: 0);
    list.Add(42);
    return;
}

Here, the call to list.Add allocates, but that allocation would have to be performed from the stack of F, which is in general not possible.

[savahmel]

@canton7

Thanks for pointing out the arena term, updated the title.

Can you clarify why you want this? "Avoiding garbage collection overheads" is very general: which specific overheads are you worried about, and how exactly will using an arena address them?

Not sure I understand your question. Using arena improves performance by reducing the pressure on the GC. A scenario where this can be useful is processing of web requests that involves deserilization for example.

Allocating objects is basically free most of the time. However presumably a new block of memory is going to have to be allocated for your arena (perhaps several, given that it's going to have to dynamically expand), which won't be free.

Memory can be reserved for the heap and maximum heap size can be specified.
For example the following syntax can be supported:

void Foo() { 
     var commit = 0xFFFF;
     var reserve = 0xFFFFF;
     isolated commit : reserve { // throws if memory cannot be committed or reserved 
	    new Bar().IsolatedMethod(); 
     } 
}

[canton7]

@savahmel The costs associated with the GC are generally not intuitive. It's not as simple as "an arena allocator will speed things up": as I said, most heap allocations are already extremely cheap, and the cost of doing a GC does not (generally) depend on the number of objects which die during that collection.

So your deserialization is probably exerting very little pressure on the GC anyway.

I think you need to show that an arena is actually faster than the GC, for some use-cases. An example using some benchmarks would be ideal, but an argument based on what exact GC costs you're hoping to reduce, and how, would be helpful.

[savahmel]

@AartBluestoke

if it was currently possible for the compiler to guarantee that

"No reference is created from the managed heap to the isolated heap.
Every reference from the isolated heap to the managed heap is pinned for the lifetime of the isolation context."

then the compiler / jit would be able to stack-allocate the objects, knowing they would be destroyed on function return. The problem is that it is "hard" to prove that no pointer escapes. ie, is it safe to call Console.Writeline("...",yourObject) in this context?, what about List.Add(yo)?, Math.Max(...)?

MyClass x=new x(y); // is y now captured past the lifetime of x in some static cache?

(my personal opinion is that that some type of intent attribute, plus a just-in-time allocation of a stack object to the heap if it escapes should be the best way to go)

Current proposal extends the language with attributes that would allow the compiler to guarantee that no reference from managed heap points to the private heap object in a trivial way, by declaration.

For example:

class A {
    public static object a; 
}
[Isolated]
class B { 
    public void Foo() {
        a = new A(); // compiler error, since B is an isolated class, method Foo is isolated, that implies that it can be used from isolated context and therefore A can be allocated on the private heap, so it cannot be assigned to a static field that belongs to the managed heap. 
    }
}

With respect to your examples, if this feature will be supported, some framework classes/methods/parameters will be decorated with [Isolated] attributes and some will not.

Those who are decorated will accept objects from the isolated heap as arguments, the rest won't.

Since Console.Writeline has no reason to keep the reference for the provided argument, either the method or the argument will have [Isolated] attribute applied to it and therefore it will be possible to use this function with objects from isolated heap.

List.Add depends on the reference to the List object. Assuming that list class is marked with [Isolated] attribute (based on the current implementation of List container I don't see why it can't be)

[Isolated]
class A { 
}

[Isolated] 
void Foo([NonIsolated] List<A> managed_heap_list, List<A> other_list) { 
     var isolated_heap_list = new List<A>(); // allocated on isolated heap
     isolated_heap_list.AddRange(managed_heap_list); // compiler error
     isolated_heap_list.AddRange(other_list); // ok, by default arguments of isolated method are assumed to belong to isolated heap
}

Math.Max(...) - all overloads accept value types, we are concerned with references only.

MyClass x=new MyClass(y); // is y now captured past the lifetime of x in some static cache?

If the class/the constructor/the argument of the constructor of MyClass is marked with [Isolated] attribute, then it is safe to pass "y" that was allocated inside [isolated] method to it. Otherwise, it will result in a compilation error.

In general, you will not be able to pass an object allocated in an isolated method to a method that is not aware of the feature.

[HaloFour]

Current proposal extends the language with attributes that would allow the compiler to guarantee that no reference from managed heap points to the private heap object in a trivial way, by declaration.

That sounds like it would make the feature significantly less useful as the types you could actually allocate in this arena would be exceptionally limited and you'd have nearly zero access to runtime types. It sounds like you'd frequently find yourself in the situation of not being able to use the arena due to the types you need not being "isolated".

It sounds like the Arena API that has already been prototyped did not require language changes. Given that it was in some kind of state where it could be discussed and benchmarked it sounds like it's probably the best place to look for answers in this field. Any language change couldn't exist without the corresponding runtime change, and if it could be surfaced as an API instead of as a language change that would be significantly better for the .NET ecosystem.

Also, it'd be interesting to compare to such an API to escape analysis and stack allocation which is currently being worked on by the runtime team. That feature should require zero changes to applications in order to take advantage.

[savahmel]

@canton7
I believe justification for arena-based allocation would be the same as for the object stack-based allocation feature.

[savahmel]

@HaloFour

The current proposal assumes that framework classes will apply the isolation attribute whenever possible and reasonable to do (at the very least on collections).

From what I understood from the link provided above, referring to the prototyped arena API, when the arena is disposed of, the GC will be triggered and reachable objects will be copied to the managed heap. The current proposal does not involve triggering GC or copying.

Avoiding changes to the language is an advantage from my perspective as well.

[savahmel]

@theunrepentantgeek

There's significant work under way to do this at runtime, where it will be able to benefit all code without needing compiler tricks, new language keywords, or generating multiple copies of each method.

See https://github.com/dotnet/coreclr/blob/master/Documentation/design-docs/object-stack-allocation.md

Stack allocation feature is addressing small objects. Current proposal does not have heap or object size limitations.

Stack allocation does not support objects with finalizers, current proposal does.

Additional limitation which does not apply to the current proposal:

Objects allocated in a loop can be stack allocated only if the allocation doesn't escape the iteration of the loop in which it is allocated. Such analysis is complicated and is beyond the scope of at least the initial implementation.

[HaloFour]

@savahmel

IMO, a proposal like this is better taken up on Runtime repo. What you're really proposing is an alternative to the prototyped Arena API. That functionality/API would need to exist in the runtime before any language proposals could target it. That group would likely also be better equipped to discuss the current and future plans for both Arena and stack allocation.

[YairHalberstadt]

@jaredpar wrote a blog post recently explaining why supporting borrowing in C# isn't worth it. His explanation applies quite strongly here too:

https://blog.paranoidcoding.com/2019/12/02/borrowing.html

[savahmel]

@jaredpar wrote a blog post recently explaining why supporting borrowing in C# isn't worth it. His explanation applies quite strongly here too:

https://blog.paranoidcoding.com/2019/12/02/borrowing.html

Thanks for the article, but I would disagree that current proposal suffers from the same limitations that the borrowing proposal implies.

While borrowing affects the type system, current proposal doesn't.
Adding [Isoalted] attributes to the existing code (also framework classes) does not break its compatibility. An "Isolated" method can be called form non-isolated context and will behave exactly as before.

[YairHalberstadt]

@savahmel

The problem is that you can't add it to virtual methods, as that would break any implementors.

[savahmel]

@YairHalberstadt

The problem is that you can't add it to virtual methods, as that would break any implementors.

You're right, however, nothing stops you from using non-isolated objects or virtual methods within an isolated context, as long as isolated heap objects are not passed as arguments.

Under an assumption, that you can still benefit from the feature by using the private heap for your domain objects + containers and the rest on the managed heap, the real limitation, in this case, would be:

calling framework virtual functions with your domain types as arguments

framework container types have mostly non-virtual methods, they can be isolated and used on the private heap.

I agree though, that the feature imposes substantial limitations on your domain types as well (not using interfaces or virtual methods). But it is not a general-purpose feature in the first place and should only be used in performance-critical cases, where sacrifices are not uncommon.