Allow seperate memory region for code

[HU90m]

Since lowRISC/sonata-system#149, we have HyperRAM execution working on the sonata system, which is mapped to a separate memory region to SRAM. The CHERIoT RTOS test suite is passing when running from HyperRAM with lowRISC@5dad635 , but this is more of a temporary fix.

What do people think would be the best way to allow boards to be configured such that code can be stored in a non-contiguous memory region?

[davidchisnall]

I think there are two kinds of thing that we'd want to support in HyperRAM initially:

• Code.
• Pre-shared objects.

The latter is possibly easier, because the entries for it are generated in the build system and so extending the current logic that takes a size to (optionally) take a size and a location would be simple, and then they could be placed in different regions within the linker script.

Code is more interesting. I think we'd need to have a per-compartment and per-library location for code. I expect that the most common use case for HyperRAM will be exporting slow-path functions.

My suggestion would be:

• Extend the board description file to describe multiple memories.
• Do a first pass process of the linker script to insert replacement points for different memories with their different locations, then a second pass to do the current relocations, with compartment code and / or globals in different memories.
• Extend the firmware header to account for multiple regions.
• Extend the loader to derive PCC and CGP pointers from one or other of the headers.
• Provide a default memory and an attribute that you can set on compartments (which can be set from in the firmware target by looking up a dependent target) to define where it goes.

This may also need some changes to the linker to make sure that the generated JSON reports are sensible, but I don't think they should need to change.

[GregAC]

There's three aspects to this discussion I think are important to separate:

1. What lowRISC is doing short-term to enable code usage in HyperRAM on Sonata. Here I think we will go for a blunt 'all code in hyperram' approach (and whatever that has to drag along in terms of globals), this enables vastly more space for code which is a useful feature for the Sonata platform. We will be enabling the Ibex Icache to provide reasonable performance with this approach (which already looks to be working fine) plus some more hardware optimizations we have yet to work on. Potentially we'll provide a couple of Sonata system images, one with hyperram and one without so those who prefer pure SRAM can still use that option.

2. Does upstream want to support separated heap and instruction memory? As I understand it right now you just define an end for the heap and the start goes wherever your instruction and global storage ends and that's what our (in lowRISC CHERIoT-RTOS repo) hack works around.

3. An appropriate long-term solution for enabling disparate memories. What @davidchisnall suggests there all looks very sensible and I agree a per-compartments and per-library location makes sense. What would be nice would be a way to specify per-function locations as well that override the compartment/library default (a typical feature of other embedded platforms that may put code by default in flash but you can specify a function should go into SRAM instead). Obviously with compartmentalization this has complications but could we do something very similar to the existing shared library mechanism it's just only one compartment has access to the library? Then within a library it just gets auto split into two or more libraries one in location X (the default one) the others in location Y, Z etc (the per-function specified ones).

[davidchisnall]

What lowRISC is doing short-term to enable code usage in HyperRAM on Sonata.

Expedience is good here, though the more that we can align it with long-term goals, the better.

Does upstream want to support separated heap and instruction memory?

Yes, we mostly combined them for expediency. We expect to have three overlapping properties of memory:

• Can be used for instruction fetch.
• Holds tags
• Is covered by the revocation bitmap.

Code requires the first two. Globals require the middle one only. Heap requires the last two. My expectation was that we'd mostly have these as three overlapping regions, so an address map would contain instruction memory (loads and stores work, but stores may be slow) at the bottom, then global data ram (loads and stores work fine, memory here is not revokable, instruction fetch may be slow or may not work), then heap memory (revocation of this memory works). You can use heap memory or instruction memory for globals, with some performance degradation. You can't substitute anything for instruction memory or heap memory.

What would be nice would be a way to specify per-function locations

Yes, and agreed on the mechanism. Single-compartment libraries are the way that you'd express this in the ABI. This needs a bit of compiler work because the functions look like compartment functions on the inside and library calls on the outside, but all of the bits are there (except determining when you're in the same compartment / library). We might want to avoid making it look like a libcall though and have a single import entry for each compartment / library in a cluster that contained the PCC of that block of code, so that we could just add the immediate offset to the PCC and go there. Otherwise, outlining a function may require a load of import entries. Definitely some interesting longer-term work to do there.