Add multi-core IPC service

[carlocaione]

With #36120 now merged we can start discussing how to extend the IPC service to support multi-core (>2) IPC.

The idea is that the IPC Service backend implementation should be able to create several RPMsg instances based on DT information for IPC (shared memory regions, target CPU, IPM instances and mechanism, etc.. ) during system initialization. The API should allow the user to use any of the instances created without worrying about hardware details but only defining target_cpu and endpoint_name.

Tagging the relevant people: @maje-emb @hubertmis @carlescufi

[hubertmis]

I believe we would need to start with defining devicetree bindings describing IPC instances targeted to connect defined CPUs in the system. Based on such bindings we could create proper type in IPC Service API to define target_cpu.

I don't know if there are any works towards such devicetree bindings.

The main challenge I see is that devicetree entry for CPU A describing connection to CPU B would need to be a mirror of devicetree entry for CPU B describing connection to CPU A. It should be somehow handled by the scripts to ensure these entries are really mirrors.

[carlescufi]

The main challenge I see is that devicetree entry for CPU A describing connection to CPU B would need to be a mirror of devicetree entry for CPU B describing connection to CPU A. It should be somehow handled by the scripts to ensure these entries are really mirrors.

@mbolivar-nordic @galak ^^

[carlocaione]

Ok, let's try to put down some preliminary info about this.

The minimum set of information we need to define an IPC instance between two CPUs is:

• The shared memory region
• The IPM device

This is the information that we should gather from the DT. The target CPU is a piece of information that is not needed per se (because the shared memory region is enough to define the two CPUs involved in the communication) but it will probably be needed for the API implementation.

In the current codebase this information is carried by the DT using some chosen entries:

/ {
	chosen {
		/*
		 * shared memory reserved for the inter-processor communication
		 */
		zephyr,ipc_shm = &sramx1;
		zephyr,ipc = &mailbox0;
	};

	sramx1:memory@4000000{
		compatible = "mmio-sram";
		reg = <0x4000000 0x8000>;
	};
};

This of course doesn't scale at all with the number of CPUs involved, so this must be reworked.

I have two proposals on how to extend this to support multi-core IPC:

Proposal 1
The shared memory regions are defined as reserved-memory nodes (see #35460) and each IPC instance is defined using a different node. So for example this is the DT for the core 1 when two IPC instances are defined to communicate with cores 2 and 3.

/ {
	reserved-memory {
		compatible = "reserved-memory";
		...

		ipc_smh_core12: ipc_smh@42000000 {
			reg = <0x42000000 0x1000>;
		};

		ipc_smh_core13: ipc_smh@43000000 {
			reg = <0x43000000 0x1000>;
		};
	};

	mailbox_core12: mailbox@4008b000 {
		compatible = "mailbox";
		reg = <0x4008b000 0xEC>;
	};

	mailbox_core13: mailbox@4008c000 {
		compatible = "mailbox";
		reg = <0x4008c000 0xEC>;
	};

	ipc_core12 {
		compatible = "ipc";
		ipc_shm = <&ipc_smh_core12>;
		ipc = <&mailbox_core12>;
	};

	ipc_core13 {
		compatible = "ipc";
		ipc_shm = <&ipc_smh_core13>;
		ipc = <&mailbox_core13>;
	};
};

Proposal 2
This is similar to the proposal 1 but all the different IPC instances are gathered together under a single node. The IPC multi-instance driver will be in charge to create and instance for each node child.

/ {
	reserved-memory {
		compatible = "reserved-memory";
		...

		ipc_smh_core12: ipc_smh@42000000 {
			reg = <0x42000000 0x1000>;
		};

		ipc_smh_core13: ipc_smh@43000000 {
			reg = <0x43000000 0x1000>;
		};
	};

	mailbox_core12: mailbox@4008b000 {
		compatible = "mailbox";
		reg = <0x4008b000 0xEC>;
	};

	mailbox_core13: mailbox@4008c000 {
		compatible = "mailbox";
		reg = <0x4008c000 0xEC>;
	};

	ipc_core1 {
		compatible = "ipc_multi_instance";

		core2 {
			ipc_shm = <&ipc_smh_core12>;
			ipc = <&mailbox_core12>;
		};

		core3 {
			ipc_shm = <&ipc_smh_core13>;
			ipc = <&mailbox_core13>;
		};
	};
};

[carlocaione]

The main challenge I see is that devicetree entry for CPU A describing connection to CPU B would need to be a mirror of devicetree entry for CPU B describing connection to CPU A. It should be somehow handled by the scripts to ensure these entries are really mirrors.

I don't think we should take the burden of verifying this. As it's the user job to be sure to use the correct regions in the DT, it should be the user job to ensure that the shared memory region between two cores are allocated correctly and matching among different DT.

[hubertmis]

I like both proposals how to extend DT.

I would rename compatible = "ipc_multi_instance"; to something not implying software architecture (one could implement a software IPC service which can connect to multiple cores in single instance). ipc_multi_core or ipc_multi_target would be OK.

Also, I'm missing in both proposals information to which core given IPC is targeted. Let's imagine we have a system with Application CPU, Radio CPU, and DSP. Engineer crating Application CPU DT should create entries describing IPCs with both Radio and DSP. Engineer creating software for App CPU should not read DT to get information which entry is targeted to Radio and which to DSP. I think it should be clearly stated in DT in a way software can retrieve it. Is the label (ipc_core12, core2) the information that should identify target CPU for given IPC entry? Or should we add something more?

[carlocaione]

Is the label (ipc_core12, core2) the information that should identify target CPU for given IPC entry? Or should we add something more?

Yes, I think we should add a new property for that, something like target-name = <...> so that we can use that to target the destination core.

[carlocaione]

@hubertmis @maje-emb I wast starting to work on this but I think we need to clarify and discuss a bit the terminology because I think I'm confused about some choices done by the current code. Please, also refers to https://www.openampproject.org/docs/whitepapers/Introduction_to_OpenAMPlib_v1.1a.pdf to match my reasoning.

So, this is my interpretation of the things:

---

An RPMsg instance is a communication channel between two and only two cores. This is characterized and identified by a shared memory region, an IPM mechanism (or device) and a name. This information is found in the DT, so for example we can define an RPMsg instance with something like this:

ipc_core12 {
    compatible = "ipc-service,rpmsg-mi";
    target = "core2";
    ipc_shm = <&ipc_smh_core12>;
    ipc = <&mailbox_core12>;
};

Now, in theory we could have multiple RPMsg instances between the same two cores but I don't see how that could be useful and the current driver doesn't support that, so I'm going to assume that we have one and only one instance for each core-to-core communication (i.e. one instance for A<->B, one for A<->C, etc...).

The RPMsg instances are allocated at boot time reading all the info from the DT.

---

An RPMsg endpoint provides a software / logical connections on top of the channel / instance.

For example if we have an instance / channel between A and B, and we have two tasks running on A that need to communicate for different reasons with B, the two tasks can allocate two different endpoints in the same channel / instance. The endpoints allocation is done at run-time by the task: for example the task can request a new endpoint for the target core2 (see the previous DT snippet).

So basically an endpoint is always associated to one and only one instance, it's what the task is using to communicate with a remote task / processor.

---

Assuming we agree on the previous points, I don't fully understand why IPC must be treated as a subsystem and not a driver / API. An instance / channel is basically a device (with its own DT entry) that is providing a service (in the form of an endpoint) to a user. So my question is: why cannot simply introduce an IPC API backed by an IPC driver to which the user can submit requests such as: allocating an endpoint, sending data through that endpoint, etc...

I envision an API such as:

__subsystem struct ipc_driver_api {
    allocate_endpoint();
    remove_endpoint();
    send_through_endpoint();
};

Each backend is implementing this API on a different compatible and an application can use this API as usual. This approach is way easier and cleaner than introducing a new subsystem, backends, etc...

Am I missing something obvious in my reasoning?

[hubertmis]

I agree with your reasoning. With one exception:

we could have multiple RPMsg instances between the same two cores but I don't see how that could be useful and the current driver doesn't support that

I understood the point of @maje-emb's patch was exactly to support multiple RPMsg instances between single pair of cores. Each instance uses separated memory region and separated IPM. The reasoning was to assign different priority to each instance. Abstract priority value can be translated to hardware IRQ priority level of IPM handler, or to software task priority level of the thread processing given instance. I don't know if there is any real use-case to create multiple RPMsg instances with different priorities between two cores. From my point of view, we can simplify IPC solution assuming there is single RPMsg instance per single pair of cores.

[mbolivar-nordic]

@mbolivar-nordic @galak ^^

I didn't review the discussion in detail, but that seems to be one of the things that system devicetree is trying to solve. Absent a big change like that, I'm not sure we can enforce this with existing tooling any more than we can enforce consistent flash partition layouts between application and bootloader.

[carlocaione]

Some work about this going on at #37361