Using RPMsg between Linux and RTOS w/o the presence of Linux remoteproc

[foxhoundsk]

I am on ZynqMP SoC, in which APU runs Linux, RPU runs an RTOS. RPU does not use conventional firmware loading mechanism, which is remoteproc. Instead, the bootloader loads RPU firmware, so the RPU starts execution prior to Linux boot.

With this in mind, it seems that to use OpenAMP RPMsg service, userspace app in Linux shall interact (using ioctl() calls) with the RPMsg interfaces (pseudo files such as /sys/bus/rpmsg/*) in order to create /dev/rpmsgXXX file which enables the use of RPMsg in Linux userspace.

Also, as mentioned in #275 (reply in thread), OpenAMP in Linux can help with doing the ioctl() calls mentioned above. [1]

However, neither did I manage to find an example doing this in openamp-system-reference, nor in the OpenAMP document/source files.

Is it possible that I can use OpenAMP RPMsg in Linux userspace to communicate with RTOS running on RPU without the use of Linux remoteproc? [2]

Thanks!

[1] Plus, it was mentioned that Linux userspace app rpmsg_lite does it but I did not able to find the source file.
[2] The RPU runs mission-critical app and need to boot very quick, so I don't want Linux to have capability to do LCM (Life Cycle Management) against RPU, meaning Linux remoteproc should not present in the system.

[foxhoundsk]

Also in #275 (reply in thread), what is for sure is that without further works, RPMsg can't be used in Linux kernelspace as it relies on remoteproc to parse the resource table and create the VirtIO device accordingly, which enables the creation of /dev/rpmsgXX.

But I'm not sure the feasibility for it in Linux userspace.

[foxhoundsk]

If I understand RPMsg utility rpmsg_export_dev correctly, creation of the RPMsg device (aka channel) requires the presence of /dev/rpmsg_ctrlX, which in my case does not present due to the lack of remoteproc doing firmware loading.

So the question can now be narrowed down to: How to generate /dev/rmpsg_ctrlX in the absence of remoteproc?

[tnmysh]

Hello Thanks for your question.

Remoteproc is setting up vrings and shared memory address in the resource table, which is needed for RPU to create virtio devices. So yes, RPMsg can't be used without remoteproc.

RPMsg in userspace with open-amp library:
Alternatively you can build linux user-space app with open-amp library that uses shared-memory and IPI interrupt via UIO framework to communicate with RPU. As it's a mission critical application, I am not sure how comfortable you are to do this, but here is the discussion about it: OpenAMP/openamp-system-reference#63

As of now, remote core LCM is not optional if remoteproc is included in the kernel.

[foxhoundsk]

Thanks for the reply!

By mentioning how confortable I'm to use UIO, could you elaborate the implications? Does it incurs additional overhead or anything else?

[foxhoundsk]

Alternatively you can build linux user-space app with open-amp library that uses shared-memory and IPI interrupt via UIO framework to communicate with RPU

Is the shared memory you mentioned refer to using the /dev/mem file? Or UIO actually provide data exchange functionality? (I'm not familiar with UIO and currently doing study to get to know it.)

Thanks!

[tnmysh]

UIO is simple shared memory and interrupt register access directly from user-space. It's not the best way to access remote core as it has limitations like you can't actually multiplex multiple RPU core IPI to same IPI on linux side.

Also, Due to direct access of hardware from userspace, the security is also debatable. But, it's very simple and does not involve remote processor life cycle management from linux side.

Your use case is valid, and I think in kernel remoteproc, we should introduce a change that allows RPMsg use without life cycle management.

As of now, it's possible to do attach-detach operations in remoteproc, and establish RPMsg communication when Remote is already running before Linux. However, this doesn't prevent Linux from stopping remote processor i.e. completely powering off remote processor.

@arnopo , @wmamills we should discuss this use case in next openamp-rp call.

If you have any inputs please let us know.

[foxhoundsk]

Regarding the security considerations, I'll use ZynqMP built-in facilities such as XPPU and XPPU, which help invalidating the access should one occurs. So Linux theoretically has no capability to interfere the RPU.

To clarify the use of shared memory, instead of using UIO, are you suggesting doing mmap call with /dev/mem for the region of interest?

As a side note, I'll try using libmetal to manipulate the shared memory (mmap'd using /dev/mem) to do data exchange. Given that libmetal uses memory fence in a conservative way (strongest memory barrier), it is intriguing to see the throughput difference between the memory fence usages. I would share the result later in the libmetal repo.

Thanks!

[tnmysh]

To clarify the use of shared memory, instead of using UIO, are you suggesting doing mmap call with /dev/mem for the region of interest?

To be clear, I am not suggesting any such thing.

I am saying that using UIO is one of the options to achieve RPMsg communication if one must avoid using remoteproc.

Here is the documentation from Xilinx: https://xilinx-wiki.atlassian.net/wiki/spaces/A/pages/2423717889/RPMsg+in+Linux+and+OpenAMP+Project#RPMsg-in-Userspace

Here is the discussion that had happened before regarding how to use UIO and RPMsg in linux userspace: OpenAMP/openamp-system-reference#63

[tnmysh]

@foxhoundsk I proposed your use-case in the community call, and we think it is a valid use case.
Linux should be able to establish RPMsg communication without life-cycle management. I will be working on that implementation. Remoteproc will be still present in the system, but will be used only for RPMsg and not Life-cycle management.

[foxhoundsk]

@foxhoundsk I proposed your use-case in the community call, and we think it is a valid use case. Linux should be able to establish RPMsg communication without life-cycle management. I will be working on that implementation. Remoteproc will be still present in the system, but will be used only for RPMsg and not Life-cycle management.

Good to know that! If possible, please Cc. me in the pull request of this work. I would like to learn the internals. Thanks!

[tnmysh]

@foxhoundsk

Just to update you, I will not be working on this immediately. I will work on it, either end of this year or beginning of next year time frame.

If you prefer to implement this feature in remoteproc core framework in linux kernel, then following approach was agreed in the community call:

• Firmware resource table should have standard resource for remoteproc features
• Just like vdev resource: https://github.com/torvalds/linux/blob/347e9f5043c89695b01e66b3ed111755afcf1911/drivers/remoteproc/remoteproc_internal.h#L35
• Remoteproc core should parse this new features resource and decide if start/stop should be supported or not.
• If start/stop is not supported, and only attach-detach is supported, then Linux won't have capabilities to manage life-cycle of remote, but RPMsg communication will be still possible.

Thanks.