iox-1002: Async API design draft #1005

pawelrutkaq · 2025-09-03T07:21:48Z

Notes for Reviewer

Pre-Review Checklist for the PR Author

Checklist for the PR Reviewer

Commits are properly organized and messages are according to the guideline
Unit tests have been written for new behavior
Public API is documented
PR title describes the changes

Post-review Checklist for the PR Author

All open points are addressed and tracked via issues

References

Closes #1002

Signed-off-by: Pawel Rutka <[email protected]>

pawelrutkaq · 2025-09-03T07:35:29Z

@elBoberido @elfenpiff here is PR for async api we dicussed. Matrix is dead for now so I ping You here

codecov · 2025-09-03T07:52:30Z

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 74.81%. Comparing base (1d3127a) to head (272f4d3).
⚠️ Report is 447 commits behind head on main.

❗ There is a different number of reports uploaded between BASE (1d3127a) and HEAD (272f4d3). Click for more details.

HEAD has 1 upload less than BASE

Flag BASE (1d3127a) HEAD (272f4d3)

1 0

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #1005      +/-   ##
==========================================
- Coverage   80.03%   74.81%   -5.22%     
==========================================
  Files         258      385     +127     
  Lines       31781    42299   +10518     
  Branches        0      360     +360     
==========================================
+ Hits        25437    31648    +6211     
- Misses       6344     7705    +1361     
- Partials        0     2946    +2946

Flag	Coverage Δ
Rust	`74.81% <ø> (?)`

Flags with carried forward coverage won't be shown. Click here to find out more.
see 227 files with indirect coverage changes

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

elfenpiff

@pawelrutkaq Thanks for writing up the async design document - it was already requested from the community right from the beginning and we are happy that you started to tackle the challenge - thank you!

I already read over it once but now I have to start sadly my meeting marathon - I left some initial questions.

doc/design-documents/drafts/async_api/async_api.md

pawelrutkaq · 2025-09-03T11:44:40Z

@elfenpiff requests are added

pawelrutkaq · 2025-09-05T09:39:27Z

@elBoberido @orecham kind remainder ;)

elBoberido

@pawelrutkaq thanks for bringing this forward and sorry for taking so long to review.

For me it's still not clear what changes in iceroyx2 would be required to implement the design. Partly also because I'm not that familiar with async and therefore some things that might be obvious for you, are unclear to me.

From my current feeling, I'm not even sure we need a port factory with a create_async method. This is something that we won't need for the other language bindings. It might be sufficient to have an AsyncListener in the userland crates, which just consumes a Listener and offers the async API.

But in order to give a better response, I need a better picture of the proposed changes and where they need to be done in iceoryx2.

doc/design-documents/drafts/async_api/assets/event_example.rs

elBoberido · 2025-09-17T22:33:19Z

doc/design-documents/drafts/async_api/assets/event_example.rs

+    Service: service::Service,
+    <Service::Event as iceoryx2_cal::event::Event>::Listener: FileDescriptorBased,
+{
+    pub(crate) fn from(listener: iceoryx2::port::listener::Listener<Service>) -> Result<Self, CommonErrors> {


Out of curiosity, would it make sense to implement the TryFrom trait here? If not, it might be a good idea to call this function try_from, since it has the same signature like the one from the trait.

one of two you said makse sense. Since it's example I dont correct it for now

@pawelrutkaq We have iceoryx2 users that use semaphores instead of unix domain sockets for signaling mechanism - since unix domain sockets are non-certifiable and not allowed in mission-critical contexts.

As you see here, the Listener implements only file_descriptor() when the underlying concept is file descriptor based: https://github.com/eclipse-iceoryx/iceoryx2/blob/main/iceoryx2/src/port/listener.rs#L128

When a user now uses the Semaphore variant, async cannot be implemented since Semaphores alone do not support event multiplexing.

Currently, we already have the issue, that not all iceoryx2 code compiles (waitset breaks) when the semaphore configuration is used. But we must avoid adding more code that has the same issue.

One way of addressing this would be to introduce another service variant called ipc_async_threadsafe and define the event: https://github.com/eclipse-iceoryx/iceoryx2/blob/main/iceoryx2/src/service/ipc.rs#L53 as socket based.

But this will wonderfully clash with the C, C++ and Python language bindings - so it would be a service variant that is not cross-language compatible.

@elBoberido what is your opinion here.

The Semaphores are not events. So in case user have system where we dont have proper notification backend (hope we dont dicuss here QNX ;)) then the only thing user can do is polling or ? In that case, using impl for specific trait (Event - FileDescriptor or Event - Semaphore) we can implement different stategy. For semaphore it would be simply timeout based polling.

For Semaphores we could implement a WaitSet that works with them and for async, a separate thread could be spawned to wait on this WaitSet and the use e.g. EventFDs to wake the async runtime.

Regarding the additional service variant. I think we need a solution that scales better than what we have today. But that is not a simple task and needs some investment.

Makse no sense really. Waitset is doing sleep (or what have you meant?) and will add new thread. The runtimes can do the same with one liner.

Also the runtimes has own IO multiplexers, so adding new one in seperate thread witll normally cause multiple thread hops to process a change instead processing in place by runtime. So I would not do it as default solution.

As we dicussed today: We can write specialized handling of API when events do use Semaphore or so. And the other option is what @elBoberido mentioned, once custom waitset will be there. This would be up to Runtime to choose an integration way for this.

elBoberido · 2025-09-17T22:34:45Z

doc/design-documents/drafts/async_api/assets/event_example.rs

+
+    /// Async wait for a new [`EventId`]. On error it returns [`ListenerWaitError`] is returned which describes
+    /// the error in detail.
+    pub async fn wait_one(&self) -> Result<EventId, ListenerWaitError> {


Clarification: Do you expect that there is a one-to-one relation between sent samples and received events?

This is Event part. so here we have one to one realtions. However second part of your questions is more into pub-sub. Here our idea is:

first implemetation - yes, each new produced sample fires event

later stage (optimization when rolled out) - one can limit number of events by implemeting special checks in pub-sub internals where we can fire event only if we know async subscriber as now awaiting new sample.

Since my approach is to first do not touch iceoryx2 implemetation details.

@pawelrutkaq I just skimmed over the comments - so this comment might be misplaced.

first implemetation - yes, each new produced sample fires event

This is an implementation flaw of iceoryx2 that will be fixed. The issue is here: #925 and we are currently writing a design doc for it.

The event is in the and a switch that is triggered when a state changes. If the same state changes multiple times, the switch is only triggered once until the counterpart resets it by handling all events.

This means for the following sequence, that only one event is emitted.

Publish sample

Notify sample-published (emits event notification)

Publish sample

Notify sample-published (does nothing)

....

When the listener is activated and acquires all events then the event is reseted and the event notified once again.

The reason for this behavior change is, that it is essential in a mission-critical context that no notification is lost. This can be best realized with some form of bitset - not a queue with a limited buffer size.

This does not matter I think. So we are doing notify for each sample produced. Whether this send event, or already have send event does not matter, and is even better since we would send less events. At the end, some event was send to listener which will wake up and process all until it has no more events, correct? ;)
Speaking in code it means AsyncSubscriber doing receive().await:

check if there is sample, if there is, its done, next cal does same.

if no samples, wait for notification

repeat

Still i wonder what You will do, counter for each event value ? At the end, it will mean less OS calls -> better preformance in both async and non async I think.

As dicussed: No issue

doc/design-documents/drafts/async_api/assets/event_example.rs

elBoberido · 2025-09-18T00:44:25Z

doc/design-documents/drafts/async_api/async_api.md

+#### AsyncSubscriber - `messaging-patter == pubsub`
+
+Pure `async` implementation not using any specifics of `runtime` shall be
+doable. In case some unexpected dependency will be needed it has to be exposed
+over defined abstraction same as `AsyncListenerTrait`


I'm a bit confused. Further above, it is stated that pub-sub always needs to be combined with the event messaging pattern.

Yes, but once You have iml AsyncListenerTrait the rest is pure async (no runtime dependent). So assuming we have somehwere type X: AsyncListenerTrait, X is becoming given and we dont depend on runtime in implementation (we only depend indirectly on concreet X type ensuring it has out trait implemented).

Maybe I'm not getting the point. Do you propose to add an separate AsyncSubscriber which has an integrated Listener?

At the end yes, Subscriber needs event source so it would have Listener.,

As discussed. This would be part of the userland crates

doc/design-documents/drafts/async_api/async_api.md

elBoberido · 2025-09-18T00:52:35Z

doc/design-documents/drafts/async_api/async_api.md

+    .open_or_create()
+    .unwrap();
+
+  let notifier = event.notifier_builder().create_async().unwrap();


For me it is not clear why there should be an async notifier?

This can be skipped at first line since blocking is unlikelly. But to be 101% accurate, notifier is a write syscall that can block ;)

Actually, it is not. The underlying socket is configured with fcntl(fd, F_SETFL, O_NONBLOCK) and will return an errno when it would need to block.

Let's assume the notifier has 2 listeners and of one listener, the buffer is full, then the listener is skipped and not notified, the other listener will receive a notification. If the user want to handle this error, a degradation-callback can be registered which is called whenever a delivery problem is encountered.

This is also, as contract, verified in the concept test suite so you can therefore safely assume that this is always non-blocking - if its blocking it is a bug.

EWOULDBLOCK is normally translated to .await doing async waiting. Here, this can happen too. So this looks like it still shall be async API

As dicussed: We may not need to do anything as per mentioned guarantess it will not block and it will do correct action (after reimplementation). I would leave still open for later Whether AsyncPublisher shall be created as proxy wrapper (or type alias) to Publisher to keep async API symetric only.

Signed-off-by: Pawel Rutka <[email protected]>

pawelrutkaq · 2025-09-18T06:59:17Z

@pawelrutkaq thanks for bringing this forward and sorry for taking so long to review.

For me it's still not clear what changes in iceroyx2 would be required to implement the design. Partly also because I'm not that familiar with async and therefore some things that might be obvious for you, are unclear to me.

From my current feeling, I'm not even sure we need a port factory with a create_async method. This is something that we won't need for the other language bindings. It might be sufficient to have an AsyncListener in the userland crates, which just consumes a Listener and offers the async API.

But in order to give a better response, I need a better picture of the proposed changes and where they need to be done in iceoryx2.

My assumption is that first API can be created without changing the code, only extending is needed.
I think in case of async it may be impossible to always have same API across languages since EACH language do this differently.
At the end, I would disagree with doing all that in userland since this will limit further performance improvements and will not keep it consistent imho - and no one wants iceoryx2-tokio API and iceoryx2-other_runtime API looking similar, but actually not.

elfenpiff

I think we should reduce the scope of this design document to the async listener and plan it. I added some more comments but I am optimistic that we can merge the document until the end of this week.

To add further async ports we need to discuss the iceoryx2-userland layer and how there data-flow and control-flow messaging patterns are merged together which would than naturally converge to an async API.

elfenpiff · 2025-10-06T06:16:54Z

doc/design-documents/drafts/async_api/assets/event_example.rs

+    Service: service::Service,
+    <Service::Event as iceoryx2_cal::event::Event>::Listener: FileDescriptorBased,
+{
+    pub(crate) fn from(listener: iceoryx2::port::listener::Listener<Service>) -> Result<Self, CommonErrors> {


@pawelrutkaq We have iceoryx2 users that use semaphores instead of unix domain sockets for signaling mechanism - since unix domain sockets are non-certifiable and not allowed in mission-critical contexts.

As you see here, the Listener implements only file_descriptor() when the underlying concept is file descriptor based: https://github.com/eclipse-iceoryx/iceoryx2/blob/main/iceoryx2/src/port/listener.rs#L128

When a user now uses the Semaphore variant, async cannot be implemented since Semaphores alone do not support event multiplexing.

Currently, we already have the issue, that not all iceoryx2 code compiles (waitset breaks) when the semaphore configuration is used. But we must avoid adding more code that has the same issue.

One way of addressing this would be to introduce another service variant called ipc_async_threadsafe and define the event: https://github.com/eclipse-iceoryx/iceoryx2/blob/main/iceoryx2/src/service/ipc.rs#L53 as socket based.

But this will wonderfully clash with the C, C++ and Python language bindings - so it would be a service variant that is not cross-language compatible.

@elBoberido what is your opinion here.

elfenpiff · 2025-10-06T06:20:51Z

doc/design-documents/drafts/async_api/assets/event_example.rs

+        loop {
+            match self.listener.try_wait_one() {
+                Ok(event) if event.is_some() => return Ok(Ok(event.unwrap())),
+                Ok(_) => {


So the assumption is that when try_wait_one() returns nothing that there was an error?

This assumption would be wrong. One-to-one relations between sample and event notification will soon no longer be possible and events will have switch semantics - otherwise we have no way of dealing with limited queue buffer sizes like every mechanism like unix domain sockets has.

Not sure what that mean, we need to dicuss.

As dicussed: Not a problem, some notification will happen underneath and this is enought to get waked in async. The implemented need to take care before waiting that it consumed all events/samples etc.

elfenpiff · 2025-10-06T06:22:54Z

doc/design-documents/drafts/async_api/assets/event_example.rs

+                Ok(event) if event.is_some() => return Ok(Ok(event.unwrap())),
+                Ok(_) => {
+                    // This is None, so there was and error, probably EAGAIN or EWOULDBLOCK
+                    if std::io::Error::last_os_error().kind() == std::io::ErrorKind::WouldBlock {


Please remove the whole error handling based on errno. This is broken since iceoryx2 resets the errno variable always to zero when it is handled - like the POSIX standard recommends.

This is esentially needed in FD case that we get EWouldBlock in any runtime. Otherwise we dont know when to wait. And You are not clearing this specific error ;) Anyway this information shall be preserved or we need side API.

We agreed to stick to a fact that try_receive will return Ok(None) only whne there would be EWOULDBLOCK error from os call in FD case.

As discussed: We may not need to do anything as per mentioned guarantee it will not block and it will do correct action (after reimplementation). I would leave still open for later Whether AsyncPublisher shall be created as proxy wrapper (or type alias) to Publisher to keep async API symetric only.

doc/design-documents/drafts/async_api/async_api.md

elfenpiff · 2025-10-06T06:42:42Z

doc/design-documents/drafts/async_api/async_api.md

+
+There are two approaches that can be chosen for implementation:
+
+#### 1. Use custom event for each pair (messaging patern, data type) based on `ServiceName`


Maybe I had more context last time, but I have currently a hard time understanding this. Let me translate this.

Create corresponding event-service for each data-flow service. So when we have a service my_pub_sub_service we create also a my_pub_sub_service_events.

Global event service, we use one event service like global_events and all data-flow services use that for notification.

If I am correct, could you please rewrite this a bit to make it more understandable.

Yes

No, but it's interesting what have you meant ;) Second is that we would use event in same service but this would mean that in that service You can only have single messaging pattern as async.
Added examples.

elfenpiff · 2025-10-06T07:00:48Z

doc/design-documents/drafts/async_api/async_api.md

+beneficial to either expose some properties (of current listener) or `add` new
+sync api with different signature.
+
+#### AsyncSubscriber - `messaging-patter == pubsub`


I would recommend removing AsyncSubscriber from the design document for now. I agree that we need some kind of async subscriber, and we have this on our radar.

This would be more like a "meta-port" instead of a subscriber located in iceoryx2-userland and it combines control-flow pattern (event) with data-flow pattern (pubsub, request-response, blackboard). It would have an async API and would enable the user to wait on a wide variety of events of the data-flow messaging pattern. But on the iceoryx2 layer the async API makes, in my opinion, only sense for the listener.

Another point is that we have to always consider the full iceoryx2 API. Even when you only require an async subscriber, still an async publisher, async client, async server, async reader (blackboard), async writer (blackboard) might be missing. They might wait until there is a subscriber, or a corresponding other part.

I think without AsyncSubscriber, the Listener is anyway implemented outside iceoryx2, so we could only do some minor things here. AsyncSubscriber would pave us a way to others. Once we are happy with implementation, we can add missing part of this messaging pattern and make it officially supported for Pub-Sub. Then, we can continue with others incrementally.

We agreed to do AsyncSubscriber in userland.

elBoberido

Just some small typos

doc/design-documents/drafts/async_api/async_api.md

elBoberido

Just a little bit of nitpicking. Looks good. Waiting for @elfenpiff for his final review.

elBoberido · 2025-10-13T17:15:55Z

doc/design-documents/drafts/async_api/async_api.md

+* limit a service to only single messaging pattern as using event will cause no
+  way to use it again
+
+Considering above, the continuation of the proposal is based on option 1. The


Considering option 1, one could also extend the Service Discovery with the service attributes. Please have a loot at https://github.com/eclipse-iceoryx/iceoryx2/tree/main/examples/rust/service_attributes

elBoberido · 2025-10-13T17:26:41Z

doc/design-documents/drafts/async_api/async_api.md

+* If unsuitable for zero-trust or real-time use, how do we prevent accidental
+  misuse?
+
+PR: To me idea, we will build async API on top of existing non-blocking, non


nitpicking: Just to have this a little bit more formal :)

Suggested change

PR: To me idea, we will build async API on top of existing non-blocking, non

PR: The author proposes to build the `async` API on top of existing non-blocking, non

elBoberido · 2025-10-13T17:27:23Z

doc/design-documents/drafts/async_api/async_api.md

+  open a way for other to implement a bridge to other runtime like `tokio` as
+  basic idea will be shown
+
+### Milestone 2 – Implement PubSub


Suggested change

### Milestone 2 – Implement PubSub

### Milestone 3 – Implement PubSub

pawelrutkaq · 2025-10-15T06:01:23Z

Just a little bit of nitpicking. Looks good. Waiting for @elfenpiff for his final review.

Kk, will fix it once @elfenpiff review

pawelrutkaq · 2025-10-30T15:41:53Z

@elfenpiff ?

WIP

99ddd82

Signed-off-by: Pawel Rutka <[email protected]>

pawelrutkaq force-pushed the iox2-1002-async-api-design branch 2 times, most recently from db32337 to ce027cb Compare September 3, 2025 07:30

pawelrutkaq changed the title ~~iox-1002: Async API design draft~~ iox-1002: Async API design Sep 3, 2025

pawelrutkaq changed the title ~~iox-1002: Async API design~~ iox-1002: Async API design draft Sep 3, 2025

elfenpiff reviewed Sep 3, 2025

View reviewed changes

doc/design-documents/drafts/async_api/async_api.md Show resolved Hide resolved

doc/design-documents/drafts/async_api/async_api.md Show resolved Hide resolved

elfenpiff reviewed Sep 3, 2025

View reviewed changes

doc/design-documents/drafts/async_api/async_api.md Show resolved Hide resolved

pawelrutkaq force-pushed the iox2-1002-async-api-design branch from 2d51086 to e48b7ae Compare September 3, 2025 11:44

elfenpiff requested review from elBoberido and orecham September 3, 2025 13:25

pawelrutkaq mentioned this pull request Sep 16, 2025

Sync for cw35-cw38 eclipse-score/orchestrator#31

Merged

10 tasks

elBoberido reviewed Sep 18, 2025

View reviewed changes

iox-1002: Async API design draft

24954ad

Signed-off-by: Pawel Rutka <[email protected]>

pawelrutkaq force-pushed the iox2-1002-async-api-design branch from e48b7ae to a92615e Compare September 18, 2025 06:55

iox-1002: Extend async documentation

c71ec25

pawelrutkaq force-pushed the iox2-1002-async-api-design branch from a92615e to c71ec25 Compare September 18, 2025 07:00

elfenpiff reviewed Oct 6, 2025

View reviewed changes

elBoberido reviewed Oct 6, 2025

View reviewed changes

doc/design-documents/drafts/async_api/async_api.md Outdated Show resolved Hide resolved

doc/design-documents/drafts/async_api/async_api.md Outdated Show resolved Hide resolved

elfenpiff reviewed Oct 6, 2025

View reviewed changes

doc/design-documents/drafts/async_api/async_api.md Show resolved Hide resolved

iox-1002: Addressing review comments

272f4d3

elBoberido reviewed Oct 13, 2025

View reviewed changes


		There are two approaches that can be chosen for implementation:

		#### 1. Use custom event for each pair (messaging patern, data type) based on `ServiceName`

	PR: To me idea, we will build async API on top of existing non-blocking, non
	PR: The author proposes to build the `async` API on top of existing non-blocking, non

	### Milestone 2 – Implement PubSub
	### Milestone 3 – Implement PubSub

iox-1002: Async API design draft #1005

Are you sure you want to change the base?

iox-1002: Async API design draft #1005

Uh oh!

Conversation

pawelrutkaq commented Sep 3, 2025

Notes for Reviewer

Pre-Review Checklist for the PR Author

Checklist for the PR Reviewer

Post-review Checklist for the PR Author

References

Uh oh!

pawelrutkaq commented Sep 3, 2025

Uh oh!

codecov bot commented Sep 3, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

elfenpiff left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

pawelrutkaq commented Sep 3, 2025

Uh oh!

pawelrutkaq commented Sep 5, 2025

Uh oh!

elBoberido left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

pawelrutkaq Oct 7, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

pawelrutkaq Oct 7, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

elfenpiff Oct 6, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

codecov bot commented Sep 3, 2025 •

edited

Loading

pawelrutkaq Oct 7, 2025 •

edited

Loading

pawelrutkaq Oct 7, 2025 •

edited

Loading

elfenpiff Oct 6, 2025 •

edited

Loading

pawelrutkaq Oct 6, 2025 •

edited

Loading

pawelrutkaq Oct 6, 2025 •

edited

Loading

pawelrutkaq Oct 7, 2025 •

edited

Loading