Pipeline 2,0 #497

marcinszkudlinski · 2024-05-31T12:32:16Z

DO NOT MERGE - DISCUSSION ONLY

After long time I could proceed with Pipeline 2.0 design

To make tracking changes and discussions easier, I translated the initial discussion to .rst format an put it on my private branch

Initial headsup:
thesofproject/sof#7261

discussion and 1st version of the document
https://github.com/orgs/thesofproject/discussions/8645

current version of the document:

https://marcinszkudlinski.github.io/sof-docs/PAGES/architectures/firmware/sof-common/pipeline_2_0/pipeline2_0_discussion.html

the above HTML is a compilation of this PR

please - read it and comment in this PR. I will add all changes as commits to this branch so we'll be able to track changes

gbernatxintel · 2024-06-27T13:23:32Z

Please include fast mode task in implementation Pipeline 2.0

marcinszkudlinski · 2024-07-09T10:38:17Z

sink/src APIs should have states. Based on a state, module would know if the sink/source has already been configured and if it is allowed to process data from if / put result to it

lgirdwood · 2024-09-13T15:49:02Z

@marcinszkudlinski I think pipeline 2.0 has made some recent progress upstream, do you think we will be able to target this PR in Q4 for merge or do you still have many code PRs pending ?

marcinszkudlinski · 2024-09-16T06:43:40Z

@lgirdwood
The main problem is changing all components/modules API to sink/src

currently the modules don't use any API for interaction with buffers, that makes any modifications extremely hard. The modules are bound hard to comp_buffer and some internal pipeline structures
I don't think we can target P2.0 for Q4, unless we'll find some resources to deal with the modules.

marcinszkudlinski · 2024-11-06T09:13:38Z

TODO: add probe support to generic sink/src - this should allow flawless work of probes at any point of pipeline

This commit contains a description of a pipeline2.0 design Signed-off-by: Marcin Szkudlinski <[email protected]>

marcinszkudlinski · 2025-09-04T12:16:10Z

RFC changed to a commit - please proceed with merge

lgirdwood

@marcinszkudlinski LGTM. I've spotted one spelling, will see if I can add co-pilot for review to pick up other spellings.

lgirdwood · 2025-09-17T09:52:08Z

architectures/firmware/sof-common/pipeline_2_0/pipeline2_0.rst

+Note that modules have access to sink or source API only, modules cannot use audio_buffer API because (as described) there are more types of data sources / data recievers than buffers. That generates a requirement: sink/source API must 
+provide all required data/functions for a module to configure and use data source. For performance reasons there may be neccessary keeping a shortcuts to certains strucures, like audio data parameters, in all APIs.
+
+Also the pipeline code must maitain buffers using audio_buffer API only, there must not be any "side calls" to buffers code. 


Copilot

Pull Request Overview

This PR introduces Pipeline 2.0 design documentation for the SOF (Sound Open Firmware) project. The proposal aims to make the audio pipeline more flexible, optimal, multi-core friendly, and better suited for IPC4 while maintaining IPC3 backward compatibility.

Introduces new scheduling types (LL and DP) with cross-core capabilities
Proposes a new sink/source API for flexible data flow between modules
Defines multiple buffer types optimized for different use cases

Reviewed Changes

Copilot reviewed 12 out of 12 changed files in this pull request and generated 10 comments.

File	Description
`architectures/firmware/sof-common/pipeline_2_0/pipeline2_0.rst`	Main document describing Pipeline 2.0 architecture, scheduling, APIs, and buffer management
`architectures/firmware/sof-common/pipeline_2_0/index.rst`	Index file for Pipeline 2.0 documentation
`architectures/firmware/sof-common/index.rst`	Updated to include Pipeline 2.0 documentation
Multiple `.pu` files	PlantUML diagram files illustrating various pipeline connection patterns

Comments suppressed due to low confidence (1)

architectures/firmware/sof-common/pipeline_2_0/images/shaerd_buffer_1.pu:1

The filename contains a typo: 'shaerd_buffer_1.pu' should be 'shared_buffer_1.pu' to match the intended 'shared buffer' terminology used in the document.

left to right direction

_{Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.}

Copilot · 2025-09-17T09:56:14Z

architectures/firmware/sof-common/pipeline_2_0/pipeline2_0.rst

+The purpose of the changes is to make the pipeline:
+
+ -  more flexible
+ -  more optimal 


Remove the trailing space after 'optimal'.

Suggested change

- more optimal

- more optimal

Copilot · 2025-09-17T09:56:15Z

architectures/firmware/sof-common/pipeline_2_0/pipeline2_0.rst

+
+LL5, LL6, LL1, LL2, LL3, LL4
+
+Agin, LL3 is able to work having data on both inputs


Typo: 'Agin' should be 'Again'.

Suggested change

Agin, LL3 is able to work having data on both inputs

Again, LL3 is able to work having data on both inputs

Copilot · 2025-09-17T09:56:15Z

architectures/firmware/sof-common/pipeline_2_0/pipeline2_0.rst

+
+LL5, LL6, LL1, LL2, LL3, LL4
+
+Agin, LL3 is able to work having data on both inputs


Typo: 'Agin' should be 'Again'.

Suggested change

Agin, LL3 is able to work having data on both inputs

Again, LL3 is able to work having data on both inputs

Copilot · 2025-09-17T09:56:15Z

architectures/firmware/sof-common/pipeline_2_0/pipeline2_0.rst

+
+Backward compatibility: legacy recurrency based algorithm should be kept in case of IPC3 and be used for creating module iteration list.
+
+DP modules are also a part of pipelies, but they are not a part of LL scheduling. That means DP modules data processing is excluded from module iteration list. DP, however, is still a part of a pipeline, so it should be included in pipeline iteration list for all other operations, like pipeline start/stop etc.


Typo: 'pipelies' should be 'pipelines'.

Suggested change

DP modules are also a part of pipelies, but they are not a part of LL scheduling. That means DP modules data processing is excluded from module iteration list. DP, however, is still a part of a pipeline, so it should be included in pipeline iteration list for all other operations, like pipeline start/stop etc.

DP modules are also a part of pipelines, but they are not a part of LL scheduling. That means DP modules data processing is excluded from module iteration list. DP, however, is still a part of a pipeline, so it should be included in pipeline iteration list for all other operations, like pipeline start/stop etc.

Copilot · 2025-09-17T09:56:16Z

architectures/firmware/sof-common/pipeline_2_0/pipeline2_0.rst

+=======================
+
+*Although SOF is written in C, not C++, lets use C++ naming conventions for classes and objects*
+Current pipeline processing is based on comp_buffer structure, wchich is providing much more than just buffering. It is a base for module binding, iteration etc. This is a source of


Typo: 'wchich' should be 'which'.

Suggested change

Current pipeline processing is based on comp_buffer structure, wchich is providing much more than just buffering. It is a base for module binding, iteration etc. This is a source of

Current pipeline processing is based on comp_buffer structure, which is providing much more than just buffering. It is a base for module binding, iteration etc. This is a source of

Copilot · 2025-09-17T09:56:16Z

architectures/firmware/sof-common/pipeline_2_0/pipeline2_0.rst

+
+(1.0 compatibility) before 2.0 is implemented, some features for module binding (ilke "is_walked" flag, etc.) must be also put into audio_buffer API. 
+
+Note that modules have access to sink or source API only, modules cannot use audio_buffer API because (as described) there are more types of data sources / data recievers than buffers. That generates a requirement: sink/source API must 


Typo: 'recievers' should be 'receivers'.

Suggested change

Note that modules have access to sink or source API only, modules cannot use audio_buffer API because (as described) there are more types of data sources / data recievers than buffers. That generates a requirement: sink/source API must

Note that modules have access to sink or source API only, modules cannot use audio_buffer API because (as described) there are more types of data sources / data receivers than buffers. That generates a requirement: sink/source API must

Copilot · 2025-09-17T09:56:17Z

architectures/firmware/sof-common/pipeline_2_0/pipeline2_0.rst

+(1.0 compatibility) before 2.0 is implemented, some features for module binding (ilke "is_walked" flag, etc.) must be also put into audio_buffer API. 
+
+Note that modules have access to sink or source API only, modules cannot use audio_buffer API because (as described) there are more types of data sources / data recievers than buffers. That generates a requirement: sink/source API must 
+provide all required data/functions for a module to configure and use data source. For performance reasons there may be neccessary keeping a shortcuts to certains strucures, like audio data parameters, in all APIs.


Multiple typos: 'neccessary' should be 'necessary', 'a shortcuts' should be 'shortcuts', 'certains strucures' should be 'certain structures'.

Suggested change

provide all required data/functions for a module to configure and use data source. For performance reasons there may be neccessary keeping a shortcuts to certains strucures, like audio data parameters, in all APIs.

provide all required data/functions for a module to configure and use data source. For performance reasons there may be necessary keeping shortcuts to certain structures, like audio data parameters, in all APIs.

Copilot · 2025-09-17T09:56:17Z

architectures/firmware/sof-common/pipeline_2_0/pipeline2_0.rst

+Note that modules have access to sink or source API only, modules cannot use audio_buffer API because (as described) there are more types of data sources / data recievers than buffers. That generates a requirement: sink/source API must 
+provide all required data/functions for a module to configure and use data source. For performance reasons there may be neccessary keeping a shortcuts to certains strucures, like audio data parameters, in all APIs.
+
+Also the pipeline code must maitain buffers using audio_buffer API only, there must not be any "side calls" to buffers code. 


Typo: 'maitain' should be 'maintain'.

Suggested change

Also the pipeline code must maitain buffers using audio_buffer API only, there must not be any "side calls" to buffers code.

Also the pipeline code must maintain buffers using audio_buffer API only, there must not be any "side calls" to buffers code.

Copilot · 2025-09-17T09:56:17Z

architectures/firmware/sof-common/pipeline_2_0/pipeline2_0.rst

+In case of a typical LL pipeline, each of the modules is processing a complete set of data on its input and produce a complete set of data on output. That typically means 16 - 48 audio frames per LL cycle. The requirement is that the input buffer(s) is always drained completely (unless explicitely requested).
+In case of LL chain of modules within a single pipeline:
+
+.. uml:: images/shaerd_buffer_1.pu


Typo in filename: 'shaerd_buffer_1.pu' should be 'shared_buffer_1.pu'.

Suggested change

.. uml:: images/shaerd_buffer_1.pu

.. uml:: images/shared_buffer_1.pu

Copilot · 2025-09-17T09:56:18Z

architectures/firmware/sof-common/pipeline_2_0/pipeline2_0.rst

+
+*implentation details - TODO*
+
+The price of course will be some additional cycles for data copying, befefits - data linearity and less memory usage. Of course if a module request for renention is close to the buffer size, it makes no sense. In this case probably a "cross core lockless data buffer" or "cross core linearization data buffer" will be more optimal. It is up to the buffer factory to decide each time what kind of buffer is more optimal and should be created. 


Typo: 'befefits' should be 'benefits'.

Suggested change

The price of course will be some additional cycles for data copying, befefits - data linearity and less memory usage. Of course if a module request for renention is close to the buffer size, it makes no sense. In this case probably a "cross core lockless data buffer" or "cross core linearization data buffer" will be more optimal. It is up to the buffer factory to decide each time what kind of buffer is more optimal and should be created.

The price of course will be some additional cycles for data copying, benefits - data linearity and less memory usage. Of course if a module request for renention is close to the buffer size, it makes no sense. In this case probably a "cross core lockless data buffer" or "cross core linearization data buffer" will be more optimal. It is up to the buffer factory to decide each time what kind of buffer is more optimal and should be created.

marcinszkudlinski force-pushed the pipeline2_0 branch from 40b384e to 16b730d Compare June 6, 2024 09:46

marcinszkudlinski changed the title ~~TEST ONLY~~ [RFC] Pipeline 2.0 discussions Jun 6, 2024

marcinszkudlinski changed the title ~~[RFC] Pipeline 2.0 discussions~~ [DNM-RFC] Pipeline 2.0 discussions Jun 6, 2024

marcinszkudlinski marked this pull request as ready for review June 6, 2024 14:07

marcinszkudlinski requested review from lgirdwood, deb-intel, intelkevinputnam, mmaka1 and mwasko as code owners June 6, 2024 14:07

marc-hb marked this pull request as draft June 6, 2024 15:10

marcinszkudlinski mentioned this pull request Sep 16, 2024

Pipeline2.0: step3, make all components using API instead of direct access to bsink_list and bsource_list thesofproject/sof#9478

Merged

pipeline 2.0 desigh

d4515f4

This commit contains a description of a pipeline2.0 design Signed-off-by: Marcin Szkudlinski <[email protected]>

marcinszkudlinski force-pushed the pipeline2_0 branch from ea3b11f to d4515f4 Compare September 4, 2025 12:15

marcinszkudlinski changed the title ~~[DNM-RFC] Pipeline 2.0 discussions~~ Pipeline 2,0 Sep 4, 2025

marcinszkudlinski marked this pull request as ready for review September 4, 2025 12:36

lgirdwood approved these changes Sep 17, 2025

View reviewed changes

lgirdwood requested a review from Copilot September 17, 2025 09:53

Copilot AI reviewed Sep 17, 2025

View reviewed changes


		LL5, LL6, LL1, LL2, LL3, LL4

		Agin, LL3 is able to work having data on both inputs

	Agin, LL3 is able to work having data on both inputs
	Again, LL3 is able to work having data on both inputs


		Backward compatibility: legacy recurrency based algorithm should be kept in case of IPC3 and be used for creating module iteration list.

		DP modules are also a part of pipelies, but they are not a part of LL scheduling. That means DP modules data processing is excluded from module iteration list. DP, however, is still a part of a pipeline, so it should be included in pipeline iteration list for all other operations, like pipeline start/stop etc.

	DP modules are also a part of pipelies, but they are not a part of LL scheduling. That means DP modules data processing is excluded from module iteration list. DP, however, is still a part of a pipeline, so it should be included in pipeline iteration list for all other operations, like pipeline start/stop etc.
	DP modules are also a part of pipelines, but they are not a part of LL scheduling. That means DP modules data processing is excluded from module iteration list. DP, however, is still a part of a pipeline, so it should be included in pipeline iteration list for all other operations, like pipeline start/stop etc.

	Current pipeline processing is based on comp_buffer structure, wchich is providing much more than just buffering. It is a base for module binding, iteration etc. This is a source of
	Current pipeline processing is based on comp_buffer structure, which is providing much more than just buffering. It is a base for module binding, iteration etc. This is a source of


		(1.0 compatibility) before 2.0 is implemented, some features for module binding (ilke "is_walked" flag, etc.) must be also put into audio_buffer API.

		Note that modules have access to sink or source API only, modules cannot use audio_buffer API because (as described) there are more types of data sources / data recievers than buffers. That generates a requirement: sink/source API must

	provide all required data/functions for a module to configure and use data source. For performance reasons there may be neccessary keeping a shortcuts to certains strucures, like audio data parameters, in all APIs.
	provide all required data/functions for a module to configure and use data source. For performance reasons there may be necessary keeping shortcuts to certain structures, like audio data parameters, in all APIs.

	Also the pipeline code must maitain buffers using audio_buffer API only, there must not be any "side calls" to buffers code.
	Also the pipeline code must maintain buffers using audio_buffer API only, there must not be any "side calls" to buffers code.

	.. uml:: images/shaerd_buffer_1.pu
	.. uml:: images/shared_buffer_1.pu


		implentation details - TODO

		The price of course will be some additional cycles for data copying, befefits - data linearity and less memory usage. Of course if a module request for renention is close to the buffer size, it makes no sense. In this case probably a "cross core lockless data buffer" or "cross core linearization data buffer" will be more optimal. It is up to the buffer factory to decide each time what kind of buffer is more optimal and should be created.

Pipeline 2,0 #497

Are you sure you want to change the base?

Pipeline 2,0 #497

Uh oh!

Conversation

marcinszkudlinski commented May 31, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

gbernatxintel commented Jun 27, 2024

Uh oh!

marcinszkudlinski commented Jul 9, 2024

Uh oh!

lgirdwood commented Sep 13, 2024

Uh oh!

marcinszkudlinski commented Sep 16, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

marcinszkudlinski commented Nov 6, 2024

Uh oh!

marcinszkudlinski commented Sep 4, 2025

Uh oh!

lgirdwood left a comment

Choose a reason for hiding this comment

Uh oh!

lgirdwood Sep 17, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Reviewed Changes

Uh oh!

Copilot AI Sep 17, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Sep 17, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Sep 17, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Sep 17, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Sep 17, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Sep 17, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Sep 17, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Sep 17, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Sep 17, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Sep 17, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

marcinszkudlinski commented May 31, 2024 •

edited

Loading

marcinszkudlinski commented Sep 16, 2024 •

edited

Loading