The 1st draft of score repository structure proposal #1475
Conversation
The 1st draft of S-CORE repository strcutre and CI/CD proposal Signed-off-by: Nikola Radakovic <[email protected]>
nradakovic
changed the title
|
The created documentation from the pull request is available at: docu-html |
|
Can we move that into the docu. Proposal as adr: #1408 |
There was a problem hiding this comment.
Can you, as experts in Bazel, consider sharing some lessons learned or best resources with SCORE community?
edit:
started discussion -> https://github.com/orgs/eclipse-score/discussions/1515
| Advanced features like toolchain extensions or module overrides enable fine-tuned control of the build graph without modifying upstream repos. | ||
|
|
||
| - 🔁 **Update via Tag or SHA** | ||
| Module versions are pinned to specific Git tags or release packages to ensure stability and auditability. |
There was a problem hiding this comment.
Can bazel version be somehow automated and connected with GH release? That would make it less misleading. See current process of updating bazel targets in tooling and bazel_registry
There was a problem hiding this comment.
I'm not sure if the highlighted reference here is correct. Bazel version will be set by the root module (reference integration). Each module can use version that they prefer as long as there are no combability issues between root version and module version. In the end, root module will override the Bazel version set in module .bazelversion file.
| The root repo defines Bazel modules using `bazel_dep()`, locking specific versions and handling updates in a controlled manner. This prevents accidental mismatches across modules. | ||
|
|
||
| - 🔐 **Version Locking & Isolation** | ||
| The configuration ensures that only compatible versions of modules are referenced together, improving reproducibility and avoiding dependency hell. |
There was a problem hiding this comment.
Please consider following situation:
ModuleA have releases: 1.0, 1.1, 1.2 in bazel registry
ModuleB have releases: 2.0, 2.1 in bazel registry
Is it possible for ModuleA that is dependent on ModuleB to create a release 2.2 that is still dependent on older release 1.1? Can version 2.2 still be integrated in root repository or it needs to internally bump dependency to 1.2 fist?
There was a problem hiding this comment.
Yes, it is possible for [email protected] to depend on [email protected] even if [email protected] is the latest version in the Bazel Registry.
And yes, [email protected] can still be integrated and tested in the root/reference repository without requiring an upgrade to [email protected], but we will probably enforce that new versions always depend on latest BR version.
What root module will do is that he will check out both versions in sandbox and use them. It's called Module Mapping.
There was a problem hiding this comment.
But down below there is written: however root repository will override all versioning down the tree. What would be the effects? Maybe this should be explained in more detail below?
There was a problem hiding this comment.
Thanks for this proposal: Just a view remarks:
- Shouldn't this be an rst file?
- So far we did not consider the integration into the a specific project. So there will be bazel modules, which need to be provided be the environment and need to be exchangeable. Don't know if we have a solution for this already e.g. via labelflags.
|
|
||
| To address the challenges of S-CORE project — such as slower development cycles, inefficient testing, and governance complexity — we propose a multirepo architecture built on Bazel build automation tool with central configuration repository. In this design, each functional unit resides in its own repository, enabling teams to work independently and iterate quickly. | ||
|
|
||
| A central configuration repository serves as the coordinator. Rather than containing application logic, it defines dependency relationships between modules and acts as the single source for system-wide integration. This setup ensures that only well-tested and validated module versions enter the production build context. |
There was a problem hiding this comment.
| A central configuration repository serves as the coordinator. Rather than containing application logic, it defines dependency relationships between modules and acts as the single source for system-wide integration. This setup ensures that only well-tested and validated module versions enter the production build context. | |
| A central integration repository serves as the coordinator. Rather than containing application logic, it defines, which modules incl. the specific module version becomes part of a S-CORE release build. This setup ensures that only well-tested and validated module versions enter the production build context. |
There was a problem hiding this comment.
Shouldn't this be an rst file?
- No, this was not supposed to be presented in this form but TL requested that we publish it anyway. Proper document still needs to be written.
So far we did not consider the integration into the a specific project. So there will be bazel modules, which need to be provided be the environment and need to be exchangeable. Don't know if we have a solution for this already e.g. via labelflags.
- I'm sorry, I don't really understand what is the question here.
There was a problem hiding this comment.
let's suppose that we do the reference integration on qemu against a sdp8 from qnx everywhere as described here. But in the series project we have a commercial license of qnx. The question is how do we support this use case where have to replace the bazel targets of the qnx everywhere with the ones from the commercial version.
But as said above we don't have to answer this right now.
There was a problem hiding this comment.
I would say you need an internal ci/cd anyway with different targets
| @@ -0,0 +1,176 @@ | |||
| ## 📘 Project Repository Structure Proposal | |||
There was a problem hiding this comment.
this should not be named "Repository Structure Proposal" IMHO
| The combination of modular development and centralized governance creates a scalable and maintainable system. Root-level CI with GitHub Actions further guarantees integration quality without slowing down individual module workflows. | ||
|
|
||
| - 🔲 **Scalability Through Modularity** | ||
| As project grow, tightly coupled monoliths slow down iteration. By segmenting the codebase into independent Bazel modules, each team can develop, test, and deploy in isolation—improving scalability and parallelism across the organization. |
There was a problem hiding this comment.
| As project grow, tightly coupled monoliths slow down iteration. By segmenting the codebase into independent Bazel modules, each team can develop, test, and deploy in isolation—improving scalability and parallelism across the organization. | |
| As projects grow, tightly coupled monoliths slow down iteration. By segmenting the codebase into independent Bazel modules, each team can develop, test, and deploy in isolation - improving scalability and parallelism across the organization. |
|
|
||
| The architecture uses one central repo for Bazel coordination and multiple surrounding module repos. Each module is a Bazel unit. The root repo runs integration builds/tests, ensuring system consistency without blocking individual module progress. | ||
|
|
||
| - ⚙️ **Central Config Repo** |
There was a problem hiding this comment.
Why not
| - ⚙️ **Central Config Repo** | |
| - ⚙️ **Central Release Repo** |
?
Since its only purpose is to collect modules in their respective version for each release, did I get this right?
| The configuration ensures that only compatible versions of modules are referenced together, improving reproducibility and avoiding dependency hell. | ||
|
|
||
| - 🔧 **Global Toolchains and Rules** | ||
| Bazel toolchains (e.g., Go, Python, C++) and common rules are configured in independent modules, but with pinpointed versions to ensuring consistency in how builds and tests are executed. |
There was a problem hiding this comment.
| Bazel toolchains (e.g., Go, Python, C++) and common rules are configured in independent modules, but with pinpointed versions to ensuring consistency in how builds and tests are executed. | |
| Bazel toolchains (e.g., Go, Python, C++) and common rules are configured in independent modules, but with pinpointed versions for ensuring consistency in how builds and tests are executed. |
| Modules run their own linting, testing, and coverage steps in isolation, ensuring correctness before integration into the system. | ||
|
|
||
| - 🧭 **Cross-Repo Dependencies** | ||
| Modules can depend on other modules however root repository will override all versioning down the tree. It's up to module owners to ensure the compatibility in overall system. |
There was a problem hiding this comment.
| Modules can depend on other modules however root repository will override all versioning down the tree. It's up to module owners to ensure the compatibility in overall system. | |
| In general, modules can depend on other modules. However, the root repository will override all versioning down the tree. It's up to module owners to ensure the compatibility within the overall system. |
| The root repo defines Bazel modules using `bazel_dep()`, locking specific versions and handling updates in a controlled manner. This prevents accidental mismatches across modules. | ||
|
|
||
| - 🔐 **Version Locking & Isolation** | ||
| The configuration ensures that only compatible versions of modules are referenced together, improving reproducibility and avoiding dependency hell. |
There was a problem hiding this comment.
But down below there is written: however root repository will override all versioning down the tree. What would be the effects? Maybe this should be explained in more detail below?
| Failures can be isolated to individual modules or to integration logic, improving debugging speed and root cause analysis. | ||
|
|
||
| - 🔄 **Upfront Overhead** | ||
| Requires investment in Bazel tooling (where BMW has already provided significant effort), developer training, and repository bootstrapping for new modules. |
There was a problem hiding this comment.
is the BMW statement really necessary here?
| Requires investment in Bazel tooling (where BMW has already provided significant effort), developer training, and repository bootstrapping for new modules. | ||
|
|
||
| - 🔗 **Tighter Coordination at Integration Points** | ||
| Integration still requires discipline and oversight to manage versions, rollouts, and test coverage. |
There was a problem hiding this comment.
| Integration still requires discipline and oversight to manage versions, rollouts, and test coverage. | |
| Integration still requires discipline and oversight to manage versions, rollouts and test coverage. |
| ### *Moving Forward with Confidence* | ||
|
|
||
| - 🔁 **Iterate and Test Gradually** | ||
| Start by migrating a few core components into Bazel modules to gain experience before scaling out. |
There was a problem hiding this comment.
| Start by migrating a few core components into Bazel modules to gain experience before scaling out. | |
| Start by migrating a few core components into Bazel modules to gain experience before full rollout. |
| Start by migrating a few core components into Bazel modules to gain experience before scaling out. | ||
|
|
||
| - 📌 **Lock Down Root Governance** | ||
| Enforce mandatory checks, code owners, and integration workflows in the root repository from day one. |
There was a problem hiding this comment.
| Enforce mandatory checks, code owners, and integration workflows in the root repository from day one. | |
| Enforce mandatory checks, code owners and integration workflows in the root repository from day one. |
| Create reusable templates and automation to help teams create compliant Bazel module repos quickly. | ||
|
|
||
| - 🏁 **Monitor and Optimize** | ||
| Use metrics like CI test pass rates, module update frequency, and build cache efficiency to guide ongoing improvement. |
There was a problem hiding this comment.
| Use metrics like CI test pass rates, module update frequency, and build cache efficiency to guide ongoing improvement. | |
| Use metrics like CI test pass rates, module update frequency and build cache efficiency to guide ongoing improvement. |
There was a problem hiding this comment.
I'm currently missing a description of the resulting workflows that this setup will cause.
For example:
- How do we announce upcoming breaking changes in the API of a module?
- What happens if a module has a breaking change between one version or the other?
- How will a developer see that he broke the integration with a change?
- How is decided which versions go into a new release of the root repo?
- ...
If you could describe these workflows and others you can think of, it would help tremendously to know what we will agree to with this proposal.
|
@nradakovic We had some integration ticket created by Anton. I cannot find it. Anyway i wanted to share this. Let's align. https://github.com/eclipse-score/docs-as-code/blob/integration.md/integration.md |
|
Looks also good from my side but coming from your proposal @AlexanderLanin https://github.com/eclipse-score/docs-as-code/blob/integration.md/integration.md there are also some differences. I would also say more alignment is needed to come to one "source of truth" document |
The 1st draft of S-CORE repository strcutre and CI/CD proposal