Package infrastructure (#7)

# Package infrastructure

- [x] Configuration Node for multiple package build backend
- [x] Package base class
- [ ] Implicit handling of kernel/runtime package 
- [x] Meson package support 
- [x] Cargo package support 

## Change in configuration
The followings nodes are renamed:
 - `sentry` --> `kernel`
- `libshield` --> `runtime` (will cover runtime for supported languages)
 - `app` --> `application`

The following properties are renamed:
- `exelist` --> `provides` (which may contain other compile product than
an executable)
 - `config_file` --> `config`
 - `deps` --> `depends`

Scm and git configuration are no longer flattened. each package has a
scm.<method>, only scm.git is supported yet.
Application has a build.backend entry with meson/cargo value. A generic
options list can be passed w/ build.options.<opt>, they can be either a
boolean, a string or a integer value.
There will be a set of "built-in" options derived according to the
backend. Others, will be translate as-is and pass to configure step.
e.g.:
```toml
build.options.with_doc = true
```
 -->
```meson
-Dwith_doc=true
```
### Configuration example

```toml
name = 'Sample project'
license = 'Apache-2.0'
license_file = ['LICENSE.txt']
dts = 'dts/sample_project.dts'
crossfile = 'cm33-none-eabi-gcc.ini'
version = 'v0.0.1'

[kernel]
scm.git.uri = 'https://github.com/outpost-os/sentry-kernel.git'
scm.git.revision = 'main'
config = 'configs/sentry/nucleo_u5a5.config'

[runtime]
scm.git.uri = 'https://github.com/outpost-os/libshield.git'
scm.git.revision = 'main'
config = 'configs/shield/shield.config'

[application.app1]
scm.git.uri = 'https://github.com/outpost-os/app1.git'
scm.git.revision = 'main'
config = 'configs/app1.config'
build.backend = 'meson'
depends = []
provides = ['app1.elf']

[application.app2]
scm.git.uri = 'https://github.com/outpost-os/app2.git'
scm.git.revision = 'main'
config = 'configs/app2.config'
build.backend = 'meson'
depends = []
provides = ['app2.elf']
```

## Package base class

Provide base package attribute and property as source directory, build
directory forge, config and dts handling, virtual methods for:
 - post download hook (e.g. meson subprojects download, cargo vendoring)
 - post update hook
- install hook (e.g. cargo install can't install in a sysroot w/ elf
suffix).
- classmethod setup hook (each package backend may need a dedicated step
before configure/compile, e.g. cargo need a config.toml to configure SDK
path, meson a cross file, cmake a toolchain file for toolchain/sysroot
configuration, etc.).

## Implicit kernel/runtime

Kernel and runtime build and/or scm config shall not appear at
integration level. A system integration should select an OS release
(usually the tuple kernel, runtime, compiler collection). This is not
supported yet but dependencies on kernel and runtime are implicit. In
future version, this will be packaged as sdk source, project will only
has to choose a version to use.

## Meson Package

- [x] Meson package class inherit from Package
- [x] Add option to enable/disable subprojects fetch/update (e.g.
download might be false by defaut, update shall be true)
- [ ] Impl virtual property for configure/compile/install args

## Cargo Package

Author: fvalette-ledger

doc/commands/internals/install.rst

@@ -0,0 +1,11 @@
+`install`
+==========
+
+.. argparse::
+   :module: outpost.barbican._internals.install
+   :func: argument_parser
+   :prog: barbican --internal install
+
+.. seealso::
+
+    :py:mod:`outpost.barbican._internals.install` module documentation

pyproject.toml

@@ -60,7 +60,7 @@ barbican = "outpost.barbican.barbican:main"
 [project.optional-dependencies]
 devel = [
     "flake8>=7.0,<8.0",
-    "mypy>=1.8.0",
+    "mypy>=1.12.0",
     "pytest>=4.6",
     "pytest-cov>=5.0.0",
     "pytest-dependency>=0.6",

src/outpost/barbican/_internals/install.py

@@ -0,0 +1,41 @@
+# SPDX-FileCopyrightText: 2024 Ledger SAS
+#
+# SPDX-License-Identifier: Apache-2.0
+
+from argparse import ArgumentParser, REMAINDER
+from pathlib import Path
+import shutil
+import typing as T
+
+from ..console import console
+
+
+def run_install(from_dir: Path, files: list[Path], suffix: str = ""):
+    for f in files:
+        src = (from_dir / f.name).resolve(strict=True)
+        dest = f.with_suffix(suffix)
+        if not dest.parent.exists():
+            dest.parent.mkdir(parents=True, exist_ok=True)
+        console.message(f"Installing [i]{str(src)}[/i]→ [i]{str(dest)}[/i]")
+        shutil.copy2(src, dest)
+
+
+def argument_parser() -> ArgumentParser:
+    parser = ArgumentParser()
+    parser.add_argument(
+        "-s",
+        "--suffix",
+        action="store",
+        type=str,
+        default="",
+        help="suffix to append to the installed file(s)",
+    )
+    parser.add_argument("from_dir", type=Path, help="directory from where files are installed")
+    parser.add_argument("files", nargs=REMAINDER, type=Path, help="file(s) install destination")
+
+    return parser
+
+
+def run(argv: T.List[str]) -> None:
+    args = argument_parser().parse_args(argv)
+    run_install(args.from_dir, args.files, args.suffix)

src/outpost/barbican/barbican.py

@@ -20,7 +20,10 @@
 from .console import console
 from .logger import logger, log_config
 from . import config
-from .package import Package
+from .package import Package, create_package, Backend
+from .package.meson import Meson
+from .package.cargo import Cargo
+
 from .buildsys import ninja_backend
 from .utils import pathhelper
 
@@ -55,14 +58,25 @@ def __init__(self, project_dir: pathlib.Path) -> None:
         # This will be, likely, false for next devel step.
 
         # Instantiate Sentry kernel
-        self._packages.append(Package("sentry", self, self._toml["sentry"]))
+        self._packages.append(
+            Meson("kernel", self, self._toml["kernel"], Package.Type.Kernel)  # type: ignore
+        )
         # Instantiate libshield
-        self._packages.append(Package("libshield", self, self._toml["libshield"]))
+        self._packages.append(
+            Meson(
+                "runtime",
+                self,
+                self._toml["runtime"],
+                Package.Type.Runtime,  # type: ignore[arg-type]
+            )
+        )
 
-        if "app" in self._toml:
+        if "application" in self._toml:
             self._noapp = False
-            for app, node in self._toml["app"].items():
-                self._packages.append(Package(app, self, node, is_app=True))
+            for app, node in self._toml["application"].items():
+                self._packages.append(
+                    create_package(app, self, node, Package.Type.Application)  # type: ignore
+                )
         else:
             self._noapp = True
 
@@ -98,10 +112,14 @@ def setup(self) -> None:
         )
 
         ninja.add_meson_rules()
+        ninja.add_cargo_rules()
 
         # Add setup/compile/install targets for meson packages
         for p in self._packages:
-            ninja.add_meson_package(p)
+            if isinstance(p, Meson):
+                ninja.add_meson_package(p)
+            elif isinstance(p, Cargo):
+                ninja.add_cargo_package(p)
 
         if self._noapp:
             ninja.close()
@@ -114,29 +132,32 @@ def setup(self) -> None:
 
         # linkerscript template file
         # XXX: hardcoded in early steps
-        linker_script_template = pathlib.Path(self._packages[1].data_dir) / "linkerscript.ld.in"
+        linker_script_template = (
+            pathlib.Path(self.path.sysroot_data_dir) / "shield" / "linkerscript.ld.in"
+        )
 
         dummy_linker_script = pathlib.Path(self.path.private_build_dir, "dummy.lds")
         ninja.add_gen_ldscript_target(
             "dummy", dummy_linker_script, linker_script_template, pathlib.Path(dummy_layout[0])
         )
 
-        # Dummy link
-        for app in self._packages[2:]:
-            ninja.add_relink_meson_target(
-                app.name,
-                app.installed_exelist[0],
-                app.dummy_linked_exelist[0],
-                dummy_linker_script,
-            )
+        # Dummy link, for non pic application
+        for package in self._packages:
+            if package.is_application and package.backend == Backend.Meson:
+                ninja.add_relink_meson_target(
+                    package.name,
+                    package.installed_targets[0],
+                    package.dummy_linked_targets[0],
+                    dummy_linker_script,
+                )
 
         layout_sys_exelist = []
         layout_app_exelist = []
         for package in self._packages:
             if package.is_sys_package:
-                layout_sys_exelist.extend(package.installed_exelist)
-            else:
-                layout_app_exelist.extend(package.dummy_linked_exelist)
+                layout_sys_exelist.extend(package.installed_targets)
+            elif package.backend == Backend.Meson:
+                layout_app_exelist.extend(package.dummy_linked_targets)
 
         firmware_layout = ninja.add_internal_gen_memory_layout_target(
             output=pathlib.Path(self.path.private_build_dir, "layout.json"),
@@ -153,10 +174,10 @@ def setup(self) -> None:
 
         # gen_ld/relink/gen_meta/objcopy app(s)
         for package in self._packages:
-            if package.is_app_package:
+            if package.is_application and package.backend == Backend.Meson:
                 # XXX: Handle multiple exe package
-                elf_in = package.installed_exelist[0]
-                elf_out = package.relocated_exelist[0]
+                elf_in = package.installed_targets[0]
+                elf_out = package.relocated_targets[0]
                 linker_script = pathlib.Path(self.path.private_build_dir, f"{elf_in.stem}.lds")
                 metadata_out = elf_out.with_suffix(".meta")
                 hex_out = elf_out.with_suffix(".hex")
@@ -185,12 +206,12 @@ def setup(self) -> None:
                 app_metadata.append(metadata_out)
 
         # Patch kernel/objcopy
-        kernel_elf = self._packages[0].installed_exelist[1]
-        kernel_patched_elf = self._packages[0].relocated_exelist[1]
+        kernel_elf = self._packages[0].installed_targets[1]
+        kernel_patched_elf = self._packages[0].relocated_targets[1]
         kernel_hex = kernel_patched_elf.with_suffix(".hex")
-        # idle_elf = self._packages[0].installed_exelist[0]
+        # idle_elf = self._packages[0].installed_targets[0]
         # XXX this is ugly (...)
-        idle_hex = self._packages[0].installed_exelist[0].with_suffix(".hex")
+        idle_hex = self._packages[0].installed_targets[0].with_suffix(".hex")
 
         ninja.add_fixup_kernel_rule(kernel_elf, kernel_patched_elf, app_metadata)
         ninja.add_objcopy_rule(kernel_patched_elf, kernel_hex, "ihex", [], self._packages[0].name)

src/outpost/barbican/buildsys/ninja_backend.py

@@ -139,7 +139,7 @@ def add_gen_ldscript_target(
         layout: Path,
         package_name: Optional[str] = None,
     ) -> None:
-        implicit_inputs = ["libshield_install.stamp"]
+        implicit_inputs = ["runtime_install.stamp"]
         if name != "dummy":
             implicit_inputs.append(f"{package_name if package_name else name}_install.stamp")
         self._ninja.newline()
@@ -271,6 +271,55 @@ def add_meson_rules(self) -> None:
             "touch $out",
         )
 
+    def add_cargo_rules(self) -> None:
+        self._ninja.newline()
+        self._ninja.variable("cargo", find_program("cargo"))
+        self._ninja.newline()
+        self._ninja.rule(
+            "cargo_compile",
+            description="cargo compile $name",
+            pool="console",
+            command="$cargo build -Z unstable-options --manifest-path=$sourcedir/Cargo.toml "
+            "--target-dir=$builddir --out-dir=$builddir && touch $out",
+        )
+        self._ninja.newline()
+        self._ninja.rule(
+            "cargo_install",
+            description="cargo install $name",
+            pool="console",
+            command="touch $out",
+        )
+
+    def add_cargo_package(self, package: "Package") -> None:
+        self._ninja.newline()
+        self._ninja.build(
+            f"{package.name}_compile.stamp",
+            "cargo_compile",
+            variables={
+                "sourcedir": package.src_dir,
+                "builddir": package.build_dir,
+                "name": package.name,
+            },
+        )
+        self._ninja.newline()
+        self._ninja.build(f"{package.name}_compile", "phony", f"{package.name}_compile.stamp")
+        self._ninja.newline()
+        self._ninja.build(
+            f"{package.name}_install.stamp",
+            "internal",
+            implicit=f"{package.name}_compile",
+            variables={
+                "cmd": "install",
+                "args": f"--suffix=.elf {str(package.build_dir)} "
+                + " ".join((str(t) for t in package.installed_targets)),  # noqa: W503
+                "description": f"cargo install {package.name}",
+            },
+        )
+
+        self._ninja.newline()
+        self._ninja.build(f"{package.name}_install", "phony", f"{package.name}_install.stamp")
+        self._ninja.newline()
+
     def add_meson_package(self, package: "Package") -> None:
         self._ninja.newline()
         self._ninja.build(
@@ -280,7 +329,7 @@ def add_meson_package(self, package: "Package") -> None:
                 "builddir": package.build_dir,
                 "sourcedir": package.src_dir,
                 "name": package.name,
-                "opts": package.build_opts,
+                "opts": package.build_options,
             },
             order_only=[f"{dep}_install.stamp" for dep in package.deps],
         )