Add infrahubctl object command

Author: dgarros

.github/workflows/ci.yml

@@ -166,7 +166,7 @@ jobs:
           python-version: "3.12"
       - name: "Setup Python environment"
         run: |
-          pipx install poetry==1.8.5
+          pipx install poetry==2.1
           poetry config virtualenvs.create true --local
           poetry env use 3.12
       - name: "Install dependencies"
@@ -230,7 +230,7 @@ jobs:
           python-version: ${{ matrix.python-version }}
       - name: "Setup environment"
         run: |
-          pipx install poetry==1.8.5 --python python${{ matrix.python-version }}
+          pipx install poetry==2.1 --python python${{ matrix.python-version }}
           poetry config virtualenvs.create true --local
           pip install invoke toml codecov
       - name: "Install Package"
@@ -283,7 +283,7 @@ jobs:
           echo "PYTEST_DEBUG_TEMPROOT=/var/lib/github/${RUNNER_NAME}/_temp" >> $GITHUB_ENV
       - name: "Setup environment"
         run: |
-          pipx install poetry==1.8.5
+          pipx install poetry==2.1
           poetry config virtualenvs.create true --local
           pip install invoke toml codecov
       - name: "Install Package"
@@ -359,7 +359,7 @@ jobs:
 
   #     - name: "Setup environment"
   #       run: |
-  #         pipx install poetry==1.8.5
+  #         pipx install poetry==2.1
   #         poetry config virtualenvs.create true --local
   #         pip install invoke toml codecov
 

.gitignore

@@ -27,4 +27,5 @@ dist/*
 **/*.csv
 
 # Generated files
-generated/
+generated/
+sandbox/

.yamllint.yml

@@ -6,6 +6,7 @@ ignore: |
   /examples
   tests/unit/sdk/test_data/schema_encoding_error.yml
   /**/node_modules/**
+  tests/unit/sdk/test_data/multiple_files_valid_not_valid.yml
 
 rules:
   new-lines: disable

docs/docs/infrahubctl/infrahubctl-object.mdx

@@ -17,6 +17,7 @@ $ infrahubctl object [OPTIONS] COMMAND [ARGS]...
 **Commands**:
 
 * `load`: Load one or multiple objects files into...
+* `validate`: Validate one or multiple objects files.
 
 ## `infrahubctl object load`
 
@@ -38,3 +39,24 @@ $ infrahubctl object load [OPTIONS] PATHS...
 * `--branch TEXT`: Branch on which to load the objects.
 * `--config-file TEXT`: [env var: INFRAHUBCTL_CONFIG; default: infrahubctl.toml]
 * `--help`: Show this message and exit.
+
+## `infrahubctl object validate`
+
+Validate one or multiple objects files.
+
+**Usage**:
+
+```console
+$ infrahubctl object validate [OPTIONS] PATHS...
+```
+
+**Arguments**:
+
+* `PATHS...`: [required]
+
+**Options**:
+
+* `--debug / --no-debug`: [default: no-debug]
+* `--branch TEXT`: Branch on which to validate the objects.
+* `--config-file TEXT`: [env var: INFRAHUBCTL_CONFIG; default: infrahubctl.toml]
+* `--help`: Show this message and exit.

docs/docs/python-sdk/topics/object_file.mdx

@@ -0,0 +1,195 @@
+---
+title: Manage data with Object Files
+---
+
+# Manage data with Object files
+
+## Introduction
+
+An Object file is a YAML file that allows you to manage data to be loaded in Infrahub based on your own custom schema. It provides a declarative way to define and manage resources in your Infrahub instance.
+
+Object files work well for models that don't change too often and/or that need to be tracked in Git. Examples include: Groups, tags, Users, etc.
+Below is an example of an Object file that defines tags (`BuiltinTag`).
+
+```yaml
+---
+apiVersion: infrahub.app/v1
+kind: Object
+spec:
+  kind: BuiltinTag
+  data:
+    - name: Blue
+    - name: Yellow
+    - name: Red
+```
+
+Object files are meant to be used in an idempotent way and as such they work better for models with a Human Friendly ID (HFID) defined. An HFID is a unique identifier that makes it easier to reference objects across different files and operations.
+
+## Load Object files into Infrahub
+
+Object files can be loaded into Infrahub using the `infrahub object load` command.
+
+```bash
+infrahub object load <path_to_object_file>
+```
+
+Multiple object files can be loaded at once by specifying the path to multiple files or by specifying a directory.
+
+The `object load` command will create/update the objects using an `Upsert` operation. All objects previously loaded will NOT be deleted in the Infrahub instance.
+Also, if some objects present in different files are identical and dependent on each other, the `object load` command will NOT calculate the dependencies between the objects and as such it's the responsibility of the users to execute the command in the right order.
+
+### Validate the format of object files
+
+The object file can be validated using the `infrahub object validate` command.
+
+```bash
+infrahub object validate <path_to_object_file>
+```
+
+## Object file format
+
+All object files must start with the following format, all other formats will be automatically ignored.
+Each file is intended for one specific top level kind, but one file can include multiple nested objects of any kind.
+The kind of the top level object must be defined in spec/kind.
+
+```yaml
+---
+apiVersion: infrahub.app/v1
+kind: Object
+spec:
+  kind: <NamespaceName>
+  data:
+    - [...]
+```
+
+> Multiple documents in a single YAML file are also supported, each document will be loaded separately. Documents are separated by `---`
+
+### Relationship of cardinality one
+
+A relationship of cardinality one can either reference an existing node via its HFID or create a new node if it doesn't exist.
+In the example below, both `site` and `primary_ip` are relationships of cardinality one.
+
+```yaml
+---
+apiVersion: infrahub.app/v1
+kind: Object
+spec:
+  kind: InfraDevice
+  data:
+    - name: edge01
+      site: "Paris"         # Reference existing node via its HFID
+      primary_ip:           # Nested object, will be created if it doesn't exist
+        data:
+            address: "192.168.1.1"
+```
+
+### Relationship of cardinality many
+
+A relationship of cardinality many can reference existing nodes via their HFID or define nested objects.
+
+#### Existing nodes referenced by their HFID
+
+Existing nodes can be referenced by their HFID in string format or in list format.
+In the example below, both `best_friends` and `tags` are relationships of cardinality many.
+
+> An HFID is composed of a single value, it's possible to use a string instead of a list
+
+```yaml
+---
+apiVersion: infrahub.app/v1
+kind: Object
+spec:
+  kind: TestingPerson
+  data:
+    - name: Mike Johnson
+      height: 175
+      best_friends: # Relationship of cardinality many that references existing nodes based on their HFID
+        - [Jane Smith, Max]
+        - [Sarah Williams, Charlie]
+      tags:
+        - Veterinarian    # Existing Node referenced by its HFID in string format
+        - [Breeder]       # Existing Node referenced by its HFID in list format
+```
+
+#### Nested objects
+
+When defining nested objects, the node will be automatically created if it doesn't exist and if the relationship between the parent object and the nested object exists, it will be automatically inserted.
+For example, in the example below, the `owner` of a `TestingDog` doesn't need to be specified because it will be automatically inserted.
+
+Two different syntax are supported:
+
+- A dictionary with multiple values under data
+- A list of objects
+
+##### Nested objects as a dictionary
+
+In the example below, `tags` is a relationship of cardinality many that is defined as a dictionary with multiple values under data.
+
+> The kind is optional here because there is only one option possible (not a generic)
+
+```yaml
+---
+apiVersion: infrahub.app/v1
+kind: Object
+spec:
+  kind: TestingPerson
+  data:
+    - name: Alex Thompson
+      tags:
+        data:
+          - name: dog-lover
+            description: "Dog Lover"
+          - name: veterinarian
+            description: "Veterinarian"
+```
+
+This format works well when all objects are of the same kind and when all objects are using the same properties.
+For more complex cases, the list of objects format is more flexible.
+
+##### Nested objects as a list of objects
+
+In the example below, `animals` is a relationship of cardinality many that is defined as a list of objects.
+Each object must contain a `data` key and each object can also define a specific `kind`.
+
+> If the kind is not specified, it will be inferred from schema
+
+```yaml
+---
+apiVersion: infrahub.app/v1
+kind: Object
+spec:
+  kind: TestingPerson
+  data:
+    - name: Alex Thompson
+      height: 180
+      animals:
+        - kind: TestingDog
+          data:
+            name: Max
+            weight: 25
+            breed: Golden Retriever
+            color: "#FFD700"
+        - kind: TestingCat
+          data:
+            name: Mimi
+            breed: Persian
+```
+
+### Support for metadata
+
+Metadata support is planned for future releases. Currently, the Object file does not support metadata on attributes or relationships.
+
+## Troubleshooting
+
+### Common issues
+
+1. **Objects not being created**: Ensure that the YAML syntax is correct and that the file follows the required format.
+2. **Dependency errors**: When objects depend on each other, load them in the correct order (dependencies first).
+3. **Validation errors**: Use the `infrahub object validate` command to check for syntax errors before loading.
+
+### Best practices
+
+1. Use Human Friendly IDs (HFIDs) for all objects to ensure consistent referencing.
+2. Keep object files organized by model type or purpose.
+3. Validate object files before loading them into production environments.
+4. Use comments in your YAML files to document complex relationships or dependencies.

docs/sidebars-python-sdk.ts

@@ -32,6 +32,7 @@ const sidebars: SidebarsConfig = {
           label: 'Topics',
           items: [
             'topics/tracking',
+            'topics/object_file',
           ],
         },
         {

infrahub_sdk/ctl/cli_commands.py

@@ -61,7 +61,7 @@
 app.add_typer(validate_app, name="validate")
 app.add_typer(repository_app, name="repository")
 app.add_typer(menu_app, name="menu")
-app.add_typer(object_app, name="object", hidden=True)
+app.add_typer(object_app, name="object")
 
 app.command(name="dump")(dump)
 app.command(name="load")(load)

infrahub_sdk/ctl/menu.py

@@ -7,6 +7,7 @@
 from ..async_typer import AsyncTyper
 from ..ctl.client import initialize_client
 from ..ctl.utils import catch_exception, init_logging
+from ..exceptions import ObjectValidationError
 from ..spec.menu import MenuFile
 from .parameters import CONFIG_PARAM
 from .utils import load_yamlfile_from_disk_and_exit
@@ -44,11 +45,16 @@ async def load(
         schema = await client.schema.get(kind=file.spec.kind, branch=branch)
 
         for idx, item in enumerate(file.spec.data):
-            await file.spec.create_node(
-                client=client,
-                schema=schema,
-                data=item,
-                branch=branch,
-                default_schema_kind=file.spec.kind,
-                context={"list_index": idx},
-            )
+            try:
+                await file.spec.create_node(
+                    client=client,
+                    schema=schema,
+                    position=[idx + 1],
+                    data=item,
+                    branch=branch,
+                    default_schema_kind=file.spec.kind,
+                    context={"list_index": idx},
+                )
+            except ObjectValidationError as exc:
+                console.print(f"[red] {exc!s}")
+                raise typer.Exit(1)