Add input argument overrides for pipeline runs via CLI and SDK

[Copilot]

Enables dynamic parameter injection for pipeline runs without creating new PipelineVersions. Users can override orchestration parameters (e.g., Argo Workflow arguments) at runtime, supporting use cases like prompt injection for Agentic AI and multi-tenant configurations.

Changes

CLI

• Added --set key=value for inline parameter overrides (repeatable)
• Added --overrides-file path.json for file-based overrides
• Inline parameters override file parameters when both provided

SDK

• Added optional input_args_override: Dict[str, str] parameter to Pipeline.run()
• Forward-compatible with upcoming proto field PipelineVersionRun.input_args_override
• Gracefully handles current proto version without the field

Utilities

• clarifai/utils/pipeline_overrides.py: Parse, load, merge, validate, and build override structures
• All values converted to strings per Argo convention

Testing

• 20 unit tests for utility functions
• 6 CLI integration tests covering inline, file, and mixed override scenarios

Usage

CLI:

clarifai pipeline run \
  --compute_cluster_id=cc1 --nodepool_id=np1 \
  --set prompt="Summarize this research paper" \
  --set temperature="0.7"

SDK:

pipeline = Pipeline(
    pipeline_id='my-pipeline',
    pipeline_version_id='v1',
    user_id='user123',
    app_id='app456',
    nodepool_id='nodepool1',
    compute_cluster_id='cluster1',
    pat='your-pat'
)

result = pipeline.run(
    input_args_override={
        "prompt": "Summarize this research paper",
        "temperature": "0.7"
    }
)

Compatibility

Fully backward compatible—input_args_override is optional. Implementation detects proto field availability at runtime and adapts accordingly.

Original prompt

---

This section details on the original issue you should resolve

<issue_title>Enable Input Arguments Override through CLI & CLI</issue_title>
<issue_description>## Overview

This document describes the required changes to the Python CLI and CLI to support input arguments override when running a pipeline via PipelineVersionRun. This enables users to dynamically override orchestration-specific input parameters (e.g., Argo Workflow arguments) for each run, supporting use-cases such as prompt injection for Agentic AI without requiring new PipelineVersions.

---

API Feature Summary (Reference Only; Already Implemented)

• Protobuf schema has been updated in the backend to support:
  • input_args_override field in PipelineVersionRun (accepts OrchestrationArgsOverride)
  • orchestration_spec field in PipelineVersionRun for the merged, final spec
  • Support for Argo parameter overrides via ArgoArgsOverride and ArgoParameterOverride
• CLI/SDK should expect these fields/messages in the API responses and requests.

---

Semantics

• If no override is provided, values from the PipelineVersion's orchestration spec are used.
• For each run, matching parameter names are replaced by those in input_args_override.
• Unknown parameters are rejected with clear error messaging.
• All values are treated as strings, per Argo convention.

---

CLI Changes

1. Command Updates

• Update clarifai pipeline run to accept input argument overrides for each run.

2. Input Override Methods

• Inline via --set:

  clarifai pipeline run \
    --compute_cluster_id=... \
    --nodepool_id=... \
    --set prompt="Summarize this research paper" \
    --set temperature="0.7"
  

• File-based via --overrides-file:

  clarifai pipeline run \
    --compute_cluster_id=... \
    --nodepool_id=... \
    --overrides-file overrides.json
  

  overrides.json:

  {
    "prompt": "Summarize this research paper",
    "temperature": "0.7"
  }

• Support both inline and file-based overrides for flexibility.

3. Output Formatting

• Update output to display the final merged orchestration spec (orchestration_spec) for the run.
• Add JSON output option for programmatic inspection.

---

SDK Changes

1. Proto/Model Updates

• Regenerate Python proto files from the updated .proto definitions to include all relevant messages and fields:
  • OrchestrationArgsOverride
  • ArgoArgsOverride
  • ArgoParameterOverride
  • updated PipelineVersionRun with new fields

2. Client Method Updates

• Update Pipeline.run() (or equivalent) to accept an input_args_override argument.
• Validate that provided overrides match parameters defined in the PipelineVersion spec.
• Merge overrides with PipelineVersion parameters (as described in Semantics).
• Add type hints and docstrings for new fields.
• Ensure PipelineVersionRun responses expose the merged orchestration_spec.

3. Helper Methods (Optional)

• Add convenience helpers for building and validating overrides:
  • build_arg_override(parameters: dict[str, str]) -> OrchestrationArgsOverride
  • validate_arg_override(override: OrchestrationArgsOverride, pipeline_version: PipelineVersion) -> bool

4. Example SDK Usage

from clarifai_grpc.grpc.api import resources_pb2
from clarifai.client import Pipeline

pipeline = Pipeline(...existing fields...)
overrides = resources_pb2.OrchestrationArgsOverride(
    argo_args_override=resources_pb2.ArgoArgsOverride(
        parameters=[
            resources_pb2.ArgoParameterOverride(name="prompt", value="Summarize this research paper"),
            resources_pb2.ArgoParameterOverride(name="temperature", value="0.7"),
        ]
    )
)
run = pipeline.run(
    ... existing params ...
    input_args_override=overrides
)

---

Backward Compatibility

• ✅ Fully Backward Compatible: Existing pipeline run commands without overrides will continue to work.
• ✅ Optional Field: input_args_override is optional.
• ✅ No Breaking Changes: No existing fields are modified or removed.

---

Testing Requirements

1. Unit Tests

• Test pipeline run with input argument overrides (both inline and file).
• Test validation and error handling for unknown parameters.
• Test backward compatibility (run with no overrides).

2. Integration Tests

• End-to-end tests: pipeline creation, run with overrides, inspect resulting workflow spec.
• CLI command testing for all override modes.
• Error scenarios (invalid override keys, malformed file).

3. Example Code & Documentation

• Add input argument override usage examples to docs.
• Update pipeline run examples to show both inline and file-based override usage.

---

Security & Traceability

• Traceability: Store the final orchestration spec in PipelineVersionRun for accurate auditing and debugging.
• **Valida...

• Fixes Enable Input Arguments Override through CLI & SDK #831

---

✨ Let Copilot coding agent set things up for you — coding agent works faster and does higher quality work when set up for your repo.