Overview

Clarifai backend now supports Pipeline Step Secrets—enabling different pipeline steps to access distinct sets of secrets with step-level isolation. This document outlines the technical changes required to adopt this feature in the Python SDK and CLI, ensuring compatibility with updated API fields and backend functionality.

API Feature Summary

1. New Proto Fields

• PipelineVersionConfig
  • step_version_secrets: map<string, StepSecretConfig>
    • Key: Step reference (step1, step2, ...)
    • Value: StepSecretConfig
• StepSecretConfig (New Message)
  • secrets: map<string, string>
    • Key: Secret name (e.g., API_KEY)
    • Value: Secret reference path (e.g., users/user123/secrets/my-api-key)

2. Updated API Endpoints

• GetPipelineVersion: Returns pipeline version with step secret config.
• PostPipelineVersions: Create pipeline version with step secrets.

3. Example API Usage

Creation

{
  "pipeline_version": {
    "config": {
      "step_version_secrets": {
        "step1": {
          "secrets": {
            "API_KEY": "users/user123/secrets/my-api-key",
            "DB_PASSWORD": "users/user123/secrets/db-secret"
          }
        },
        "step2": {
          "secrets": {
            "EMAIL_TOKEN": "users/user123/secrets/email-token"
          }
        }
      }
    }
  }
}

Python SDK Changes

1. Proto/Model Updates

• Regenerate Python proto files from updated .proto definitions.
• Ensure PipelineVersionConfig includes the step_version_secrets field.
• Add StepSecretConfig model/class for mapping secrets per step.

2. Client Method Updates

• Update get_pipeline_version() to handle new step secrets field.
• Update create_pipeline_version() to accept step secrets.
• Add type hints for new fields.

3. Helper Methods (Recommended)

• Add convenience methods:
  • add_step_secret(step_ref, secret_name, secret_ref)
  • remove_step_secret(step_ref, secret_name)
  • list_step_secrets(step_ref)

CLI Changes

1. Command Updates

• Update clarifai pipeline version create to accept step secrets specified in the orchestration spec (config.yaml), not as CLI parameters.
• Update clarifai pipeline version get to display step secrets.

2. Config File Integration (Orchestration Spec)

• Step secrets should be defined within the config.yaml file as part of the pipeline orchestration specification, matching the API schema.

• Example config.yaml:

  pipeline_version:
    config:
      step_version_secrets:
        step1:
          secrets:
            API_KEY: users/user123/secrets/my-api-key
            DB_PASSWORD: users/user123/secrets/db-secret
        step2:
          secrets:
            EMAIL_TOKEN: users/user123/secrets/email-token

• CLI usage examples:

  clarifai pipeline version create --pipeline-id "my-pipeline" --config config.yaml
  

• All step secrets should be managed via the spec file; do not support CLI flags for individual secrets.

3. Output Formatting

• Update list/get commands to display step secrets in human-readable/table format.
• Add JSON output option for programmatic use.

Backward Compatibility

• ✅ Fully Backward Compatible: All existing API calls will continue to work.
• ✅ Optional Field: step_version_secrets is optional; existing pipelines unaffected.
• ✅ No Breaking Changes: No existing fields were modified or removed.

Testing Requirements

1. Unit Tests

• Test pipeline version creation with step secrets from config file.
• Test step secrets validation and error handling.
• Test backward compatibility (existing pipelines without secrets).

2. Integration Tests

• End-to-end testing for pipeline creation, execution, and CLI commands with step secrets defined in config file.
• Error scenarios (invalid secret references, missing secrets).

3. Example Code & Documentation

• Add step secrets usage examples to docs.
• Update any existing pipeline examples to show step secrets usage in orchestration spec.

Security Considerations

• Step-Level Isolation: Each step only accesses explicitly configured secrets.
• Secret References Only: API handles references, not actual secret values.
• Kubernetes Integration: Backend mounts secrets securely as environment variables.
• No Secret Value Leakage: Only references are stored in config; values are injected at runtime.

Additional Context

• Backend uses env var injection and Kubernetes SecretKeyRef for secure secret mounting.
• Isolation is guaranteed for each step.
• Multiple secrets per step and multiple steps per pipeline are supported.

Implementation Guidance

• Verify Model Implementation: Review how secrets are handled for models (see clarifai/client/model.py, clarifai/utils/secrets.py) and match the approach for pipelines.
• Proto Changes: Regenerate and update Python proto files and dependent classes.
• Client & CLI Consistency: Ensure CLI and SDK methods for pipeline step secrets mirror model secret handling (env vars, helper functions, API fields), with config file as the source of truth for secrets.
• Tests and Docs: Provide comprehensive tests and update documentation to reflect step secrets support via orchestration spec.

References

• clarifai/client/model.py — Model secret handling
• clarifai/utils/secrets.py — Secret loading/injection utilities
• Backend proto changes: PipelineVersionConfig, StepSecretConfig