CLI tool for running checks

[linear]

Summary

Add a command-line interface for running Giskard checks without writing Python code. Enable YAML/JSON-defined test suites that can be run from the terminal.

Motivation

promptfoo is CLI-first and this is a major UX advantage — users can define tests in YAML and run them with a single command. A CLI lowers the barrier for non-Python-native users and enables scripted evaluation workflows.

Scope

Core commands

• giskard run <scenario.yaml> — run a scenario from a YAML definition
• giskard run <suite.yaml> — run a test suite
• giskard list checks — list available built-in checks
• giskard validate <scenario.yaml> — validate a scenario definition without running

YAML scenario format

name: basic_test
target: "python:my_module.my_function"
steps:
  - interact:
      inputs: "What is Python?"
    check:
      - kind: string_matching
        expected: "programming language"
      - kind: not_empty

Output

• Rich terminal output (already exists for results)
• JUnit XML output option (added EE license #29)
• JSON output for programmatic consumption

Dependencies

• Serialization support already exists (Pydantic discriminated unions)
• added EE license #29 (JUnit XML) for CI-friendly output

Acceptance Criteria

• Run scenarios from YAML/JSON files
• List available checks
• Validate scenario definitions
• Multiple output formats (rich, JUnit XML, JSON)
• Clear error messages for invalid configurations
• giskard --help documents all commands

[linear]

OSS-60

[raghavtries]

Hi! I really wanted to work on this and was just able to finish, so I opened a PR.

I added a giskard CLI for running and validating serialized scenarios and suites from YAML/ JSON, including run, validate, and list checks, with rich, JSON, and JUnit outputs. I also validated the CLI flow locally, including trace_type handling, target resolution from the definition directory, and the JUnit export path.

Please let me know if you'd like any changes. Thanks!