feat(cli): add shell completion for bash, zsh, fish, and PowerShell

[ruivieira]

What and why

Adds a evalhub completion command group that generates shell completion scripts for bash, zsh, fish, and PowerShell. This improves CLI discoverability and usability by enabling tab-completion for all commands, subcommands, and options.

Type

• feat
• fix
• docs
• refactor / chore
• test / ci

Testing

• Tests added or updated
• Tested manually

Summary by CodeRabbit

• New Features
  • Added shell completion support (bash, zsh, fish, PowerShell) with generated registration scripts.
• Bug Fixes
  • Ensure MLflow run IDs are handled reliably when logging runs.
• Tests
  • Added unit tests covering CLI completion scripts and PowerShell completion behavior.
• Chores
  • Updated development tooling version constraints for type-checking (mypy).

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: 09f40a7f-7084-4e63-a560-ccb4bbf9a3df

📥 Commits

Reviewing files that changed from the base of the PR and between f2eae94 and 77c1a38.

📒 Files selected for processing (1)

• src/evalhub/cli/completion.py

🚧 Files skipped from review as they are similar to previous changes (1)

• src/evalhub/cli/completion.py

---

📝 Walkthrough

Walkthrough

Adds a new completion Click command group that emits shell completion scripts for bash, zsh, fish, and powershell, registers it on the root evalhub CLI, and includes a custom PowerShell completion integration and unit tests.

Changes

Cohort / File(s)	Summary
Shell Completion Infrastructure src/evalhub/cli/completion.py	New module adding _get_completion_script(shell: str), a completion Click group with bash, zsh, fish, and powershell subcommands, and a PowerShellComplete class implementing PowerShell completion template, argument reconstruction, and output formatting.
CLI Entry Point Integration src/evalhub/cli/main.py	Imports and registers the new completion command group on the root main Click group.
Tests tests/unit/test_cli_completion.py	New unit tests covering completion help/registration, each shell subcommand output, PowerShell env-var behavior and cleanup, and direct tests for PowerShellComplete methods and registration.
Build / Tooling .pre-commit-config.yaml, .github/workflows/pre-commit.yml, pyproject.toml	Bumped mypy version/rev in pre-commit and pyproject constraints (mirror/rev updates only).
Adapter Fix src/evalhub/adapter/callbacks.py	Force-casts MLflow run_id to str(...) before use/return.

Sequence Diagram(s)

mermaid
sequenceDiagram
participant User
participant Shell
participant EvalhubCLI
participant ClickCore
User->>Shell: run "evalhub completion "
Shell->>EvalhubCLI: invoke command
EvalhubCLI->>ClickCore: request completion script for shell
ClickCore-->>EvalhubCLI: return script text
EvalhubCLI-->>Shell: print completion script

mermaid
sequenceDiagram
participant User
participant PowerShell
participant EvalhubProcess
participant EvalhubCLI
User->>PowerShell: press Tab (registered completer)
PowerShell->>EvalhubProcess: launch evalhub with env vars
EvalhubProcess->>EvalhubCLI: Click completion logic runs and outputs items
EvalhubCLI-->>PowerShell: completion items (type,value,help)
PowerShell-->>User: map items to suggestions

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~25 minutes

Poem

🐇 I hopped to the keys with a twitch and a grin,
Taught shells to finish the words they begin.
Bash, zsh, fish, PowerShell — now answer my taps,
I nudged in some scripts and polished the gaps.
Hooray — tab-complete magic, carrots for snacks! 🥕✨

🚥 Pre-merge checks | ✅ 2 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 20.59% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (2 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title 'feat(cli): add shell completion for bash, zsh, fish, and PowerShell' directly and clearly describes the main change: adding shell completion functionality for multiple shells to the CLI.
Description check	✅ Passed	The description covers all required sections from the template (What and why, Type, Testing) with appropriate detail, though it lacks an issue reference (Closes #).

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[coderabbitai]

Actionable comments posted: 1

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: 028c7c8a-37f4-487d-aad5-11dbaae87fc9

📥 Commits

Reviewing files that changed from the base of the PR and between 8f15396 and 0c31da3.

📒 Files selected for processing (1)

• src/evalhub/cli/completion.py

[coderabbitai]

🧹 Nitpick comments (1)

pyproject.toml (1)

75-75: Inconsistent version constraints for mypy.

Line 75 pins mypy exactly with ==1.19.1, while Line 202 uses a minimum constraint >=1.19.1. This inconsistency can cause:

• Different mypy versions being installed depending on which dependency path is resolved
• Potential conflicts if the >= constraint resolves to a newer version with breaking changes

Consider using the same constraint style in both locations for consistency.

♻️ Proposed fix to use consistent pinning

 [dependency-groups]
 dev = [
     "eval-hub-server @ git+https://github.com/eval-hub/eval-hub#subdirectory=python-server",  # TODO: this should be moved to a pypi release
     "fastapi>=0.124.4",
     "httpx>=0.28.1",
-    "mypy>=1.19.1",
+    "mypy==1.19.1",
     "olot>=0.1.15",

Also applies to: 202-202

🤖 Prompt for AI Agents

Verify each finding against the current code and only fix it if needed.

In `@pyproject.toml` at line 75, The mypy version constraint is inconsistent: one
dependency entry pins "mypy==1.19.1" while another uses "mypy>=1.19.1"; pick a
single constraint style and make both entries match (either change the
"mypy==1.19.1" entry to "mypy>=1.19.1" or change the "mypy>=1.19.1" entry to
"mypy==1.19.1") so that all occurrences of the mypy dependency in pyproject.toml
use the same version specifier, updating both the exact "mypy==1.19.1" token and
the "mypy>=1.19.1" token to the chosen form.

🤖 Prompt for all review comments with AI agents

Verify each finding against the current code and only fix it if needed.

Nitpick comments:
In `@pyproject.toml`:
- Line 75: The mypy version constraint is inconsistent: one dependency entry
pins "mypy==1.19.1" while another uses "mypy>=1.19.1"; pick a single constraint
style and make both entries match (either change the "mypy==1.19.1" entry to
"mypy>=1.19.1" or change the "mypy>=1.19.1" entry to "mypy==1.19.1") so that all
occurrences of the mypy dependency in pyproject.toml use the same version
specifier, updating both the exact "mypy==1.19.1" token and the "mypy>=1.19.1"
token to the chosen form.

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: 4a06e4be-0132-4dd2-b196-311bd589b922

📥 Commits

Reviewing files that changed from the base of the PR and between 0c31da3 and f2eae94.

⛔ Files ignored due to path filters (1)

• uv.lock is excluded by !**/*.lock

📒 Files selected for processing (6)

• .github/workflows/pre-commit.yml
• .pre-commit-config.yaml
• pyproject.toml
• src/evalhub/adapter/callbacks.py
• src/evalhub/cli/completion.py
• tests/unit/test_cli_completion.py

✅ Files skipped from review due to trivial changes (2)

• .pre-commit-config.yaml
• .github/workflows/pre-commit.yml

🚧 Files skipped from review as they are similar to previous changes (1)

• src/evalhub/cli/completion.py

[tarilabs]

LGTM for as far as I can see
I have 2 main doubts:

• Powershell is an append, so how does an update work for them?
• is there a way to provide a mechanism to the user to get informed when they need to refresh the completions? (ie new/changed commands?)

[ruivieira]

LGTM for as far as I can see I have 2 main doubts:

• Powershell is an append, so how does an update work for them?

@tarilabs thanks for the review. I've changed the PowerShell completions to not append, but use a dedicated file.

• is there a way to provide a mechanism to the user to get informed when they need to refresh the completions? (ie new/changed commands?)

I was planning on adding this to the main documentation. Typically if the completion is loaded at config read time (e.g. ~/.zshrc) no update would be needed. If using a dedicated file, it would be the user's responsibility to re-generate the file after a pip upgrade of the SDK.

[tarilabs]

it would be the user's responsibility to re-generate the file after a pip upgrade of the SDK

makes sense, my question is more how can we inform the user they need to perform this operation?
not sure if there's a way to

• hook on pipx install or the likes
• check if there are any shell completions
• if yes, advise the user to update their shell completions

but could be done in followup PR

[ruivieira]

@coderabbitai could you open an issue for the comments in #100 (comment) ?

[coderabbitai]

Tip

For best results, initiate chat on the files or code changes.

🧩 Analysis chain

🏁 Script executed:

#!/bin/bash
gh issue create \
  --repo eval-hub/eval-hub-sdk \
  --title "feat(cli): inform users to refresh shell completions after SDK upgrade" \
  --body "## Summary

When users install or upgrade the \`evalhub\` SDK via \`pip\` or \`pipx\`, their previously generated shell completion scripts may become stale if commands, subcommands, or options have changed. There is currently no mechanism to inform users that they need to re-generate their completions.

## Context

This was raised as a follow-up item in PR `#100` (https://github.com/eval-hub/eval-hub-sdk/pull/100) by `@tarilabs` in https://github.com/eval-hub/eval-hub-sdk/pull/100#issuecomment-4153709434.

## Possible Approaches

- Hook into \`pipx install\` / \`pip install\` post-install step to notify users
- On CLI startup, detect whether shell completions are already installed and compare them against the currently installed version
- If completions are found but appear stale (e.g. version mismatch), advise the user to run \`evalhub completion <shell>\` again to refresh

## Notes

- Users who source completions dynamically at config read time (e.g. via \`~/.zshrc\`) may not be affected.
- Users who use a dedicated completions file will need to manually re-generate after an upgrade.

Requested by `@ruivieira`." \
  --assignee ruivieira

Length of output: 51

---

@ruivieira I've opened a follow-up issue to track this: #103

The issue captures the discussion from @tarilabs around possible approaches to notify users when their shell completions may be stale after a SDK upgrade, including:

• Hooking into pip/pipx install post-install steps
• Detecting installed completions at CLI startup and comparing against the current version
• Advising users to re-run evalhub completion <shell> when completions are found to be outdated