chore(evalhub): update garak provider configmap with supported benchmarks

[saichandrapandraju]

This PR updates the evalhub garak provider configmap to have supported pre-defined benchmarks by garak evalhub adapter

Summary by CodeRabbit

• New Features

  • Expanded benchmark suite replacing a single toxicity test with multiple evaluations (OWASP LLM Top 10, AVID variants, CWE, Quality, Quick Scan) covering security, safety, performance, and ethics.
  • Standardized on a metrics-centric evaluation using a consistent attack_success_rate metric.

• Refactor

  • Reorganized benchmark taxonomy, categories, and tags; removed legacy benchmark fields for a streamlined configuration.

[openshift-ci]

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

Configuration used: defaults

Review profile: CHILL

Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between a884fc9 and 046a147.

📒 Files selected for processing (1)

• config/configmaps/evalhub/provider-garak.yaml

---

📝 Walkthrough

Walkthrough

Replaces the Garak provider container entrypoint to run a Python module and restructures the evaluation benchmarks in the provider config from a single toxicity benchmark to a taxonomy-aligned suite (OWASP, AVID variants, CWE, quality/quick) with standardized metrics and tags.

Changes

Cohort / File(s)

Summary

Garak Provider Configuration config/configmaps/evalhub/provider-garak.yaml

Changed entrypoint from running main.py to module invocation (-m llama_stack_provider_trustyai_garak.evalhub). Replaced the single toxicity benchmark with multiple taxonomy-aligned benchmarks (owasp_llm_top10, avid, avid_security, avid_ethics, avid_performance, quality, cwe, quick). Standardized metrics to attack_success_rate, removed legacy fields (num_few_shot, dataset_size), and updated categories and tags (security, safety, performance, avid, red_team, ethics, cwe, quick).

Estimated code review effort

🎯 2 (Simple) | ⏱️ ~10 minutes

Poem

🐰 I swapped one test for many hops and bounds,
OWASP, AVID, CWE in tidy rounds.
Metrics aligned, tags all in view,
A burrow of checks—fresh, precise, and new.
Nibble the config, let the audits pursue! 🥕

🚥 Pre-merge checks | ✅ 3

✅ Passed checks (3 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title accurately describes the main change: updating the garak provider configmap with supported benchmarks, which aligns with the primary objective of the PR.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Post copyable unit tests in a comment

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[openshift-ci]

@saichandrapandraju: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
ci/prow/trustyai-service-operator-e2e	046a147	link	true	/test trustyai-service-operator-e2e

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

[openshift-ci]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: ruivieira

The full list of commands accepted by this bot can be found here.

Details

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment