Skip to content

feat: add accuracy tag to behave scenarios#436

Merged
rgonalo merged 2 commits intomasterfrom
feat/accuracy_tag
Oct 29, 2025
Merged

feat: add accuracy tag to behave scenarios#436
rgonalo merged 2 commits intomasterfrom
feat/accuracy_tag

Conversation

@rgonalo
Copy link
Member

@rgonalo rgonalo commented Oct 28, 2025

This pull request introduces support for accuracy tags in Behave scenarios, allowing tests involving AI-generated content to specify required accuracy levels and the number of retries. It adds documentation, implementation, and tests for this feature, and integrates it into the Behave environment setup. The changes also include minor improvements to logging and documentation for clarity.

Accuracy Tag Feature for Behave Scenarios

  • Added support for accuracy tags (e.g., @accuracy_80_10) in Behave scenarios to specify required accuracy percentage and number of retries for AI-based tests. Scenarios are automatically retried and marked as failed if the desired accuracy is not met. [1] [2]
  • Updated documentation in docs/bdd_integration.rst and docs/ai_utils.rst to describe the usage and format of accuracy tags, including examples and integration details. [1] [2]
  • Integrated accuracy tag patching into the Behave environment by calling patch_feature_scenarios_with_accuracy in the before_feature hook. [1] [2]

Testing and Logging Improvements

  • Added unit tests for accuracy tag parsing and scenario patching in toolium/test/utils/ai_utils/test_accuracy.py.
  • Improved logging in text similarity checks to separate similarity and explanation messages for better clarity.

Documentation Updates

  • Updated the changelog to announce the new accuracy tag feature

@rgonalo rgonalo merged commit f93d205 into master Oct 29, 2025
25 of 26 checks passed
@rgonalo rgonalo deleted the feat/accuracy_tag branch October 29, 2025 09:14
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants