You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Currently, IntentGuard uses a simple majority voting system to determine assertion results. This can lead to inconsistent results, especially when dealing with complex or ambiguous assertions.
Different aggregation modes will provide more flexibility and control over the assertion process, allowing developers to fine-tune the balance between strictness and leniency.
What:
Introduce three aggregation modes:
Strict: Requires unanimous agreement among all LLM evaluations for an assertion to pass.
Balanced: Uses the current majority voting system (default).
Relaxed: Requires only one LLM evaluation to agree for an assertion to pass.
Add a new parameter to the IntentGuardOptions class to allow users to specify the desired aggregationmode.
Update the assertion evaluation logic to incorporate the selected aggregationmode.
Update documentation and examples to reflect the new aggregation modes functionality.
Acceptance Criteria:
Strict mode: Assertions only pass if all LLM evaluations agree (e.g., if num_evaluations is 5, all 5 evaluations must agree).
Balanced mode: Assertions pass if the majority of LLM evaluations agree (e.g., if num_evaluations is 5, at least 3 evaluations must agree).
Relaxed mode: Assertions pass if at least one LLM evaluation agrees, regardless of the num_evaluations.
Users can select the aggregation mode through the IntentGuardOptions class using a dedicated parameter (e.g., aggregation_mode).
Documentation is updated to include details about the new aggregation modes and their usage.
Example code snippets are provided demonstrating how to use each aggregation mode.
Why:
What:
IntentGuardOptionsclass to allow users to specify the desired aggregationmode.Acceptance Criteria:
num_evaluationsis 5, all 5 evaluations must agree).num_evaluationsis 5, at least 3 evaluations must agree).num_evaluations.IntentGuardOptionsclass using a dedicated parameter (e.g.,aggregation_mode).