Should validator matching be agentic and semantic instead of ID-based?

[cgoncalves94]

While reviewing the rule engine, I noticed that validator selection is still largely tied to rule ID mapping. This approach works, but it introduces rigidity and can make the system less adaptable as new rules and validators are introduced.

Given that the project already leverages agents/LLMs for rule evaluation, would it make sense to shift validator selection to a more agentic, semantic process?

Specifically:

• Let the agent decide which validator (if any) to use, based on the rule's description, parameters, and event type, rather than relying on static ID matching.
• Keep validators as modular, high-performance tools, but have the agent orchestrate their use based on context.
• Maintain current performance optimizations by using validators for standard/common rules and LLMs for complex or custom logic.

---

Potential Benefits

• Increased flexibility and extensibility for rule/validator management
• Reduced coupling between rule definitions and validator logic
• Smoother onboarding for new rule types and validation strategies

---

Questions

• Are there technical or performance trade-offs to a fully agent-driven validator selection?
• Would a hybrid approach (semantic first, ID fallback) be preferable for legacy support?
• Any prior experiments or benchmarks on this?

---

Curious to hear thoughts from maintainers and contributors!

[tomups]

Yeah I think having validators as tools and letting the LLM choose the right one to use is the way to go in terms of flexibility.

We do have to be very careful about managing non-determinism though, as it's not uncommon for models to "randomly" choose to use a tool or not even with the same prompt. But I think with proper prompt engineering this can be kept consistent.

We should also add a flag to the config to force using a certain validator without any LLM choice if that's what the user wants, but keep it disabled by default.

[dkargatzis]

Great points from both of you!

Initially the logic relied on a rule-based system which was super fast but not extensible. Then I tried a fully prompt-driven solution which was flexible but slow (>10s for violation detection). While the current approach balances performance, accuracy, and flexibility, it has important limitations we should address ASAP.

Based on your suggestions, I believe we should consider:

• Semantic matching - Agent analyzes rule intent to choose validator
• Configurable fallback - Flag to force specific validators when needed
• Complete migration - Move away from ID-based mapping entirely

This would give us the flexibility we need while maintaining the performance gains from a hybrid system. The non-determinism concern @tomups raised is valid - we'll need proper prompt engineering to ensure consistent validator selection.