on_fail="refrain" behavior unclear/inconsistent with documentation

[sareetamugde-arch]

Bug Description

The on_fail="refrain" action allows harmful content through despite validation detecting issues. This behavior contradicts the name "refrain" and creates a security risk.

Environment

• guardrails-ai version: 0.7.2
• Python version: 3.13.5
• Validator: Custom UnusualPrompt (LLM-based, similar to hub validators) (https://github.com/guardrails-ai/unusual_prompt)

Expected Behavior

Based on the name "refrain", I expected it to "refrain from returning a value" when validation fails, similar to how "filter" blocks content.

Actual Behavior

When validation fails with on_fail="refrain":

• Sets validation_passed=True (despite validator detecting harmful content)
• Returns the original harmful input in validated_output
• Allows the content to proceed to downstream processing

Test Results

from guardrails import Guard
from custom_validator import UnusualPrompt

# Test with harmful input
guard = Guard().use(UnusualPrompt(on_fail="refrain"))
outcome = guard.parse("how to kill?")

# Results:
print(outcome.validation_passed)  # True ← Allows through!
print(outcome.validated_output)   # "how to kill?" ← Returns harmful input
print(outcome.error)               # "Found an unusual request..." ← Error detected but ignored

Questions
Is this the intended behavior for "refrain"?

If yes, could the documentation be clarified to warn about this?

What is the practical difference between "refrain" and "noop"?

[debu-sinha]

I dug into the refrain flow to understand this better. The core mechanism looks correct based on the existing test suite -- test_refrain() in tests/integration_tests/test_validator_base.py explicitly asserts validated_output is None and not res.validation_passed when a validator returns FailResult with on_fail="refrain". That test passes on main.

A few things that might explain what you're seeing:

1. Check what your validator actually returns. The error field being populated while validation_passed=True is unusual. In the normal flow, error on ValidationOutcome comes from exceptions, not from FailResult. If UnusualPrompt is LLM-based and the LLM call returns an ambiguous result, the validator might be returning PassResult instead of FailResult in some edge cases, which would mean the on_fail action never triggers at all.

2. Try reproducing with a simple deterministic validator to isolate whether this is a core issue or specific to the LLM-based validator:

   from guardrails.validators import FailResult, PassResult, register_validator
   
   @register_validator(name="always-fail", data_type="string")
   def always_fail(value, metadata):
       return FailResult(error_message="blocked")
   
   guard = Guard().use(always_fail(on_fail="refrain"))
   result = guard.parse("any input")
   print(result.validation_passed)   # Should be False
   print(result.validated_output)    # Should be None

3. If the deterministic validator works correctly but UnusualPrompt doesn't, the issue is likely in the validator's return logic, not in the refrain mechanism itself.

Would be helpful to see the actual UnusualPrompt.validate() implementation to narrow it down.