[Bug]: Responses API does not support guardrails? Documenting a limitation (?) #14152
Description
What happened?
When using Responses API, I don't see pre_call guardrails (GuardrailsAI) hooks being invoked at all. With same LiteLLM config, same model, /chat/completion requests do invoke the pre_call guardrails but /responses does not.
Upon reviewing the documentation page: https://docs.litellm.ai/docs/response_api I see that guardrails support is not listed with a checkmark there. So perhaps that's a well-known limitation for Responses API (?) If that's the case, the documentation page is misleading. For major features that are not supported by Responses API it should explicitly call it out as unsupported, rather then quietly omitting.
Relevant log output
LiteLLM config:
guardrails:
- guardrail_name: "secrets-detection-fail"
litellm_params:
guardrail: guardrails_ai
guard_name: "secrets-detection-fail"
mode: "pre_call"
api_base: os.environ/GUARDRAILS_API_BASE_URL
guardrails_ai_api_input_format: "llmOutput"
default_on: true
- guardrail_name: "toxic-language-0.5-fail"
litellm_params:
guardrail: guardrails_ai
guard_name: "toxic-language-0.5-fail"
mode: "pre_call"
api_base: os.environ/GUARDRAILS_API_BASE_URL
guardrails_ai_api_input_format: "llmOutput"
default_on: trueRelevant log (debug level)
20:47:33 - LiteLLM:DEBUG: utils.py:349 - Initialized litellm callbacks, Async Success Callbacks: [<bound method Router.deployment_callback_on_success of <litellm.router.Router object at 0xffff7fc0c050>>, <litellm.proxy.hooks.model_max_budget_limiter._PROXY_VirtualKeyModelMaxBudgetLimiter object at 0xffff81107e00>, <litellm.router_utils.pre_call_checks.prompt_caching_deployment_check.PromptCachingDeploymentCheck object at 0xffff7fc0c830>, <litellm.router_utils.forward_clientside_headers_by_model_group.ForwardClientSideHeadersByModelGroup object at 0xffff7fc0cad0>, <litellm.proxy.guardrails.guardrail_hooks.guardrails_ai.guardrails_ai.GuardrailsAI object at 0xffff7fc0cd70>, <litellm.proxy.guardrails.guardrail_hooks.guardrails_ai.guardrails_ai.GuardrailsAI object at 0xffff7fbfca50>, <litellm.proxy.hooks.proxy_track_cost_callback._ProxyDBLogger object at 0xffff7f1d0440>, <litellm.proxy.hooks.max_budget_limiter._PROXY_MaxBudgetLimiter object at 0xffff7fbfee90>, <litellm.proxy.hooks.parallel_request_limiter._PROXY_MaxParallelRequestsHandler object at 0xffff7ece3d90>, <litellm.proxy.hooks.cache_control_check._PROXY_CacheControlCheck object at 0xffff7eb78550>, <litellm_enterprise.proxy.hooks.managed_files._PROXY_LiteLLMManagedFiles object at 0xffff7f1d02f0>, <litellm._service_logger.ServiceLogging object at 0xffff80c443e0>]
20:47:33 - LiteLLM:DEBUG: litellm_logging.py:474 - self.optional_params: {}
20:47:33 - LiteLLM Proxy:DEBUG: utils.py:851 - Inside Proxy Logging Pre-call hook!
20:47:33 - LiteLLM:DEBUG: custom_guardrail.py:237 - inside should_run_guardrail for guardrail=secrets-detection-fail event_type= GuardrailEventHooks.pre_call guardrail_supported_event_hooks= pre_call requested_guardrails= [] self.default_on= True
20:47:33 - LiteLLM:DEBUG: custom_guardrail.py:237 - inside should_run_guardrail for guardrail=toxic-language-0.5-fail event_type= GuardrailEventHooks.pre_call guardrail_supported_event_hooks= pre_call requested_guardrails= [] self.default_on= True
20:47:33 - LiteLLM Proxy:DEBUG: max_budget_limiter.py:23 - Inside Max Budget Limiter Pre-Call Hook
20:47:33 - LiteLLM Proxy:DEBUG: parallel_request_limiter.py:48 - Inside Max Parallel Request Pre-Call Hook
20:47:33 - LiteLLM Proxy:INFO: parallel_request_limiter.py:68 - Current Usage of key in this minute: None
20:47:33 - LiteLLM Proxy:INFO: parallel_request_limiter.py:68 - Current Usage of user in this minute: None
20:47:33 - LiteLLM Proxy:DEBUG: cache_control_check.py:27 - Inside Cache Control Check Pre-Call Hook
20:47:33 - LiteLLM:DEBUG: custom_guardrail.py:237 - inside should_run_guardrail for guardrail=secrets-detection-fail event_type= GuardrailEventHooks.during_call guardrail_supported_event_hooks= pre_call requested_guardrails= [] self.default_on= True
20:47:33 - LiteLLM:DEBUG: custom_guardrail.py:237 - inside should_run_guardrail for guardrail=toxic-language-0.5-fail event_type= GuardrailEventHooks.during_call guardrail_supported_event_hooks= pre_call requested_guardrails= [] self.default_on= True
It does see the registered guardrails and evaluates for each, but actual guardrail invocation does not happen - no call arrives to the Guardrails API server.
When using /chat/completions, the LiteLLM log is similar but at that moment (inside should_run_guardrail for guardrail=secrets-detection-fail) the call goes to Guardrails API and it actually executes.
Are you a ML Ops Team?
No
What LiteLLM version are you on ?
v1.75.8
Twitter / LinkedIn details
No response