Skip to content

Commit 7a6413f

Browse files
zsimjeezsimjee
committed
add how to guide for on fail actions
1 parent 9b7cfc3 commit 7a6413f

File tree

3 files changed

+359
-0
lines changed

3 files changed

+359
-0
lines changed

‎docs/concepts/error_remediation.md‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -37,6 +37,8 @@ Guardrails provides a number of `OnFailActions` for when a validator fails. The
3737
| `OnFailAction.FIX_REASK` | First, fix the generated output deterministically, and then rerun validation with the deterministically fixed output. If validation fails, then perform reasking. | No |
3838

3939

40+
Custom OnFailActions can also be implemented, see the `custom` section in the [how to use on fail actions guide](../how_to_guides/use_on_fail_actions).
41+
4042
## Guidance on dealing with Validator errors
4143

4244
When a validator fails, Guardrails does two things

‎docs/how_to_guides/use_on_fail_actions.ipynb‎

Lines changed: 356 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,356 @@
1+
{
2+
"cells": [
3+
{
4+
"cell_type": "markdown",
5+
"metadata": {},
6+
"source": [
7+
"# Use `on_fail` Actions\n",
8+
"\n",
9+
"`on_fail` actions are instructions to Guardrails that direct action when a validator fails. They are set at the validator level, not the guard level. \n",
10+
"\n",
11+
"The full set of on_fail actions are avialable in the [Error remediation concepts doc](https://www.guardrailsai.com/docs/concepts/error_remediation), and will not all be covered here.\n",
12+
"\n",
13+
"Instead, this interactive doc will serve to guide you on how and when to use different `on_fail` actions."
14+
]
15+
},
16+
{
17+
"cell_type": "code",
18+
"execution_count": 1,
19+
"metadata": {},
20+
"outputs": [
21+
{
22+
"name": "stderr",
23+
"output_type": "stream",
24+
"text": [
25+
"/home/zayd/workspace/testbench/.venv/lib/python3.11/site-packages/torch/cuda/__init__.py:654: UserWarning: Can't initialize NVML\n",
26+
" warnings.warn(\"Can't initialize NVML\")\n"
27+
]
28+
}
29+
],
30+
"source": [
31+
"# setup, run imports\n",
32+
"from guardrails import Guard, install\n",
33+
"try:\n",
34+
" from guardrails.hub import DetectPII\n",
35+
"except ImportError:\n",
36+
" install(\"hub://guardrails/detect_pii\")\n",
37+
" from guardrails.hub import DetectPII"
38+
]
39+
},
40+
{
41+
"cell_type": "markdown",
42+
"metadata": {
43+
"vscode": {
44+
"languageId": "plaintext"
45+
}
46+
},
47+
"source": [
48+
"## Raise Exceptions\n",
49+
"\n",
50+
"If you do not want to change the outputs of an LLM when validation fails, you can instead wrap your guard in a try/catch block.\n",
51+
"\n",
52+
"This works particularly well for input validation."
53+
]
54+
},
55+
{
56+
"cell_type": "code",
57+
"execution_count": 6,
58+
"metadata": {},
59+
"outputs": [
60+
{
61+
"name": "stderr",
62+
"output_type": "stream",
63+
"text": [
64+
"/home/zayd/workspace/testbench/.venv/lib/python3.11/site-packages/guardrails/validator_service/__init__.py:85: UserWarning: Could not obtain an event loop. Falling back to synchronous validation.\n",
65+
" warnings.warn(\n"
66+
]
67+
},
68+
{
69+
"name": "stdout",
70+
"output_type": "stream",
71+
"text": [
72+
"output validation failed\n",
73+
"Validation failed for field with errors: The following text in your response contains PII:\n",
74+
"Hello, my name is John Doe and my email is [email protected]\n",
75+
"input validation failed\n",
76+
"Validation failed for field with errors: The following text in your response contains PII:\n",
77+
"Hello, my name is John Doe and my email is [email protected]\n"
78+
]
79+
}
80+
],
81+
"source": [
82+
"guard = Guard().use(\n",
83+
" DetectPII(pii_entities=\"pii\", on_fail=\"exception\")\n",
84+
")\n",
85+
"\n",
86+
"try:\n",
87+
" guard.validate(\"Hello, my name is John Doe and my email is [email protected]\")\n",
88+
"except Exception as e:\n",
89+
" print(\"output validation failed\")\n",
90+
" print(e)\n",
91+
"\n",
92+
"\n",
93+
"# on input validation\n",
94+
"\n",
95+
"guard = Guard().use(\n",
96+
" DetectPII(pii_entities=\"pii\", on_fail=\"exception\"),\n",
97+
" on=\"msg_history\"\n",
98+
")\n",
99+
"\n",
100+
"\n",
101+
"# on input validation\n",
102+
"\n",
103+
"guard = Guard().use(\n",
104+
" DetectPII(pii_entities=\"pii\", on_fail=\"exception\"),\n",
105+
" on=\"msg_history\"\n",
106+
")\n",
107+
"\n",
108+
"try:\n",
109+
" guard(\n",
110+
" model='gpt-4o-mini',\n",
111+
" messages=[{\n",
112+
" \"role\": \"user\",\n",
113+
" \"content\": \"Hello, my name is John Doe and my email is [email protected]\"\n",
114+
" }]\n",
115+
" )\n",
116+
"except Exception as e:\n",
117+
" print(\"input validation failed\")\n",
118+
" print(e)"
119+
]
120+
},
121+
{
122+
"cell_type": "markdown",
123+
"metadata": {},
124+
"source": [
125+
"## `noop` to log and continue\n",
126+
"\n",
127+
"If you want to log the error and continue, you can use the `noop` action. This is useful for when you want to log the error, but not stop the LLM from running."
128+
]
129+
},
130+
{
131+
"cell_type": "code",
132+
"execution_count": 10,
133+
"metadata": {},
134+
"outputs": [
135+
{
136+
"name": "stdout",
137+
"output_type": "stream",
138+
"text": [
139+
"guarded just fine\n",
140+
"Check if validation passed: False\n",
141+
"Show that the validated text and raw text remain the same: True\n"
142+
]
143+
}
144+
],
145+
"source": [
146+
"# the on_fail parameter does not have to be set, as the default is \"noop\"\n",
147+
"\n",
148+
"guard = Guard().use(\n",
149+
" DetectPII(pii_entities=\"pii\", on_fail=\"noop\")\n",
150+
")\n",
151+
"\n",
152+
"res = guard.validate(\"Hello, my name is John Doe and my email is [email protected]\")\n",
153+
"print(\"guarded just fine\")\n",
154+
"print(\"Check if validation passed: \", res.validation_passed)\n",
155+
"print(\"Show that the validated text and raw text remain the same: \",\n",
156+
" res.validated_output == res.raw_llm_output)"
157+
]
158+
},
159+
{
160+
"cell_type": "markdown",
161+
"metadata": {},
162+
"source": [
163+
"## `fix` to automatically fix the error\n",
164+
"\n",
165+
"Note, not all validators implement a 'fix' value. You can view the FailResult implementation in a validator to see if it has a fix value.\n",
166+
"\n",
167+
"Here's [an example](https://github.com/guardrails-ai/detect_pii/blob/48b15716460fe9e4e5b83ba9607ce3764d9b6d2e/validator/main.py#L188) that shows how Detect PII is written to return anonymized text as a fix value"
168+
]
169+
},
170+
{
171+
"cell_type": "code",
172+
"execution_count": 11,
173+
"metadata": {},
174+
"outputs": [
175+
{
176+
"name": "stderr",
177+
"output_type": "stream",
178+
"text": [
179+
"/home/zayd/workspace/testbench/.venv/lib/python3.11/site-packages/guardrails/validator_service/__init__.py:85: UserWarning: Could not obtain an event loop. Falling back to synchronous validation.\n",
180+
" warnings.warn(\n"
181+
]
182+
},
183+
{
184+
"name": "stdout",
185+
"output_type": "stream",
186+
"text": [
187+
"Check if validated_output is valid text: True\n",
188+
"Scrubbed text: Hello, my name is <PERSON> and my email is <EMAIL_ADDRESS>\n"
189+
]
190+
}
191+
],
192+
"source": [
193+
"guard = Guard().use(\n",
194+
" DetectPII(pii_entities=\"pii\", on_fail=\"fix\")\n",
195+
")\n",
196+
"\n",
197+
"res = guard.validate(\"Hello, my name is John Doe and my email is [email protected]\")\n",
198+
"\n",
199+
"print(\"Check if validated_output is valid text: \", res.validation_passed)\n",
200+
"print(\"Scrubbed text: \", res.validated_output)"
201+
]
202+
},
203+
{
204+
"cell_type": "markdown",
205+
"metadata": {},
206+
"source": [
207+
"## `reask` to automatically ask for an output that passes validation\n",
208+
"\n",
209+
"This reask prompt is computed from the validators themselves. It's an interpolation \n",
210+
"\n",
211+
"In order for the reask prompt to work, the following additional params must be provided:\n",
212+
"\n",
213+
"- messages\n",
214+
"- llm_api OR model"
215+
]
216+
},
217+
{
218+
"cell_type": "code",
219+
"execution_count": 41,
220+
"metadata": {},
221+
"outputs": [
222+
{
223+
"name": "stdout",
224+
"output_type": "stream",
225+
"text": [
226+
"Validated output: Sure! Here's a fictional person without any personal identifiable information:\n",
227+
"\n",
228+
"**Name:** <PERSON> \n",
229+
"**Email:** <EMAIL_ADDRESS> \n",
230+
"\n",
231+
"Feel free to use this for any creative purposes!\n",
232+
"Number of reasks: 1\n"
233+
]
234+
}
235+
],
236+
"source": [
237+
"guard = Guard().use(\n",
238+
" DetectPII(pii_entities=\"pii\", on_fail=\"reask\"),\n",
239+
")\n",
240+
"\n",
241+
"res = guard(\n",
242+
" messages=[{\n",
243+
" \"role\": \"user\",\n",
244+
" \"content\": \"Make up a fake person and email address\",\n",
245+
" }],\n",
246+
" model='gpt-4o-mini',\n",
247+
" num_reasks=1\n",
248+
")\n",
249+
"\n",
250+
"print(\"Validated output: \", res.validated_output)\n",
251+
"print(\"Number of reasks: \", len(guard.history.last.iterations) - 1)"
252+
]
253+
},
254+
{
255+
"cell_type": "markdown",
256+
"metadata": {},
257+
"source": [
258+
"## `custom` to anything else"
259+
]
260+
},
261+
{
262+
"cell_type": "code",
263+
"execution_count": 53,
264+
"metadata": {},
265+
"outputs": [
266+
{
267+
"name": "stderr",
268+
"output_type": "stream",
269+
"text": [
270+
"/home/zayd/workspace/testbench/.venv/lib/python3.11/site-packages/guardrails/validator_service/__init__.py:85: UserWarning: Could not obtain an event loop. Falling back to synchronous validation.\n",
271+
" warnings.warn(\n"
272+
]
273+
},
274+
{
275+
"name": "stdout",
276+
"output_type": "stream",
277+
"text": [
278+
"CUSTOM LOGIC COMPLETE!\n"
279+
]
280+
}
281+
],
282+
"source": [
283+
"# A custom on_fail can be as simple as a function\n",
284+
"\n",
285+
"def custom_on_fail(value, fail_result):\n",
286+
" # This will turn up in validated output\n",
287+
" return \"CUSTOM LOGIC COMPLETE!\"\n",
288+
"\n",
289+
"guard = Guard().use(\n",
290+
" DetectPII(pii_entities=\"pii\", on_fail=custom_on_fail),\n",
291+
")\n",
292+
"res = guard.validate(\"Hello, my name is John Doe and my email is [email protected]\")\n",
293+
"\n",
294+
"print(res.validated_output)"
295+
]
296+
},
297+
{
298+
"cell_type": "code",
299+
"execution_count": 57,
300+
"metadata": {},
301+
"outputs": [
302+
{
303+
"name": "stdout",
304+
"output_type": "stream",
305+
"text": [
306+
"\n",
307+
"String validated: Hello, my name is John Doe and my email is [email protected]\n",
308+
"\n",
309+
"Reason it failed: [ErrorSpan(start=18, end=26, reason='PII detected in John Doe')]\n",
310+
"\n"
311+
]
312+
}
313+
],
314+
"source": [
315+
"# Of course, the function also has access to the fail_result and source text, \n",
316+
"# so interesting logic/formatting over those is also possible\n",
317+
"# Here, we show the specific char spans where the validator detected malfeasance\n",
318+
"\n",
319+
"def custom_on_fail(value, fail_result):\n",
320+
" return f\"\"\"\n",
321+
"String validated: {value}\n",
322+
"\n",
323+
"Reasons it failed: {fail_result.error_spans}\n",
324+
"\"\"\"\n",
325+
"\n",
326+
"guard = Guard().use(\n",
327+
" DetectPII(pii_entities=\"pii\", on_fail=custom_on_fail),\n",
328+
")\n",
329+
"res = guard.validate(\"Hello, my name is John Doe and my email is [email protected]\")\n",
330+
"\n",
331+
"print(res.validated_output)"
332+
]
333+
}
334+
],
335+
"metadata": {
336+
"kernelspec": {
337+
"display_name": ".venv",
338+
"language": "python",
339+
"name": "python3"
340+
},
341+
"language_info": {
342+
"codemirror_mode": {
343+
"name": "ipython",
344+
"version": 3
345+
},
346+
"file_extension": ".py",
347+
"mimetype": "text/x-python",
348+
"name": "python",
349+
"nbconvert_exporter": "python",
350+
"pygments_lexer": "ipython3",
351+
"version": "3.11.7"
352+
}
353+
},
354+
"nbformat": 4,
355+
"nbformat_minor": 2
356+
}

‎docusaurus/sidebars.js‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -81,6 +81,7 @@ const sidebars = {
8181
"how_to_guides/enable_streaming",
8282
"how_to_guides/generate_structured_data",
8383
"how_to_guides/custom_validators",
84+
"how_to_guides/use_on_fail_actions",
8485
"how_to_guides/hosting_validator_models",
8586
"how_to_guides/hosting_with_docker",
8687
"how_to_guides/continuous_integration_continuous_deployment",

0 commit comments

Comments
 (0)