diff --git a/docs/concepts/error_remediation.md b/docs/concepts/error_remediation.md index e9178ed52..05f03d60b 100644 --- a/docs/concepts/error_remediation.md +++ b/docs/concepts/error_remediation.md @@ -37,6 +37,8 @@ Guardrails provides a number of `OnFailActions` for when a validator fails. The | `OnFailAction.FIX_REASK` | First, fix the generated output deterministically, and then rerun validation with the deterministically fixed output. If validation fails, then perform reasking. | No | +Custom OnFailActions can also be implemented, see the `custom` section in the [how to use on fail actions guide](../how_to_guides/use_on_fail_actions). + ## Guidance on dealing with Validator errors When a validator fails, Guardrails does two things diff --git a/docs/how_to_guides/use_on_fail_actions.ipynb b/docs/how_to_guides/use_on_fail_actions.ipynb new file mode 100644 index 000000000..7d04ead8f --- /dev/null +++ b/docs/how_to_guides/use_on_fail_actions.ipynb @@ -0,0 +1,356 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "# Use `on_fail` Actions\n", + "\n", + "`on_fail` actions are instructions to Guardrails that direct action when a validator fails. They are set at the validator level, not the guard level. \n", + "\n", + "The full set of on_fail actions are avialable in the [Error remediation concepts doc](https://www.guardrailsai.com/docs/concepts/error_remediation), and will not all be covered here.\n", + "\n", + "Instead, this interactive doc will serve to guide you on how and when to use different `on_fail` actions." + ] + }, + { + "cell_type": "code", + "execution_count": 1, + "metadata": {}, + "outputs": [ + { + "name": "stderr", + "output_type": "stream", + "text": [ + "/home/zayd/workspace/testbench/.venv/lib/python3.11/site-packages/torch/cuda/__init__.py:654: UserWarning: Can't initialize NVML\n", + " warnings.warn(\"Can't initialize NVML\")\n" + ] + } + ], + "source": [ + "# setup, run imports\n", + "from guardrails import Guard, install\n", + "try:\n", + " from guardrails.hub import DetectPII\n", + "except ImportError:\n", + " install(\"hub://guardrails/detect_pii\")\n", + " from guardrails.hub import DetectPII" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "vscode": { + "languageId": "plaintext" + } + }, + "source": [ + "## Raise Exceptions\n", + "\n", + "If you do not want to change the outputs of an LLM when validation fails, you can instead wrap your guard in a try/catch block.\n", + "\n", + "This works particularly well for input validation." + ] + }, + { + "cell_type": "code", + "execution_count": 6, + "metadata": {}, + "outputs": [ + { + "name": "stderr", + "output_type": "stream", + "text": [ + "/home/zayd/workspace/testbench/.venv/lib/python3.11/site-packages/guardrails/validator_service/__init__.py:85: UserWarning: Could not obtain an event loop. Falling back to synchronous validation.\n", + " warnings.warn(\n" + ] + }, + { + "name": "stdout", + "output_type": "stream", + "text": [ + "output validation failed\n", + "Validation failed for field with errors: The following text in your response contains PII:\n", + "Hello, my name is John Doe and my email is john.doe@example.com\n", + "input validation failed\n", + "Validation failed for field with errors: The following text in your response contains PII:\n", + "Hello, my name is John Doe and my email is john.doe@example.com\n" + ] + } + ], + "source": [ + "guard = Guard().use(\n", + " DetectPII(pii_entities=\"pii\", on_fail=\"exception\")\n", + ")\n", + "\n", + "try:\n", + " guard.validate(\"Hello, my name is John Doe and my email is john.doe@example.com\")\n", + "except Exception as e:\n", + " print(\"output validation failed\")\n", + " print(e)\n", + "\n", + "\n", + "# on input validation\n", + "\n", + "guard = Guard().use(\n", + " DetectPII(pii_entities=\"pii\", on_fail=\"exception\"),\n", + " on=\"msg_history\"\n", + ")\n", + "\n", + "\n", + "# on input validation\n", + "\n", + "guard = Guard().use(\n", + " DetectPII(pii_entities=\"pii\", on_fail=\"exception\"),\n", + " on=\"msg_history\"\n", + ")\n", + "\n", + "try:\n", + " guard(\n", + " model='gpt-4o-mini',\n", + " messages=[{\n", + " \"role\": \"user\",\n", + " \"content\": \"Hello, my name is John Doe and my email is john.doe@example.com\"\n", + " }]\n", + " )\n", + "except Exception as e:\n", + " print(\"input validation failed\")\n", + " print(e)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## `noop` to log and continue\n", + "\n", + "If you want to log the error and continue, you can use the `noop` action. This is useful for when you want to log the error, but not stop the LLM from running." + ] + }, + { + "cell_type": "code", + "execution_count": 10, + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "guarded just fine\n", + "Check if validation passed: False\n", + "Show that the validated text and raw text remain the same: True\n" + ] + } + ], + "source": [ + "# the on_fail parameter does not have to be set, as the default is \"noop\"\n", + "\n", + "guard = Guard().use(\n", + " DetectPII(pii_entities=\"pii\", on_fail=\"noop\")\n", + ")\n", + "\n", + "res = guard.validate(\"Hello, my name is John Doe and my email is john.doe@example.com\")\n", + "print(\"guarded just fine\")\n", + "print(\"Check if validation passed: \", res.validation_passed)\n", + "print(\"Show that the validated text and raw text remain the same: \",\n", + " res.validated_output == res.raw_llm_output)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## `fix` to automatically fix the error\n", + "\n", + "Note, not all validators implement a 'fix' value. You can view the FailResult implementation in a validator to see if it has a fix value.\n", + "\n", + "Here's [an example](https://github.com/guardrails-ai/detect_pii/blob/48b15716460fe9e4e5b83ba9607ce3764d9b6d2e/validator/main.py#L188) that shows how Detect PII is written to return anonymized text as a fix value" + ] + }, + { + "cell_type": "code", + "execution_count": 11, + "metadata": {}, + "outputs": [ + { + "name": "stderr", + "output_type": "stream", + "text": [ + "/home/zayd/workspace/testbench/.venv/lib/python3.11/site-packages/guardrails/validator_service/__init__.py:85: UserWarning: Could not obtain an event loop. Falling back to synchronous validation.\n", + " warnings.warn(\n" + ] + }, + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Check if validated_output is valid text: True\n", + "Scrubbed text: Hello, my name is and my email is \n" + ] + } + ], + "source": [ + "guard = Guard().use(\n", + " DetectPII(pii_entities=\"pii\", on_fail=\"fix\")\n", + ")\n", + "\n", + "res = guard.validate(\"Hello, my name is John Doe and my email is john.doe@example.com\")\n", + "\n", + "print(\"Check if validated_output is valid text: \", res.validation_passed)\n", + "print(\"Scrubbed text: \", res.validated_output)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## `reask` to automatically ask for an output that passes validation\n", + "\n", + "This reask prompt is computed from the validators themselves. It's an interpolation \n", + "\n", + "In order for the reask prompt to work, the following additional params must be provided:\n", + "\n", + "- messages\n", + "- llm_api OR model" + ] + }, + { + "cell_type": "code", + "execution_count": 41, + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Validated output: Sure! Here's a fictional person without any personal identifiable information:\n", + "\n", + "**Name:** \n", + "**Email:** \n", + "\n", + "Feel free to use this for any creative purposes!\n", + "Number of reasks: 1\n" + ] + } + ], + "source": [ + "guard = Guard().use(\n", + " DetectPII(pii_entities=\"pii\", on_fail=\"reask\"),\n", + ")\n", + "\n", + "res = guard(\n", + " messages=[{\n", + " \"role\": \"user\",\n", + " \"content\": \"Make up a fake person and email address\",\n", + " }],\n", + " model='gpt-4o-mini',\n", + " num_reasks=1\n", + ")\n", + "\n", + "print(\"Validated output: \", res.validated_output)\n", + "print(\"Number of reasks: \", len(guard.history.last.iterations) - 1)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## `custom` to anything else" + ] + }, + { + "cell_type": "code", + "execution_count": 53, + "metadata": {}, + "outputs": [ + { + "name": "stderr", + "output_type": "stream", + "text": [ + "/home/zayd/workspace/testbench/.venv/lib/python3.11/site-packages/guardrails/validator_service/__init__.py:85: UserWarning: Could not obtain an event loop. Falling back to synchronous validation.\n", + " warnings.warn(\n" + ] + }, + { + "name": "stdout", + "output_type": "stream", + "text": [ + "CUSTOM LOGIC COMPLETE!\n" + ] + } + ], + "source": [ + "# A custom on_fail can be as simple as a function\n", + "\n", + "def custom_on_fail(value, fail_result):\n", + " # This will turn up in validated output\n", + " return \"CUSTOM LOGIC COMPLETE!\"\n", + "\n", + "guard = Guard().use(\n", + " DetectPII(pii_entities=\"pii\", on_fail=custom_on_fail),\n", + ")\n", + "res = guard.validate(\"Hello, my name is John Doe and my email is john.doe@example.com\")\n", + "\n", + "print(res.validated_output)" + ] + }, + { + "cell_type": "code", + "execution_count": 57, + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "\n", + "String validated: Hello, my name is John Doe and my email is john.doe@example.com\n", + "\n", + "Reason it failed: [ErrorSpan(start=18, end=26, reason='PII detected in John Doe')]\n", + "\n" + ] + } + ], + "source": [ + "# Of course, the function also has access to the fail_result and source text, \n", + "# so interesting logic/formatting over those is also possible\n", + "# Here, we show the specific char spans where the validator detected malfeasance\n", + "\n", + "def custom_on_fail(value, fail_result):\n", + " return f\"\"\"\n", + "String validated: {value}\n", + "\n", + "Reasons it failed: {fail_result.error_spans}\n", + "\"\"\"\n", + "\n", + "guard = Guard().use(\n", + " DetectPII(pii_entities=\"pii\", on_fail=custom_on_fail),\n", + ")\n", + "res = guard.validate(\"Hello, my name is John Doe and my email is john.doe@example.com\")\n", + "\n", + "print(res.validated_output)" + ] + } + ], + "metadata": { + "kernelspec": { + "display_name": ".venv", + "language": "python", + "name": "python3" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.11.7" + } + }, + "nbformat": 4, + "nbformat_minor": 2 +} diff --git a/docusaurus/sidebars.js b/docusaurus/sidebars.js index 3314d40f1..e1cd48414 100644 --- a/docusaurus/sidebars.js +++ b/docusaurus/sidebars.js @@ -81,6 +81,7 @@ const sidebars = { "how_to_guides/enable_streaming", "how_to_guides/generate_structured_data", "how_to_guides/custom_validators", + "how_to_guides/use_on_fail_actions", "how_to_guides/hosting_validator_models", "how_to_guides/hosting_with_docker", "how_to_guides/continuous_integration_continuous_deployment",