|
1 | 1 | { |
2 | 2 | "cells": [ |
| 3 | + { |
| 4 | + "cell_type": "code", |
| 5 | + "execution_count": null, |
| 6 | + "metadata": {}, |
| 7 | + "outputs": [], |
| 8 | + "source": [ |
| 9 | + "!guardrails hub install hub://guardrails/provenance_embeddings\n", |
| 10 | + "!guardrails hub install hub://guardrails/provenance_llm" |
| 11 | + ] |
| 12 | + }, |
3 | 13 | { |
4 | 14 | "attachments": {}, |
5 | 15 | "cell_type": "markdown", |
|
26 | 36 | "cell_type": "markdown", |
27 | 37 | "metadata": {}, |
28 | 38 | "source": [ |
29 | | - "## Provenance: v0\n", |
| 39 | + "## ProvenanceEmbeddings\n", |
30 | 40 | "\n", |
31 | | - "When you need to confirm that an LLM response is supported by a source text, you can use the `provenance-v0` Guardrails validator function. This function takes a list source articles, an LLM response (or prompt), and a threshold. It validates whether the response is \"close\" enough to the source articles. The threshold is a number between 0 and 1, where 1 means the response is not close to the source articles, and 0 means the response is very close to the source articles.\n", |
| 41 | + "When you need to confirm that an LLM response is supported by a source text, you can use the ```ProvenanceEmbeddings``` Guardrails validator function. This function takes a list source articles, an LLM response (or prompt), and a threshold. It validates whether the response is \"close\" enough to the source articles. The threshold is a number between 0 and 1, where 1 means the response is not close to the source articles, and 0 means the response is very close to the source articles.\n", |
32 | 42 | "\n", |
33 | 43 | "The provenance validator function can also wrap an LLM response (similar to what we saw in the first example). It can also wrap a query function to interact with your vector database as a source. These two uses are out-of-scope for this document.\n", |
34 | 44 | "\n", |
35 | 45 | "For the sake of this post, let's use the following configuration for our validator:\n", |
36 | 46 | "\n", |
37 | 47 | "1. `sources`: List of strings, each representing a source.\n", |
38 | | - "2. `embed function`: A function that takes a list of strings of length k and returns a 2D numpy array of shape (k, embedding_dim).\n" |
| 48 | + "2. `embed function`: A function that takes a list of strings of length k and returns a 2D numpy array of shape (k, embedding_dim)." |
39 | 49 | ] |
40 | 50 | }, |
41 | 51 | { |
|
268 | 278 | "import cohere\n", |
269 | 279 | "import numpy as np\n", |
270 | 280 | "from guardrails import Guard\n", |
271 | | - "from guardrails.validators import ProvenanceV0\n", |
| 281 | + "from guardrails.hub import ProvenanceEmbeddings\n", |
272 | 282 | "from typing import List, Union\n", |
273 | 283 | "import os\n", |
274 | 284 | "\n", |
|
295 | 305 | "cell_type": "markdown", |
296 | 306 | "metadata": {}, |
297 | 307 | "source": [ |
298 | | - "Set up the rail with the `provenance-v0` validator on a single string output. The threshold set for this example is 0.3, which is more sensitive than usual. In this example, since some of the passed-in text is strongly supported by the source text, distance for those chunks is small and the result remains unchanged by Guardrails. However, some other information is not directly supporteds by the source. Guardrails took the Raw LLM output and removed the areas it deemed hallucinated.\n" |
| 308 | + "Set up the rail with the ```ProvenanceEmbeddings``` validator on a single string output. The threshold set for this example is 0.3, which is more sensitive than usual. In this example, since some of the passed-in text is strongly supported by the source text, distance for those chunks is small and the result remains unchanged by Guardrails. However, some other information is not directly supporteds by the source. Guardrails took the Raw LLM output and removed the areas it deemed hallucinated.\n" |
299 | 309 | ] |
300 | 310 | }, |
301 | 311 | { |
|
307 | 317 | "# Initialize the guard object\n", |
308 | 318 | "guard = Guard.from_string(\n", |
309 | 319 | " validators=[\n", |
310 | | - " ProvenanceV0(threshold=0.3, validation_method=\"sentence\", on_fail=\"fix\")\n", |
| 320 | + " ProvenanceEmbeddings(threshold=0.3, validation_method=\"sentence\", on_fail=\"fix\")\n", |
311 | 321 | " ],\n", |
312 | 322 | " description=\"testmeout\",\n", |
313 | 323 | ")" |
|
473 | 483 | "cell_type": "markdown", |
474 | 484 | "metadata": {}, |
475 | 485 | "source": [ |
476 | | - "## Provenance: v1\n", |
| 486 | + "## ProvenanceLLM\n", |
477 | 487 | "\n", |
478 | | - "Guardrails also provides a provenance: v1 validator that uses evaluates provenance for LLM-generated text using an LLM. Currently, to perform this self-evaluation-based provenance check, you can pass in a name of any OpenAI ChatCompletion model like `gpt-3.5-turbo` or `gpt-4`, or pass in a callable that handles LLM calls. This callable can use any LLM, that you define. For simplicity purposes, we show here a demo of using OpenAI's `gpt-3.5-turbo` model.\n", |
| 488 | + "Guardrails also provides a ```ProvenanceLLM``` validator that uses evaluates provenance for LLM-generated text using an LLM. Currently, to perform this self-evaluation-based provenance check, you can pass in a name of any OpenAI ChatCompletion model like `gpt-3.5-turbo` or `gpt-4`, or pass in a callable that handles LLM calls. This callable can use any LLM, that you define. For simplicity purposes, we show here a demo of using OpenAI's `gpt-3.5-turbo` model.\n", |
479 | 489 | "\n", |
480 | 490 | "To use the OpenAI API, you have 3 options:\n", |
481 | 491 | "\n", |
482 | 492 | "- Set the OPENAI_API_KEY environment variable: os.environ[\"OPENAI_API_KEY\"] = \"[OpenAI_API_KEY]\"\n", |
483 | 493 | "- Set the OPENAI_API_KEY using openai.api_key=\"[OpenAI_API_KEY]\"\n", |
484 | | - "- Pass the api_key as a parameter to the parse function as done below, in this example\n" |
| 494 | + "- Pass the api_key as a parameter to the parse function as done below, in this example" |
485 | 495 | ] |
486 | 496 | }, |
487 | 497 | { |
|
491 | 501 | "outputs": [], |
492 | 502 | "source": [ |
493 | 503 | "# Initialize the guard object\n", |
494 | | - "from guardrails.validators import ProvenanceV1\n", |
| 504 | + "from guardrails.hub import ProvenanceLLM\n", |
495 | 505 | "\n", |
496 | 506 | "guard_1 = Guard.from_string(\n", |
497 | 507 | " validators=[\n", |
498 | | - " ProvenanceV1(\n", |
| 508 | + " ProvenanceLLM(\n", |
499 | 509 | " validation_method=\"sentence\", # can be \"sentence\" or \"full\"\n", |
500 | 510 | " llm_callable=\"gpt-3.5-turbo\", # as explained above\n", |
501 | 511 | " top_k=3, # number of chunks to retrieve\n", |
|
0 commit comments