Merge pull request #506 from guardrails-ai/index-on-topic

index on topic nb in docs

Author: zsimjee

docs/examples/response_is_on_topic.ipynb

@@ -14,17 +14,18 @@
     "This validator checks if a text is related with a topic. Using a list of valid topics (which can include one or many) and optionally a list of invalid topics, it validates that the text's main topic is one of the valid ones. If none of the valid topics are relevant, the topic 'Other' will be considered as the most relevant one and the validator will fail.\n",
     "\n",
     "The validator supports 3 different variants:\n",
-    "- Using an ensemble of Zero-Shot classifier + LLM fallback: if the original classification score is less than 0.5, an LLM is used to classify the main topic. This is the default behavior, setting `disable_classifier = False` and `disable_llm = False`.\n",
-    "- Using just a Zero-Shot classifier to get the main topic (`disable_classifier = False` and `disable_llm = True`).\n",
-    "- Using just an LLM to classify the main topic (`disable_classifier = True` and `disable_llm = False`).\n",
+    "\n",
+    "1. Using an ensemble of Zero-Shot classifier + LLM fallback: if the original classification score is less than 0.5, an LLM is used to classify the main topic. This is the default behavior, setting `disable_classifier = False` and `disable_llm = False`.\n",
+    "2. Using just a Zero-Shot classifier to get the main topic (`disable_classifier = False` and `disable_llm = True`).\n",
+    "3. Using just an LLM to classify the main topic (`disable_classifier = True` and `disable_llm = False`).\n",
     "\n",
     "To use the LLM, you can pass in a name of any OpenAI ChatCompletion model like `gpt-3.5-turbo` or `gpt-4` as the `llm_callable`, or pass in a callable that handles LLM calls. This callable can use any LLM, that you define. For simplicity purposes, we show here a demo of using OpenAI's gpt-3.5-turbo model.\n",
     "\n",
     "To use the OpenAI API, you have 3 options:\n",
     "\n",
-    "- Set the OPENAI_API_KEY environment variable: os.environ[\"OPENAI_API_KEY\"] = \"<OpenAI_API_KEY>\"\n",
-    "- Set the OPENAI_API_KEY using openai.api_key=\"<OpenAI_API_KEY>\"\n",
-    "- Pass the api_key as a parameter to the parse function as done below, in this example"
+    "1. Set the OPENAI_API_KEY environment variable: os.environ[\"OPENAI_API_KEY\"] = \"<OpenAI_API_KEY>\"\n",
+    "2. Set the OPENAI_API_KEY using openai.api_key=\"<OpenAI_API_KEY>\"\n",
+    "3. Pass the api_key as a parameter to the parse function"
    ]
   },
   {

guardrails/validators/on_topic.py

@@ -220,7 +220,9 @@ def get_topic_zero_shot(
         score = result["scores"][0]  # type: ignore
         return topic, score  # type: ignore
 
-    def validate(self, value: str, metadata: Dict[str, Any]) -> ValidationResult:
+    def validate(
+        self, value: str, metadata: Optional[Dict[str, Any]]
+    ) -> ValidationResult:
         valid_topics = set(self._valid_topics)
         invalid_topics = set(self._invalid_topics)
 
@@ -236,7 +238,7 @@ def validate(self, value: str, metadata: Dict[str, Any]) -> ValidationResult:
 
         # Add 'other' to the invalid topics list
         if "other" not in invalid_topics:
-            self._invalid_topics.append("other")
+            invalid_topics.add("other")
 
         # Combine valid and invalid topics
         candidate_topics = valid_topics.union(invalid_topics)

mkdocs.yml

@@ -63,6 +63,7 @@ nav:
     - 'Check whether a value is similar to a set of other values': examples/value_within_distribution.ipynb
     - 'Using GuardrailsOutputParser in LlamaIndex': examples/llamaindex-output-parsing.ipynb
     - 'Check if a competitor is named': examples/competitors_check.ipynb
+    - 'Assure that responses are on topic': examples/response_is_on_topic.ipynb
   - 'Integrations':
     - 'Azure OpenAI': integrations/azure_openai.ipynb
     - 'OpenAI Functions': integrations/openai_functions.ipynb

tests/integration_tests/validators/test_on_topic.py

@@ -24,7 +24,6 @@ def test_validate_invalid_topic_cpu_disable_llm(self):
             valid_topics=["sports", "politics"],
             disable_classifier=False,
             disable_llm=True,
-            model_threshold=0.6,
         )
         text = "This is an article about music."
         expected_result = FailResult(error_message="Most relevant topic is other.")