Skip to content

Commit d6e7428

Browse files
authored
Improved quality of custom evaluators notebook (Azure-Samples#171)
* Improved quality of the notebook - `BlocklistEvaluator` wasn't detecting words in `blocklist`. - `blocklist` init parameter for `BlocklistEvaluator` was fixed to be an array of words. * Update Custom_Evaluators.ipynb Ran nbclean
1 parent 19918a3 commit d6e7428

File tree

2 files changed

+11
-3
lines changed

2 files changed

+11
-3
lines changed

scenarios/evaluate/Supported_Evaluation_Metrics/Custom_Evaluators/Custom_Evaluators.ipynb

Lines changed: 10 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -175,7 +175,7 @@
175175
"source": [
176176
"from blocklist import BlocklistEvaluator\n",
177177
"\n",
178-
"blocklist_evaluator = BlocklistEvaluator(blocklist=[\"bad, worst, terrible\"])\n",
178+
"blocklist_evaluator = BlocklistEvaluator(blocklist=[\"bad\", \"worst\", \"terrible\"])\n",
179179
"\n",
180180
"blocklist_evaluator(response=\"New Delhi is Capital of India\")"
181181
]
@@ -231,11 +231,19 @@
231231
"source": [
232232
"pd.DataFrame(results[\"rows\"])"
233233
]
234+
},
235+
{
236+
"cell_type": "code",
237+
"execution_count": null,
238+
"id": "ef355fa6",
239+
"metadata": {},
240+
"outputs": [],
241+
"source": []
234242
}
235243
],
236244
"metadata": {
237245
"kernelspec": {
238-
"display_name": "venv-azureai-samples",
246+
"display_name": ".venv",
239247
"language": "python",
240248
"name": "python3"
241249
},

scenarios/evaluate/Supported_Evaluation_Metrics/Custom_Evaluators/blocklist.py

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -8,5 +8,5 @@ def __init__(self, blocklist) -> None: # noqa: ANN101, ANN001
88
self._blocklist = blocklist
99

1010
def __call__(self: "BlocklistEvaluator", *, response: str): # noqa: ANN204
11-
score = any(word in response for word in self._blocklist)
11+
score = any(word in self._blocklist for word in response.split())
1212
return {"score": score}

0 commit comments

Comments
 (0)