Skip to content

Refactor presets for attacks and manual tests #154

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
RomiconEZ merged 6 commits into from
Jul 15, 2025
Merged

Refactor presets for attacks and manual tests #154

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
edfba18
Add 'model:llm' tag to various attack descriptions and refactor test …
RomiconEZ Jul 6, 2025
5c75e87
Run pre-commit and small refactor
RomiconEZ Jul 6, 2025
2b95922
Refactor test parameter naming from 'basic_tests_params' to 'basic_te…
RomiconEZ Jul 10, 2025
8d91c07
Run pre-commit
RomiconEZ Jul 10, 2025
cbb908f
add available presets in notebooks
RomiconEZ Jul 10, 2025
2fad915
split cell
RomiconEZ Jul 10, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 4 additions & 4 deletions docs/code_documentation.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -52,20 +52,20 @@ Available Clients
Additional Utility Functions
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

.. autofunction:: llamator.utils.params_example.get_preset_tests_params_example
.. autofunction:: llamator.utils.test_presets.get_test_preset
:noindex:

.. note::

This function generates an example code snippet for configuring basic_tests_params based on a preset configuration.
This function generates an example code snippet for configuring basic_tests based on a preset configuration.
It returns a code snippet as a string.

.. autofunction:: llamator.utils.params_example.print_preset_tests_params_example
.. autofunction:: llamator.utils.test_presets.print_test_preset
:noindex:

.. note::

This function prints an example configuration for basic_tests_params based on a preset to the console.
This function prints an example configuration for basic_tests based on a preset to the console.

.. autofunction:: llamator.client.langchain_integration.print_chat_models_info
:noindex:
Expand Down
15 changes: 9 additions & 6 deletions docs/howtos.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -104,26 +104,29 @@ print(test_result_dict)

## Helper Functions

### `print_preset_tests_params_example`
### `print_test_preset`
Prints example configuration for presets to the console.

Available presets: `all`, `eng`, `llm`, `owasp:llm01`, `owasp:llm07`, `owasp:llm09`, `rus`, `vlm`

**Usage:**

```python
from llamator import print_preset_tests_params_example
from llamator import print_test_preset

# Print configuration for all available tests
print_preset_tests_params_example("all")
print_test_preset("all")
```

### `get_preset_tests_params_example`
### `get_test_preset`
Returns a string containing example configurations for presets.

**Usage:**
```python
from llamator import get_preset_tests_params_example
from llamator import get_test_preset

# Get example for all available tests
all_tests_preset = get_preset_tests_params_example("all")
all_tests_preset = get_test_preset("all")
print(all_tests_preset)
```

Expand Down
72 changes: 40 additions & 32 deletions examples/llamator-api.ipynb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,13 +10,13 @@
]
},
{
"cell_type": "code",
"metadata": {
"ExecuteTime": {
"end_time": "2025-05-29T09:44:45.903323Z",
"start_time": "2025-05-29T09:44:44.049249Z"
"end_time": "2025-07-06T13:58:57.453098Z",
"start_time": "2025-07-06T13:58:57.059519Z"
}
},
"cell_type": "code",
"source": [
"%pip install llamator python-dotenv requests --upgrade --quiet\n",
"%pip show llamator"
Expand All @@ -26,12 +26,11 @@
"name": "stdout",
"output_type": "stream",
"text": [
"Note: you may need to restart the kernel to use updated packages.\n",
"Name: llamator\r\n",
"Version: 3.2.0\r\n",
"Version: 3.3.0\r\n",
"Summary: Framework for testing vulnerabilities of GenAI systems.\r\n",
"Home-page: \r\n",
"Author: \r\n",
"Home-page: https://github.com/LLAMATOR-Core/llamator\r\n",
"Author: Roman Neronov, Timur Nizamov, Nikita Ivanov\r\n",
"Author-email: \r\n",
"License: Attribution 4.0 International\r\n",
"Location: /Users/roman/Projects/PycharmProjects/LLAMATORS/llamator/src\r\n",
Expand All @@ -41,21 +40,19 @@
]
}
],
"execution_count": 6
"execution_count": 1
},
{
"cell_type": "code",
"metadata": {
"ExecuteTime": {
"end_time": "2025-05-29T10:23:28.569845Z",
"start_time": "2025-05-29T10:23:22.843545Z"
"end_time": "2025-07-10T20:48:57.401571Z",
"start_time": "2025-07-10T20:48:57.399176Z"
}
},
"source": [
"import llamator"
],
"cell_type": "code",
"source": "import llamator",
"outputs": [],
"execution_count": 3
"execution_count": 2
},
{
"cell_type": "code",
Expand Down Expand Up @@ -181,39 +178,44 @@
]
},
{
"cell_type": "markdown",
"metadata": {},
"cell_type": "markdown",
"source": [
"## Start testing\n",
"\n",
"The `start_testing()` function starts the execution of vulnerability detection tests in language models (LLM).\n",
"\n",
"Refer to the [code documentation](https://LLAMATOR-Core.github.io/llamator/code_documentation.html#main-functions) for more details.\n",
"Refer to the [code documentation](https://LLAMATOR-Core.github.io/llamator/code_documentation.html#main-functions) for more details."
]
},
{
"metadata": {},
"cell_type": "markdown",
"source": [
"\n",
"### Available Attacks\n",
"\n",
"Check out the `llamator.print_preset_tests_params_example(\"all\")` and [documentation](https://LLAMATOR-Core.github.io/llamator/attack_descriptions.html)."
"Check out the `llamator.print_test_preset(\"all\")` and [documentation](https://LLAMATOR-Core.github.io/llamator/attack_descriptions.html).\n",
"\n",
"Available presets: `all`, `eng`, `llm`, `owasp:llm01`, `owasp:llm07`, `owasp:llm09`, `rus`, `vlm`"
]
},
{
"cell_type": "code",
"metadata": {
"ExecuteTime": {
"end_time": "2025-05-29T10:23:31.662181Z",
"start_time": "2025-05-29T10:23:31.655940Z"
"end_time": "2025-07-10T20:49:03.961794Z",
"start_time": "2025-07-10T20:49:03.955671Z"
}
},
"source": [
"llamator.print_preset_tests_params_example(\"all\")"
],
"cell_type": "code",
"source": "llamator.print_test_preset(\"all\")",
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"# Example configuration for preset 'all':\n",
"basic_tests_params = [\n",
" (\"suffix\", { \"custom_dataset\": None, \"num_attempts\": 0 }),\n",
"basic_tests = [\n",
" (\"aim_jailbreak\", { \"num_attempts\": 0 }),\n",
" (\"autodan_turbo\", {\n",
" \"custom_dataset\": None,\n",
Expand All @@ -236,15 +238,14 @@
" \"multistage_depth\": 5,\n",
" \"num_attempts\": 0\n",
" }),\n",
" (\"dan\", { \"language\": \"any\", \"num_attempts\": 0 }),\n",
" (\"deceptive_delight\", { \"custom_dataset\": None, \"num_attempts\": 0 }),\n",
" (\"dialogue_injection_continuation\", { \"custom_dataset\": None, \"language\": \"any\", \"num_attempts\": 0 }),\n",
" (\"dialogue_injection_devmode\", { \"custom_dataset\": None, \"num_attempts\": 0 }),\n",
" (\"dan\", { \"language\": \"any\", \"num_attempts\": 0 }),\n",
" (\"dialogue_injection_continuation\", { \"custom_dataset\": None, \"language\": \"any\", \"num_attempts\": 0 }),\n",
" (\"ethical_compliance\", { \"custom_dataset\": None, \"num_attempts\": 0 }),\n",
" (\"harmbench\", { \"custom_dataset\": None, \"language\": \"any\", \"num_attempts\": 0 }),\n",
" (\"linguistic_evasion\", { \"num_attempts\": 0 }),\n",
" (\"logical_inconsistencies\", { \"multistage_depth\": 20, \"num_attempts\": 0 }),\n",
" (\"past_tense\", { \"num_attempts\": 0 }),\n",
" (\"pair\", {\n",
" \"custom_dataset\": None,\n",
" \"language\": \"any\",\n",
Expand All @@ -257,8 +258,15 @@
" \"num_attempts\": 0,\n",
" \"num_transformations\": 5\n",
" }),\n",
" (\"suffix\", { \"custom_dataset\": None, \"num_attempts\": 0 }),\n",
" (\"sycophancy\", { \"multistage_depth\": 20, \"num_attempts\": 0 }),\n",
" (\"system_prompt_leakage\", { \"custom_dataset\": None, \"multistage_depth\": 20, \"num_attempts\": 0 }),\n",
" (\"time_machine\", {\n",
" \"custom_dataset\": None,\n",
" \"language\": \"any\",\n",
" \"num_attempts\": 0,\n",
" \"time_context\": \"any\"\n",
" }),\n",
" (\"ucar\", { \"language\": \"any\", \"num_attempts\": 0 }),\n",
" (\"vlm_lowres_docs\", {\n",
" \"custom_pdf_dir\": None,\n",
Expand All @@ -279,7 +287,7 @@
]
}
],
"execution_count": 4
"execution_count": 3
},
{
"cell_type": "code",
Expand Down Expand Up @@ -455,7 +463,7 @@
" }\n",
")\n",
"\n",
"basic_tests_params = [\n",
"basic_tests = [\n",
" (\"shuffle\", {\"num_attempts\": 3, \"num_transformations\": 5, \"custom_dataset\": custom_dataset}),\n",
" (\"harmbench\", {\"language\": \"ru\", \"num_attempts\": 3}),\n",
" (\"pair\", {\"language\": \"ru\", \"num_attempts\": 3, \"multistage_depth\": 5}),\n",
Expand All @@ -475,7 +483,7 @@
" judge_model=judge_model,\n",
" tested_model=tested_client,\n",
" config=config,\n",
" basic_tests=basic_tests_params,\n",
" basic_tests=basic_tests,\n",
")"
]
},
Expand Down
68 changes: 38 additions & 30 deletions examples/llamator-langchain-custom-attack.ipynb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,13 +10,13 @@
]
},
{
"cell_type": "code",
"metadata": {
"ExecuteTime": {
"end_time": "2025-05-29T09:49:52.411435Z",
"start_time": "2025-05-29T09:49:51.206995Z"
"end_time": "2025-07-06T13:59:29.416433Z",
"start_time": "2025-07-06T13:59:29.025054Z"
}
},
"cell_type": "code",
"source": [
"%pip install llamator python-dotenv --upgrade --quiet\n",
"%pip show llamator"
Expand All @@ -26,12 +26,11 @@
"name": "stdout",
"output_type": "stream",
"text": [
"Note: you may need to restart the kernel to use updated packages.\n",
"Name: llamator\r\n",
"Version: 3.2.0\r\n",
"Version: 3.3.0\r\n",
"Summary: Framework for testing vulnerabilities of GenAI systems.\r\n",
"Home-page: \r\n",
"Author: \r\n",
"Home-page: https://github.com/LLAMATOR-Core/llamator\r\n",
"Author: Roman Neronov, Timur Nizamov, Nikita Ivanov\r\n",
"Author-email: \r\n",
"License: Attribution 4.0 International\r\n",
"Location: /Users/roman/Projects/PycharmProjects/LLAMATORS/llamator/src\r\n",
Expand All @@ -41,21 +40,19 @@
]
}
],
"execution_count": 7
"execution_count": 1
},
{
"cell_type": "code",
"metadata": {
"ExecuteTime": {
"end_time": "2025-05-29T10:24:22.519124Z",
"start_time": "2025-05-29T10:24:16.999210Z"
"end_time": "2025-07-10T20:48:39.812221Z",
"start_time": "2025-07-10T20:48:39.810058Z"
}
},
"source": [
"import llamator"
],
"cell_type": "code",
"source": "import llamator",
"outputs": [],
"execution_count": 1
"execution_count": 2
},
{
"cell_type": "code",
Expand Down Expand Up @@ -1184,39 +1181,44 @@
"execution_count": 16
},
{
"cell_type": "markdown",
"metadata": {},
"cell_type": "markdown",
"source": [
"## Start testing\n",
"\n",
"The `start_testing()` function starts the execution of vulnerability detection tests in language models (LLM).\n",
"\n",
"Refer to the [code documentation](https://LLAMATOR-Core.github.io/llamator/code_documentation.html#main-functions) for more details.\n",
"Refer to the [code documentation](https://LLAMATOR-Core.github.io/llamator/code_documentation.html#main-functions) for more details."
]
},
{
"metadata": {},
"cell_type": "markdown",
"source": [
"\n",
"### Available Attacks\n",
"\n",
"Check out the `llamator.print_preset_tests_params_example(\"all\")` and [documentation](https://LLAMATOR-Core.github.io/llamator/attack_descriptions.html)."
"Check out the `llamator.print_test_preset(\"all\")` and [documentation](https://LLAMATOR-Core.github.io/llamator/attack_descriptions.html).\n",
"\n",
"Available presets: `all`, `eng`, `llm`, `owasp:llm01`, `owasp:llm07`, `owasp:llm09`, `rus`, `vlm`"
]
},
{
"cell_type": "code",
"metadata": {
"ExecuteTime": {
"end_time": "2025-05-29T10:24:23.256008Z",
"start_time": "2025-05-29T10:24:23.251225Z"
"end_time": "2025-07-10T20:48:44.130751Z",
"start_time": "2025-07-10T20:48:44.126703Z"
}
},
"source": [
"llamator.print_preset_tests_params_example(\"all\")"
],
"cell_type": "code",
"source": "llamator.print_test_preset(\"all\")",
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"# Example configuration for preset 'all':\n",
"basic_tests_params = [\n",
" (\"suffix\", { \"custom_dataset\": None, \"num_attempts\": 0 }),\n",
"basic_tests = [\n",
" (\"aim_jailbreak\", { \"num_attempts\": 0 }),\n",
" (\"autodan_turbo\", {\n",
" \"custom_dataset\": None,\n",
Expand All @@ -1239,15 +1241,14 @@
" \"multistage_depth\": 5,\n",
" \"num_attempts\": 0\n",
" }),\n",
" (\"dan\", { \"language\": \"any\", \"num_attempts\": 0 }),\n",
" (\"deceptive_delight\", { \"custom_dataset\": None, \"num_attempts\": 0 }),\n",
" (\"dialogue_injection_continuation\", { \"custom_dataset\": None, \"language\": \"any\", \"num_attempts\": 0 }),\n",
" (\"dialogue_injection_devmode\", { \"custom_dataset\": None, \"num_attempts\": 0 }),\n",
" (\"dan\", { \"language\": \"any\", \"num_attempts\": 0 }),\n",
" (\"dialogue_injection_continuation\", { \"custom_dataset\": None, \"language\": \"any\", \"num_attempts\": 0 }),\n",
" (\"ethical_compliance\", { \"custom_dataset\": None, \"num_attempts\": 0 }),\n",
" (\"harmbench\", { \"custom_dataset\": None, \"language\": \"any\", \"num_attempts\": 0 }),\n",
" (\"linguistic_evasion\", { \"num_attempts\": 0 }),\n",
" (\"logical_inconsistencies\", { \"multistage_depth\": 20, \"num_attempts\": 0 }),\n",
" (\"past_tense\", { \"num_attempts\": 0 }),\n",
" (\"pair\", {\n",
" \"custom_dataset\": None,\n",
" \"language\": \"any\",\n",
Expand All @@ -1260,8 +1261,15 @@
" \"num_attempts\": 0,\n",
" \"num_transformations\": 5\n",
" }),\n",
" (\"suffix\", { \"custom_dataset\": None, \"num_attempts\": 0 }),\n",
" (\"sycophancy\", { \"multistage_depth\": 20, \"num_attempts\": 0 }),\n",
" (\"system_prompt_leakage\", { \"custom_dataset\": None, \"multistage_depth\": 20, \"num_attempts\": 0 }),\n",
" (\"time_machine\", {\n",
" \"custom_dataset\": None,\n",
" \"language\": \"any\",\n",
" \"num_attempts\": 0,\n",
" \"time_context\": \"any\"\n",
" }),\n",
" (\"ucar\", { \"language\": \"any\", \"num_attempts\": 0 }),\n",
" (\"vlm_lowres_docs\", {\n",
" \"custom_pdf_dir\": None,\n",
Expand All @@ -1282,7 +1290,7 @@
]
}
],
"execution_count": 2
"execution_count": 3
},
{
"cell_type": "code",
Expand Down
Loading
Loading