Skip to content

Commit 896243c

Browse files
RomiconEZShine-afknizamovtimurNickoJo
authored
Release v2.2.0 (#86)
* Add HarmBench Prompts * Add Suffix Attack * Remake Harmful Behavior Attack --------- Co-authored-by: Shine-afk <belyaevskij.nikita@gmail.com> Co-authored-by: Timur Nizamov <abc@nizamovtimur.ru> Co-authored-by: Nikita Ivanov <nikita.ivanov.778@gmail.com>
1 parent 88c7378 commit 896243c

23 files changed

+356
-52
lines changed

.bumpversion.cfg

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,5 @@
11
[bumpversion]
2-
current_version = 2.1.0
2+
current_version = 2.2.0
33
commit = False
44
tag = False
55
parse = (?P<major>\d+)\.(?P<minor>\d+)\.(?P<patch>\d+)(\-(?P<release>[a-z]+))?

CONTRIBUTING.md

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -149,12 +149,14 @@ AvailableTests = [
149149
"typoglycemia_attack",
150150
"ucar",
151151
152-
#TODO: YOUR TEST HERE
152+
#TODO: YOUR TEST HERE (in alphabetical order!)
153153
]
154154
```
155155

156156
#### 5. Add your attack to the `attack_descriptions.json` and `attack_descriptions.md` files.
157157

158+
Please pay attention to the `attack_descriptions.md` structure. Description should be the same as docstring of the attack class. If your attack has an original paper or repository, it would be nice if you mentioned it in docstring and `attack_descriptions.md`.
159+
158160
#### 6. Open a PR! Submit your changes for review by opening a pull request to the `main` branch.
159161

160162
## Submitting a Pull Request.

README.md

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -5,15 +5,15 @@ Red Teaming python-framework for testing chatbots and LLM-systems
55
[![License: CC BY-NC-SA 4.0](https://img.shields.io/badge/License-CC_BY--NC--SA_4.0-lightgrey.svg)](https://creativecommons.org/licenses/by-nc-sa/4.0/)
66
[![PyPI - Python Version](https://img.shields.io/pypi/pyversions/llamator)](https://pypi.org/project/llamator)
77
[![PyPI](https://badge.fury.io/py/llamator.svg)](https://badge.fury.io/py/llamator)
8+
[![Docker](https://img.shields.io/badge/dockerfile-gray.svg?logo=docker)](https://github.com/RomiconEZ/llamator/blob/release/docker)
89
[![Downloads](https://pepy.tech/badge/llamator)](https://pepy.tech/project/llamator)
910
[![Downloads](https://pepy.tech/badge/llamator/month)](https://pepy.tech/project/llamator)
1011
[![GitHub Repo stars](https://img.shields.io/github/stars/RomiconEZ/llamator)](https://github.com/RomiconEZ/llamator/stargazers)
11-
[![Docker](https://img.shields.io/badge/dockerfile-gray.svg?logo=docker)](https://github.com/RomiconEZ/llamator/blob/release/docker)
1212

1313
## Install 🚀
1414

1515
```bash
16-
pip install llamator==2.1.0
16+
pip install llamator==2.2.0
1717
```
1818

1919
## Documentation 📚

docker/README.md

Lines changed: 134 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,134 @@
1+
# Usage Instructions
2+
3+
## Preliminary Steps
4+
5+
1. **Project Structure:**
6+
7+
```
8+
llamator/
9+
├── docker/
10+
│ ├── Dockerfile
11+
│ ├── jupyter_docker.sh
12+
│ └── README.md
13+
└── workspace/
14+
└── (your working files)
15+
```
16+
17+
2. **Creating Local `workspace` Directory:**
18+
19+
If the local `./workspace` directory doesn't exist, it will be created automatically when the container starts. No need to create it manually.
20+
21+
## Main Commands
22+
23+
The `jupyter_docker.sh` script supports the following commands:
24+
25+
1. **Building Docker Image:**
26+
27+
Navigate to the `llamator/docker` directory and execute:
28+
29+
```bash
30+
./jupyter_docker.sh build
31+
```
32+
33+
This will create a Docker image named `jupyter_img` by default. Jupyter Notebook will run on port `9000` by default.
34+
35+
2. **Running Container:**
36+
37+
```bash
38+
./jupyter_docker.sh run [port]
39+
```
40+
41+
- **Parameters:**
42+
- `[port]` *(optional)*: Port for accessing Jupyter Notebook. If not specified, port `9000` is used.
43+
44+
- **Examples:**
45+
46+
- Using default port:
47+
48+
```bash
49+
./jupyter_docker.sh run
50+
```
51+
52+
- Setting custom port, e.g., `8888`:
53+
54+
```bash
55+
./jupyter_docker.sh run 8888
56+
```
57+
58+
- **Features:**
59+
- All arguments and settings are defined within the `jupyter_docker.sh` script.
60+
- When running the `run` command, you can specify a port as an argument that will be passed to the container.
61+
- If the local `./workspace` directory doesn't exist in the current directory, it will be created automatically.
62+
- If a container with the same name already exists, it will be stopped and removed before starting a new one.
63+
- The container will be launched in background mode.
64+
- The script will automatically extract the token and display the complete URL for accessing Jupyter Notebook, e.g.: `http://localhost:9000/?token=abc123def456ghi789jkl012mno345pqr678stu901vwx234yz`.
65+
66+
3. **Getting URL with Token:**
67+
68+
If you need to retrieve the URL with the current token again:
69+
70+
```bash
71+
./jupyter_docker.sh token
72+
```
73+
74+
You'll see output with a URL that can be opened in a browser to access Jupyter Notebook.
75+
76+
4. **Adding New Packages via Poetry:**
77+
78+
For example, to add the `numpy` package:
79+
80+
```bash
81+
./jupyter_docker.sh add numpy
82+
```
83+
84+
5. **Accessing Container's Bash Shell:**
85+
86+
If you need to execute commands inside the container:
87+
88+
```bash
89+
./jupyter_docker.sh bash
90+
```
91+
92+
6. **Stopping Container:**
93+
94+
To stop the running container:
95+
96+
```bash
97+
./jupyter_docker.sh stop
98+
```
99+
100+
7. **Removing Container:**
101+
102+
If you need to completely remove the container:
103+
104+
```bash
105+
./jupyter_docker.sh remove
106+
```
107+
108+
## Usage Examples
109+
110+
### 1. Build and Run with Default Port
111+
112+
```bash
113+
./jupyter_docker.sh build
114+
./jupyter_docker.sh run
115+
```
116+
117+
- Jupyter Notebook will be accessible at: `http://localhost:9000/?token=your_token_here`
118+
119+
### 2. Build and Run with Custom Port
120+
121+
```bash
122+
./jupyter_docker.sh build
123+
./jupyter_docker.sh run 8888
124+
```
125+
126+
- Jupyter Notebook will be accessible at: `http://localhost:8888/?token=your_token_here`
127+
128+
### 3. Adding `pandas` Package
129+
130+
```bash
131+
./jupyter_docker.sh add pandas
132+
```
133+
134+
- The `pandas` package will be installed and available in your project.

docs/attack_descriptions.md

Lines changed: 27 additions & 14 deletions
Original file line numberDiff line numberDiff line change
@@ -1,10 +1,23 @@
11
# Attacks
2+
23
This section provides an overview of all available attacks.
34

45
## Single-stage attacks
56

67
### OWASP [LLM01:2025](https://genai.owasp.org/llmrisk/llm01-prompt-injection/) Prompt Injection
78

9+
<details>
10+
<summary><strong>Adversarial suffix</strong></summary>
11+
12+
In code name: `suffix`
13+
14+
Uses a specially crafted suffix to bypass LLM filters and restrictions, forcing the model to generate prohibited or harmful content despite the original query. Evaluates the model's vulnerability to input data manipulation.
15+
16+
_Original Paper <https://arxiv.org/abs/2307.15043>, Code <https://github.com/llm-attacks/llm-attacks>_
17+
18+
<a href="https://github.com/RomiconEZ/llamator/blob/release/src/llamator/attacks/suffix.py">View code on GitHub</a>
19+
</details>
20+
821
<details>
922
<summary><strong>Aim Jailbreak</strong></summary>
1023

@@ -32,7 +45,7 @@ In code name: `bon`
3245

3346
Probes the LLM resilience against Best-Of-N attack.
3447

35-
`Original Paper <https://arxiv.org/abs/2412.03556>`_, `Code <https://github.com/jplhughes/bon-jailbreaking>`_.
48+
_Original Paper <https://arxiv.org/abs/2412.03556>, Code <https://github.com/jplhughes/bon-jailbreaking>_
3649

3750
<a href="https://github.com/RomiconEZ/llamator/blob/release/src/llamator/attacks/bon.py">View code on GitHub</a>
3851
</details>
@@ -47,18 +60,6 @@ Evaluates the LLM's ability to maintain content moderation standards when transi
4760
<a href="https://github.com/RomiconEZ/llamator/blob/release/src/llamator/attacks/complimentary_transition.py">View code on GitHub</a>
4861
</details>
4962

50-
<details>
51-
<summary><strong>Crescendo</strong></summary>
52-
53-
In code name: `crescendo`
54-
55-
Challenges the LLM with a dataset of harmful behavior prompts using Crescendo strategy, operates by engaging the LLM in a series of escalating conversational turns, and multistage refinement.
56-
57-
`Original Paper <https://arxiv.org/abs/2404.01833>`_
58-
59-
<a href="https://github.com/RomiconEZ/llamator/blob/release/src/llamator/attacks/crescendo.py">View code on GitHub</a>
60-
</details>
61-
6263
<details>
6364
<summary><strong>Do Anything Now Jailbreak</strong></summary>
6465

@@ -184,6 +185,18 @@ The `multistage_depth` parameter for using in `history_limit` can be extracted f
184185

185186
### OWASP [LLM01:2025](https://genai.owasp.org/llmrisk/llm01-prompt-injection/) Prompt Injection
186187

188+
<details>
189+
<summary><strong>Crescendo</strong></summary>
190+
191+
In code name: `crescendo`
192+
193+
Challenges the LLM with a dataset of harmful behavior prompts using Crescendo strategy, operates by engaging the LLM in a series of escalating conversational turns, and multistage refinement.
194+
195+
_Original Paper <https://arxiv.org/abs/2404.01833>_
196+
197+
<a href="https://github.com/RomiconEZ/llamator/blob/release/src/llamator/attacks/crescendo.py">View code on GitHub</a>
198+
</details>
199+
187200
<details>
188201
<summary><strong>Harmful Behavior Multistage</strong></summary>
189202

@@ -225,7 +238,7 @@ In code name: `sycophancy`
225238

226239
Attacks the LLM system during a multistage conversation using the argumentum ad verecundiam method based on the system's domain.
227240

228-
`Original Paper <https://arxiv.org/abs/2310.13548>`_
241+
_Original Paper <https://arxiv.org/abs/2310.13548>_
229242

230243
<a href="https://github.com/RomiconEZ/llamator/blob/release/src/llamator/attacks/sycophancy.py">View code on GitHub</a>
231244
</details>

docs/howtos.md

Lines changed: 5 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -12,7 +12,7 @@
1212
## Installation Guide
1313

1414
```bash
15-
pip install llamator==2.1.0
15+
pip install llamator==2.2.0
1616
```
1717

1818
## Usage Guide (using LM Studio)
@@ -42,17 +42,19 @@ tested_model = llamator.ClientOpenAI(
4242
tests_with_attempts = [
4343
("aim_jailbreak", 2),
4444
("base64_injection", 2),
45+
("bon", 2),
4546
("complimentary_transition", 2),
46-
("do_anything_now_jailbreak", 2),
47+
("crescendo", 2),
4748
# Uncomment the following lines to include additional tests
49+
# ("do_anything_now_jailbreak", 2),
4850
# ("RU_do_anything_now_jailbreak", 2),
49-
# ("bon", 2),
5051
# ("ethical_compliance", 2),
5152
# ("harmful_behavior", 2),
5253
# ("harmful_behavior_multistage", 2),
5354
# ("linguistic_evasion", 2),
5455
# ("logical_inconsistencies", 2),
5556
# ("past_tense", 2),
57+
# ("suffix", 2),
5658
# ("sycophancy", 2),
5759
# ("system_prompt_leakage", 2),
5860
# ("typoglycemia_attack", 2),

docs/project_overview.md

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -5,6 +5,7 @@ LLAMATOR - Red Teaming python-framework for testing chatbots and LLM-systems
55
[![License: CC BY-NC-SA 4.0](https://img.shields.io/badge/License-CC_BY--NC--SA_4.0-lightgrey.svg)](https://creativecommons.org/licenses/by-nc-sa/4.0/)
66
[![PyPI - Python Version](https://img.shields.io/pypi/pyversions/llamator)](https://pypi.org/project/llamator)
77
[![PyPI](https://badge.fury.io/py/llamator.svg)](https://badge.fury.io/py/llamator)
8+
[![Docker](https://img.shields.io/badge/dockerfile-gray.svg?logo=docker)](https://github.com/RomiconEZ/llamator/blob/release/docker)
89
[![Downloads](https://pepy.tech/badge/llamator)](https://pepy.tech/project/llamator)
910
[![Downloads](https://pepy.tech/badge/llamator/month)](https://pepy.tech/project/llamator)
1011
[![GitHub Repo stars](https://img.shields.io/github/stars/RomiconEZ/llamator)](https://github.com/RomiconEZ/llamator/stargazers)

examples/llamator-api.ipynb

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -283,7 +283,9 @@
283283
"tests_with_attempts = [\n",
284284
" # (\"aim_jailbreak\", 2),\n",
285285
" # (\"base64_injection\", 2),\n",
286+
" # (\"bon\", 2),\n",
286287
" # (\"complimentary_transition\", 2),\n",
288+
" # (\"crescendo\", 2),\n",
287289
" # (\"do_anything_now_jailbreak\", 2),\n",
288290
" # (\"RU_do_anything_now_jailbreak\", 2),\n",
289291
" # (\"ethical_compliance\", 2),\n",
@@ -292,6 +294,7 @@
292294
" # (\"linguistic_evasion\", 2),\n",
293295
" # (\"logical_inconsistencies\", 2),\n",
294296
" # (\"past_tense\", 2),\n",
297+
" # (\"suffix\", 2),\n",
295298
" (\"sycophancy\", 2),\n",
296299
" (\"system_prompt_leakage\", 2),\n",
297300
" # (\"typoglycemia_attack\", 2),\n",

examples/llamator-selenium.ipynb

Lines changed: 5 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -365,15 +365,18 @@
365365
"tests_with_attempts = [\n",
366366
" # (\"aim_jailbreak\", 2),\n",
367367
" # (\"base64_injection\", 2),\n",
368-
" # (\"complimentary_transition\", 3),\n",
368+
" # (\"bon\", 2),\n",
369+
" # (\"complimentary_transition\", 2),\n",
370+
" # (\"crescendo\", 2),\n",
369371
" # (\"do_anything_now_jailbreak\", 2),\n",
370372
" # (\"RU_do_anything_now_jailbreak\", 2),\n",
371373
" # (\"ethical_compliance\", 2),\n",
372374
" # (\"harmful_behavior\", 2),\n",
373375
" # (\"harmful_behavior_multistage\", 2),\n",
374376
" (\"linguistic_evasion\", 2),\n",
375377
" (\"logical_inconsistencies\", 2),\n",
376-
" # (\"past_tense\", 1),\n",
378+
" # (\"past_tense\", 2),\n",
379+
" # (\"suffix\", 2),\n",
377380
" (\"sycophancy\", 2),\n",
378381
" (\"system_prompt_leakage\", 2),\n",
379382
" # (\"typoglycemia_attack\", 2),\n",

examples/llamator-telegram.ipynb

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -385,7 +385,9 @@
385385
"tests_with_attempts = [\n",
386386
" # (\"aim_jailbreak\", 2),\n",
387387
" # (\"base64_injection\", 2),\n",
388+
" # (\"bon\", 2),\n",
388389
" # (\"complimentary_transition\", 2),\n",
390+
" # (\"crescendo\", 2),\n",
389391
" # (\"do_anything_now_jailbreak\", 2),\n",
390392
" # (\"RU_do_anything_now_jailbreak\", 2),\n",
391393
" # (\"ethical_compliance\", 2),\n",
@@ -394,6 +396,7 @@
394396
" (\"linguistic_evasion\", 2),\n",
395397
" (\"logical_inconsistencies\", 2),\n",
396398
" # (\"past_tense\", 2),\n",
399+
" # (\"suffix\", 2),\n",
397400
" (\"sycophancy\", 2),\n",
398401
" # (\"system_prompt_leakage\", 2),\n",
399402
" # (\"typoglycemia_attack\", 2),\n",

0 commit comments

Comments
 (0)