Skip to content

Commit c898881

Browse files
committed
Release v3.3.0
2 parents d20351b + 78e5346 commit c898881

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

55 files changed

+5695
-5130
lines changed

.bumpversion.cfg

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,5 @@
11
[bumpversion]
2-
current_version = 3.2.0
2+
current_version = 3.3.0
33
commit = False
44
tag = False
55
parse = (?P<major>\d+)\.(?P<minor>\d+)\.(?P<patch>\d+)(\-(?P<release>[a-z]+))?

README.md

Lines changed: 8 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -14,7 +14,7 @@ Red Teaming python-framework for testing chatbots and GenAI systems
1414
## Install 🚀
1515

1616
```bash
17-
pip install llamator==3.2.0
17+
pip install llamator==3.3.0
1818
```
1919

2020
## Documentation 📚
@@ -59,15 +59,16 @@ Documentation Link: [https://LLAMATOR-Core.github.io/llamator](https://LLAMATOR-
5959

6060
## Community 🌍
6161

62-
* 💬 [Telegram Chat — LLAMATOR | AI Red Team Community](https://t.me/llamator)
63-
* 📣 [Telegram Channel — AI Security Lab](https://t.me/aisecuritylab)
62+
* 💬 Telegram Chat — [LLAMATOR | AI Red Team Community](https://t.me/llamator)
63+
* 🖤 Supported by [AI Talent Hub and ITMO University](https://ai.itmo.ru)
6464

65+
## Contacts 📬
6566

66-
## Supported by 🚀
67+
✉️ **Email**: [team@llamator.org](mailto:team@llamator.org)
6768

68-
* [AI Security Lab ITMO](https://ai.itmo.ru/aisecuritylab)
69-
* [Raft Security](https://raftds.ru/)
70-
* [AI Talent Hub](https://ai.itmo.ru/)
69+
## Enterprise Version 🏢
70+
71+
⚡️<span style="color:#FFC107;"><em>Coming Soon</em></span>⚡️
7172

7273
## License 📜
7374

docs/attack_descriptions.md

Lines changed: 22 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -13,7 +13,7 @@ In code name: `suffix`
1313

1414
Uses a specially crafted suffix to bypass LLM filters and restrictions, forcing the model to generate prohibited or harmful content despite the original query. Evaluates the model's vulnerability to input data manipulation.
1515

16-
_Original Paper: <https://arxiv.org/abs/2307.15043>, Original Code: <https://github.com/llm-attacks/llm-attacks>_
16+
_Original Paper: <https://arxiv.org/abs/2307.15043>, Code: <https://github.com/llm-attacks/llm-attacks>_
1717

1818
<a href="https://github.com/LLAMATOR-Core/llamator/blob/release/src/llamator/attacks/suffix.py">View code on GitHub</a>
1919
</details>
@@ -45,7 +45,7 @@ In code name: `bon`
4545

4646
Probes the LLM resilience against Best-Of-N (BoN) attack by randomly transform symbols in input harmful prompt.
4747

48-
_Original Paper: <https://arxiv.org/abs/2412.03556>, Original Code: <https://github.com/jplhughes/bon-jailbreaking>_
48+
_Original Paper: <https://arxiv.org/abs/2412.03556>, Code: <https://github.com/jplhughes/bon-jailbreaking>_
4949

5050
<a href="https://github.com/LLAMATOR-Core/llamator/blob/release/src/llamator/attacks/bon.py">View code on GitHub</a>
5151
</details>
@@ -127,13 +127,15 @@ Leverages the use of a non-Russian language to circumvent the restrictions or fo
127127
</details>
128128

129129
<details>
130-
<summary><strong>Past Tense Attack</strong></summary>
130+
<summary><strong>Linguistic Sandwich</strong></summary>
131131

132-
In code name: `past_tense`
132+
In code name: `linguistic_sandwich`
133133

134-
Challenges the LLM with a dataset of harmful behavior prompts in past tense form.
134+
Exploits attention blink phenomena by hiding adversarial questions between innocent questions in low-resource languages. The attack leverages the concept that LLMs may have reduced attention when processing mixed low-resource languages.
135135

136-
<a href="https://github.com/LLAMATOR-Core/llamator/blob/release/src/llamator/attacks/past_tense.py">View code on GitHub</a>
136+
_Original Paper: <https://arxiv.org/abs/2404.07242v1>_
137+
138+
<a href="https://github.com/LLAMATOR-Core/llamator/blob/release/src/llamator/attacks/linguistic_sandwich.py">View code on GitHub</a>
137139
</details>
138140

139141
<details>
@@ -143,11 +145,21 @@ In code name: `shuffle`
143145

144146
Attempts to bypass LLM's defense mechanism by randomly shuffling elements in input harmful prompt.
145147

146-
_Original Paper: <https://arxiv.org/html/2501.04931>_
148+
_Original Paper: <https://arxiv.org/abs/2501.04931>_
147149

148150
<a href="https://github.com/RomiconEZ/llamator/blob/release/src/llamator/attacks/shuffle.py">View code on GitHub</a>
149151
</details>
150152

153+
<details>
154+
<summary><strong>Time Machine Attack</strong></summary>
155+
156+
In code name: `time_machine`
157+
158+
Uses an attack model to dynamically transform prompts into historical or future contexts to test temporal framing vulnerabilities.
159+
160+
<a href="https://github.com/LLAMATOR-Core/llamator/blob/release/src/llamator/attacks/time_machine.py">View code on GitHub</a>
161+
</details>
162+
151163
<details>
152164
<summary><strong>UCAR Jailbreak</strong></summary>
153165

@@ -163,7 +175,7 @@ Probes an LLM's adherence to content restrictions when explicitly requested to i
163175
<details>
164176
<summary><strong>What Drives the Multi-stage?</strong></summary>
165177

166-
Multi-stage attacks are inspired by the [Jailbreaking Black Box Large Language Models in Twenty Queries (PAIR)](https://arxiv.org/html/2310.08419) paper.
178+
Multi-stage attacks are inspired by the [Jailbreaking Black Box Large Language Models in Twenty Queries (PAIR)](https://arxiv.org/abs/2310.08419) paper.
167179

168180
For managing a multi-stage interaction between an attacker and tested chat clients, the `MultiStageInteractionSession` class is available [[source]](https://github.com/LLAMATOR-Core/llamator/blob/release/src/llamator/client/chat_client.py). It contains the following properties:
169181
* `attacker_session` is the session for the attacker.
@@ -185,7 +197,7 @@ In code name: `autodan_turbo`
185197

186198
Implements the AutoDAN-Turbo attack methodology which uses a lifelong agent for strategy self-exploration to jailbreak LLMs. This attack automatically discovers jailbreak strategies without human intervention and combines them for more effective attacks.
187199

188-
_Original Paper: <https://arxiv.org/abs/2410.05295v3>, Original Code: <https://github.com/SaFoLab-WISC/AutoDAN-Turbo>_
200+
_Original Paper: <https://arxiv.org/abs/2410.05295v3>, Code: <https://github.com/SaFoLab-WISC/AutoDAN-Turbo>_
189201

190202
<a href="https://github.com/LLAMATOR-Core/llamator/blob/release/src/llamator/attacks/autodan_turbo.py">View code on GitHub</a>
191203
</details>
@@ -207,7 +219,7 @@ _Original Paper: <https://arxiv.org/abs/2404.01833>_
207219

208220
In code name: `pair`
209221

210-
Challenges the LLM with a dataset of harmful behavior prompts using multistage refinement with judge model scoring.
222+
Challenges the LLM with a dataset of adversarial prompts using multistage refinement with judge model scoring.
211223
Original Paper: https://arxiv.org/abs/2310.08419v4, Code: https://github.com/patrickrchao/JailbreakingLLMs
212224

213225
<a href="https://github.com/LLAMATOR-Core/llamator/blob/release/src/llamator/attacks/pair.py">View code on GitHub</a>

docs/code_documentation.rst

Lines changed: 4 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -52,20 +52,20 @@ Available Clients
5252
Additional Utility Functions
5353
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
5454

55-
.. autofunction:: llamator.utils.params_example.get_preset_tests_params_example
55+
.. autofunction:: llamator.utils.test_presets.get_test_preset
5656
:noindex:
5757

5858
.. note::
5959

60-
This function generates an example code snippet for configuring basic_tests_params based on a preset configuration.
60+
This function generates an example code snippet for configuring basic_tests based on a preset configuration.
6161
It returns a code snippet as a string.
6262

63-
.. autofunction:: llamator.utils.params_example.print_preset_tests_params_example
63+
.. autofunction:: llamator.utils.test_presets.print_test_preset
6464
:noindex:
6565

6666
.. note::
6767

68-
This function prints an example configuration for basic_tests_params based on a preset to the console.
68+
This function prints an example configuration for basic_tests based on a preset to the console.
6969

7070
.. autofunction:: llamator.client.langchain_integration.print_chat_models_info
7171
:noindex:

docs/howtos.md

Lines changed: 10 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -13,7 +13,7 @@
1313
## Installation Guide
1414

1515
```bash
16-
pip install llamator==3.2.0
16+
pip install llamator==3.3.0
1717
```
1818

1919
## Usage Guide (using LM Studio)
@@ -104,26 +104,29 @@ print(test_result_dict)
104104

105105
## Helper Functions
106106

107-
### `print_preset_tests_params_example`
107+
### `print_test_preset`
108108
Prints example configuration for presets to the console.
109109

110+
Available presets: `all`, `eng`, `llm`, `owasp:llm01`, `owasp:llm07`, `owasp:llm09`, `rus`, `vlm`
111+
110112
**Usage:**
113+
111114
```python
112-
from llamator import print_preset_tests_params_example
115+
from llamator import print_test_preset
113116

114117
# Print configuration for all available tests
115-
print_preset_tests_params_example("all")
118+
print_test_preset("all")
116119
```
117120

118-
### `get_preset_tests_params_example`
121+
### `get_test_preset`
119122
Returns a string containing example configurations for presets.
120123

121124
**Usage:**
122125
```python
123-
from llamator import get_preset_tests_params_example
126+
from llamator import get_test_preset
124127

125128
# Get example for all available tests
126-
all_tests_preset = get_preset_tests_params_example("all")
129+
all_tests_preset = get_test_preset("all")
127130
print(all_tests_preset)
128131
```
129132

docs/project_overview.md

Lines changed: 2 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -33,15 +33,8 @@ LLAMATOR - Red Teaming python-framework for testing chatbots and GenAI systems
3333

3434
## Community
3535

36-
* 💬 [Telegram Chat — LLAMATOR | AI Red Team Community](https://t.me/llamator)
37-
* 📣 [Telegram Channel — AI Security Lab](https://t.me/aisecuritylab)
38-
39-
40-
## Supported by
41-
42-
* [AI Security Lab ITMO](https://ai.itmo.ru/aisecuritylab)
43-
* [Raft Security](https://raftds.ru/)
44-
* [AI Talent Hub](https://ai.itmo.ru/aisecuritylab)
36+
* 💬 Telegram Chat — [LLAMATOR | AI Red Team Community](https://t.me/llamator)
37+
* 🖤 Supported by [AI Talent Hub and ITMO University](https://ai.itmo.ru)
4538

4639
## License
4740

0 commit comments

Comments
 (0)