Release v3.2.0 (#144)

* Added Deceptive Delight
* Added Dialogue Injection Continuation
* Added VLM Lowres PDFs Attack
* Added VLM M-Attack
* Added VLM Text Hallucination Attack
* Introduced support for Vision Language Model (VLM) attacks, expanding the framework’s multimodal testing capabilities
* Added Dialogue Injection Developer Mode*(formerly "Dialog Injection")
* Renamed Harmful Behavior Multistage to PAIR
* Added scoring to PAIR attack via the Judge Model 
* Revised and translated Harmbench dataset into Russian
* Added `language` column to datasets and enabled filtering attacks by language
* Updated `start_testing` to return a dictionary object with test results
* Removed Complimentary Transition
* Removed Typoglycemia Attack
* Removed legacy `RU_*` attacks (now handled via language-based dataset filtering)

---------

Co-authored-by: Timur Nizamov <abc@nizamovtimur.ru>
Co-authored-by: Nikita Ivanov <nikita.ivanov.778@gmail.com>
Co-authored-by: 3ndetz <jayrawrr3@gmail.com>
Co-authored-by: ti3c2 <ti3c2@yandex.com>
Co-authored-by: svyatocheck <svyatwork2@gmail.com>
Co-authored-by: Egorov, Michil <michil.egorov@x5.ru>

Author: 7 people

.bumpversion.cfg

@@ -1,5 +1,5 @@
 [bumpversion]
-current_version = 3.1.0
+current_version = 3.2.0
 commit = False
 tag = False
 parse = (?P<major>\d+)\.(?P<minor>\d+)\.(?P<patch>\d+)(\-(?P<release>[a-z]+))?

.gitignore

@@ -92,3 +92,5 @@ cmake-build-*/
 /venv/
 /docker/workspace/
 .DS_Store
+
+__MACOSX

CONTRIBUTING.md

@@ -11,7 +11,7 @@ This document provides guidelines and instructions for contributing to this proj
 ### Prerequisites
 
 Before you begin, ensure you have the following installed:
-- Python 3.10
+- Python 3.9, 3.10, or 3.11
 - Git
 
 ### Setting Up Your Development Environment
@@ -159,11 +159,15 @@ from ..attacks import (
 )
 ```
 
-### 4. Test Your Attack
+### 4. Add Your Attack to Documentation
+
+Add info about your attack to `docs/attack_descriptions.md`, observing the structure and alphabetical order.
+
+### 5. Test Your Attack
 
 Run your attack locally using your test setup to verify it works as intended.
 
-### 5. Open a Pull Request
+### 6. Open a Pull Request
 
 Submit your changes for review by opening a pull request to the `main` branch.
 
@@ -182,4 +186,4 @@ Submit your changes for review by opening a pull request to the `main` branch.
 
 4. Incorporate feedback as needed.
 
-5. Once approved, your changes will be merged.
+5. Once approved, your changes will be merged.

README-dev.md README.dev.mdREADME-dev.md renamed to README.dev.md

@@ -1,33 +1,37 @@
-# LLAMATOR
+# LLAMATOR development instructions
 
 # Setup developer environment
 
-To start, you need to setup your local machine.
-
 ## Setup venv
 
-You need to setup virtual environment, simplest way is to run from project root directory:
+Run from project root directory:
 
 ```bash
-$ . ./setup_dev_env.sh
-$ source venv/bin/activate
+. ./setup_dev_env.sh
+```
+```bash
+source venv/bin/activate
 ```
+
 This will create a new venv and run `pip install -r requirements-dev.txt`.
+
 Last line shows how to activate the environment.
 
 ## Install pre-commit
 
 To ensure code quality we use pre-commit hook with several checks. Setup it by:
 
-```
+```bash
 pre-commit install
 ```
 
 All updated files will be reformatted and linted before the commit.
 
-To reformat and lint all files in the project, use:
+Reformat and lint all files in the project:
 
-`pre-commit run --all-files`
+```bash
+pre-commit run --all-files
+```
 
 The used linters are configured in `.pre-commit-config.yaml`. You can use `pre-commit autoupdate` to bump tools to the latest versions.
 
@@ -44,14 +48,15 @@ Read more about different modes in [documentation](https://ipython.org/ipython-d
 
 All code should be in `src/` to make reusability and review straightforward, keep notebooks simple for exploratory data analysis.
 See also [Cookiecutter Data Science opinion](https://drivendata.github.io/cookiecutter-data-science/#notebooks-are-for-exploration-and-communication).
+
 # Project documentation
 
 In `docs/` directory are Sphinx RST/Markdown files.
 
-To build documentation locally, in your configured environment, you can use `build_docs.sh` script:
+Build documentation locally:
 
 ```bash
-$ ./build_docs.sh
+./build_docs.sh
 ```
 
 Then open `public/index.html` file.
@@ -61,13 +66,12 @@ Please read the official [Sphinx documentation](https://www.sphinx-doc.org/en/ma
 
 ### Github Actions Documentation
 
-By default **Github Actions** pipelines have `documentation` workflow which will build sphinx documentation automatically on main branch - and it will push it to a branch - it can be hosted on **Github Pages** if you enable it.
+**Github Actions** pipelines have `documentation` workflow which will build sphinx documentation automatically on release branch - and it will push it to a branch - it can be hosted on **Github Pages** if you enable it.
 
 To access it, you need to enable it, on **Github repository -> Settings -> Pages** page select **Deploy from a branch** and select **gh-pages**. Link will appear here after deployment.
 
-**WARNING:** Only on Github Enterprise you can make it private so only people with repository access can view it.
-
 Please read more about it [here](https://docs.github.com/en/pages/quickstart).
+
 # Semantic version bump
 
 To bump version of the library please use `bump2version` which will update all version strings.
@@ -76,19 +80,25 @@ NOTE: Configuration is in `.bumpversion.cfg` and **this is a main file defining
 
 For convenience there is bash script which will create commit, to use it call:
 
+```bash
+./bump_version.sh minor
+```
+```bash
+./bump_version.sh major
+```
+```bash
+./bump_version.sh patch
+```
+
 <img src="assets/img.png" alt="img" width="250"/>
 
+to see what is going to change run:
+
 ```bash
-# to create a new commit by increasing one semvar:
-$ ./bump_version.sh minor
-$ ./bump_version.sh major
-$ ./bump_version.sh patch
-# to see what is going to change run:
-$ ./bump_version.sh --dry-run major
+./bump_version.sh --dry-run major
 ```
-Script updates **VERSION** file and setup.cfg automatically uses that version.
 
-You can configure it to update version string in other files as well - please check out the bump2version configuration file.
+Script updates **VERSION** file and setup.cfg automatically uses that version.
 
 ## Publishing a New Version to PyPI
 

README.md

@@ -1,6 +1,6 @@
 # ![LLAMATOR](assets/LLAMATOR.svg)
 
-Red Teaming python-framework for testing chatbots and LLM systems
+Red Teaming python-framework for testing chatbots and GenAI systems
 
 [![License: CC BY-NC-SA 4.0](https://img.shields.io/badge/License-CC_BY--NC--SA_4.0-lightgrey.svg)](https://creativecommons.org/licenses/by-nc-sa/4.0/)
 [![PyPI - Python Version](https://img.shields.io/pypi/pyversions/llamator)](https://pypi.org/project/llamator)
@@ -14,7 +14,7 @@ Red Teaming python-framework for testing chatbots and LLM systems
 ## Install 🚀
 
 ```bash
-pip install llamator==3.1.0
+pip install llamator==3.2.0
 ```
 
 ## Documentation 📚
@@ -33,6 +33,7 @@ Documentation Link: [https://LLAMATOR-Core.github.io/llamator](https://LLAMATOR-
 * 💬 [Telegram bot testing via Telethon](https://github.com/LLAMATOR-Core/llamator/blob/release/examples/llamator-telegram.ipynb)
 * 📱 [WhatsApp bot testing via Selenium](https://github.com/LLAMATOR-Core/llamator/blob/release/examples/llamator-whatsapp.ipynb)
 * 🔗 [LangChain client testing with custom attack](https://github.com/LLAMATOR-Core/llamator/blob/release/examples/llamator-langchain-custom-attack.ipynb)
+* 📷 [Vision Model testing via LangChain](https://github.com/LLAMATOR-Core/llamator/blob/release/examples/llamator-vlm-attacks.ipynb)
 * 🐋 [Running LLAMATOR in Docker](https://github.com/LLAMATOR-Core/llamator/blob/release/docker)
 
 
@@ -44,8 +45,8 @@ Documentation Link: [https://LLAMATOR-Core.github.io/llamator](https://LLAMATOR-
 
 ## Unique Features 🌟
 
-* ️🗡 Support for custom attacks from the user
-* 👜 Large selection of attacks on RAG / Agent / Prompt in English and Russian
+* ️🗡 Support for custom attacks and datasets from the user
+* 👜 Large selection of attacks on LLMs, RAGs, Agents, VLMs in English and Russian
 * 🛡 Custom configuration of chat clients
 * 📊 History of attack requests and responses in Excel and CSV format
 * 📄 Test report document in DOCX format