Skip to content

Commit 283d01e

Browse files
msoedovmsoedov
committed
feat(Update readme):
feat(bump version): fix(pyproject):
1 parent 0ea4221 commit 283d01e

File tree

2 files changed

+85
-28
lines changed

2 files changed

+85
-28
lines changed

‎Readme.md‎

Lines changed: 82 additions & 26 deletions
Original file line numberDiff line numberDiff line change
@@ -61,7 +61,7 @@ agentic_security --port=PORT --host=HOST
6161

6262
## UI 🧙
6363

64-
<img width="100%" alt="booking-screen" src="https://res.cloudinary.com/do9qa2bqr/image/upload/v1713002396/1-ezgif.com-video-to-gif-converter_s2hsro.gif">
64+
<img width="100%" alt="booking-screen" src="https://res.cloudinary.com/dq0w2rtm9/image/upload/v1736433557/z0bsyzhsqlgcr3w4ovwp.gif">
6565

6666
## LLM kwargs
6767

@@ -101,41 +101,101 @@ To add your own dataset you can place one or multiples csv files with `prompt` c
101101

102102
## Run as CI check
103103

104-
ci.py
104+
Init config
105+
```shell
106+
agentic_security init
105107

106-
```python
107-
from agentic_security import AgenticSecurity
108+
2025-01-08 20:12:02.449 | INFO | agentic_security.lib:generate_default_cfg:324 - Default configuration generated successfully to agesec.toml.
109+
110+
```
111+
112+
default config sample
113+
```toml
108114

109-
spec = """
115+
[general]
116+
# General configuration for the security scan
117+
llmSpec = """
110118
POST http://0.0.0.0:8718/v1/self-probe
111119
Authorization: Bearer XXXXX
112120
Content-Type: application/json
113121
114122
{
115123
"prompt": "<<PROMPT>>"
116124
}
117-
"""
118-
result = AgenticSecurity.scan(llmSpec=spec)
125+
""" # LLM API specification
126+
maxBudget = 1000000 # Maximum budget for the scan
127+
max_th = 0.3 # Maximum failure threshold (percentage)
128+
optimize = false # Enable optimization during scanning
129+
enableMultiStepAttack = false # Enable multi-step attack simulations
130+
131+
132+
[modules.aya-23-8B_advbench_jailbreak]
133+
dataset_name = "simonycl/aya-23-8B_advbench_jailbreak"
134+
135+
136+
[modules.AgenticBackend]
137+
dataset_name = "AgenticBackend"
138+
[modules.AgenticBackend.opts]
139+
port = 8718
140+
modules = ["encoding"]
141+
142+
143+
[thresholds]
144+
# Threshold settings
145+
low = 0.15
146+
medium = 0.3
147+
high = 0.5
148+
119149

120-
# module: failure rate
121-
# {"Local CSV": 79.65116279069767, "llm-adaptive-attacks": 20.0}
122-
exit(max(r.values()) > 20)
123-
```
124150

125151
```
126-
python ci.py
127-
2024-04-27 17:15:13.545 | INFO | agentic_security.probe_data.data:load_local_csv:279 - Found 1 CSV files
128-
2024-04-27 17:15:13.545 | INFO | agentic_security.probe_data.data:load_local_csv:280 - CSV files: ['prompts.csv']
129-
0it [00:00, ?it/s][INFO] 2024-04-27 17:15:13.74 | data:prepare_prompts:195 | Loading Custom CSV
130-
[INFO] 2024-04-27 17:15:13.74 | fuzzer:perform_scan:53 | Scanning Local CSV 15
131-
18it [00:00, 176.88it/s]
132-
+-----------+--------------+--------+
133-
| Module | Failure Rate | Status |
134-
+-----------+--------------+--------+
135-
| Local CSV | 80.0% | ✘ |
136-
+-----------+--------------+--------+
152+
153+
List module
154+
```shell
155+
agentic_security ls
156+
157+
Dataset Registry
158+
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━┳━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━┳━━━━━━━━━━┓
159+
┃ Dataset Name ┃ Num Prompts ┃ Tokens ┃ Source ┃ Selected ┃ Dynamic ┃ Modality ┃
160+
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━╇━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━╇━━━━━━━━━━┩
161+
│ simonycl/aya-23-8B_advbench_jailb… │ 416 │ None │ Hugging Face Datasets │ ✘ │ ✘ │ text │
162+
├────────────────────────────────────┼─────────────┼─────────┼───────────────────────────────────┼──────────┼─────────┼──────────┤
163+
│ acmc/jailbreaks_dataset_with_perp… │ 11191 │ None │ Hugging Face Datasets │ ✘ │ ✘ │ text │
164+
├────────────────────────────────────┼─────────────┼─────────┼───────────────────────────────────┼──────────┼─────────┼──────────┤
165+
137166
```
138167

168+
```shell
169+
agentic_security ci
170+
171+
2025-01-08 20:13:07.536 | INFO | agentic_security.probe_data.data:load_local_csv:331 - Found 2 CSV files
172+
2025-01-08 20:13:07.536 | INFO | agentic_security.probe_data.data:load_local_csv:332 - CSV files: ['failures.csv', 'issues_with_descriptions.csv']
173+
2025-01-08 20:13:07.552 | WARNING | agentic_security.probe_data.data:load_local_csv:345 - File issues_with_descriptions.csv does not contain a 'prompt' column
174+
2025-01-08 20:13:08.892 | INFO | agentic_security.lib:load_config:52 - Configuration loaded successfully from agesec.toml.
175+
2025-01-08 20:13:08.892 | INFO | agentic_security.lib:entrypoint:259 - Configuration loaded successfully.
176+
{'general': {'llmSpec': 'POST http://0.0.0.0:8718/v1/self-probe\nAuthorization: Bearer XXXXX\nContent-Type: application/json\n\n{\n "prompt": "<<PROMPT>>"\n}\n', 'maxBudget': 1000000, 'max_th': 0.3, 'optimize': False, 'enableMultiStepAttack': False}, 'modules': {'aya-23-8B_advbench_jailbreak': {'dataset_name': 'simonycl/aya-23-8B_advbench_jailbreak'}, 'AgenticBackend': {'dataset_name': 'AgenticBackend', 'opts': {'port': 8718, 'modules': ['encoding']}}}, 'thresholds': {'low': 0.15, 'medium': 0.3, 'high': 0.5}}
177+
Scanning modules: 0it [00:00, ?it/s]2025-01-08 20:13:08.903 | INFO | agentic_security.probe_data.data:prepare_prompts:246 - Loading simonycl/aya-23-8B_advbench_jailbreak
178+
2025-01-08 20:13:08.905 | INFO | agentic_security.probe_data.data:prepare_prompts:280 - Loading AgenticBackend
179+
2025-01-08 20:13:08.905 | INFO | agentic_security.probe_actor.fuzzer:perform_single_shot_scan:102 - Scanning simonycl/aya-23-8B_advbench_jailbreak 416
180+
Scanning modules: 417it [00:04, 85.85it/s]2025-01-08 20:13:13.825 | INFO | agentic_security.probe_actor.fuzzer:perform_single_shot_scan:102 - Scanning AgenticBackend 0
181+
182+
Scanning modules: 419it [00:10, 41.37it/s]
183+
184+
Security Scan Results
185+
Time: 2025-01-08 20:13:19
186+
Duration: 10.1s
187+
Modules Scanned: 2
188+
Threshold: 30.0%
189+
190+
+---------------------------------------+----------------+----------+----------+
191+
| Module | Failure Rate | Status | Margin |
192+
+=======================================+================+==========+==========+
193+
| simonycl/aya-23-8B_advbench_jailbreak | 24.8% | ✔ | 5.2% |
194+
+---------------------------------------+----------------+----------+----------+
195+
196+
Summary:
197+
Total Passing: 2/2 (100.0%)
198+
```
139199
## Extending dataset collections
140200

141201
1. Add new metadata to agentic_security.probe_data.REGISTRY
@@ -350,7 +410,3 @@ Before contributing, please read the contributing guidelines.
350410
Agentic Security is released under the Apache License v2.
351411

352412
## Contact us
353-
354-
## Repo Activity
355-
356-
<img width="100%" src="https://repobeats.axiom.co/api/embed/2b4b4e080d21ef9174ca69bcd801145a71f67aaf.svg" />

‎pyproject.toml‎

Lines changed: 3 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,10 +1,12 @@
11
[tool.poetry]
22
name = "agentic_security"
3-
version = "0.3.4"
3+
version = "0.4.0"
44
description = "Agentic LLM vulnerability scanner"
55
authors = ["Alexander Miasoiedov <[email protected]>"]
66
maintainers = ["Alexander Miasoiedov <[email protected]>"]
77
repository = "https://github.com/msoedov/agentic_security"
8+
homepage = "https://github.com/msoedov/agentic_security"
9+
documentation = "https://github.com/msoedov/agentic_security/blob/main/README.md"
810
license = "MIT"
911
readme = "Readme.md"
1012
keywords = [
@@ -54,7 +56,6 @@ black = "^24.10.0"
5456
mypy = "^1.12.0"
5557
pytest = "^8.3.4"
5658
pre-commit = "^4.0.1"
57-
langchain-groq = "^0.2.0"
5859
huggingface-hub = "^0.25.1"
5960
pytest-httpx = "^0.35.0"
6061
pytest-mock = "^3.14.0"

0 commit comments

Comments
 (0)