feat(llm): add LLM provider (#8555)

Co-authored-by: Andoni A. <14891798+andoniaf@users.noreply.github.com>

Author: MrCloudSec

.pre-commit-config.yaml

@@ -6,6 +6,7 @@ repos:
       - id: check-merge-conflict
       - id: check-yaml
         args: ["--unsafe"]
+        exclude: prowler/config/llm_config.yaml
       - id: check-json
       - id: end-of-file-fixer
       - id: trailing-whitespace

README.md

@@ -90,6 +90,7 @@ prowler dashboard
 | M365 | 70 | 7 | 3 | 2 | Official | Stable | UI, API, CLI |
 | IaC | [See `trivy` docs.](https://trivy.dev/latest/docs/coverage/iac/) | N/A | N/A | N/A | Official | Beta | CLI |
 | MongoDB Atlas | 10 | 3 | 0 | 0 | Official | Beta | CLI |
+| LLM | [See `promptfoo` docs.](https://www.promptfoo.dev/docs/red-team/plugins/) | N/A | N/A | N/A | Official | Beta | CLI |
 | NHN | 6 | 2 | 1 | 0 | Unofficial | Beta | CLI |
 
 > [!Note]

docs/developer-guide/llm-details.md

@@ -0,0 +1,102 @@
+# LLM Provider
+
+This page details the [Large Language Model (LLM)](https://en.wikipedia.org/wiki/Large_language_model) provider implementation in Prowler.
+
+The LLM provider enables security testing of language models using red team techniques. By default, Prowler uses the built-in LLM configuration that targets OpenAI models with comprehensive security test suites. To configure it, follow the [LLM getting started guide](../tutorials/llm/getting-started-llm.md).
+
+## LLM Provider Classes Architecture
+
+The LLM provider implementation follows the general [Provider structure](./provider.md). This section focuses on the LLM-specific implementation, highlighting how the generic provider concepts are realized for LLM security testing in Prowler. For a full overview of the provider pattern, base classes, and extension guidelines, see [Provider documentation](./provider.md).
+
+### Main Class
+
+- **Location:** [`prowler/providers/llm/llm_provider.py`](https://github.com/prowler-cloud/prowler/blob/master/prowler/providers/llm/llm_provider.py)
+- **Base Class:** Inherits from `Provider` (see [base class details](https://github.com/prowler-cloud/prowler/blob/master/prowler/providers/common/provider.py)).
+- **Purpose:** Central orchestrator for LLM-specific logic, configuration management, and integration with promptfoo for red team testing.
+- **Key LLM Responsibilities:**
+    - Initializes and manages LLM configuration using promptfoo.
+    - Validates configuration and sets up the LLM testing context.
+    - Loads and manages red team test configuration, plugins, and target models.
+    - Provides properties and methods for downstream LLM security testing.
+    - Integrates with promptfoo for comprehensive LLM security evaluation.
+
+### Data Models
+
+- **Location:** [`prowler/providers/llm/models.py`](https://github.com/prowler-cloud/prowler/blob/master/prowler/providers/llm/models.py)
+- **Purpose:** Define structured data for LLM output options and configuration.
+- **Key LLM Models:**
+    - `LLMOutputOptions`: Customizes output filename logic for LLM-specific reporting.
+
+### LLM Security Testing Integration
+
+- **Location:** [`prowler/providers/llm/llm_provider.py`](https://github.com/prowler-cloud/prowler/blob/master/prowler/providers/llm/llm_provider.py)
+- **Purpose:** Integrates with promptfoo for comprehensive LLM security testing.
+- **Key LLM Responsibilities:**
+    - Executes promptfoo red team evaluations against target LLMs.
+    - Processes security test results and converts them to Prowler reports.
+    - Manages test concurrency and progress tracking.
+    - Handles real-time streaming of test results.
+
+### Configuration Management
+
+The LLM provider uses promptfoo configuration files to define:
+
+- **Target Models**: The LLM models to test (e.g., OpenAI GPT, Anthropic Claude)
+- **Red Team Plugins**: Security test suites (OWASP, MITRE, NIST, EU AI Act)
+- **Test Parameters**: Concurrency, test counts, and evaluation criteria
+
+### Default Configuration
+
+Prowler includes a comprehensive default LLM configuration that:
+
+- Targets OpenAI models by default
+- Includes multiple security test frameworks (OWASP, MITRE, NIST, EU AI Act)
+- Provides extensive test coverage for LLM security vulnerabilities
+- Supports custom configuration for specific testing needs
+
+## Specific Patterns in LLM Security Testing
+
+The LLM provider implements security testing through integration with promptfoo, following these patterns:
+
+### Red Team Testing Framework
+
+- **Plugin-based Architecture**: Uses promptfoo plugins for different security test categories
+- **Comprehensive Coverage**: Includes OWASP LLM Top 10, MITRE ATLAS, NIST AI Risk Management, and EU AI Act compliance
+- **Real-Time Evaluation**: Streams test results as they are generated
+- **Progress Tracking**: Provides detailed progress information during test execution
+
+### Test Execution Flow
+
+1. **Configuration Loading**: Loads promptfoo configuration with target models and test plugins
+2. **Test Generation**: Generates security test cases based on configured plugins
+3. **Concurrent Execution**: Runs tests with configurable concurrency limits
+4. **Result Processing**: Converts promptfoo results to Prowler security reports
+5. **Progress Monitoring**: Tracks and displays test execution progress
+
+### Security Test Categories
+
+The LLM provider supports comprehensive security testing across multiple frameworks:
+
+- **OWASP LLM Top 10**: Covers prompt injection, data leakage, and model security
+- **MITRE ATLAS**: Adversarial threat landscape for AI systems
+- **NIST AI Risk Management**: AI system risk assessment and mitigation
+- **EU AI Act**: European Union AI regulation compliance
+- **Custom Tests**: Support for organization-specific security requirements
+
+## Error Handling and Validation
+
+The LLM provider includes comprehensive error handling for:
+
+- **Configuration Validation**: Ensures valid promptfoo configuration files
+- **Model Access**: Handles authentication and access issues with target LLMs
+- **Test Execution**: Manages test failures and timeout scenarios
+- **Result Processing**: Handles malformed or incomplete test results
+
+## Integration with Prowler Ecosystem
+
+The LLM provider seamlessly integrates with Prowler's existing infrastructure:
+
+- **Output Formats**: Supports all Prowler output formats (JSON, CSV, HTML, etc.)
+- **Compliance Frameworks**: Integrates with Prowler's compliance reporting
+- **Fixer Integration**: Supports automated remediation recommendations
+- **Dashboard Integration**: Compatible with Prowler App for centralized management

docs/index.md

@@ -14,6 +14,7 @@ The official supported providers right now are:
 | **Github** | Official | Stable | UI, API, CLI |
 | **IaC** | Official | Beta | CLI |
 | **MongoDB Atlas** | Official | Beta | CLI |
+| **LLM** | Official | Beta | CLI |
 | **NHN** | Unofficial | Beta | CLI |
 
 Prowler supports **auditing, incident response, continuous monitoring, hardening, forensic readiness, and remediation**.

docs/tutorials/llm/getting-started-llm.md

@@ -0,0 +1,142 @@
+# Getting Started With LLM on Prowler
+
+## Overview
+
+Prowler's LLM provider enables comprehensive security testing of large language models using red team techniques. It integrates with [promptfoo](https://promptfoo.dev/) to provide extensive security evaluation capabilities.
+
+## Prerequisites
+
+Before using the LLM provider, ensure the following requirements are met:
+
+- **promptfoo installed**: The LLM provider requires promptfoo to be installed on the system
+- **LLM API access**: Valid API keys for the target LLM models to test
+- **Email verification**: promptfoo requires email verification for red team evaluations
+
+## Installation
+
+### Install promptfoo
+
+Install promptfoo using one of the following methods:
+
+**Using npm:**
+```bash
+npm install -g promptfoo
+```
+
+**Using Homebrew (macOS):**
+```bash
+brew install promptfoo
+```
+
+**Using other package managers:**
+See the [promptfoo installation guide](https://promptfoo.dev/docs/installation/) for additional installation methods.
+
+### Verify Installation
+
+```bash
+promptfoo --version
+```
+
+## Configuration
+
+### Step 1: Email Verification
+
+promptfoo requires email verification for red team evaluations. Set the email address:
+
+```bash
+promptfoo config set email your-email@company.com
+```
+
+### Step 2: Configure LLM API Keys
+
+Set up API keys for the target LLM models. For OpenAI (default configuration):
+
+```bash
+export OPENAI_API_KEY="your-openai-api-key"
+```
+
+For other providers, see the [promptfoo documentation](https://promptfoo.dev/docs/providers/) for specific configuration requirements.
+
+### Step 3: Generate Test Cases (Optional)
+
+Prowler provides a default suite of red team tests but to customize the test cases, generate them first:
+
+```bash
+promptfoo redteam generate
+```
+
+This creates test cases based on your configuration.
+
+## Usage
+
+### Basic Usage
+
+Run LLM security testing with the default configuration:
+
+```bash
+prowler llm
+```
+
+### Custom Configuration
+
+Use a custom promptfoo configuration file:
+
+```bash
+prowler llm --config-path /path/to/your/config.yaml
+```
+
+### Output Options
+
+Generate reports in various formats:
+
+```bash
+# JSON output
+prowler llm --output-format json
+
+# CSV output
+prowler llm --output-format csv
+
+# HTML report
+prowler llm --output-format html
+```
+
+### Concurrency Control
+
+Adjust the number of concurrent tests:
+
+```bash
+prowler llm --max-concurrency 5
+```
+
+## Default Configuration
+
+Prowler includes a comprehensive default LLM configuration that provides:
+
+- **Target Models**: OpenAI GPT models by default
+- **Security Frameworks**:
+  - OWASP LLM Top 10
+  - OWASP API Top 10
+  - MITRE ATLAS
+  - NIST AI Risk Management Framework
+  - EU AI Act compliance
+- **Test Coverage**: Over 5,000 security test cases
+- **Plugin Support**: Multiple security testing plugins
+
+## Advanced Configuration
+
+### Custom Test Suites
+
+Create custom test configurations by modifying the promptfoo config file in `prowler/config/llm_config.yaml` or pass a custom configuration with `--config-file` flag:
+
+```yaml
+description: Custom LLM Security Tests
+targets:
+  - id: openai:gpt-4
+redteam:
+  plugins:
+    - id: owasp:llm
+      numTests: 10
+    - id: mitre:atlas
+      numTests: 5
+```
+

mkdocs.yml

@@ -138,7 +138,8 @@ nav:
         - MongoDB Atlas:
             - Getting Started: tutorials/mongodbatlas/getting-started-mongodbatlas.md
             - Authentication: tutorials/mongodbatlas/authentication.md
-
+        - LLM:
+            - Getting Started: tutorials/llm/getting-started-llm.md
       - Compliance:
         - ThreatScore: tutorials/compliance/threatscore.md
 
@@ -159,6 +160,7 @@ nav:
         - Kubernetes: developer-guide/kubernetes-details.md
         - Microsoft 365: developer-guide/m365-details.md
         - GitHub: developer-guide/github-details.md
+        - LLM: developer-guide/llm-details.md
       - Miscellaneous:
         - Documentation: developer-guide/documentation.md
         - Testing:

prowler/CHANGELOG.md

@@ -7,6 +7,7 @@ All notable changes to the **Prowler SDK** are documented in this file.
 ### Added
 - Support for AdditionalURLs in outputs [(#8651)](https://github.com/prowler-cloud/prowler/pull/8651)
 - Support for markdown metadata fields in Dashboard [(#8667)](https://github.com/prowler-cloud/prowler/pull/8667)
+- LLM provider using `promptfoo` [(#8555)](https://github.com/prowler-cloud/prowler/pull/8555)
 - Documentation for renaming checks [(#8717)](https://github.com/prowler-cloud/prowler/pull/8717)
 - Add explicit "name" field for each compliance framework and include "FRAMEWORK" and "NAME" in CSV output [(#7920)](https://github.com/prowler-cloud/prowler/pull/7920)
 

prowler/__main__.py

@@ -89,7 +89,7 @@
 from prowler.lib.outputs.finding import Finding
 from prowler.lib.outputs.html.html import HTML
 from prowler.lib.outputs.ocsf.ocsf import OCSF
-from prowler.lib.outputs.outputs import extract_findings_statistics
+from prowler.lib.outputs.outputs import extract_findings_statistics, report
 from prowler.lib.outputs.slack.slack import Slack
 from prowler.lib.outputs.summary_table import display_summary_table
 from prowler.providers.aws.lib.s3.s3 import S3
@@ -102,6 +102,7 @@
 from prowler.providers.github.models import GithubOutputOptions
 from prowler.providers.iac.models import IACOutputOptions
 from prowler.providers.kubernetes.models import KubernetesOutputOptions
+from prowler.providers.llm.models import LLMOutputOptions
 from prowler.providers.m365.models import M365OutputOptions
 from prowler.providers.mongodbatlas.models import MongoDBAtlasOutputOptions
 from prowler.providers.nhn.models import NHNOutputOptions
@@ -180,8 +181,8 @@ def prowler():
     # Load compliance frameworks
     logger.debug("Loading compliance frameworks from .json files")
 
-    # Skip compliance frameworks for IAC provider
-    if provider != "iac":
+    # Skip compliance frameworks for IAC and LLM providers
+    if provider != "iac" and provider != "llm":
         bulk_compliance_frameworks = Compliance.get_bulk(provider)
         # Complete checks metadata with the compliance framework specification
         bulk_checks_metadata = update_checks_metadata_with_compliance(
@@ -238,8 +239,8 @@ def prowler():
     if not args.only_logs:
         global_provider.print_credentials()
 
-    # Skip service and check loading for IAC provider
-    if provider != "iac":
+    # Skip service and check loading for IAC and LLM providers
+    if provider != "iac" and provider != "llm":
         # Import custom checks from folder
         if checks_folder:
             custom_checks = parse_checks_from_folder(global_provider, checks_folder)
@@ -322,6 +323,8 @@ def prowler():
         )
     elif provider == "iac":
         output_options = IACOutputOptions(args, bulk_checks_metadata)
+    elif provider == "llm":
+        output_options = LLMOutputOptions(args, bulk_checks_metadata)
 
     # Run the quick inventory for the provider if available
     if hasattr(args, "quick_inventory") and args.quick_inventory:
@@ -331,9 +334,20 @@ def prowler():
     # Execute checks
     findings = []
 
-    if provider == "iac":
-        # For IAC provider, run the scan directly
-        findings = global_provider.run()
+    if provider == "iac" or provider == "llm":
+        # For IAC and LLM providers, run the scan directly
+        if provider == "llm":
+
+            def streaming_callback(findings_batch):
+                """Callback to report findings as they are processed in real-time."""
+                report(findings_batch, global_provider, output_options)
+
+            findings = global_provider.run_scan(streaming_callback=streaming_callback)
+        else:
+            # Original behavior for IAC or non-verbose LLM
+            findings = global_provider.run()
+            # Report findings for verbose output
+            report(findings, global_provider, output_options)
     elif len(checks_to_execute):
         findings = execute_checks(
             checks_to_execute,

prowler/compliance/llm/__init__.py

@@ -76,6 +76,9 @@ def get_available_compliance_frameworks(provider=None):
 default_fixer_config_file_path = (
     f"{pathlib.Path(os.path.dirname(os.path.realpath(__file__)))}/fixer_config.yaml"
 )
+default_redteam_config_file_path = (
+    f"{pathlib.Path(os.path.dirname(os.path.realpath(__file__)))}/llm_config.yaml"
+)
 encoding_format_utf_8 = "utf-8"
 available_output_formats = ["csv", "json-asff", "json-ocsf", "html"]