Inline link and ### tweaks (#471)

Author: Setotet

2_0_vulns/LLM01_PromptInjection.md

@@ -10,9 +10,11 @@ While prompt injection and jailbreaking are related concepts in LLM security, th
 
 ### Types of Prompt Injection Vulnerabilities
 
-**Direct Prompt Injections** occur when a user's prompt input directly alters the behavior of the model in unintended or unexpected ways. The input can be either intentional (i.e., a malicious actor deliberately crafting a prompt to exploit the model) or unintentional (i.e., a user inadvertently providing input that triggers unexpected behavior).
+#### Direct Prompt Injections
+  Direct prompt injections occur when a user's prompt input directly alters the behavior of the model in unintended or unexpected ways. The input can be either intentional (i.e., a malicious actor deliberately crafting a prompt to exploit the model) or unintentional (i.e., a user inadvertently providing input that triggers unexpected behavior).
 
-**Indirect Prompt Injections** occur when an LLM accepts input from external sources, such as websites or files. The content may have in the external content data that when interpreted by the model, alters the behavior of the model in unintended or unexpected ways. Like direct injections, indirect injections can be either intentional or unintentional.
+#### Indirect Prompt Injections
+  Indirect prompt injections occur when an LLM accepts input from external sources, such as websites or files. The content may have in the external content data that when interpreted by the model, alters the behavior of the model in unintended or unexpected ways. Like direct injections, indirect injections can be either intentional or unintentional.
 
 The severity and nature of the impact of a successful prompt injection attack can vary greatly and are largely dependent on both the business context the model operates in, and the agency with which the model is architected. Generally, however, prompt injection can lead to unintended outcomes, including but not limited to:
 
@@ -29,25 +31,41 @@ The rise of multimodal AI, which processes multiple data types simultaneously, i
 
 Prompt injection vulnerabilities are possible due to the nature of generative AI. Given the stochastic influence at the heart of the way models work, it is unclear if there are fool-proof methods of prevention for prompt injection. However, the following measures can mitigate the impact of prompt injections:
 
-1. Constrain model behavior: Provide specific instructions about the model's role, capabilities, and limitations within the system prompt. Enforce strict context adherence, limit responses to specific tasks or topics, and instruct the model to ignore attempts to modify core instructions.
-2. Define and validate expected output formats: Specify clear output formats, request detailed reasoning and source citations, and use deterministic code to validate adherence to these formats.
-3. Implement input and output filtering: Define sensitive categories and construct rules for identifying and handling such content. Apply semantic filters and use string-checking to scan for non-allowed content. Evaluate responses using the RAG Triad: Assess context relevance, groundedness, and question/answer relevance to identify potentially malicious outputs.
-4. Enforce privilege control and least privilege access: Provide the application with its own API tokens for extensible functionality, and handle these functions in code rather than providing them to the model. Restrict the model's access privileges to the minimum necessary for its intended operations.
-5. Require human approval for high-risk actions: Implement human-in-the-loop controls for privileged operations to prevent unauthorized actions.
-6. Segregate and identify external content: Separate and clearly denote untrusted content to limit its influence on user prompts.
-7. Conduct adversarial testing and attack simulations: Perform regular penetration testing and breach simulations, treating the model as an untrusted user to test the effectiveness of trust boundaries and access controls.
+#### 1. Constrain model behavior
+  Provide specific instructions about the model's role, capabilities, and limitations within the system prompt. Enforce strict context adherence, limit responses to specific tasks or topics, and instruct the model to ignore attempts to modify core instructions.
+#### 2. Define and validate expected output formats
+  Specify clear output formats, request detailed reasoning and source citations, and use deterministic code to validate adherence to these formats.
+#### 3. Implement input and output filtering
+  Define sensitive categories and construct rules for identifying and handling such content. Apply semantic filters and use string-checking to scan for non-allowed content. Evaluate responses using the RAG Triad: Assess context relevance, groundedness, and question/answer relevance to identify potentially malicious outputs.
+#### 4. Enforce privilege control and least privilege access
+  Provide the application with its own API tokens for extensible functionality, and handle these functions in code rather than providing them to the model. Restrict the model's access privileges to the minimum necessary for its intended operations.
+#### 5. Require human approval for high-risk actions
+  Implement human-in-the-loop controls for privileged operations to prevent unauthorized actions.
+#### 6. Segregate and identify external content
+  Separate and clearly denote untrusted content to limit its influence on user prompts.
+#### 7. Conduct adversarial testing and attack simulations
+  Perform regular penetration testing and breach simulations, treating the model as an untrusted user to test the effectiveness of trust boundaries and access controls.
 
 ### Example Attack Scenarios
 
-1. Direct Injection: An attacker injects a prompt into a customer support chatbot, instructing it to ignore previous guidelines, query private data stores, and send emails, leading to unauthorized access and privilege escalation.
-2. Indirect Injection: A user employs an LLM to summarize a webpage containing hidden instructions that cause the LLM to insert an image linking to a URL, leading to exfiltration of the the private conversation.
-3. Unintentional Injection: A company includes an instruction in a job description to identify AI-generated applications. An applicant, unaware of this instruction, uses an LLM to optimize their resume, inadvertently triggering the AI detection.
-4. Intentional Model Influence: An attacker modifies a document in a repository used by a Retrieval-Augmented Generation (RAG) application. When a user's query returns the modified content, the malicious instructions alter the LLM's output, generating misleading results.
-5. Code Injection: An attacker exploits a vulnerability (CVE-2024-5184) in an LLM-powered email assistant to inject malicious prompts, allowing access to sensitive information and manipulation of email content.
-6. Payload Splitting: An attacker uploads a resume with split malicious prompts. When an LLM is used to evaluate the candidate, the combined prompts manipulate the model's response, resulting in a positive recommendation despite the actual resume contents.
-7. Multimodal Injection: An attacker embeds a malicious prompt within an image that accompanies benign text. When a multimodal AI processes the image and text concurrently, the hidden prompt alters the model's behavior, potentially leading to unauthorized actions or disclosure of sensitive information.
-8. Adversarial Suffix: An attacker appends a seemingly meaningless string of characters to a prompt, which influences the LLM's output in a malicious way, bypassing safety measures.
-9. Multilingual/Obfuscated Attack: An attacker uses multiple languages or encodes malicious instructions (e.g., using Base64 or emojis) to evade filters and manipulate the LLM's behavior.
+#### Scenario #1: Direct Injection
+  An attacker injects a prompt into a customer support chatbot, instructing it to ignore previous guidelines, query private data stores, and send emails, leading to unauthorized access and privilege escalation.
+#### Scenario #2: Indirect Injection
+  A user employs an LLM to summarize a webpage containing hidden instructions that cause the LLM to insert an image linking to a URL, leading to exfiltration of the the private conversation.
+#### Scenario #3: Unintentional Injection
+  A company includes an instruction in a job description to identify AI-generated applications. An applicant, unaware of this instruction, uses an LLM to optimize their resume, inadvertently triggering the AI detection.
+#### Scenario #4: Intentional Model Influence
+  An attacker modifies a document in a repository used by a Retrieval-Augmented Generation (RAG) application. When a user's query returns the modified content, the malicious instructions alter the LLM's output, generating misleading results.
+#### Scenario #5: Code Injection
+  An attacker exploits a vulnerability (CVE-2024-5184) in an LLM-powered email assistant to inject malicious prompts, allowing access to sensitive information and manipulation of email content.
+#### Scenario #6: Payload Splitting
+  An attacker uploads a resume with split malicious prompts. When an LLM is used to evaluate the candidate, the combined prompts manipulate the model's response, resulting in a positive recommendation despite the actual resume contents.
+#### Scenario #7: Multimodal Injection
+  An attacker embeds a malicious prompt within an image that accompanies benign text. When a multimodal AI processes the image and text concurrently, the hidden prompt alters the model's behavior, potentially leading to unauthorized actions or disclosure of sensitive information.
+#### Scenario #8: Adversarial Suffix
+  An attacker appends a seemingly meaningless string of characters to a prompt, which influences the LLM's output in a malicious way, bypassing safety measures.
+#### Scenario #9: Multilingual/Obfuscated Attack
+  An attacker uses multiple languages or encodes malicious instructions (e.g., using Base64 or emojis) to evade filters and manipulate the LLM's behavior.
 
 ### Reference Links
 
@@ -63,7 +81,7 @@ Prompt injection vulnerabilities are possible due to the nature of generative AI
 11. [Adversarial Machine Learning: A Taxonomy and Terminology of Attacks and Mitigations (nist.gov)](https://nvlpubs.nist.gov/nistpubs/ai/NIST.AI.100-2e2023.pdf)
 12. [2407.07403 A Survey of Attacks on Large Vision-Language Models: Resources, Advances, and Future Trends (arxiv.org)](https://arxiv.org/abs/2407.07403)
 13. [Exploiting Programmatic Behavior of LLMs: Dual-Use Through Standard Security Attacks](https://ieeexplore.ieee.org/document/10579515)
-14. [Universal and Transferable Adversarial Attacks on Aligned Language Models (arxiv.org)](https://arxiv.org/abs/2307.15043_
+14. [Universal and Transferable Adversarial Attacks on Aligned Language Models (arxiv.org)](https://arxiv.org/abs/2307.15043_)
 15. [From ChatGPT to ThreatGPT: Impact of Generative AI in Cybersecurity and Privacy (arxiv.org)](https://arxiv.org/abs/2307.00691)
 
 ### Related Frameworks and Taxonomies

2_0_vulns/LLM02_SensitiveInformationDisclosure.md

@@ -10,46 +10,66 @@ To reduce this risk, LLM applications should perform adequate data sanitization
 
 ### Common Examples of Vulnerability
 
-1. **PII Leakage:** Personal identifiable information (PII) may be disclosed during interactions with the LLM.
-2. **Proprietary Algorithm Exposure:** Poorly configured model outputs can reveal proprietary algorithms or data. Revealing training data can expose models to inversion attacks, where attackers extract sensitive information or reconstruct inputs. For instance, as demonstrated in the 'Proof Pudding' attack (CVE-2019-20634), disclosed training data facilitated model extraction and inversion, allowing attackers to circumvent security controls in machine learning algorithms and bypass email filters.
-3. **Sensitive Business Data Disclosure:** Generated responses might inadvertently include confidential business information.
+#### 1. PII Leakage
+  Personal identifiable information (PII) may be disclosed during interactions with the LLM.
+#### 2. Proprietary Algorithm Exposure
+  Poorly configured model outputs can reveal proprietary algorithms or data. Revealing training data can expose models to inversion attacks, where attackers extract sensitive information or reconstruct inputs. For instance, as demonstrated in the 'Proof Pudding' attack (CVE-2019-20634), disclosed training data facilitated model extraction and inversion, allowing attackers to circumvent security controls in machine learning algorithms and bypass email filters.
+#### 3. Sensitive Business Data Disclosure
+  Generated responses might inadvertently include confidential business information.
 
 ### Prevention and Mitigation Strategies
 
-**Sanitization**:
+###@ Sanitization:
 
-1. **Integrate Data Sanitization Techniques:** Implement data sanitization to prevent user data from entering the training model. This includes scrubbing or masking sensitive content before it is used in training.
-2. **Robust Input Validation:** Apply strict input validation methods to detect and filter out potentially harmful or sensitive data inputs, ensuring they do not compromise the model.
+#### 1. Integrate Data Sanitization Techniques
+  Implement data sanitization to prevent user data from entering the training model. This includes scrubbing or masking sensitive content before it is used in training.
+#### 2. Robust Input Validation
+  Apply strict input validation methods to detect and filter out potentially harmful or sensitive data inputs, ensuring they do not compromise the model.
 
-**Access Controls**:
+###@ Access Controls:
 
-1. **Enforce Strict Access Controls:** Limit access to sensitive data based on the principle of least privilege. Only grant access to data that is necessary for the specific user or process.
-2. **Restrict Data Sources:** Limit model access to external data sources, and ensure runtime data orchestration is securely managed to avoid unintended data leakage.
+#### 1. Enforce Strict Access Controls
+  Limit access to sensitive data based on the principle of least privilege. Only grant access to data that is necessary for the specific user or process.
+#### 2. Restrict Data Sources
+  Limit model access to external data sources, and ensure runtime data orchestration is securely managed to avoid unintended data leakage.
 
-**Federated Learning and Privacy Techniques**:
+###@ Federated Learning and Privacy Techniques:
 
-1. **Utilize Federated Learning:** Train models using decentralized data stored across multiple servers or devices. This approach minimizes the need for centralized data collection and reduces exposure risks.
-2. **Incorporate Differential Privacy:** Apply techniques that add noise to the data or outputs, making it difficult for attackers to reverse-engineer individual data points.
+#### 1. Utilize Federated Learning
+  Train models using decentralized data stored across multiple servers or devices. This approach minimizes the need for centralized data collection and reduces exposure risks.
+#### 2. Incorporate Differential Privacy
+  Apply techniques that add noise to the data or outputs, making it difficult for attackers to reverse-engineer individual data points.
 
-**User Education and Transparency**
+###@ User Education and Transparency:
 
-1. **Educate Users on Safe LLM Usage:** Provide guidance on avoiding the input of sensitive information. Offer training on best practices for interacting with LLMs securely.
-2. **Ensure Transparency in Data Usage:** Maintain clear policies about data retention, usage, and deletion. Allow users to opt out of having their data included in training processes.
+#### 1. Educate Users on Safe LLM Usage
+  Provide guidance on avoiding the input of sensitive information. Offer training on best practices for interacting with LLMs securely.
+#### 2. Ensure Transparency in Data Usage
+  Maintain clear policies about data retention, usage, and deletion. Allow users to opt out of having their data included in training processes.
 
-**Secure System Configuration**
+###@ Secure System Configuration:
 
-1. **Conceal System Preamble:** Limit the ability for users to override or access the system's initial settings, reducing the risk of exposure to internal configurations.
-2. **Reference Security Misconfiguration Best Practices:** Follow guidelines like [OWASP API8:2023 Security Misconfiguration](https://owasp.org/API-Security/editions/2023/en/0xa8-security-misconfiguration/) to prevent leaking sensitive information through error messages or configuration details.  
-**Advanced Techniques**
+#### 1. Conceal System Preamble
+  Limit the ability for users to override or access the system's initial settings, reducing the risk of exposure to internal configurations.
+#### 2. Reference Security Misconfiguration Best Practices
+  Follow guidelines like "OWASP API8:2023 Security Misconfiguration" to prevent leaking sensitive information through error messages or configuration details.
+  (Ref. link:[OWASP API8:2023 Security Misconfiguration](https://owasp.org/API-Security/editions/2023/en/0xa8-security-misconfiguration/))
 
-1. **Homomorphic Encryption:** Use homomorphic encryption to enable secure data analysis and privacy-preserving machine learning. This ensures data remains confidential while being processed by the model.
-2. **Tokenization and Redaction:** Implement tokenization to preprocess and sanitize sensitive information. Techniques like pattern matching can detect and redact confidential content before processing.
+###@ Advanced Techniques:
+
+#### 1. Homomorphic Encryption
+  Use homomorphic encryption to enable secure data analysis and privacy-preserving machine learning. This ensures data remains confidential while being processed by the model.
+#### 2. Tokenization and Redaction
+  Implement tokenization to preprocess and sanitize sensitive information. Techniques like pattern matching can detect and redact confidential content before processing.
 
 ### Example Attack Scenarios
 
-1. **Unintentional Data Exposure:** A user receives a response containing another user's personal data due to inadequate data sanitization.
-2. **Targeted Prompt Injection:** An attacker bypasses input filters to extract sensitive information.
-3. **Data Leak via Training Data:** Negligent data inclusion in training leads to sensitive information disclosure.
+#### Scenario #1: Unintentional Data Exposure
+  A user receives a response containing another user's personal data due to inadequate data sanitization.
+#### Scenario #2: Targeted Prompt Injection
+  An attacker bypasses input filters to extract sensitive information.
+#### Scenario #3: Data Leak via Training Data
+  Negligent data inclusion in training leads to sensitive information disclosure.
 
 ### Reference Links
 
@@ -61,6 +81,8 @@ To reduce this risk, LLM applications should perform adequate data sanitization
 
 ### Related Frameworks and Taxonomies
 
+Refer to this section for comprehensive information, scenarios strategies relating to infrastructure deployment, applied environment controls and other best practices.
+
 - [AML.T0024.000 - Infer Training Data Membership](https://atlas.mitre.org/techniques/AML.T0024.000) **MITRE ATLAS**
 - [AML.T0024.001 - Invert ML Model](https://atlas.mitre.org/techniques/AML.T0024.001) **MITRE ATLAS**
-- [AML.T0024.002 - Extract ML Model](https://atlas.mitre.org/techniques/AML.T0024.002) **MITRE ATLAS**
+- [AML.T0024.002 - Extract ML Model](https://atlas.mitre.org/techniques/AML.T0024.002) **MITRE ATLAS**