Merge pull request #674 from talesh/cleaned-markdown

Author: virtualsteve-star

2_0_vulns/LLM00_Preface.md

@@ -20,13 +20,14 @@ Like the technology itself, this list is a product of the open-source community
 
 Thank you to everyone who helped bring this together and those who continue to use and improve it. We’re grateful to be part of this work with you.
 
-
 #### Steve Wilson
+
 Project Lead
 OWASP Top 10 for Large Language Model Applications
-LinkedIn: https://www.linkedin.com/in/wilsonsd/
+LinkedIn: <https://www.linkedin.com/in/wilsonsd/>
 
 #### Ads Dawson
+
 Technical Lead & Vulnerability Entries Lead
 OWASP Top 10 for Large Language Model Applications
-LinkedIn: https://www.linkedin.com/in/adamdawson0/
+LinkedIn: <https://www.linkedin.com/in/adamdawson0/>

2_0_vulns/LLM01_PromptInjection.md

@@ -11,9 +11,11 @@ While prompt injection and jailbreaking are related concepts in LLM security, th
 ### Types of Prompt Injection Vulnerabilities
 
 #### Direct Prompt Injections
+
   Direct prompt injections occur when a user's prompt input directly alters the behavior of the model in unintended or unexpected ways. The input can be either intentional (i.e., a malicious actor deliberately crafting a prompt to exploit the model) or unintentional (i.e., a user inadvertently providing input that triggers unexpected behavior).
 
 #### Indirect Prompt Injections
+
   Indirect prompt injections occur when an LLM accepts input from external sources, such as websites or files. The external source may have content data that when interpreted by the model, alters the behavior of the model in unintended or unexpected ways. Like direct injections, indirect injections can be either intentional or unintentional.
 
 The severity and nature of the impact of a successful prompt injection attack can vary greatly and are largely dependent on both the business context the model operates in, and the agency with which the model is architected. Generally, however, prompt injection can lead to unintended outcomes, including but not limited to:
@@ -32,39 +34,69 @@ The rise of multimodal AI, which processes multiple data types simultaneously, i
 Prompt injection vulnerabilities are possible due to the nature of generative AI. Given the stochastic influence at the heart of the way models work, it is unclear if there are fool-proof methods of prevention for prompt injection. However, the following measures can mitigate the impact of prompt injections:
 
 #### 1. Constrain model behavior
+
   Provide specific instructions about the model's role, capabilities, and limitations within the system prompt. Enforce strict context adherence, limit responses to specific tasks or topics, and instruct the model to ignore attempts to modify core instructions.
+
 #### 2. Define and validate expected output formats
+
   Specify clear output formats, request detailed reasoning and source citations, and use deterministic code to validate adherence to these formats.
+
 #### 3. Implement input and output filtering
+
   Define sensitive categories and construct rules for identifying and handling such content. Apply semantic filters and use string-checking to scan for non-allowed content. Evaluate responses using the RAG Triad: Assess context relevance, groundedness, and question/answer relevance to identify potentially malicious outputs.
+
 #### 4. Enforce privilege control and least privilege access
+
   Provide the application with its own API tokens for extensible functionality, and handle these functions in code rather than providing them to the model. Restrict the model's access privileges to the minimum necessary for its intended operations.
+
 #### 5. Require human approval for high-risk actions
+
   Implement human-in-the-loop controls for privileged operations to prevent unauthorized actions.
+
 #### 6. Segregate and identify external content
+
   Separate and clearly denote untrusted content to limit its influence on user prompts.
+
 #### 7. Conduct adversarial testing and attack simulations
+
   Perform regular penetration testing and breach simulations, treating the model as an untrusted user to test the effectiveness of trust boundaries and access controls.
 
 ### Example Attack Scenarios
 
 #### Scenario #1: Direct Injection
+
   An attacker injects a prompt into a customer support chatbot, instructing it to ignore previous guidelines, query private data stores, and send emails, leading to unauthorized access and privilege escalation.
+
 #### Scenario #2: Indirect Injection
+
   A user employs an LLM to summarize a webpage containing hidden instructions that cause the LLM to insert an image linking to a URL, leading to exfiltration of the the private conversation.
+
 #### Scenario #3: Unintentional Injection
+
   A company includes an instruction in a job description to identify AI-generated applications. An applicant, unaware of this instruction, uses an LLM to optimize their resume, inadvertently triggering the AI detection.
+
 #### Scenario #4: Intentional Model Influence
+
   An attacker modifies a document in a repository used by a Retrieval-Augmented Generation (RAG) application. When a user's query returns the modified content, the malicious instructions alter the LLM's output, generating misleading results.
+
 #### Scenario #5: Code Injection
+
   An attacker exploits a vulnerability (CVE-2024-5184) in an LLM-powered email assistant to inject malicious prompts, allowing access to sensitive information and manipulation of email content.
+
 #### Scenario #6: Payload Splitting
+
   An attacker uploads a resume with split malicious prompts. When an LLM is used to evaluate the candidate, the combined prompts manipulate the model's response, resulting in a positive recommendation despite the actual resume contents.
+
 #### Scenario #7: Multimodal Injection
+
   An attacker embeds a malicious prompt within an image that accompanies benign text. When a multimodal AI processes the image and text concurrently, the hidden prompt alters the model's behavior, potentially leading to unauthorized actions or disclosure of sensitive information.
+
 #### Scenario #8: Adversarial Suffix
+
   An attacker appends a seemingly meaningless string of characters to a prompt, which influences the LLM's output in a malicious way, bypassing safety measures.
+
 #### Scenario #9: Multilingual/Obfuscated Attack
+
   An attacker uses multiple languages or encodes malicious instructions (e.g., using Base64 or emojis) to evade filters and manipulate the LLM's behavior.
 
 ### Reference Links

2_0_vulns/LLM02_SensitiveInformationDisclosure.md

@@ -11,64 +11,92 @@ To reduce this risk, LLM applications should perform adequate data sanitization
 ### Common Examples of Vulnerability
 
 #### 1. PII Leakage
+
   Personal identifiable information (PII) may be disclosed during interactions with the LLM.
+
 #### 2. Proprietary Algorithm Exposure
+
   Poorly configured model outputs can reveal proprietary algorithms or data. Revealing training data can expose models to inversion attacks, where attackers extract sensitive information or reconstruct inputs. For instance, as demonstrated in the 'Proof Pudding' attack (CVE-2019-20634), disclosed training data facilitated model extraction and inversion, allowing attackers to circumvent security controls in machine learning algorithms and bypass email filters.
+
 #### 3. Sensitive Business Data Disclosure
+
   Generated responses might inadvertently include confidential business information.
 
 ### Prevention and Mitigation Strategies
 
-#### Sanitization:
+#### Sanitization
 
 #### 1. Integrate Data Sanitization Techniques
+
   Implement data sanitization to prevent user data from entering the training model. This includes scrubbing or masking sensitive content before it is used in training.
+
 #### 2. Robust Input Validation
+
   Apply strict input validation methods to detect and filter out potentially harmful or sensitive data inputs, ensuring they do not compromise the model.
 
-#### Access Controls:
+#### Access Controls
 
 #### 1. Enforce Strict Access Controls
+
   Limit access to sensitive data based on the principle of least privilege. Only grant access to data that is necessary for the specific user or process.
+
 #### 2. Restrict Data Sources
+
   Limit model access to external data sources, and ensure runtime data orchestration is securely managed to avoid unintended data leakage.
 
-#### Federated Learning and Privacy Techniques:
+#### Federated Learning and Privacy Techniques
 
 #### 1. Utilize Federated Learning
+
   Train models using decentralized data stored across multiple servers or devices. This approach minimizes the need for centralized data collection and reduces exposure risks.
+
 #### 2. Incorporate Differential Privacy
+
   Apply techniques that add noise to the data or outputs, making it difficult for attackers to reverse-engineer individual data points.
 
-#### User Education and Transparency:
+#### User Education and Transparency
 
 #### 1. Educate Users on Safe LLM Usage
+
   Provide guidance on avoiding the input of sensitive information. Offer training on best practices for interacting with LLMs securely.
+
 #### 2. Ensure Transparency in Data Usage
+
   Maintain clear policies about data retention, usage, and deletion. Allow users to opt out of having their data included in training processes.
 
-#### Secure System Configuration:
+#### Secure System Configuration
 
 #### 1. Conceal System Preamble
+
   Limit the ability for users to override or access the system's initial settings, reducing the risk of exposure to internal configurations.
+
 #### 2. Reference Security Misconfiguration Best Practices
+
   Follow guidelines like "OWASP API8:2023 Security Misconfiguration" to prevent leaking sensitive information through error messages or configuration details.
   (Ref. link:[OWASP API8:2023 Security Misconfiguration](https://owasp.org/API-Security/editions/2023/en/0xa8-security-misconfiguration/))
 
-#### Advanced Techniques:
+#### Advanced Techniques
 
 #### 1. Homomorphic Encryption
+
   Use homomorphic encryption to enable secure data analysis and privacy-preserving machine learning. This ensures data remains confidential while being processed by the model.
+
 #### 2. Tokenization and Redaction
+
   Implement tokenization to preprocess and sanitize sensitive information. Techniques like pattern matching can detect and redact confidential content before processing.
 
 ### Example Attack Scenarios
 
 #### Scenario #1: Unintentional Data Exposure
+
   A user receives a response containing another user's personal data due to inadequate data sanitization.
+
 #### Scenario #2: Targeted Prompt Injection
+
   An attacker bypasses input filters to extract sensitive information.
+
 #### Scenario #3: Data Leak via Training Data
+
   Negligent data inclusion in training leads to sensitive information disclosure.
 
 ### Reference Links

2_0_vulns/LLM03_SupplyChain.md

@@ -14,23 +14,40 @@ A simple threat model can be found [here](https://github.com/jsotiro/ThreatModel
 ### Common Examples of Risks
 
 #### 1. Traditional Third-party Package Vulnerabilities
+
   Such as outdated or deprecated components, which attackers can exploit to compromise LLM applications. This is similar to "A06:2021 – Vulnerable and Outdated Components" with increased risks when components are used during model development or fine-tuning.
   (Ref. link: [A06:2021 – Vulnerable and Outdated Components](https://owasp.org/Top10/A06_2021-Vulnerable_and_Outdated_Components/))
+
 #### 2. Licensing Risks
+
   AI development often involves diverse software and dataset licenses, creating risks if not properly managed. Different open-source and proprietary licenses impose varying legal requirements. Dataset licenses may restrict usage, distribution, or commercialization.
+
 #### 3. Outdated or Deprecated Models
+
   Using outdated or deprecated models that are no longer maintained leads to security issues.
+
 #### 4. Vulnerable Pre-Trained Model
+
   Models are binary black boxes and unlike open source, static inspection can offer little to security assurances. Vulnerable pre-trained models can contain hidden biases, backdoors, or other malicious features that have not been identified through the safety evaluations of model repository. Vulnerable models can be created by both poisoned datasets and direct model tampering using techniques such as ROME also known as lobotomisation.
+
 #### 5. Weak Model Provenance
+
   Currently there are no strong provenance assurances in published models. Model Cards and associated documentation provide model information and relied upon users, but they offer no guarantees on the origin of the model. An attacker can compromise supplier account on a model repo or create a similar one and combine it with social engineering techniques to compromise the supply-chain of an LLM application.
+
 #### 6. Vulnerable LoRA adapters
+
   LoRA is a popular fine-tuning technique that enhances modularity by allowing pre-trained layers to be bolted onto an existing LLM. The method increases efficiency but create new risks, where a malicious LorA adapter compromises the integrity and security of the pre-trained base model. This can happen both in collaborative model merge environments but also exploiting the support for LoRA from popular inference deployment platforms such as vLMM and OpenLLM where adapters can be downloaded and applied to a deployed model.
+
 #### 7. Exploit Collaborative Development Processes
+
   Collaborative model merge and model handling services (e.g. conversions) hosted in shared environments can be exploited to introduce vulnerabilities in shared models. Model merging is is very popular on Hugging Face with model-merged models topping the OpenLLM leaderboard and can be exploited to bypass reviews. Similarly, services such as conversation bot have been proved to be vulnerable to manipulation and introduce malicious code in models.
+
 #### 8. LLM Model on Device supply-chain vulnerabilities
+
   LLM models on device increase the supply attack surface with compromised manufactured processes and exploitation of device OS or firmware vulnerabilities to compromise models. Attackers can reverse engineer and re-package applications with tampered models.
+
 #### 9. Unclear T&Cs and Data Privacy Policies
+
   Unclear T&Cs and data privacy policies of the model operators lead to the application's sensitive data being used for model training and subsequent sensitive information exposure. This may also apply to risks from using copyrighted material by the model supplier.
 
 ### Prevention and Mitigation Strategies
@@ -51,31 +68,56 @@ A simple threat model can be found [here](https://github.com/jsotiro/ThreatModel
 ### Sample Attack Scenarios
 
 #### Scenario #1: Vulnerable Python Library
+
   An attacker exploits a vulnerable Python library to compromise an LLM app. This happened in the first Open AI data breach. Attacks on the PyPi package registry tricked model developers into downloading a compromised PyTorch dependency with malware in a model development environment. A more sophisticated example of this type of attack is Shadow Ray attack on the Ray AI framework used by many vendors to manage AI infrastructure. In this attack, five vulnerabilities are believed to have been exploited in the wild affecting many servers.
+
 #### Scenario #2: Direct Tampering
+
   Direct Tampering and publishing a model to spread misinformation. This is an actual attack with PoisonGPT bypassing Hugging Face safety features by directly changing model parameters.
+
 #### Scenario #3: Fine-tuning Popular Model
+
   An attacker fine-tunes a popular open access model to remove key safety features and perform high in a specific domain (insurance). The model is fine-tuned to score highly on safety benchmarks but has very targeted triggers. They deploy it on Hugging Face for victims to use it exploiting their trust on benchmark assurances.
+
 #### Scenario #4: Pre-Trained Models
+
   An LLM system deploys pre-trained models from a widely used repository without thorough verification. A compromised model introduces malicious code, causing biased outputs in certain contexts and leading to harmful or manipulated outcomes
+
 #### Scenario #5: Compromised Third-Party Supplier
+
   A compromised third-party supplier provides a vulnerable LorA adapter that is being merged to an LLM using model merge on Hugging Face.
+
 #### Scenario #6: Supplier Infiltration
+
   An attacker infiltrates a third-party supplier and compromises the production of a LoRA (Low-Rank Adaptation) adapter intended for integration with an on-device LLM deployed using frameworks like vLLM or OpenLLM. The compromised LoRA adapter is subtly altered to include hidden vulnerabilities and malicious code. Once this adapter is merged with the LLM, it provides the attacker with a covert entry point into the system. The malicious code can activate during model operations, allowing the attacker to manipulate the LLM’s outputs.
+
 #### Scenario #7: CloudBorne and CloudJacking Attacks
+
   These attacks target cloud infrastructures, leveraging shared resources and vulnerabilities in the virtualization layers. CloudBorne involves exploiting firmware vulnerabilities in shared cloud environments, compromising the physical servers hosting virtual instances. CloudJacking refers to malicious control or misuse of cloud instances, potentially leading to unauthorized access to critical LLM deployment platforms. Both attacks represent significant risks for supply chains reliant on cloud-based ML models, as compromised environments could expose sensitive data or facilitate further attacks.
+
 #### Scenario #8: LeftOvers (CVE-2023-4969)
+
   LeftOvers exploitation of leaked GPU local memory to recover sensitive data. An attacker can use this attack to exfiltrate sensitive data in production servers and development workstations or laptops.
+
 #### Scenario #9: WizardLM
+
   Following the removal of WizardLM, an attacker exploits the interest in this model and publish a fake version of the model with the same name but containing malware and backdoors.
+
 #### Scenario #10: Model Merge/Format Conversion Service
+
   An attacker stages an attack with a model merge or format conversation service to compromise a publicly available access model to inject malware. This is an actual attack published by vendor HiddenLayer.
+
 #### Scenario #11: Reverse-Engineer Mobile App
+
   An attacker reverse-engineers an mobile app to replace the model with a tampered version that leads the user to scam sites. Users are encouraged to download the app directly via social engineering techniques. This is a "real attack on predictive AI" that affected 116 Google Play apps including popular security and safety-critical applications used for as cash recognition, parental control, face authentication, and financial service.
   (Ref. link: [real attack on predictive AI](https://arxiv.org/abs/2006.08131))
+
 #### Scenario #12: Dataset Poisoning
+
   An attacker poisons publicly available datasets to help create a back door when fine-tuning models. The back door subtly favors certain companies in different markets.
+
 #### Scenario #13: T&Cs and Privacy Policy
+
   An LLM operator changes its T&Cs and Privacy Policy to require an explicit opt out from using application data for model training, leading to the memorization of sensitive data.
 
 ### Reference Links

2_0_vulns/LLM04_DataModelPoisoning.md

@@ -34,14 +34,23 @@ Moreover, models distributed through shared repositories or open-source platform
 ### Example Attack Scenarios
 
 #### Scenario #1
+
   An attacker biases the model's outputs by manipulating training data or using prompt injection techniques, spreading misinformation.
+
 #### Scenario #2
+
   Toxic data without proper filtering can lead to harmful or biased outputs, propagating dangerous information.
-#### Scenario # 3
+
+#### Scenario #3
+
   A malicious actor or competitor creates falsified documents for training, resulting in model outputs that reflect these inaccuracies.
+
 #### Scenario #4
+
   Inadequate filtering allows an attacker to insert misleading data via prompt injection, leading to compromised outputs.
+
 #### Scenario #5
+
   An attacker uses poisoning techniques to insert a backdoor trigger into the model. This could leave you open to authentication bypass, data exfiltration or hidden command execution.
 
 ### Reference Links