Merge branch 'main' into translations/el-GR

Author: talesh

2_0_vulns/LLM01_PromptInjection.md

@@ -69,7 +69,7 @@ Prompt injection vulnerabilities are possible due to the nature of generative AI
 
 #### Scenario #2: Indirect Injection
 
-  A user employs an LLM to summarize a webpage containing hidden instructions that cause the LLM to insert an image linking to a URL, leading to exfiltration of the the private conversation.
+  A user employs an LLM to summarize a webpage containing hidden instructions that cause the LLM to insert an image linking to a URL, leading to exfiltration of the private conversation.
 
 #### Scenario #3: Unintentional Injection
 

2_0_vulns/LLM02_SensitiveInformationDisclosure.md

@@ -12,105 +12,104 @@ To reduce this risk, LLM applications should perform adequate data sanitization
 
 #### 1. PII Leakage
 
-  Personal identifiable information (PII) may be disclosed during interactions with the LLM.
+Personal identifiable information (PII) may be disclosed during interactions with the LLM.
 
 #### 2. Proprietary Algorithm Exposure
 
-  Poorly configured model outputs can reveal proprietary algorithms or data. Revealing training data can expose models to inversion attacks, where attackers extract sensitive information or reconstruct inputs. For instance, as demonstrated in the 'Proof Pudding' attack (CVE-2019-20634), disclosed training data facilitated model extraction and inversion, allowing attackers to circumvent security controls in machine learning algorithms and bypass email filters.
+Poorly configured model outputs can reveal proprietary algorithms or data. Revealing training data can expose models to inversion attacks, where attackers extract sensitive information or reconstruct inputs. For instance, as demonstrated in the 'Proof Pudding' attack (CVE-2019-20634), disclosed training data facilitated model extraction and inversion, allowing attackers to circumvent security controls in machine learning algorithms and bypass email filters.
 
 #### 3. Sensitive Business Data Disclosure
 
-  Generated responses might inadvertently include confidential business information.
+Generated responses might inadvertently include confidential business information.
 
 ### Prevention and Mitigation Strategies
-
 #### Sanitization
 
-#### 1. Integrate Data Sanitization Techniques
+##### 1. Integrate Data Sanitization Techniques
 
-  Implement data sanitization to prevent user data from entering the training model. This includes scrubbing or masking sensitive content before it is used in training.
+Implement data sanitization to prevent user data from entering the training model. This includes scrubbing or masking sensitive content before it is used in training.
 
-#### 2. Robust Input Validation
+##### 2. Robust Input Validation
 
-  Apply strict input validation methods to detect and filter out potentially harmful or sensitive data inputs, ensuring they do not compromise the model.
+Apply strict input validation methods to detect and filter out potentially harmful or sensitive data inputs, ensuring they do not compromise the model.
 
 #### Access Controls
 
-#### 1. Enforce Strict Access Controls
+##### 1. Enforce Strict Access Controls
 
-  Limit access to sensitive data based on the principle of least privilege. Only grant access to data that is necessary for the specific user or process.
+Limit access to sensitive data based on the principle of least privilege. Only grant access to data that is necessary for the specific user or process.
 
-#### 2. Restrict Data Sources
+##### 2. Restrict Data Sources
 
-  Limit model access to external data sources, and ensure runtime data orchestration is securely managed to avoid unintended data leakage.
+Limit model access to external data sources, and ensure runtime data orchestration is securely managed to avoid unintended data leakage.
 
 #### Federated Learning and Privacy Techniques
 
-#### 1. Utilize Federated Learning
+##### 1. Utilize Federated Learning
 
-  Train models using decentralized data stored across multiple servers or devices. This approach minimizes the need for centralized data collection and reduces exposure risks.
+Train models using decentralized data stored across multiple servers or devices. This approach minimizes the need for centralized data collection and reduces exposure risks.
 
-#### 2. Incorporate Differential Privacy
+##### 2. Incorporate Differential Privacy
 
-  Apply techniques that add noise to the data or outputs, making it difficult for attackers to reverse-engineer individual data points.
+Apply techniques that add noise to the data or outputs, making it difficult for attackers to reverse-engineer individual data points.
 
 #### User Education and Transparency
 
-#### 1. Educate Users on Safe LLM Usage
+##### 1. Educate Users on Safe LLM Usage
 
-  Provide guidance on avoiding the input of sensitive information. Offer training on best practices for interacting with LLMs securely.
+Provide guidance on avoiding the input of sensitive information. Offer training on best practices for interacting with LLMs securely. 
 
-#### 2. Ensure Transparency in Data Usage
+##### 2. Ensure Transparency in Data Usage
 
-  Maintain clear policies about data retention, usage, and deletion. Allow users to opt out of having their data included in training processes.
+Maintain clear policies about data retention, usage, and deletion. Allow users to opt out of having their data included in training processes.
 
 #### Secure System Configuration
 
-#### 1. Conceal System Preamble
+##### 1. Conceal System Preamble
 
-  Limit the ability for users to override or access the system's initial settings, reducing the risk of exposure to internal configurations.
+Limit the ability for users to override or access the system's initial settings, reducing the risk of exposure to internal configurations.
 
-#### 2. Reference Security Misconfiguration Best Practices
+##### 2. Reference Security Misconfiguration Best Practices
 
-  Follow guidelines like "OWASP API8:2023 Security Misconfiguration" to prevent leaking sensitive information through error messages or configuration details.
-  (Ref. link:[OWASP API8:2023 Security Misconfiguration](https://owasp.org/API-Security/editions/2023/en/0xa8-security-misconfiguration/))
+Follow guidelines like "OWASP API8:2023 Security Misconfiguration" to prevent leaking sensitive information through error messages or configuration details. (Ref. link: [OWASP API8:2023 Security Misconfiguration](https://owasp.org/API-Security/editions/2023/en/0xa8-security-misconfiguration/)).
 
 #### Advanced Techniques
 
-#### 1. Homomorphic Encryption
+##### 1. Homomorphic Encryption
 
-  Use homomorphic encryption to enable secure data analysis and privacy-preserving machine learning. This ensures data remains confidential while being processed by the model.
+Use homomorphic encryption to enable secure data analysis and privacy-preserving machine learning. This ensures data remains confidential while being processed by the model.
 
-#### 2. Tokenization and Redaction
+##### 2. Tokenization and Redaction
 
-  Implement tokenization to preprocess and sanitize sensitive information. Techniques like pattern matching can detect and redact confidential content before processing.
+Implement tokenization to preprocess and sanitize sensitive information. Techniques like pattern matching can detect and redact confidential content before processing.
 
 ### Example Attack Scenarios
 
-#### Scenario #1: Unintentional Data Exposure
+#### Scenario 1: Unintentional Data Exposure
 
-  A user receives a response containing another user's personal data due to inadequate data sanitization.
+A user receives a response containing another user's personal data due to inadequate data sanitization.
 
-#### Scenario #2: Targeted Prompt Injection
+#### Scenario 2: Targeted Prompt Injection
 
-  An attacker bypasses input filters to extract sensitive information.
+An attacker bypasses input filters to extract sensitive information.
 
-#### Scenario #3: Data Leak via Training Data
+#### Scenario 3: Data Leak via Training Data
 
-  Negligent data inclusion in training leads to sensitive information disclosure.
+Negligent data inclusion in training leads to sensitive information disclosure.
 
-### Reference Links
+#### Reference Links
 
-1. [Lessons learned from ChatGPT’s Samsung leak](https://cybernews.com/security/chatgpt-samsung-leak-explained-lessons/): **Cybernews**
-2. [AI data leak crisis: New tool prevents company secrets from being fed to ChatGPT](https://www.foxbusiness.com/politics/ai-data-leak-crisis-prevent-company-secrets-chatgpt): **Fox Business**
-3. [ChatGPT Spit Out Sensitive Data When Told to Repeat ‘Poem’ Forever](https://www.wired.com/story/chatgpt-poem-forever-security-roundup/): **Wired**
-4. [Using Differential Privacy to Build Secure Models](https://neptune.ai/blog/using-differential-privacy-to-build-secure-models-tools-methods-best-practices): **Neptune Blog**
-5. [Proof Pudding (CVE-2019-20634)](https://avidml.org/database/avid-2023-v009/) **AVID** (`moohax` & `monoxgas`)
+- *Cybernews*: [Lessons learned from ChatGPT’s Samsung leak](https://cybernews.com/security/chatgpt-samsung-leak-explained-lessons/)
+- *Fox Business*: [AI data leak crisis: New tool prevents company secrets from being fed to ChatGPT](https://www.foxbusiness.com/politics/ai-data-leak-crisis-prevent-company-secrets-chatgpt)
+- *Wired*: [ChatGPT Spit Out Sensitive Data When Told to Repeat ‘Poem’ Forever](https://www.wired.com/story/chatgpt-poem-forever-security-roundup/)
+- *Neptune Blog*: [sing Differential Privacy to Build Secure Models](https://neptune.ai/blog/using-differential-privacy-to-build-secure-models-tools-methods-best-practices)
+- *AVID 2023-009*: [Proof Pudding (CVE-2019-20634)](https://avidml.org/database/avid-2023-v009/)
+- *Will Pearce \(Moo_hax\) & Nick Landers \(Monoxgas\)*: [Proof Pudding research](https://github.com/moohax/Proof-Pudding)
 
-### Related Frameworks and Taxonomies
+#### Related Frameworks and Taxonomies
 
 Refer to this section for comprehensive information, scenarios strategies relating to infrastructure deployment, applied environment controls and other best practices.
 
-- [AML.T0024.000 - Infer Training Data Membership](https://atlas.mitre.org/techniques/AML.T0024.000) **MITRE ATLAS**
-- [AML.T0024.001 - Invert ML Model](https://atlas.mitre.org/techniques/AML.T0024.001) **MITRE ATLAS**
-- [AML.T0024.002 - Extract ML Model](https://atlas.mitre.org/techniques/AML.T0024.002) **MITRE ATLAS**
+- *MITRE ATLAS*: [AML.T0024.000 - Infer Training Data Membership](https://atlas.mitre.org/techniques/AML.T0024.000)
+- *MITRE ATLAS*: [AML.T0024.001 - Invert ML Model](https://atlas.mitre.org/techniques/AML.T0024.001)
+- *MITRE ATLAS*: [AML.T0024.002 - Extract ML Model](https://atlas.mitre.org/techniques/AML.T0024.002)

2_0_vulns/LLM03_SupplyChain.md

@@ -28,19 +28,19 @@ A simple threat model can be found [here](https://github.com/jsotiro/ThreatModel
 
 #### 4. Vulnerable Pre-Trained Model
 
-  Models are binary black boxes and unlike open source, static inspection can offer little to security assurances. Vulnerable pre-trained models can contain hidden biases, backdoors, or other malicious features that have not been identified through the safety evaluations of model repository. Vulnerable models can be created by both poisoned datasets and direct model tampering using techniques such as ROME also known as lobotomisation.
+  Models are binary black boxes and unlike open source, static inspection can offer little to no security assurances. Vulnerable pre-trained models can contain hidden biases, backdoors, or other malicious features that have not been identified through the safety evaluations of model repositories. Vulnerable models can be created by both poisoned datasets and direct model tampering using techniques such as ROME also known as lobotomisation.
 
 #### 5. Weak Model Provenance
 
-  Currently there are no strong provenance assurances in published models. Model Cards and associated documentation provide model information and relied upon users, but they offer no guarantees on the origin of the model. An attacker can compromise supplier account on a model repo or create a similar one and combine it with social engineering techniques to compromise the supply-chain of an LLM application.
+  Currently there are no strong provenance assurances in published models. Model Cards and associated documentation provide model information and relied upon users, but they offer no guarantees on the origin of the model. An attacker can compromise a supplier account on a model repo or create a similar one and combine it with social engineering techniques to compromise the supply-chain of an LLM application.
 
 #### 6. Vulnerable LoRA adapters
 
-  LoRA is a popular fine-tuning technique that enhances modularity by allowing pre-trained layers to be bolted onto an existing LLM. The method increases efficiency but create new risks, where a malicious LorA adapter compromises the integrity and security of the pre-trained base model. This can happen both in collaborative model merge environments but also exploiting the support for LoRA from popular inference deployment platforms such as vLMM and OpenLLM where adapters can be downloaded and applied to a deployed model.
+  LoRA is a popular fine-tuning technique that enhances modularity by allowing pre-trained layers to be bolted onto an existing LLM. The method increases efficiency but creates new risks, where a malicious LorA adapter compromises the integrity and security of the pre-trained base model. This can happen both in collaborative model merge environments but also exploiting the support for LoRA from popular inference deployment platforms such as vLMM and OpenLLM where adapters can be downloaded and applied to a deployed model.
 
 #### 7. Exploit Collaborative Development Processes
 
-  Collaborative model merge and model handling services (e.g. conversions) hosted in shared environments can be exploited to introduce vulnerabilities in shared models. Model merging is is very popular on Hugging Face with model-merged models topping the OpenLLM leaderboard and can be exploited to bypass reviews. Similarly, services such as conversation bot have been proved to be vulnerable to manipulation and introduce malicious code in models.
+  Collaborative model merge and model handling services (e.g. conversions) hosted in shared environments can be exploited to introduce vulnerabilities in shared models. Model merging is very popular on Hugging Face with model-merged models topping the OpenLLM leaderboard and can be exploited to bypass reviews. Similarly, services such as a conversation bot have been proved to be vulnerable to manipulation and introduce malicious code in models.
 
 #### 8. LLM Model on Device supply-chain vulnerabilities
 
@@ -55,7 +55,7 @@ A simple threat model can be found [here](https://github.com/jsotiro/ThreatModel
 1. Carefully vet data sources and suppliers, including T&Cs and their privacy policies, only using trusted suppliers. Regularly review and audit supplier Security and Access, ensuring no changes in their security posture or T&Cs.
 2. Understand and apply the mitigations found in the OWASP Top Ten's "A06:2021 – Vulnerable and Outdated Components." This includes vulnerability scanning, management, and patching components. For development environments with access to sensitive data, apply these controls in those environments, too.
   (Ref. link: [A06:2021 – Vulnerable and Outdated Components](https://owasp.org/Top10/A06_2021-Vulnerable_and_Outdated_Components/))
-3. Apply comprehensive AI Red Teaming and Evaluations when selecting a third party model. Decoding Trust is an example of a Trustworthy AI benchmark for LLMs but models can fine-tuned to by pass published benchmarks. Use extensive AI Red Teaming to evaluate the model, especially in the use cases you are planning to use the model for.
+3. Apply comprehensive AI Red Teaming and Evaluations when selecting a third party model. Decoding Trust is an example of a Trustworthy AI benchmark for LLMs but models can be fine-tuned to bypass published benchmarks. Use extensive AI Red Teaming to evaluate the model, especially in the use cases you are planning to use the model for.
 4. Maintain an up-to-date inventory of components using a Software Bill of Materials (SBOM) to ensure you have an up-to-date, accurate, and signed inventory, preventing tampering with deployed packages. SBOMs can be used to detect and alert for new, zero-date vulnerabilities quickly. AI BOMs and ML SBOMs are an emerging area and you should evaluate options starting with OWASP CycloneDX
 5. To mitigate AI licensing risks, create an inventory of all types of licenses involved using BOMs and conduct regular audits of all software, tools, and datasets, ensuring compliance and transparency through BOMs. Use automated license management tools for real-time monitoring and train teams on licensing models. Maintain detailed licensing documentation in BOMs and leverage tools such as [Dyana](https://github.com/dreadnode/dyana) to perform dynamic analysis of third-party software.
 6. Only use models from verifiable sources and use third-party model integrity checks with signing and file hashes to compensate for the lack of strong model provenance. Similarly, use code signing for externally supplied code.
@@ -101,15 +101,15 @@ A simple threat model can be found [here](https://github.com/jsotiro/ThreatModel
 
 #### Scenario #9: WizardLM
 
-  Following the removal of WizardLM, an attacker exploits the interest in this model and publish a fake version of the model with the same name but containing malware and backdoors.
+  Following the removal of WizardLM, an attacker exploits the interest in this model and publishes a fake version of the model with the same name but containing malware and backdoors.
 
 #### Scenario #10: Model Merge/Format Conversion Service
 
   An attacker stages an attack with a model merge or format conversation service to compromise a publicly available access model to inject malware. This is an actual attack published by vendor HiddenLayer.
 
 #### Scenario #11: Reverse-Engineer Mobile App
 
-  An attacker reverse-engineers an mobile app to replace the model with a tampered version that leads the user to scam sites. Users are encouraged to download the app directly via social engineering techniques. This is a "real attack on predictive AI" that affected 116 Google Play apps including popular security and safety-critical applications used for as cash recognition, parental control, face authentication, and financial service.
+  An attacker reverse-engineers an mobile app to replace the model with a tampered version that leads the user to scam sites. Users are encouraged to download the app directly via social engineering techniques. This is a "real attack on predictive AI" that affected 116 Google Play apps including popular security and safety-critical applications used for cash recognition, parental control, face authentication, and financial service.
   (Ref. link: [real attack on predictive AI](https://arxiv.org/abs/2006.08131))
 
 #### Scenario #12: Dataset Poisoning