Data Gathering Methodology

OWASP Top 10 for LLM GenAI Apps - Data Gathering Methodology

Overview

Welcome to the GitHub wiki dedicated to understanding and advancing the data-gathering methodology for OWASP's Top 10 for LLM AI Applications. As AI and deep learning technologies evolve rapidly, securing these systems is crucial. This wiki serves as a central repository for methodologies, strategies, and tools aimed at identifying and prioritizing vulnerabilities in LLMs based on real-world data.

The Data Gathering Methodology, Mapping, Risk, and Exploit initiative is designed to collect real-world data on vulnerabilities and risks associated with Large Language Models (LLMs). This effort supports the ongoing update of the OWASP Top 10 for LLMs list while maintaining mappings to major cybersecurity frameworks. By employing a comprehensive data collection approach, the initiative aims to enhance AI security guidelines and offer insights to help organizations secure their LLM-based systems.

Why this Wiki?

• Centralized Knowledge Base: With the complex nature of LLM vulnerabilities, having a single repository where developers, researchers, and security experts can find and contribute the most recent methodologies is invaluable.
• Collaborative Environment: GitHub offers an interactive platform for community members to collaborate, provide insights, updates, and refinements to the existing methodology.
• Transparency & Open-Source Spirit: In line with OWASP and open-source principles, this wiki promotes transparency in the data-gathering process, making best practices in vulnerability assessment widely accessible.
• Adapting to Emerging Threats: AI security is a rapidly growing field. This wiki will act as a live document, continuously evolving to capture the latest threats and vulnerabilities.

How to Contribute

• Join our Slack channel: #team-llm-datagathering-methodology
• Reach out to Emmanuel via email

Methodology

1. Data Collection

   • Sources:
     • Industry reports, academic papers, vulnerability databases, and real-world exploit analysis.
     • Partner organizations contributing vulnerability disclosures and risk assessments.
   • Approach:
     • Manual review and automated data scraping.
     • Use of standardized templates for data consistency.
     • Prioritization based on impact, exploitability, and prevalence.
   • We seek collaboration with organizations and individuals willing to share relevant datasets for LLM protection.

2. Data Analysis

   • Initial Review:
     • Classification of vulnerabilities by type, origin, and potential impact.
   • Statistical Analysis:
     • Python scripts validate and analyze gathered data for accuracy and completeness.
   • Risk Scoring:
     • Application of scoring frameworks (e.g., CVSS) to rank vulnerabilities based on severity.

3. Data Validation

   • Python Code:
     • Automated scripts ensure the integrity and accuracy of collected data.
   • Peer Review:
     • Involvement of cybersecurity experts for manual verification and risk assessment.

Datasets

The initiative will maintain several datasets, including:

• Vulnerability Dataset: Real-world vulnerabilities affecting LLM applications.
• Exploit Dataset: Documented exploits and attack techniques targeting LLMs.
• Risk Assessment Dataset: Mapped risk assessments for various LLM deployments.

Mapping to Cybersecurity Frameworks

Collected data is mapped to widely recognized security frameworks, including:

1. NIST Cybersecurity Framework CFS 2.0
   • Provides guidelines for managing cybersecurity risk.
2. ISO/IEC Standards
   • ISO/IEC 27001 (Information Security Management)
   • ISO/IEC 20547-4:2020 (Big Data Reference Architecture Security and Privacy)
3. MITRE ATT&CK
   • Knowledge base for understanding and defending against cyber attacks.
4. CIS Controls
   • Practical, actionable controls for strengthening cybersecurity defenses.
5. CVEs and CWEs
   • Standard for identifying and cataloging vulnerabilities.
6. FAIR
   • Risk quantification and management framework for cybersecurity.
7. STRIDE
   • Threat modeling methodology used in early software development.
8. ENISA
   • EU's agency for network and information security, focusing on compliance and best practices.
9. ASVS
   • Application Security Verification Standard, important for web application security.
10. SAMM
    • Software Assurance Maturity Model, helps in integrating security into software development.
11. MITRE ATLAS
    • Focuses on adversarial behaviors in threat modeling and analysis.
12. BSIMM
    • Tool for measuring and improving software security initiatives.
13. OPENCRE
    • Facilitates understanding and implementing cybersecurity controls across standards.
14. CycloneDX Machine Learning SBOM
    • Provides advanced supply chain capabilities for cyber risk reduction.

This mapping ensures compatibility and compliance, facilitating broader adoption across organizations.

LLM Data Security Best Practices

This initiative has produced a white paper detailing best practices for securing data in LLM-based systems, focusing on:

• Data Architecture
• Risk mitigation strategies
• Secure LLM deployment architectures
• Governance models for AI security

Given the rapid evolution of this technology, always monitor new TTPs, frameworks, regulations, and tools.

Additional Resources

• Data Validation Python Code Repository:
• White Paper: "LLM Data Security Best Practices"

---

For more details or contributions, please reach out to the OWASP Top 10 for LLM GenAI Apps team.