From 139cb837003d286f4dac674395cb3b0475fdee94 Mon Sep 17 00:00:00 2001 From: balteravishay Date: Tue, 29 Jul 2025 18:16:01 +0100 Subject: [PATCH 01/31] adding citations and tldr Signed-off-by: balteravishay --- ...uide-for-AI-Code-Assistant-Instructions.md | 181 +++++++++++++++--- docs/developers.md | 3 +- docs/index.md | 1 + 3 files changed, 154 insertions(+), 31 deletions(-) diff --git a/docs/Security-Focused-Guide-for-AI-Code-Assistant-Instructions.md b/docs/Security-Focused-Guide-for-AI-Code-Assistant-Instructions.md index dc788854..ca47da3b 100644 --- a/docs/Security-Focused-Guide-for-AI-Code-Assistant-Instructions.md +++ b/docs/Security-Focused-Guide-for-AI-Code-Assistant-Instructions.md @@ -1,6 +1,28 @@ -# **Security-Focused Guide for AI Code Assistant Instructions** +# Security-Focused Guide for AI Code Assistant Instructions -AI code assistants can significantly speed up development, but they need guidance to produce **secure** and robust code. This guide explains how to create custom instructions (e.g. [GitHub Copilot instructions file](https://docs.github.com/en/copilot/how-tos/custom-instructions/adding-repository-custom-instructions-for-github-copilot), [Cline instructions file](https://docs.cline.bot/enterprise-solutions/custom-instructions), [Cursor rules](https://docs.cursor.com/context/rules), [Claude markdown](https://docs.anthropic.com/en/docs/claude-code/common-workflows#create-an-effective-claude-md-file), etc.) that ensure the AI assistant accounts for application code security, supply chain safety, and platform or language-specific considerations and to embed a “security conscience” into the tool. In practice, this means fewer vulnerabilities making it into your codebase. Remember that these instructions should be kept concise, specific, and actionable – the goal is to influence the AI’s behaviour without overwhelming it. +AI code assistants can significantly speed up development, but they need guidance to produce **secure** and robust code. This guide explains how to create custom instructions (e.g. [GitHub Copilot instructions file](https://docs.github.com/en/copilot/how-tos/custom-instructions/adding-repository-custom-instructions-for-github-copilot), [Cline instructions file](https://docs.cline.bot/enterprise-solutions/custom-instructions), [Cursor rules](https://docs.cursor.com/context/rules), [Claude markdown](https://docs.anthropic.com/en/docs/claude-code/common-workflows#create-an-effective-claude-md-file), etc.) that ensure the AI assistant accounts for application code security, supply chain safety, and platform or language-specific considerations and to embed a "security conscience" into the tool. In practice, this means fewer vulnerabilities making it into your codebase. Remember that these instructions should be kept concise, specific, and actionable – the goal is to influence the AI's behaviour without overwhelming it. [[1]](#1) + +--- + +## TL;DR + +Short on time? Here's what really matters: + +* **You Are the Pilot – AI is the Co-pilot:** The developer (you) remains in full control of the code. Critically evaluate and edit AI-generated code just as you would code written by a human colleague and never blindly accept suggestions. [[2]](#2) +* **Apply Engineering Best Practices Always:** AI-generated code isn't a shortcut around engineering processes such as code reviews, testing, static analysis, documentation, and version control discipline. +* **Be Security-Conscious:** Assume AI-written code can have bugs or vulnerabilities, because it often does. AI coding assistants can introduce security issues like using outdated cryptography or outdated dependencies, ignoring error handling, or leaking secrets. Check for any secrets or sensitive data in the suggested code. Make sure dependency suggestions are safe and not pulling in known vulnerable packages. [[3]](#3), [[4]](#4) +* **Guide the AI:** AI is a powerful assistant, but it works best with your guidance. Write clear prompts that specify security requirements. Don't hesitate to modify or reject AI outputs. Direct your AI tool to build its own instructions file based on this guide. [[5]](#5) [[6]](#6) + +By keeping these points in mind, you can harness AI code assistants effectively without sacrificing quality or security. + +Additionally, you can use the following ready-to-use instructions files as a starting point: + +- [Python](TODO) +- [Golang](TODO) +- [Java](TODO) +- [JavaScript](TODO) +- [C/C++](TODO) +- [Rust](TODO) --- @@ -8,57 +30,156 @@ AI code assistants can significantly speed up development, but they need guidanc One of the first sections in your instructions should reinforce general secure coding best practices. These principles apply to all languages and frameworks, and you want the AI to **always** keep them in mind when generating code: -* **Input Validation & Output Encoding:** Instruct the AI to treat all external inputs as untrusted and to validate them. *Example*: “user inputs should be checked for expected format and length”. Any output should be properly encoded to prevent injection attacks such as SQL injection or cross-site scripting (XSS). *Example:* “Always validate function arguments and use parameterized queries for database access” and "Escape special characters in user-generated content before rendering it in HTML”. Similarly, specify that when generating output contexts such as HTML or SQL, the assistant should use safe frameworks or encoding functions to avoid vulnerabilities. -* **Authentication, Authorization & Secrets Management:** Emphasize that **credentials and sensitive tokens must never be hard-coded** or exposed. Your instructions can say: *“Never include API keys, passwords, or secrets in code output, and use environment variables or secure vault references instead”*. Also instruct the AI to use secure authentication flows (for instance, using industry-standard libraries for handling passwords or tokens) and to enforce role-based access checks where appropriate. -* **Error Handling & Logging:** Guide the AI to implement errors securely. That means **catching exceptions and failures without revealing sensitive info** (stack traces, server paths, etc.) to the end-user. In your instructions, you might include: *“When generating code, handle errors gracefully and log them, but do not expose internal details or secrets in error messages.”* This ensures the assistant’s suggestions include secure error-handling patterns (like generic user-facing messages and detailed logs only on the server side). Additionally, instruct the AI to use logging frameworks that can be configured for security (e.g. avoiding logging of personal data or secrets). -* **Secure Defaults & Configurations:** Include guidance such as: *“Prefer safe defaults in configurations – for example, use HTTPS by default, require strong encryption algorithms, and disable insecure protocols or options.”* By specifying this, the AI will be more likely to generate code that opts-in to security features. Always instruct the AI to follow the principle of least privilege (e.g. minimal file system permissions, least-privileged user accounts for services, etc.) in any configuration or code it proposes. -* **Testing for Security:** Encourage the AI to produce or suggest tests for critical code paths. In your instructions, add: *“When applicable, generate unit tests for security-critical functions (including negative tests to ensure the code fails safely).”* Automated testing is a recognized best practice – guides recommend including negative tests to verify that what *shouldn’t* happen, doesn’t happen. -* **Call out and review stubbed code:** If the AI generates code that is stubbed or incomplete, instruct it to flag these areas for review. For example: *“If you generate placeholder code (e.g., `TODO` comments), ensure it is marked for security review before deployment.”* This will help ensure that any incomplete code does not inadvertently introduce vulnerabilities. +* **Input Validation & Output Encoding:** Instruct the AI to treat all external inputs as untrusted and to validate them. *Example: "user inputs should be checked for expected format and length*". Any output should be properly encoded to prevent injection attacks such as SQL injection or cross-site scripting (XSS). *Example: "Always validate function arguments and use parameterized queries for database access"* and *"Escape special characters in user-generated content before rendering it in HTML"*. Similarly, specify that when generating output contexts such as HTML or SQL, the assistant should use safe frameworks or encoding functions to avoid vulnerabilities. [[7]](#7) [[8]](#8) +* **Authentication, Authorization & Secrets Management:** Emphasize that credentials and sensitive tokens must never be hard-coded or exposed. Your instructions can say: *"Never include API keys, passwords, or secrets in code output, and use environment variables or secure vault references instead"*. Also instruct the AI to use secure authentication flows (for instance, using industry-standard libraries for handling passwords or tokens) and to enforce role-based access checks where appropriate. [[9]](#9) [[10]](#10) [[11]](#11) +* **Error Handling & Logging:** Guide the AI to implement errors securely by catching exceptions and failures without revealing sensitive info (stack traces, server paths, etc.) to the end-user. In your instructions, you might include: *"When generating code, handle errors gracefully and log them, but do not expose internal details or secrets in error messages".* This ensures the assistant's suggestions include secure error-handling patterns (like generic user-facing messages and detailed logs only on the server side). Additionally, instruct the AI to use logging frameworks that can be configured for security (e.g. avoiding logging of personal data or secrets). [[12]](#12) +* **Secure Defaults & Configurations:** Include guidance such as: *"Prefer safe defaults in configurations – for example, use HTTPS by default, require strong encryption algorithms, and disable insecure protocols or options".* By specifying this, the AI will be more likely to generate code that opts-in to security features. Always instruct the AI to follow the principle of least privilege (e.g. minimal file system permissions, least-privileged user accounts for services, etc.) in any configuration or code it proposes. [[13]](#13) [[14]](#14) +* **Testing for Security:** Encourage the AI to produce or suggest tests for critical code paths including negative tests that verify that what shouldn't happen, doesn't happen. In your instructions, add: *"When applicable, generate unit tests for security-critical functions (including negative tests to ensure the code fails safely)"*. [[15]](#15) +* **Call out and review stubbed code:** If the AI generates code that is stubbed or incomplete, instruct it to flag these areas for review. For example: *"If you generate placeholder code (e.g., `TODO` comments), ensure it is marked for security review before deployment".* This will help ensure that any incomplete code does not inadvertently introduce vulnerabilities. [[16]](#16) [[17]](#17) +* **Data Protection:** When generating code, always prioritize data minimization and avoid storing or processing confidential or otherwise sensitive information (like personal data - PII) unless absolutely necessary. For that case, suggest strong encryption at rest and in transit, and recommend techniques like anonymization. For example: *"Generate a function that securely handles user input for a registration form, asking for necessary fields to avoid logging sensitive information of PII. Ensure that no sensitive or PII is stored in plaintext"*. [[18]](#18) --- ## **Addressing Software Supply Chain Security** -Modern software heavily relies on third-party libraries and dependencies. It’s crucial that your AI assistant’s instructions cover supply chain security, ensuring that suggested dependencies and build processes are secure: - -* **Safe Dependency Selection:** Instruct the AI to **prefer well-vetted, reputable libraries** when suggesting code that pulls in external packages. For example, *“Use popular, community-trusted libraries for common tasks (and avoid adding obscure dependencies if a standard library or well-known package can do the same job).”* Emphasize evaluating packages before use – as a developer would manually. -* **Use Package Managers & Lock Versions:** Your instructions should tell the AI to use proper package management. For instance: *“Always use the official package manager for the given language (npm, pip, Maven, etc.) to install libraries, rather than copying code snippets.”* Also, instruct it to specify version ranges or exact versions that are known to be secure. By doing so, the AI will generate code that, for example, uses a `requirements.txt` or `package.json` entry, which aids in maintaining supply chain integrity. -* **Stay Updated & Monitor Vulnerabilities:** Include guidance for keeping dependencies up-to-date. For example: *"When suggesting dependency versions, prefer the latest stable release and mention updating dependencies regularly to patch vulnerabilities"*. -* **Generate Software Bill of Materials (SBOM):** Instruct the AI to create and maintaining SBOMs for better visibility into your software supply chain. For example: *"Generate a Software Bill of Materials (SBOM) by using tools that support standard formats like SPDX or CycloneDX."* You can also mention provenance tracking: *"Where applicable, use in-toto attestations or similar frameworks to create verifiable records of your build and deployment processes."* This ensures comprehensive tracking of what goes into your software and provides the foundation for ongoing vulnerability monitoring and incident response. -* **Integrity Verification:** To further secure the supply chain, you can instruct the assistant to show how to verify what it uses. For instance: *"When adding important external resources (scripts, containers, etc.), include steps to verify integrity (like checksum verification or signature validation) if applicable."* +Modern software heavily relies on third-party libraries and dependencies. It's crucial that your AI assistant's instructions cover supply chain security, ensuring that suggested dependencies and build processes are secure: +* **Safe Dependency Selection:** Instruct the AI to prefer well-vetted, reputable libraries when suggesting code that pulls in external packages. For example: *"Use popular, community-trusted libraries for common tasks (and avoid adding obscure dependencies if a standard library or well-known package can do the same job)".* Emphasize evaluating packages before use – as a developer would manually. [[19]](#19) +* **Use Package Managers & Lock Versions:** Your instructions should tell the AI to use proper package management. For instance: *"Always use the official package manager for the given language (npm, pip, Maven, etc.) to install libraries, rather than copying code snippets".* Also, instruct it to specify version ranges or exact versions that are known to be secure. By doing so, the AI will generate code that, for example, uses a `requirements.txt` or `package.json` entry, which aids in maintaining supply chain integrity. [[20]](#20) +* **Stay Updated & Monitor Vulnerabilities:** Include guidance for keeping dependencies up-to-date. For example: *"When suggesting dependency versions, prefer the latest stable release and mention updating dependencies regularly to patch vulnerabilities"*. [[21]](#21) [[22]](#22) +* **Generate Software Bill of Materials (SBOM):** Instruct the AI to create and maintaining SBOMs for better visibility into your software supply chain. For example: *"Generate a Software Bill of Materials (SBOM) by using tools that support standard formats like SPDX or CycloneDX".* You can also mention provenance tracking: *"Where applicable, use in-toto attestations or similar frameworks to create verifiable records of your build and deployment processes".* This ensures comprehensive tracking of what goes into your software and provides the foundation for ongoing vulnerability monitoring and incident response. [[23]](#23) +* **Integrity Verification:** To further secure the supply chain, you can instruct the assistant to show how to verify what it uses. For instance: *"When adding important external resources (scripts, containers, etc.), include steps to verify integrity (like checksum verification or signature validation) if applicable".* [[24]](#24) --- ## **Platform and Runtime Security Considerations** Every software runs somewhere – be it a web browser, a cloud platform, an IoT device, or an OS. Your guide for the AI assistant should have a section addressing platform-specific and runtime security concerns: -* **Operating System and Environment:** If your code will run on a server or user machine, instruct the AI on safe system interactions. *“When writing file or OS-level operations, use safe functions and check for errors (e.g., use secure file modes, avoid temp files without proper randomness, etc.). If running as a service, drop privileges when possible.”* This ensures the AI’s output respects security at the OS level (for example, not recommending running as root unless absolutely necessary). Emphasize least privilege: for instance, code for a service should use the minimal privileges needed. The assistant could then suggest using system APIs that limit permissions or show how to configure a Linux capability instead of full root access. -* **Web and Cloud Platforms:** For web applications, direct the AI to include web security best practices. *“Always include appropriate security headers (Content Security Policy, X-Frame-Options, etc.) in web responses, and use frameworks’ built-in protections for cookies and sessions.”* You can also mention using HTTPS for any client-server communication. On cloud platforms, instruct it on secure use of cloud services: *“When generating code for cloud services (AWS/Azure/GCP), follow the provider’s security guidelines (e.g., use parameterized queries for cloud databases, encrypt data at rest and in transit, handle keys via cloud KMS).”* -* **Container and Deployment Considerations:** If your project uses containers or orchestrators (like Docker, Kubernetes), include guidance like: *"Use minimal base images and avoid running containers with the root user. Use official images from trusted sources, and pin image versions using immutable digests (e.g., SHA256 hashes) instead of mutable tags like latest".* This will cause the AI to generate Dockerfiles or deployment scripts that adhere to container security best practices and ties into supply chain concerns as well – using only trusted images and specific versions that ensure consistency and prevent supply chain attacks. Additionally, instruct the AI on container signature verification: *"When working with container images, verify both the integrity and authenticity of images using container signing tools like cosign or notation. Include steps to verify signatures from trusted publishers and implement admission controllers in Kubernetes to enforce signature verification policies."* Instruct the AI to further control deployment artifacts. For instance, by ensuring production websites only load resources from trusted domains to avoid subversion: *"When generating HTML/JS, do not include direct links to untrusted third-party hosts for critical libraries; use our locally hosted or CDN with integrity checks."* -* **Mobile and Desktop App Security:** If relevant, instruct on platform-specific secure coding. For mobile apps, you might say: *“Do not suggest storing sensitive data in plaintext on the device; use the platform’s secure storage APIs.”* For desktop apps, *“Prefer high-level libraries for cryptography rather than rolling your own.”* These platform notes ensure the AI isn’t blind to the context in which code runs. It will include, for example, usage of Android’s `SharedPreferences` with encryption or Apple’s Keychain for iOS if dealing with credentials, reflecting platform best practices. +* **Operating System and Environment:** If your code will run on a server or user machine, instruct the AI on safe system interactions. *"When writing file or OS-level operations, use safe functions and check for errors (e.g., use secure file modes, avoid temp files without proper randomness, etc.). If running as a service, drop privileges when possible".* This ensures the AI's output respects security at the OS level (for example, not recommending running as root unless absolutely necessary). Emphasize least privilege: for instance, code for a service should use the minimal privileges needed. The assistant could then suggest using system APIs that limit permissions or show how to configure a Linux capability instead of full root access. [[25]](#25) [[26]](#26) +* **Web and Cloud Platforms:** For web applications, direct the AI to include web security best practices. *"Always include appropriate security headers (Content Security Policy, X-Frame-Options, etc.) in web responses, and use frameworks' built-in protections for cookies and sessions".* You can also mention using HTTPS for any client-server communication. On cloud platforms, instruct it on secure use of cloud services: *"When generating code for cloud services (AWS/Azure/GCP), follow the provider's security guidelines (e.g., use parameterized queries for cloud databases, encrypt data at rest and in transit, handle keys via cloud KMS)".* [[27]](#27) [[28]](#28) +* **Container and Deployment Considerations:** If your project uses containers or orchestrators (like Docker, Kubernetes), include guidance like: *"Use minimal base images and avoid running containers with the root user. Use official images from trusted sources, and pin image versions using immutable digests (e.g., SHA256 hashes) instead of mutable tags like latest".* This will cause the AI to generate Dockerfiles or deployment scripts that adhere to container security best practices and ties into supply chain concerns as well – using only trusted images and specific versions that ensure consistency and prevent supply chain attacks. Additionally, instruct the AI on container signature verification: *"When working with container images, verify both the integrity and authenticity of images using container signing tools like cosign or notation. Include steps to verify signatures from trusted publishers and implement admission controllers in Kubernetes to enforce signature verification policies".* Instruct the AI to further control deployment artifacts. For instance, by ensuring production websites only load resources from trusted domains to avoid subversion: *"When generating HTML/JS, do not include direct links to untrusted third-party hosts for critical libraries; use our locally hosted or CDN with integrity checks".* [[29]](#29) [[30]](#30) +* **Mobile and Desktop App Security:** If relevant, instruct on platform-specific secure coding. For mobile apps, you might say: *"Do not suggest storing sensitive data in plaintext on the device; use the platform's secure storage APIs".* For desktop apps, *"Prefer high-level libraries for cryptography rather than rolling your own".* These platform notes ensure the AI isn't blind to the context in which code runs. It will include, for example, usage of Android's `SharedPreferences` with encryption or Apple's Keychain for iOS if dealing with credentials, reflecting platform best practices. [[31]](#31) +* **Operational Security:** Instruct the AI to consider devops security aspects. For example: *"When generating github actions or CI/CD pipelines, ensure secrets are stored securely (e.g., using GitHub Secrets or environment variables) and not hard-coded in the workflow files"*. This will help the AI generate secure CI/CD configurations that prevent accidental exposure of sensitive information. You can also instruct it to include security checks in the pipeline: *"Include steps to run security scans (SAST/DAST) and dependency checks in the CI/CD pipeline to catch vulnerabilities early"*. This ensures that the AI's generated code integrates security into the development lifecycle, rather than treating it as an afterthought. Treat devops tools with the same security rigor as application code. For example, instruct the AI to use secure coding practices in infrastructure-as-code (IaC) scripts, such as Terraform or CloudFormation: *"When generating IaC scripts, ensure they follow security best practices (e.g., restrict access to resources, use secure storage for secrets, and validate inputs)"* and *"use the latest versions of devops dependencies such as GitHub actions and lock the version to specific SHA"*. [[32]](#32) [[33]](#33) --- ## **Programming Language-Specific Security Examples** -It’s valuable to dedicate individual paragraphs to language-specific security considerations that are relevant to your project. The following section serves as examples, or special cases the AI should handle to highlight unique vulnerabilities or features of that language and how the AI should approach them: +It's valuable to dedicate individual paragraphs to language-specific security considerations that are relevant to your project. The following section serves as examples, or special cases the AI should handle to highlight unique vulnerabilities or features of that language and how the AI should approach them: -* **C/C++ (Memory-Unsafe Languages):** For languages without automatic memory safety, instruct the AI to be extra cautious with memory management. *“In C or C++ code, always use bounds-checked functions (e.g., `strncpy` over `strcpy`), avoid dangerous functions like `gets`, and include buffer size constants to prevent overflow. Enable compiler defenses (stack canaries, fortify source, DEP/NX) in any build configurations you suggest”*. By giving such instructions, the assistant might prefer safer standard library calls or even suggest modern C++ classes (`std::vector` instead of raw arrays) to reduce manual memory handling. It will also acknowledge when an operation is risky, possibly inserting comments like “// ensure no buffer overflow”. -* **Rust, Go, and Memory-Safe Languages:** If the project involves memory-safe languages (Rust, Go, Java, C\#, etc.), you can note that the AI should leverage their safety features. *“In Rust code, avoid using `unsafe` blocks unless absolutely necessary and document any `unsafe` usage with justification.”* Memory-safe-by-default languages enforce a lot at compile time, but you should still have the AI follow best practices of those ecosystems. For example, instruct: *“In any memory-safe language, prefer using safe library functions and types; don’t circumvent their safety without cause.”* If a language offers any tools to verify memory access, direct the AI assistant to use them while building or testing your code. For Example “*In go code, use the data race detector when building the application*”. -* **Python and Dynamic Languages:** Python, JavaScript, and other high-level languages manage memory for you, but come with their own security pitfalls. In your instructions, emphasize things like avoiding exec/eval with untrusted input in Python and being careful with command execution. *“For Python, do not use `exec`/`eval` on user input and prefer safe APIs (e.g., use the `subprocess` module with `shell=False` to avoid shell injection).”* Additionally, mention type checking or the use of linters: *“Follow PEP 8 and use type hints, as this can catch misuse early.”* For JavaScript/TypeScript, you might add: *“When generating Node.js code, use prepared statements for database queries (just like any other language) and encode any data that goes into HTML to prevent XSS”.* These instructions incorporate known best practices (like those from OWASP cheat sheets) directly into the AI’s behavior. -* **Java/C\# and Enterprise Languages:** In languages often used for large applications, you might focus on frameworks and configurations. *“For Java, when suggesting web code (e.g., using Spring), ensure to use built-in security annotations and avoid old, vulnerable libraries (e.g., use `BCryptPasswordEncoder` rather than writing a custom password hash).”* For C\#, similarly: *“Use .NET’s cryptography and identity libraries instead of custom solutions.”* Also instruct about managing object deserialization (both Java and C\# have had vulnerabilities in this area): *“Never suggest turning off security features like XML entity security or type checking during deserialization.”* These language-specific notes guide the AI to incorporate the well-known secure patterns of each ecosystem. +* **C/C++ (Memory-Unsafe Languages):** For languages without automatic memory safety, instruct the AI to be extra cautious with memory management. *"In C or C++ code, always use bounds-checked functions (e.g., `strncpy` over `strcpy`), avoid dangerous functions like `gets`, and include buffer size constants to prevent overflow. Enable compiler defenses (stack canaries, fortify source, DEP/NX) in any build configurations you suggest"*. By giving such instructions, the assistant might prefer safer standard library calls or even suggest modern C++ classes (`std::vector` instead of raw arrays) to reduce manual memory handling. It will also acknowledge when an operation is risky, possibly inserting comments like "// ensure no buffer overflow". [[34]](#34) [[35]](#35) +* **Rust, Go, and Memory-Safe Languages:** If the project involves memory-safe languages (Rust, Go, Java, C\#, etc.), you can note that the AI should leverage their safety features. *"In Rust code, avoid using `unsafe` blocks unless absolutely necessary and document any `unsafe` usage with justification".* Memory-safe-by-default languages enforce a lot at compile time, but you should still have the AI follow best practices of those ecosystems. For example, instruct: *"In any memory-safe language, prefer using safe library functions and types; don't circumvent their safety without cause".* If a language offers any tools to verify memory access, direct the AI assistant to use them while building or testing your code. For Example "*In go code, use the data race detector when building the application*". [[36]](#36) +* **Python and Dynamic Languages:** Python, JavaScript, and other high-level languages manage memory for you, but come with their own security pitfalls. In your instructions, emphasize things like avoiding exec/eval with untrusted input in Python and being careful with command execution. *"For Python, do not use `exec`/`eval` on user input and prefer safe APIs (e.g., use the `subprocess` module with `shell=False` to avoid shell injection)".* Additionally, mention type checking or the use of linters: *"Follow PEP 8 and use type hints, as this can catch misuse early".* For JavaScript/TypeScript, you might add: *"When generating Node.js code, use prepared statements for database queries (just like any other language) and encode any data that goes into HTML to prevent XSS".* These instructions incorporate known best practices (like those from OWASP cheat sheets) directly into the AI's behavior. [[37]](#37) +* **Java/C\# and Enterprise Languages:** In languages often used for large applications, you might focus on frameworks and configurations. *"For Java, when suggesting web code (e.g., using Spring), ensure to use built-in security annotations and avoid old, vulnerable libraries (e.g., use `BCryptPasswordEncoder` rather than writing a custom password hash)".* For C\#, similarly: *"Use .NET's cryptography and identity libraries instead of custom solutions".* Also instruct about managing object deserialization (both Java and C\# have had vulnerabilities in this area): *"Never suggest turning off security features like XML entity security or type checking during deserialization".* These language-specific notes guide the AI to incorporate the well-known secure patterns of each ecosystem. [[38]](#38) --- ## **Referencing Security Standards and Frameworks** -To strengthen your AI assistant’s guidance, you should point it toward established security standards and frameworks. By referencing these in the instructions, the AI will be biased toward solutions that comply with widely accepted practices: +To strengthen your AI assistant's guidance, you should point it toward established security standards and frameworks. By referencing these in the instructions, the AI will be biased toward solutions that comply with widely accepted practices: -* **Use Industry Standards as Keywords:** Mention standards like **OWASP Top 10**, **OWASP ASVS**, **CWE/SANS Top 25**, or language-specific guidelines (Java’s SEI CERT secure coding standards, for instance). For example: *“Code suggestions should adhere to OWASP Top 10 principles (e.g., avoid injection, enforce access control) and follow the OWASP ASVS requirements where applicable”.* -* **Reference SAFECode and Other Guides:** SAFECode (Software Assurance Forum for Excellence in Code) publishes **Fundamental Practices for Secure Software Development**, and including a note about it can be beneficial. *“Our project follows SAFECode’s secure development practices – the AI should prioritize those (e.g., proper validation, authentication, cryptography usage per SAFECode guidance).”* This ties the assistant’s suggestions to a comprehensive set of secure development principles. Similarly, you can reference the **Secure Software Development Lifecycle (SSDLC)** or standards like **ISO 27034** if your organization uses them, to align AI outputs with those processes. -* **Compliance and Regulatory Security:** If you operate in an industry with specific regulations (like healthcare or finance), instruct the AI with those in mind. For instance: *“When generating code, consider compliance requirements (e.g., HIPAA privacy rules for medical data, PCI-DSS for credit card info) – do not output code that logs or transmits sensitive data in insecure ways.”* This ensures the AI’s suggestions are compliant with external standards, for example, it might suggest encryption for personal data fields by default. -* **Continuous Improvement Hooks:** Encourage practices that integrate with ongoing security processes. For example: *“Include comments or TODOs in code suggesting security reviews for complex logic, and note if any third-party component might need a future update or audit.”* This way, the AI not only writes code but also flags areas for human scrutiny or future action. If the AI outputs a piece of cryptography code, for instance, it might add a comment “// TODO: security review for cryptographic implementation” because you instructed it that critical code should not go un-reviewed. -* **Integrate and Act on Automated Security Tooling:** Instruct the AI to proactively run automated security tools (e.g., SAST, DAST, SCA) during code generation and refinement. Example: “When writing or reviewing code, run or simulate the use of tools like CodeQL, Bandit, Semgrep, or OWASP Dependency-Check. Identify any flagged vulnerabilities or outdated dependencies and revise the code accordingly. Repeat this process until the code passes all simulated scans.” This positions the AI not just as a code generator but as a security-aware assistant capable of iterating based on tool feedback. +* **Use Industry Standards as Keywords:** Mention standards like **OWASP Top 10**, **OWASP ASVS**, **CWE/SANS Top 25**, or language-specific guidelines (Java's SEI CERT secure coding standards, for instance). For example: *"Code suggestions should adhere to OWASP Top 10 principles (e.g., avoid injection, enforce access control) and follow the OWASP ASVS requirements where applicable"*. [[39]](#39) +* **Reference SAFECode and Other Guides:** SAFECode (Software Assurance Forum for Excellence in Code) publishes **Fundamental Practices for Secure Software Development**, and including a note about it can be beneficial. *"Our project follows SAFECode's secure development practices – the AI should prioritize those (e.g., proper validation, authentication, cryptography usage per SAFECode guidance)"*. This ties the assistant's suggestions to a comprehensive set of secure development principles. Similarly, you can reference the **Secure Software Development Lifecycle (SSDLC)** or standards like **ISO 27034** if your organization uses them, to align AI outputs with those processes. [[40]](#40) +* **Compliance and Regulatory Security:** If you operate in an industry with specific regulations (like healthcare or finance), instruct the AI with those in mind. For instance: *"When generating code, consider compliance requirements (e.g., HIPAA privacy rules for medical data, PCI-DSS for credit card info) – do not output code that logs or transmits sensitive data in insecure ways"*. This ensures the AI's suggestions are compliant with external standards, for example, it might suggest encryption for personal data fields by default. [[41]](#41) +* **Continuous Improvement Hooks:** Encourage practices that integrate with ongoing security processes. For example: *"Include comments or TODOs in code suggesting security reviews for complex logic, and note if any third-party component might need a future update or audit".* This way, the AI not only writes code but also flags areas for human scrutiny or future action. If the AI outputs a piece of cryptography code, for instance, it might add a comment "// TODO: security review for cryptographic implementation" because you instructed it that critical code should not go un-reviewed. [[42]](#42) +* **Integrate and Act on Automated Security Tooling:** Instruct the AI to proactively run automated security tools (e.g., SAST, DAST, SCA) during code generation and refinement. Example: "When writing or reviewing code, run or simulate the use of tools like CodeQL, Bandit, Semgrep, or OWASP Dependency-Check. Identify any flagged vulnerabilities or outdated dependencies and revise the code accordingly. Repeat this process until the code passes all simulated scans". This positions the AI not just as a code generator but as a security-aware assistant capable of iterating based on tool feedback. [[43]](#43) [[44]](#44) --- + +## Appendix: Citations and References + +[1] "Rules files are an emerging pattern to allow you to provide standard guidance to AI coding assistants. You can use these rules to establish project, company, or developer specific context, preferences, or workflows" (Wiz Research, [Rules Files for Safer Vibe Coding](https://www.wiz.io/blog/safer-vibe-coding-rules-files)) + +[2] "AI coding assistants are no substitute for experienced developers. An unrestrained use of the tools can have severe security implications." (ANSSI, BSI, [AI_Coding_Assistants](https://www.bsi.bund.de/SharedDocs/Downloads/EN/BSI/KI/ANSSI_BSI_AI_Coding_Assistants.pdf?__blob=publicationFile&v=7)) + +[3] "One of the major concerns regarding LLM-generated code is its security. As code snippets generated by LLMs are increasingly incorporated into industrial-level software and systems, it is critical to ensure that LLM-generated code is free of vulnerabilities that could be exploited by attackers" ([A Comprehensive Study of LLM Secure Code Generation](https://arxiv.org/abs/2503.15554)) + +[4] "About 40% of the programs had security vulnerabilities... One cause of these security flaws is the use of outdated programs in the training data of the AI models, leading to the suggestion of outdated and insecure best practices. This is evident, for example, in password encryption, where insecure methods such as MD5 or a single iteration of SHA-256 are still often used" (ANSSI, BSI, [AI Coding Assistants](https://www.bsi.bund.de/SharedDocs/Downloads/EN/BSI/KI/ANSSI_BSI_AI_Coding_Assistants.pdf?__blob=publicationFile&v=7)) + +[5] "Improve the prompt: The user should improve the prompt by indicating each and every aspect of the security parameters to derive the secure web application code from the LLMs" ([The Hidden Risks of LLM-Generated Web Application Code](https://arxiv.org/abs/2504.20612)) + +[6] "Proactive prompts, such as explicitly asking GPT-4 to secure the code ... led to A measurable drop in the number of severe vulnerabilities and improved consistency in applying security practices" ([Guiding AI to Fix Its Own Flaws: An Empirical Study on LLM-Driven Secure Code Generation](https://arxiv.org/abs/2506.23034)]) + +[7] "Injection attacks remain one of the most critical vulnerabilities in web applications ([OWASP Top 10](https://owasp.org/Top10/A03_2021-Injection/))... LLM- +generated code must handle user input securely to prevent +exploitation"([The Hidden Risks of LLM-Generated Web Application Code](https://arxiv.org/abs/2504.20612)) + +[8]"A systematic review shows that code injection (CWE-94) and OS command injection (CWE-78) are common in AI-generated code." (Wiz Research, [Rules Files for Safer Vibe Coding](https://www.wiz.io/blog/safer-vibe-coding-rules-files)) + +[9]"Overall, Copilot performed poorly in this CWE (CWE-522: Insufficiently Protected Credentials)... it frequently tried to generate code which +was based on the insecure 'MD5' hashing algorithm" ([Asleep at the Keyboard? Assessing the +Security of GitHub Copilot's Code Contributions +](https://arxiv.org/abs/2108.09293)) + +[10] "Participants with access to an AI assistant were ... significantly more likely to write an insecure solution ... also less likely to authenticate the +final returned value" ([Do Users Write More Insecure Code with AI Assistants?](https://arxiv.org/abs/2211.03622)) + +[11] "The analysis reveals critical vulnerabilities in authentication +mechanisms, session management..." ([The Hidden Risks of LLM-Generated Web Application Code](https://arxiv.org/abs/2504.20612)) + +[12] "Generic Error Messages: Ensuring error messages do not disclose username existence or password policies prevents attackers from gaining insights during brute-force attempts... Leaking system information through verbose error messages can provide attackers valuable insights into potential vulnerabilities within authentication systems" ([A Comprehensive Study of LLM Secure Code Generation](https://arxiv.org/abs/2503.15554)) + +[13] "It's clear that currently AI-Generated code is not secure by default... (Wiz Research, [Rules Files for Safer Vibe Coding](https://www.wiz.io/blog/safer-vibe-coding-rules-files)) + +[14] "All models require substantial improvements in authentication security, session management, error handling and HTTP security headers to align with current industry best practices and established frameworks, such as the NIST cybersecurity guidelines " ([BAXBENCH: Can LLMs Generate Correct and Secure Backends](https://arxiv.org/abs/2502.11844)) + +[15] "The generation of test cases is another area where coding assistants can offer support. They can automatically propose test cases and unit tests that also cover edge cases" (ANSSI, BSI, [AI_Coding_Assistants](https://www.bsi.bund.de/SharedDocs/Downloads/EN/BSI/KI/ANSSI_BSI_AI_Coding_Assistants.pdf?__blob=publicationFile&v=7)) + +[16] "It's important to pay particularly close attention to auto-generated comments... make sure you understand and agree with them... It's a good practice to consider implementing a code-tagging system to differentiate between AI- and human-created code." ([5 Ways to Reduce GitHub Copilot Security and Legal Risks](https://fossa.com/blog/5-ways-to-reduce-github-copilot-security-and-legal-risks/)) + +[17] "It might be beneficial to flag AI generated code blocks and to document the used AI tools. This might help security testing and it can also be a useful information for third-party auditors, e.g. in the context of a security certification process" (ANSSI, BSI, [AI_Coding_Assistants](https://www.bsi.bund.de/SharedDocs/Downloads/EN/BSI/KI/ANSSI_BSI_AI_Coding_Assistants.pdf?__blob=publicationFile&v=7)) + +[18] "Encryption safeguards sensitive data at rest and in transit. Weak encryption methods or lack of +encryption can expose passwords, personal information, and financial data... If passwords are stored in plain text or hashed without salting, attackers with database access can easily decrypt credentials, leading to mass account breaches." ([The Hidden Risks of LLM-Generated Web Application Code](https://arxiv.org/abs/2504.20612)) + +[19] "Package hallucination occurs when an LLM generates code that recommends or contains a reference to a package that does not actually exist. An adversary can exploit package hallucinations... by publishing a package to an open-source repository with the same name as the hallucinated or fictitious package and containing some malicious code/functionality" (ANSSI, BSI, [AI_Coding_Assistants](https://www.bsi.bund.de/SharedDocs/Downloads/EN/BSI/KI/ANSSI_BSI_AI_Coding_Assistants.pdf?__blob=publicationFile&v=7)) + +[20] "Avoid... copy reused code from other packages... instead of using a package manager to automate identifying and updating reused packages... Use a package manager to manage it, one that records the specific version numbers" ([Secure Software Development Fundamentals](https://github.com/ossf/secure-sw-dev-fundamentals/blob/main/docs/lfd121.md#preparing-for-the-inevitable-vulnerabilities-in-your-dependencies)) + +[21] "LLM sometimes produces code that uses older ... libraries, which are not compliant with modern security. This increases the security risk... they are not updated with the latest security vulnerabilities and threats, leaving the generated code open to new attacks." ([Exploring Hallucinations and Security Risks in AI-Assisted Software Development with Insights for LLM Deployment](https://arxiv.org/abs/2502.18468v1)) + +[22] "LLMs cannot update themselves with new information after release and have no knowledge of the world beyond their training data cutoff date." ([We Have a Package for You! A Comprehensive Analysis of Package Hallucinations](https://arxiv.org/abs/2406.10279)) + +[23] ""The creation of a Software Bill of Materials (SBOM) allows you to retrospectively understand whether vulnerable libraries were used and enables a targeted response if a vulnerability of certain components becomes known." (ANSSI, BSI, [AI_Coding_Assistants](https://www.bsi.bund.de/SharedDocs/Downloads/EN/BSI/KI/ANSSI_BSI_AI_Coding_Assistants.pdf?__blob=publicationFile&v=7)) + +[24] "Public OSS repositories such as PyPI and npm have implemented various measures, including two-factor authentication, namespace protection, and software signing to mitigate the distribution of malicious packages... Self-refinement strategies... utilize the model itself to detect and refine potential hallucinations" ([We Have a Package for You! A Comprehensive Analysis of Package Hallucinations](https://arxiv.org/abs/2406.10279)) + +[25] "LLM often lacks the deep contextual understanding needed to create secure code, resulting in sometimes irrelevant advice for specific security contexts" ([Exploring Hallucinations and Security Risks in AI-Assisted Software Development with Insights for LLM Deployment](https://arxiv.org/abs/2502.18468v1)) + +[26] "line 15 in the code directly executes the user-provided shell script without checking for malicious commands... It uses the system call os.system() to execute the copy command, which is passed as a string. If the original_location or new_location variables contain malicious input, this could lead to command injection" ([A Comprehensive Study of LLM Secure Code Generation](https://arxiv.org/abs/2503.15554)) + +[27] "Although some models implement security measures to a limited extent, none fully align with industry best practices" ([The Hidden Risks of LLM-Generated Web Application Code](https://arxiv.org/abs/2504.20612)) + +[28] "This code exposes a SQL injection vulnerability by directly attaching untested data to SQL queries, specifically the username variable. This allows hackers to manipulate the input and execute SQL commands" ([A Comprehensive Study of LLM Secure Code Generation](https://arxiv.org/abs/2503.15554)) + +[29] "... detailed information about the infrastructure and resource configuration can be sensitive. The last part is particularly relevant if approaches like Infrastructure as Code (IaC) are used" (ANSSI, BSI, [AI_Coding_Assistants](https://www.bsi.bund.de/SharedDocs/Downloads/EN/BSI/KI/ANSSI_BSI_AI_Coding_Assistants.pdf?__blob=publicationFile&v=7)) + +[30] "The open nature of these repositories... makes them an attractive platform for malware distribution. For instance, a total of 245,000 malicious code packages were discovered in open-source software repositories in 2023 alone ([We Have a Package for You! A Comprehensive Analysis of Package Hallucinations](https://arxiv.org/abs/2406.10279)) + +[31] "In total, we produce 89 different scenarios for Copilot to complete, producing 1,689 programs. Of these, we found approximately 40% to be vulnerable" ([Asleep at the Keyboard? Assessing the Security of GitHub Copilot's Code Contributions](https://arxiv.org/abs/2108.09293)) + +[32] "The use of uncontrolled access to coding assistants ... in the cloud should be prohibited for business software development." (ANSSI, BSI, [AI_Coding_Assistants](https://www.bsi.bund.de/SharedDocs/Downloads/EN/BSI/KI/ANSSI_BSI_AI_Coding_Assistants.pdf?__blob=publicationFile&v=7)) + +[33] "Many projects apply a ... DevSecOps approach, which intentionally blend these processes together with development... having secure distribution, fielding, operations, and disposal is critical for software to be secure in the real world." ([Secure Software Development Fundamentals](https://github.com/ossf/secure-sw-dev-fundamentals/blob/main/docs/lfd121.md#preparing-for-the-inevitable-vulnerabilities-in-your-dependencies)) + +[34] "The security vulnerabilities in LLM-generated correct code generally stem from missing conditional checks, incorrect memory allocations, or incorrect conditional checks" ([SecRepoBench: Benchmarking LLMs for Secure Code Generation in Real-World Repositories](https://arxiv.org/abs/2504.21205)) + +[35] "we observe that participants in the experiment group were significantly more likely to introduce integer overflow mistakes in their solutions" ([Do Users Write More Insecure Code with AI Assistants?](https://arxiv.org/abs/2211.03622)) + +[36] "Even when a language is memory safe by default, it is still vital to follow that language ecosystem's best practices and to use external tools to ensure safety not only within your code" ([The Memory Safety Continuum](https://memorysafety.openssf.org/memory-safety-continuum/#writing-software-in-memory-safe-by-default-languages)) + +[37] "we reveal that although LLMs are prone to generating insecure code, advanced models can benefit from vulnerability hints and fine-grained feedback to avoid or fix vulnerabilities." ([Guiding AI to Fix Its Own Flaws: An Empirical Study on LLM-Driven Secure Code Generation](https://arxiv.org/abs/2506.23034)) + +[38] "Often times, responses from the AI assistant use libraries that explicitly flag that they are insecure in the documentation for the library" ([Do Users Write More Insecure Code with AI Assistants?](https://arxiv.org/abs/2211.03622)) + +[39] "Additionally, we can reduce security risks by using OWASP guidelines and using trusted libraries" ([Exploring Hallucinations and Security Risks in AI-Assisted Software Development with Insights for LLM Deployment](https://arxiv.org/abs/2502.18468v1)) + +[40] "We found that those who specified task instructions... generated more secure code." ([Do Users Write More Insecure Code with AI Assistants?](https://arxiv.org/abs/2211.03622)) + +[41] "Compliance with regulations like GDPR and HIPAA is essential to minimize legal risks and uphold organizational reputation in sensitive environments" ([Exploring Hallucinations and Security Risks in AI-Assisted Software Development with Insights for LLM Deployment](https://arxiv.org/abs/2502.18468v1)) + +[42] "Generated content, in particular source code, should generally be reviewed and understood by the developers... It might be beneficial to 'deconstruct' AI-generated code and the used prompts in public code reviews within the company. This allows knowledge and experiences to be shared across teams." (ANSSI, BSI, [AI_Coding_Assistants](https://www.bsi.bund.de/SharedDocs/Downloads/EN/BSI/KI/ANSSI_BSI_AI_Coding_Assistants.pdf?__blob=publicationFile&v=7)) + +[43] "Automated vulnerability scanners or approaches like chatbots that critically question the generated source code ('source code critics') can reduce the risk" ([ANSSI, BSI, AI_Coding_Assistants](https://www.bsi.bund.de/SharedDocs/Downloads/EN/BSI/KI/ANSSI_BSI_AI_Coding_Assistants.pdf?__blob=publicationFile&v=7)) + +[44] "... post-processing the output ... has a measurable impact on code quality, and is LLM-agnostic... Presumably, non-LLM static analyzers or linters may be integrated as part of the code generation procedure to provide checks along the way and avoid producing code that is visibly incorrect or dangerous" ([A systematic literature review on the impact of AI models on the security of code generation](https://www.frontiersin.org/journals/big-data/articles/10.3389/fdata.2024.1386720/full)) diff --git a/docs/developers.md b/docs/developers.md index 8456c3a0..ac537c23 100644 --- a/docs/developers.md +++ b/docs/developers.md @@ -12,7 +12,8 @@ If you develop or build software, here are some ready-to-go resources from the O * [Concise Guide for Developing More Secure Software](https://best.openssf.org/Concise-Guide-for-Developing-More-Secure-Software) - a short "start here" page for how to develop secure software. * [Guide to Implementing a Coordinated Vulnerability Disclosure Process for Open Source Projects](https://github.com/ossf/oss-vulnerability-guide/blob/main/maintainer-guide.md#readme) - we recommend OSS projects use this _before_ you get a vulnerability report! -* [Source Code Management Best Practices Guide](https://best.openssf.org/SCM-BestPractices/) - Guide for securing and implementing best practices for SCM platforms, including GitHub and GitLab. +* [Source Code Management Best Practices Guide](https://best.openssf.org/SCM-BestPractices/) - Guide for securing and implementing best practices for SCM platforms, including GitHub and GitL +* [Secure AI Assistant Instructions](https://best.openssf.org/Security-Focused-Guide-for-AI-Code-Assistant-Instructions) - a guide for how to write instructions for AI code assistants so that they generate more secure code. You can also see the full list of [Guides released by the OpenSSF](https://openssf.org/resources/guides/). diff --git a/docs/index.md b/docs/index.md index 75c9bde7..9c778163 100644 --- a/docs/index.md +++ b/docs/index.md @@ -15,6 +15,7 @@ This is a list of materials (documents, services, and so on) released by the * [Simplifying Software Component Updates](https://best.openssf.org/Simplifying-Software-Component-Updates) * [The Memory Safety Continuum](https://memorysafety.openssf.org/memory-safety-continuum) * [Cyber Resilience Act (CRA) Brief Guide for Open Source Software (OSS) Developers](https://best.openssf.org/CRA-Brief-Guide-for-OSS-Developers) +* [Security Focused Guide for AI Code Assistant Instructions](https://best.openssf.org/Security-Focused-Guide-for-AI-Code-Assistant-Instructions) Note: You can also see the larger list of [Guides released by the OpenSSF](https://openssf.org/resources/guides/). From 7e6ed6280249c313ae3b3a9a23eb4823dfaea5f3 Mon Sep 17 00:00:00 2001 From: balteravishay Date: Wed, 30 Jul 2025 08:48:46 +0100 Subject: [PATCH 02/31] updated Signed-off-by: balteravishay --- ...uide-for-AI-Code-Assistant-Instructions.md | 99 +++++++++---------- 1 file changed, 44 insertions(+), 55 deletions(-) diff --git a/docs/Security-Focused-Guide-for-AI-Code-Assistant-Instructions.md b/docs/Security-Focused-Guide-for-AI-Code-Assistant-Instructions.md index ca47da3b..67c9a410 100644 --- a/docs/Security-Focused-Guide-for-AI-Code-Assistant-Instructions.md +++ b/docs/Security-Focused-Guide-for-AI-Code-Assistant-Instructions.md @@ -15,15 +15,6 @@ Short on time? Here's what really matters: By keeping these points in mind, you can harness AI code assistants effectively without sacrificing quality or security. -Additionally, you can use the following ready-to-use instructions files as a starting point: - -- [Python](TODO) -- [Golang](TODO) -- [Java](TODO) -- [JavaScript](TODO) -- [C/C++](TODO) -- [Rust](TODO) - --- ## **Secure Coding Principles in AI Instructions** @@ -88,98 +79,96 @@ To strengthen your AI assistant's guidance, you should point it toward establish ## Appendix: Citations and References -[1] "Rules files are an emerging pattern to allow you to provide standard guidance to AI coding assistants. You can use these rules to establish project, company, or developer specific context, preferences, or workflows" (Wiz Research, [Rules Files for Safer Vibe Coding](https://www.wiz.io/blog/safer-vibe-coding-rules-files)) +[1] "Rules files are an emerging pattern to allow you to provide standard guidance to AI coding assistants. You can use these rules to establish project, company, or developer specific context, preferences, or workflows" (Wiz Research - [Rules Files for Safer Vibe Coding](https://www.wiz.io/blog/safer-vibe-coding-rules-files)) -[2] "AI coding assistants are no substitute for experienced developers. An unrestrained use of the tools can have severe security implications." (ANSSI, BSI, [AI_Coding_Assistants](https://www.bsi.bund.de/SharedDocs/Downloads/EN/BSI/KI/ANSSI_BSI_AI_Coding_Assistants.pdf?__blob=publicationFile&v=7)) +[2] "AI coding assistants are no substitute for experienced developers. An unrestrained use of the tools can have severe security implications." (ANSSI, BSI - [AI Coding Assistants](https://www.bsi.bund.de/SharedDocs/Downloads/EN/BSI/KI/ANSSI_BSI_AI_Coding_Assistants.pdf?__blob=publicationFile&v=7)) -[3] "One of the major concerns regarding LLM-generated code is its security. As code snippets generated by LLMs are increasingly incorporated into industrial-level software and systems, it is critical to ensure that LLM-generated code is free of vulnerabilities that could be exploited by attackers" ([A Comprehensive Study of LLM Secure Code Generation](https://arxiv.org/abs/2503.15554)) +[3] "One of the major concerns regarding LLM-generated code is its security. As code snippets generated by LLMs are increasingly incorporated into industrial-level software and systems, it is critical to ensure that LLM-generated code is free of vulnerabilities that could be exploited by attackers" (Shih-Chieh Dai, Jun Xu, Guanhong Tao - [A Comprehensive Study of LLM Secure Code Generation](https://arxiv.org/abs/2503.15554)) -[4] "About 40% of the programs had security vulnerabilities... One cause of these security flaws is the use of outdated programs in the training data of the AI models, leading to the suggestion of outdated and insecure best practices. This is evident, for example, in password encryption, where insecure methods such as MD5 or a single iteration of SHA-256 are still often used" (ANSSI, BSI, [AI Coding Assistants](https://www.bsi.bund.de/SharedDocs/Downloads/EN/BSI/KI/ANSSI_BSI_AI_Coding_Assistants.pdf?__blob=publicationFile&v=7)) +[4] "About 40% of the programs had security vulnerabilities... One cause of these security flaws is the use of outdated programs in the training data of the AI models, leading to the suggestion of outdated and insecure best practices. This is evident, for example, in password encryption, where insecure methods such as MD5 or a single iteration of SHA-256 are still often used" (ANSSI, BSI - [AI Coding Assistants](https://www.bsi.bund.de/SharedDocs/Downloads/EN/BSI/KI/ANSSI_BSI_AI_Coding_Assistants.pdf?__blob=publicationFile&v=7)) -[5] "Improve the prompt: The user should improve the prompt by indicating each and every aspect of the security parameters to derive the secure web application code from the LLMs" ([The Hidden Risks of LLM-Generated Web Application Code](https://arxiv.org/abs/2504.20612)) +[5] "Improve the prompt: The user should improve the prompt by indicating each and every aspect of the security parameters to derive the secure web application code from the LLMs" (Swaroop Dora, Deven Lunkad, Naziya Aslam, S. Venkatesan, Sandeep Kumar Shukla - [The Hidden Risks of LLM-Generated Web Application Code: A Security-Centric Evaluation of Code Generation Capabilities in Large Language Models](https://arxiv.org/abs/2504.20612)) -[6] "Proactive prompts, such as explicitly asking GPT-4 to secure the code ... led to A measurable drop in the number of severe vulnerabilities and improved consistency in applying security practices" ([Guiding AI to Fix Its Own Flaws: An Empirical Study on LLM-Driven Secure Code Generation](https://arxiv.org/abs/2506.23034)]) +[6] "These reductions suggest that newer models can effectively leverage security hints to improve code safety" (Hao Yan, Swapneel Suhas Vaidya, Xiaokuan Zhang, Ziyu Yao - [Guiding AI to Fix Its Own Flaws: An Empirical Study on LLM-Driven Secure Code Generation](https://arxiv.org/abs/2506.23034)]) [7] "Injection attacks remain one of the most critical vulnerabilities in web applications ([OWASP Top 10](https://owasp.org/Top10/A03_2021-Injection/))... LLM- -generated code must handle user input securely to prevent -exploitation"([The Hidden Risks of LLM-Generated Web Application Code](https://arxiv.org/abs/2504.20612)) +generated code must handle user input securely to prevent exploitation"(Swaroop Dora, Deven Lunkad, Naziya Aslam, S. Venkatesan, Sandeep Kumar Shukla - [The Hidden Risks of LLM-Generated Web Application Code](https://arxiv.org/abs/2504.20612)) -[8]"A systematic review shows that code injection (CWE-94) and OS command injection (CWE-78) are common in AI-generated code." (Wiz Research, [Rules Files for Safer Vibe Coding](https://www.wiz.io/blog/safer-vibe-coding-rules-files)) +[8]"... systematic review shows that code injection (CWE-94) and OS command injection (CWE-78) are common in AI-generated code." (Wiz Research - [Rules Files for Safer Vibe Coding](https://www.wiz.io/blog/safer-vibe-coding-rules-files)) -[9]"Overall, Copilot performed poorly in this CWE (CWE-522: Insufficiently Protected Credentials)... it frequently tried to generate code which -was based on the insecure 'MD5' hashing algorithm" ([Asleep at the Keyboard? Assessing the +[9]"Overall, Copilot performed poorly in this CWE (CWE-522: Insufficiently Protected Credentials)... it frequently tried to generate code which was based on the insecure 'MD5' hashing algorithm" (Hammond Pearce, Baleegh Ahmad, Benjamin Tan, Brendan Dolan-Gavitt, Ramesh Karri - [Asleep at the Keyboard? Assessing the Security of GitHub Copilot's Code Contributions ](https://arxiv.org/abs/2108.09293)) [10] "Participants with access to an AI assistant were ... significantly more likely to write an insecure solution ... also less likely to authenticate the -final returned value" ([Do Users Write More Insecure Code with AI Assistants?](https://arxiv.org/abs/2211.03622)) +final returned value" (Neil Perry, Megha Srivastava, Deepak Kumar, Dan Boneh - [Do Users Write More Insecure Code with AI Assistants?](https://arxiv.org/abs/2211.03622)) [11] "The analysis reveals critical vulnerabilities in authentication -mechanisms, session management..." ([The Hidden Risks of LLM-Generated Web Application Code](https://arxiv.org/abs/2504.20612)) +mechanisms, session management..." (Swaroop Dora, Deven Lunkad, Naziya Aslam, S. Venkatesan, Sandeep Kumar Shukla - [The Hidden Risks of LLM-Generated Web Application Code](https://arxiv.org/abs/2504.20612)) -[12] "Generic Error Messages: Ensuring error messages do not disclose username existence or password policies prevents attackers from gaining insights during brute-force attempts... Leaking system information through verbose error messages can provide attackers valuable insights into potential vulnerabilities within authentication systems" ([A Comprehensive Study of LLM Secure Code Generation](https://arxiv.org/abs/2503.15554)) +[12] "Generic Error Messages: Ensuring error messages do not disclose username existence or password policies prevents attackers from gaining insights during brute-force attempts... Leaking system information through verbose error messages can provide attackers valuable insights into potential vulnerabilities within authentication systems" (Swaroop Dora, Deven Lunkad, Naziya Aslam, S. Venkatesan, Sandeep Kumar Shukla - [The Hidden Risks of LLM-Generated Web Application Code](https://arxiv.org/abs/2504.20612)) -[13] "It's clear that currently AI-Generated code is not secure by default... (Wiz Research, [Rules Files for Safer Vibe Coding](https://www.wiz.io/blog/safer-vibe-coding-rules-files)) +[13] "It's clear that currently AI-Generated code is not secure by default... (Wiz Research - [Rules Files for Safer Vibe Coding](https://www.wiz.io/blog/safer-vibe-coding-rules-files)) -[14] "All models require substantial improvements in authentication security, session management, error handling and HTTP security headers to align with current industry best practices and established frameworks, such as the NIST cybersecurity guidelines " ([BAXBENCH: Can LLMs Generate Correct and Secure Backends](https://arxiv.org/abs/2502.11844)) +[14] "All models require substantial improvements in authentication security, session management, error handling and HTTP security headers to align with current industry best practices and established frameworks, such as the NIST cybersecurity guidelines " (Swaroop Dora, Deven Lunkad, Naziya Aslam, S. Venkatesan, Sandeep Kumar Shukla - [The Hidden Risks of LLM-Generated Web Application Code](https://arxiv.org/abs/2504.20612)) -[15] "The generation of test cases is another area where coding assistants can offer support. They can automatically propose test cases and unit tests that also cover edge cases" (ANSSI, BSI, [AI_Coding_Assistants](https://www.bsi.bund.de/SharedDocs/Downloads/EN/BSI/KI/ANSSI_BSI_AI_Coding_Assistants.pdf?__blob=publicationFile&v=7)) +[15] "The generation of test cases is another area where coding assistants can offer support. They can automatically propose test cases and unit tests that also cover edge cases" (ANSSI, BSI - [AI Coding Assistants](https://www.bsi.bund.de/SharedDocs/Downloads/EN/BSI/KI/ANSSI_BSI_AI_Coding_Assistants.pdf?__blob=publicationFile&v=7)) -[16] "It's important to pay particularly close attention to auto-generated comments... make sure you understand and agree with them... It's a good practice to consider implementing a code-tagging system to differentiate between AI- and human-created code." ([5 Ways to Reduce GitHub Copilot Security and Legal Risks](https://fossa.com/blog/5-ways-to-reduce-github-copilot-security-and-legal-risks/)) +[16] "It's important to pay particularly close attention to auto-generated comments... make sure you understand and agree with them... It's a good practice to consider implementing a code-tagging system to differentiate between AI- and human-created code." (FOSSA - [5 Ways to Reduce GitHub Copilot Security and Legal Risks](https://fossa.com/blog/5-ways-to-reduce-github-copilot-security-and-legal-risks/)) -[17] "It might be beneficial to flag AI generated code blocks and to document the used AI tools. This might help security testing and it can also be a useful information for third-party auditors, e.g. in the context of a security certification process" (ANSSI, BSI, [AI_Coding_Assistants](https://www.bsi.bund.de/SharedDocs/Downloads/EN/BSI/KI/ANSSI_BSI_AI_Coding_Assistants.pdf?__blob=publicationFile&v=7)) +[17] "It might be beneficial to flag AI generated code blocks and to document the used AI tools. This might help security testing and it can also be a useful information for third-party auditors, e.g. in the context of a security certification process" (ANSSI, BSI - [AI Coding Assistants](https://www.bsi.bund.de/SharedDocs/Downloads/EN/BSI/KI/ANSSI_BSI_AI_Coding_Assistants.pdf?__blob=publicationFile&v=7)) [18] "Encryption safeguards sensitive data at rest and in transit. Weak encryption methods or lack of -encryption can expose passwords, personal information, and financial data... If passwords are stored in plain text or hashed without salting, attackers with database access can easily decrypt credentials, leading to mass account breaches." ([The Hidden Risks of LLM-Generated Web Application Code](https://arxiv.org/abs/2504.20612)) +encryption can expose passwords, personal information, and financial data... If passwords are stored in plain text or hashed without salting, attackers with database access can easily decrypt credentials, leading to mass account breaches." (Swaroop Dora, Deven Lunkad, Naziya Aslam, S. Venkatesan, Sandeep Kumar Shukla - [The Hidden Risks of LLM-Generated Web Application Code](https://arxiv.org/abs/2504.20612)) -[19] "Package hallucination occurs when an LLM generates code that recommends or contains a reference to a package that does not actually exist. An adversary can exploit package hallucinations... by publishing a package to an open-source repository with the same name as the hallucinated or fictitious package and containing some malicious code/functionality" (ANSSI, BSI, [AI_Coding_Assistants](https://www.bsi.bund.de/SharedDocs/Downloads/EN/BSI/KI/ANSSI_BSI_AI_Coding_Assistants.pdf?__blob=publicationFile&v=7)) +[19] "Package hallucination occurs when an LLM generates code that recommends or contains a reference to a package that does not actually exist. An adversary can exploit package hallucinations... by publishing a package to an open-source repository with the same name as the hallucinated or fictitious package and containing some malicious code/functionality" (Joseph Spracklen, Raveen Wijewickrama, A H M Nazmus Sakib, Anindya Maiti, Bimal Viswanath, Murtuza Jadliwala - [We Have a Package for You! A Comprehensive Analysis of Package Hallucinations by Code Generating LLMs](https://arxiv.org/abs/2406.10279)) -[20] "Avoid... copy reused code from other packages... instead of using a package manager to automate identifying and updating reused packages... Use a package manager to manage it, one that records the specific version numbers" ([Secure Software Development Fundamentals](https://github.com/ossf/secure-sw-dev-fundamentals/blob/main/docs/lfd121.md#preparing-for-the-inevitable-vulnerabilities-in-your-dependencies)) +[20] "Avoid... copy reused code from other packages... instead of using a package manager to automate identifying and updating reused packages... Use a package manager to manage it, one that records the specific version numbers" (OpenSSF - [Secure Software Development Fundamentals](https://github.com/ossf/secure-sw-dev-fundamentals/blob/main/docs/lfd121.md#preparing-for-the-inevitable-vulnerabilities-in-your-dependencies)) -[21] "LLM sometimes produces code that uses older ... libraries, which are not compliant with modern security. This increases the security risk... they are not updated with the latest security vulnerabilities and threats, leaving the generated code open to new attacks." ([Exploring Hallucinations and Security Risks in AI-Assisted Software Development with Insights for LLM Deployment](https://arxiv.org/abs/2502.18468v1)) +[21] "LLM sometimes produces code that uses older ... libraries, which are not compliant with modern security. This increases the security risk... they are not updated with the latest security vulnerabilities and threats, leaving the generated code open to new attacks." (Ariful Haque, Sunzida Siddique, Md. Mahfuzur Rahman, Ahmed Rafi Hasan, Laxmi Rani Das, Marufa Kamal, Tasnim Masura, Kishor Datta Gupta - [Exploring Hallucinations and Security Risks in AI-Assisted Software Development with Insights for LLM Deployment](https://arxiv.org/abs/2502.18468v1)) -[22] "LLMs cannot update themselves with new information after release and have no knowledge of the world beyond their training data cutoff date." ([We Have a Package for You! A Comprehensive Analysis of Package Hallucinations](https://arxiv.org/abs/2406.10279)) +[22] "LLMs cannot update themselves with new information after release and have no knowledge of the world beyond their training data cutoff date." (Joseph Spracklen, Raveen Wijewickrama, A H M Nazmus Sakib, Anindya Maiti, Bimal Viswanath, Murtuza Jadliwala - [We Have a Package for You! A Comprehensive Analysis of Package Hallucinations](https://arxiv.org/abs/2406.10279)) -[23] ""The creation of a Software Bill of Materials (SBOM) allows you to retrospectively understand whether vulnerable libraries were used and enables a targeted response if a vulnerability of certain components becomes known." (ANSSI, BSI, [AI_Coding_Assistants](https://www.bsi.bund.de/SharedDocs/Downloads/EN/BSI/KI/ANSSI_BSI_AI_Coding_Assistants.pdf?__blob=publicationFile&v=7)) +[23] "The creation of a Software Bill of Materials (SBOM) allows you to retrospectively understand whether vulnerable libraries were used and enables a targeted response if a vulnerability of certain components becomes known." (ANSSI, BSI - [AI Coding Assistants](https://www.bsi.bund.de/SharedDocs/Downloads/EN/BSI/KI/ANSSI_BSI_AI_Coding_Assistants.pdf?__blob=publicationFile&v=7)) -[24] "Public OSS repositories such as PyPI and npm have implemented various measures, including two-factor authentication, namespace protection, and software signing to mitigate the distribution of malicious packages... Self-refinement strategies... utilize the model itself to detect and refine potential hallucinations" ([We Have a Package for You! A Comprehensive Analysis of Package Hallucinations](https://arxiv.org/abs/2406.10279)) +[24] "Public OSS repositories ... have implemented various measures, including ... software signing to mitigate the distribution of malicious packages... Self-refinement strategies... utilize the model itself to detect and refine potential hallucinations" (Joseph Spracklen, Raveen Wijewickrama, A H M Nazmus Sakib, Anindya Maiti, Bimal Viswanath, Murtuza Jadliwala - [We Have a Package for You! A Comprehensive Analysis of Package Hallucinations](https://arxiv.org/abs/2406.10279)) -[25] "LLM often lacks the deep contextual understanding needed to create secure code, resulting in sometimes irrelevant advice for specific security contexts" ([Exploring Hallucinations and Security Risks in AI-Assisted Software Development with Insights for LLM Deployment](https://arxiv.org/abs/2502.18468v1)) +[25] "LLM often lacks the deep contextual understanding needed to create secure code, resulting in sometimes irrelevant advice for specific security contexts" (Ariful Haque, Sunzida Siddique, Md. Mahfuzur Rahman, Ahmed Rafi Hasan, Laxmi Rani Das, Marufa Kamal, Tasnim Masura, Kishor Datta Gupta - [Exploring Hallucinations and Security Risks in AI-Assisted Software Development with Insights for LLM Deployment](https://arxiv.org/abs/2502.18468v1)) -[26] "line 15 in the code directly executes the user-provided shell script without checking for malicious commands... It uses the system call os.system() to execute the copy command, which is passed as a string. If the original_location or new_location variables contain malicious input, this could lead to command injection" ([A Comprehensive Study of LLM Secure Code Generation](https://arxiv.org/abs/2503.15554)) +[26] "... code directly executes the user-provided shell script without checking for malicious commands... It uses the system call os.system() to execute the copy command, which is passed as a string. If the original_location or new_location variables contain malicious input, this could lead to command injection" (Shih-Chieh Dai, Jun Xu, Guanhong Tao - [A Comprehensive Study of LLM Secure Code Generation](https://arxiv.org/abs/2503.15554)) -[27] "Although some models implement security measures to a limited extent, none fully align with industry best practices" ([The Hidden Risks of LLM-Generated Web Application Code](https://arxiv.org/abs/2504.20612)) +[27] "Although some models implement security measures to a limited extent, none fully align with industry best practices" (Swaroop Dora, Deven Lunkad, Naziya Aslam, S. Venkatesan, Sandeep Kumar Shukla - [The Hidden Risks of LLM-Generated Web Application Code: A Security-Centric Evaluation of Code Generation Capabilities in Large Language Models](https://arxiv.org/abs/2504.20612)) -[28] "This code exposes a SQL injection vulnerability by directly attaching untested data to SQL queries, specifically the username variable. This allows hackers to manipulate the input and execute SQL commands" ([A Comprehensive Study of LLM Secure Code Generation](https://arxiv.org/abs/2503.15554)) +[28] "36% of participants with access to the AI assistant writing solutions that are vulnerable to SQL injections compared to 7% of the control group" (Neil Perry, Megha Srivastava, Deepak Kumar, Dan Boneh - [Do Users Write More Insecure Code with AI Assistants?](https://arxiv.org/abs/2211.03622)) -[29] "... detailed information about the infrastructure and resource configuration can be sensitive. The last part is particularly relevant if approaches like Infrastructure as Code (IaC) are used" (ANSSI, BSI, [AI_Coding_Assistants](https://www.bsi.bund.de/SharedDocs/Downloads/EN/BSI/KI/ANSSI_BSI_AI_Coding_Assistants.pdf?__blob=publicationFile&v=7)) +[29] "... detailed information about the infrastructure and resource configuration can be sensitive. The last part is particularly relevant if approaches like Infrastructure as Code (IaC) are used" (ANSSI, BSI - [AI Coding Assistants](https://www.bsi.bund.de/SharedDocs/Downloads/EN/BSI/KI/ANSSI_BSI_AI_Coding_Assistants.pdf?__blob=publicationFile&v=7)) -[30] "The open nature of these repositories... makes them an attractive platform for malware distribution. For instance, a total of 245,000 malicious code packages were discovered in open-source software repositories in 2023 alone ([We Have a Package for You! A Comprehensive Analysis of Package Hallucinations](https://arxiv.org/abs/2406.10279)) +[30] "The open nature of these repositories... makes them an attractive platform for malware distribution. For instance, a total of 245,000 malicious code packages were discovered in open-source software repositories in 2023 alone (Joseph Spracklen, Raveen Wijewickrama, A H M Nazmus Sakib, Anindya Maiti, Bimal Viswanath, Murtuza Jadliwala - [We Have a Package for You! A Comprehensive Analysis of Package Hallucinations](https://arxiv.org/abs/2406.10279)) -[31] "In total, we produce 89 different scenarios for Copilot to complete, producing 1,689 programs. Of these, we found approximately 40% to be vulnerable" ([Asleep at the Keyboard? Assessing the Security of GitHub Copilot's Code Contributions](https://arxiv.org/abs/2108.09293)) +[31] "... we produce 89 different scenarios for Copilot to complete, producing 1,689 programs. Of these, we found approximately 40% to be vulnerable" (Hammond Pearce, Baleegh Ahmad, Benjamin Tan, Brendan Dolan-Gavitt, Ramesh Karri - [Asleep at the Keyboard? Assessing the Security of GitHub Copilot's Code Contributions](https://arxiv.org/abs/2108.09293)) -[32] "The use of uncontrolled access to coding assistants ... in the cloud should be prohibited for business software development." (ANSSI, BSI, [AI_Coding_Assistants](https://www.bsi.bund.de/SharedDocs/Downloads/EN/BSI/KI/ANSSI_BSI_AI_Coding_Assistants.pdf?__blob=publicationFile&v=7)) +[32] "The use of uncontrolled access to coding assistants ... in the cloud should be prohibited for business software development." (ANSSI, BSI - [AI Coding Assistants](https://www.bsi.bund.de/SharedDocs/Downloads/EN/BSI/KI/ANSSI_BSI_AI_Coding_Assistants.pdf?__blob=publicationFile&v=7)) -[33] "Many projects apply a ... DevSecOps approach, which intentionally blend these processes together with development... having secure distribution, fielding, operations, and disposal is critical for software to be secure in the real world." ([Secure Software Development Fundamentals](https://github.com/ossf/secure-sw-dev-fundamentals/blob/main/docs/lfd121.md#preparing-for-the-inevitable-vulnerabilities-in-your-dependencies)) +[33] "Many projects apply a ... DevSecOps approach, which intentionally blend these processes together with development... having secure distribution, fielding, operations, and disposal is critical for software to be secure in the real world." (OpenSSF - [Secure Software Development Fundamentals](https://github.com/ossf/secure-sw-dev-fundamentals/blob/main/docs/lfd121.md#preparing-for-the-inevitable-vulnerabilities-in-your-dependencies)) -[34] "The security vulnerabilities in LLM-generated correct code generally stem from missing conditional checks, incorrect memory allocations, or incorrect conditional checks" ([SecRepoBench: Benchmarking LLMs for Secure Code Generation in Real-World Repositories](https://arxiv.org/abs/2504.21205)) +[34] "The security vulnerabilities in LLM-generated correct code generally stem from missing conditional checks, incorrect memory allocations, or incorrect conditional checks" (Connor Dilgren, Purva Chiniya, Luke Griffith, Yu Ding, Yizheng Chen - [Benchmarking LLMs for Secure Code Generation in Real-World Repositories](https://arxiv.org/abs/2504.21205)) -[35] "we observe that participants in the experiment group were significantly more likely to introduce integer overflow mistakes in their solutions" ([Do Users Write More Insecure Code with AI Assistants?](https://arxiv.org/abs/2211.03622)) +[35] "... we observe that participants in the experiment group were significantly more likely to introduce integer overflow mistakes in their solutions" (Neil Perry, Megha Srivastava, Deepak Kumar, Dan Boneh - [Do Users Write More Insecure Code with AI Assistants?](https://arxiv.org/abs/2211.03622)) -[36] "Even when a language is memory safe by default, it is still vital to follow that language ecosystem's best practices and to use external tools to ensure safety not only within your code" ([The Memory Safety Continuum](https://memorysafety.openssf.org/memory-safety-continuum/#writing-software-in-memory-safe-by-default-languages)) +[36] "Even when a language is memory safe by default, it is still vital to follow that language ecosystem's best practices and to use external tools to ensure safety not only within your code" (OpenSSF - [The Memory Safety Continuum](https://memorysafety.openssf.org/memory-safety-continuum/#writing-software-in-memory-safe-by-default-languages)) -[37] "we reveal that although LLMs are prone to generating insecure code, advanced models can benefit from vulnerability hints and fine-grained feedback to avoid or fix vulnerabilities." ([Guiding AI to Fix Its Own Flaws: An Empirical Study on LLM-Driven Secure Code Generation](https://arxiv.org/abs/2506.23034)) +[37] "... we reveal that although LLMs are prone to generating insecure code, advanced models can benefit from vulnerability hints and fine-grained feedback to avoid or fix vulnerabilities." (Hao Yan, Swapneel Suhas Vaidya, Xiaokuan Zhang, Ziyu Yao - [Guiding AI to Fix Its Own Flaws: An Empirical Study on LLM-Driven Secure Code Generation](https://arxiv.org/abs/2506.23034)) -[38] "Often times, responses from the AI assistant use libraries that explicitly flag that they are insecure in the documentation for the library" ([Do Users Write More Insecure Code with AI Assistants?](https://arxiv.org/abs/2211.03622)) +[38] "Often times, responses from the AI assistant use libraries that explicitly flag that they are insecure in the documentation for the library" (Neil Perry, Megha Srivastava, Deepak Kumar, Dan Boneh - [Do Users Write More Insecure Code with AI Assistants?](https://arxiv.org/abs/2211.03622)) -[39] "Additionally, we can reduce security risks by using OWASP guidelines and using trusted libraries" ([Exploring Hallucinations and Security Risks in AI-Assisted Software Development with Insights for LLM Deployment](https://arxiv.org/abs/2502.18468v1)) +[39] "Additionally, we can reduce security risks by using OWASP guidelines and using trusted libraries" (Ariful Haque, Sunzida Siddique, Md. Mahfuzur Rahman, Ahmed Rafi Hasan, Laxmi Rani Das, Marufa Kamal, Tasnim Masura, Kishor Datta Gupta - [Exploring Hallucinations and Security Risks in AI-Assisted Software Development with Insights for LLM Deployment](https://arxiv.org/abs/2502.18468v1)) -[40] "We found that those who specified task instructions... generated more secure code." ([Do Users Write More Insecure Code with AI Assistants?](https://arxiv.org/abs/2211.03622)) +[40] "We found that those who specified task instructions... generated more secure code." (Neil Perry, Megha Srivastava, Deepak Kumar, Dan Boneh - [Do Users Write More Insecure Code with AI Assistants?](https://arxiv.org/abs/2211.03622)) -[41] "Compliance with regulations like GDPR and HIPAA is essential to minimize legal risks and uphold organizational reputation in sensitive environments" ([Exploring Hallucinations and Security Risks in AI-Assisted Software Development with Insights for LLM Deployment](https://arxiv.org/abs/2502.18468v1)) +[41] "Compliance with regulations like GDPR and HIPAA is essential to minimize legal risks and uphold organizational reputation in sensitive environments" (Ariful Haque, Sunzida Siddique, Md. Mahfuzur Rahman, Ahmed Rafi Hasan, Laxmi Rani Das, Marufa Kamal, Tasnim Masura, Kishor Datta Gupta - [Exploring Hallucinations and Security Risks in AI-Assisted Software Development with Insights for LLM Deployment](https://arxiv.org/abs/2502.18468v1)) -[42] "Generated content, in particular source code, should generally be reviewed and understood by the developers... It might be beneficial to 'deconstruct' AI-generated code and the used prompts in public code reviews within the company. This allows knowledge and experiences to be shared across teams." (ANSSI, BSI, [AI_Coding_Assistants](https://www.bsi.bund.de/SharedDocs/Downloads/EN/BSI/KI/ANSSI_BSI_AI_Coding_Assistants.pdf?__blob=publicationFile&v=7)) +[42] "Generated content, in particular source code, should generally be reviewed and understood by the developers... It might be beneficial to 'deconstruct' AI-generated code and the used prompts in public code reviews within the company. This allows knowledge and experiences to be shared across teams." (ANSSI, BSI - [AI Coding Assistants](https://www.bsi.bund.de/SharedDocs/Downloads/EN/BSI/KI/ANSSI_BSI_AI_Coding_Assistants.pdf?__blob=publicationFile&v=7)) -[43] "Automated vulnerability scanners or approaches like chatbots that critically question the generated source code ('source code critics') can reduce the risk" ([ANSSI, BSI, AI_Coding_Assistants](https://www.bsi.bund.de/SharedDocs/Downloads/EN/BSI/KI/ANSSI_BSI_AI_Coding_Assistants.pdf?__blob=publicationFile&v=7)) +[43] "Automated vulnerability scanners or approaches like chatbots that critically question the generated source code ('source code critics') can reduce the risk" (ANSSI, BSI - [AI Coding Assistants](https://www.bsi.bund.de/SharedDocs/Downloads/EN/BSI/KI/ANSSI_BSI_AI_Coding_Assistants.pdf?__blob=publicationFile&v=7)) -[44] "... post-processing the output ... has a measurable impact on code quality, and is LLM-agnostic... Presumably, non-LLM static analyzers or linters may be integrated as part of the code generation procedure to provide checks along the way and avoid producing code that is visibly incorrect or dangerous" ([A systematic literature review on the impact of AI models on the security of code generation](https://www.frontiersin.org/journals/big-data/articles/10.3389/fdata.2024.1386720/full)) +[44] "... post-processing the output ... has a measurable impact on code quality, and is LLM-agnostic... Presumably, non-LLM static analyzers or linters may be integrated as part of the code generation procedure to provide checks along the way and avoid producing code that is visibly incorrect or dangerous" (Frontiers - [A systematic literature review on the impact of AI models on the security of code generation](https://www.frontiersin.org/journals/big-data/articles/10.3389/fdata.2024.1386720/full)) From c8e45183b6110a51182e3f39047223de0659c390 Mon Sep 17 00:00:00 2001 From: balteravishay Date: Wed, 30 Jul 2025 11:01:19 +0100 Subject: [PATCH 03/31] citations Signed-off-by: balteravishay --- ...uide-for-AI-Code-Assistant-Instructions.md | 151 ++++++++++-------- 1 file changed, 80 insertions(+), 71 deletions(-) diff --git a/docs/Security-Focused-Guide-for-AI-Code-Assistant-Instructions.md b/docs/Security-Focused-Guide-for-AI-Code-Assistant-Instructions.md index 67c9a410..b6b7214c 100644 --- a/docs/Security-Focused-Guide-for-AI-Code-Assistant-Instructions.md +++ b/docs/Security-Focused-Guide-for-AI-Code-Assistant-Instructions.md @@ -9,9 +9,9 @@ AI code assistants can significantly speed up development, but they need guidanc Short on time? Here's what really matters: * **You Are the Pilot – AI is the Co-pilot:** The developer (you) remains in full control of the code. Critically evaluate and edit AI-generated code just as you would code written by a human colleague and never blindly accept suggestions. [[2]](#2) -* **Apply Engineering Best Practices Always:** AI-generated code isn't a shortcut around engineering processes such as code reviews, testing, static analysis, documentation, and version control discipline. -* **Be Security-Conscious:** Assume AI-written code can have bugs or vulnerabilities, because it often does. AI coding assistants can introduce security issues like using outdated cryptography or outdated dependencies, ignoring error handling, or leaking secrets. Check for any secrets or sensitive data in the suggested code. Make sure dependency suggestions are safe and not pulling in known vulnerable packages. [[3]](#3), [[4]](#4) -* **Guide the AI:** AI is a powerful assistant, but it works best with your guidance. Write clear prompts that specify security requirements. Don't hesitate to modify or reject AI outputs. Direct your AI tool to build its own instructions file based on this guide. [[5]](#5) [[6]](#6) +* **Apply Engineering Best Practices Always:** AI-generated code isn't a shortcut around engineering processes such as code reviews, testing, static analysis, documentation, and version control discipline. [[3]](#3) +* **Be Security-Conscious:** Assume AI-written code can have bugs or vulnerabilities, because it often does. AI coding assistants can introduce security issues like using outdated cryptography or outdated dependencies, ignoring error handling, or leaking secrets. Check for any secrets or sensitive data in the suggested code. Make sure dependency suggestions are safe and not pulling in known vulnerable packages. [[4]](#4), [[5]](#5) +* **Guide the AI:** AI is a powerful assistant, but it works best with your guidance. Write clear prompts that specify security requirements. Don't hesitate to modify or reject AI outputs. Direct your AI tool to build its own instructions file based on this guide. [[6]](#6) [[7]](#7) By keeping these points in mind, you can harness AI code assistants effectively without sacrificing quality or security. @@ -21,13 +21,13 @@ By keeping these points in mind, you can harness AI code assistants effectively One of the first sections in your instructions should reinforce general secure coding best practices. These principles apply to all languages and frameworks, and you want the AI to **always** keep them in mind when generating code: -* **Input Validation & Output Encoding:** Instruct the AI to treat all external inputs as untrusted and to validate them. *Example: "user inputs should be checked for expected format and length*". Any output should be properly encoded to prevent injection attacks such as SQL injection or cross-site scripting (XSS). *Example: "Always validate function arguments and use parameterized queries for database access"* and *"Escape special characters in user-generated content before rendering it in HTML"*. Similarly, specify that when generating output contexts such as HTML or SQL, the assistant should use safe frameworks or encoding functions to avoid vulnerabilities. [[7]](#7) [[8]](#8) -* **Authentication, Authorization & Secrets Management:** Emphasize that credentials and sensitive tokens must never be hard-coded or exposed. Your instructions can say: *"Never include API keys, passwords, or secrets in code output, and use environment variables or secure vault references instead"*. Also instruct the AI to use secure authentication flows (for instance, using industry-standard libraries for handling passwords or tokens) and to enforce role-based access checks where appropriate. [[9]](#9) [[10]](#10) [[11]](#11) -* **Error Handling & Logging:** Guide the AI to implement errors securely by catching exceptions and failures without revealing sensitive info (stack traces, server paths, etc.) to the end-user. In your instructions, you might include: *"When generating code, handle errors gracefully and log them, but do not expose internal details or secrets in error messages".* This ensures the assistant's suggestions include secure error-handling patterns (like generic user-facing messages and detailed logs only on the server side). Additionally, instruct the AI to use logging frameworks that can be configured for security (e.g. avoiding logging of personal data or secrets). [[12]](#12) -* **Secure Defaults & Configurations:** Include guidance such as: *"Prefer safe defaults in configurations – for example, use HTTPS by default, require strong encryption algorithms, and disable insecure protocols or options".* By specifying this, the AI will be more likely to generate code that opts-in to security features. Always instruct the AI to follow the principle of least privilege (e.g. minimal file system permissions, least-privileged user accounts for services, etc.) in any configuration or code it proposes. [[13]](#13) [[14]](#14) -* **Testing for Security:** Encourage the AI to produce or suggest tests for critical code paths including negative tests that verify that what shouldn't happen, doesn't happen. In your instructions, add: *"When applicable, generate unit tests for security-critical functions (including negative tests to ensure the code fails safely)"*. [[15]](#15) -* **Call out and review stubbed code:** If the AI generates code that is stubbed or incomplete, instruct it to flag these areas for review. For example: *"If you generate placeholder code (e.g., `TODO` comments), ensure it is marked for security review before deployment".* This will help ensure that any incomplete code does not inadvertently introduce vulnerabilities. [[16]](#16) [[17]](#17) -* **Data Protection:** When generating code, always prioritize data minimization and avoid storing or processing confidential or otherwise sensitive information (like personal data - PII) unless absolutely necessary. For that case, suggest strong encryption at rest and in transit, and recommend techniques like anonymization. For example: *"Generate a function that securely handles user input for a registration form, asking for necessary fields to avoid logging sensitive information of PII. Ensure that no sensitive or PII is stored in plaintext"*. [[18]](#18) +* **Input Validation & Output Encoding:** Instruct the AI to treat all external inputs as untrusted and to validate them. *Example: "user inputs should be checked for expected format and length*". Any output should be properly encoded to prevent injection attacks such as SQL injection or cross-site scripting (XSS). *Example: "Always validate function arguments and use parameterized queries for database access"* and *"Escape special characters in user-generated content before rendering it in HTML"*. Similarly, specify that when generating output contexts such as HTML or SQL, the assistant should use safe frameworks or encoding functions to avoid vulnerabilities. [[8]](#8) [[9]](#9) [[10]](#10) +* **Authentication, Authorization & Secrets Management:** Emphasize that credentials and sensitive tokens must never be hard-coded or exposed. Your instructions can say: *"Never include API keys, passwords, or secrets in code output, and use environment variables or secure vault references instead"*. Also instruct the AI to use secure authentication flows (for instance, using industry-standard libraries for handling passwords or tokens) and to enforce role-based access checks where appropriate. [[11]](#11) [[12]](#12) [[13]](#13) +* **Error Handling & Logging:** Guide the AI to implement errors securely by catching exceptions and failures without revealing sensitive info (stack traces, server paths, etc.) to the end-user. In your instructions, you might include: *"When generating code, handle errors gracefully and log them, but do not expose internal details or secrets in error messages".* This ensures the assistant's suggestions include secure error-handling patterns (like generic user-facing messages and detailed logs only on the server side). Additionally, instruct the AI to use logging frameworks that can be configured for security (e.g. avoiding logging of personal data or secrets). [[14]](#14) +* **Secure Defaults & Configurations:** Include guidance such as: *"Prefer safe defaults in configurations – for example, use HTTPS by default, require strong encryption algorithms, and disable insecure protocols or options".* By specifying this, the AI will be more likely to generate code that opts-in to security features. Always instruct the AI to follow the principle of least privilege (e.g. minimal file system permissions, least-privileged user accounts for services, etc.) in any configuration or code it proposes. [[13]](#15) [[14]](#16) +* **Testing for Security:** Encourage the AI to produce or suggest tests for critical code paths including negative tests that verify that what shouldn't happen, doesn't happen. In your instructions, add: *"When applicable, generate unit tests for security-critical functions (including negative tests to ensure the code fails safely)"*. [[17]](#17) [[18]](#18) +* **Call out and review stubbed code:** If the AI generates code that is stubbed or incomplete, instruct it to flag these areas for review. For example: *"If you generate placeholder code (e.g., `TODO` comments), ensure it is marked for security review before deployment".* This will help ensure that any incomplete code does not inadvertently introduce vulnerabilities. [[19]](#19) [[20]](#20) +* **Data Protection:** When generating code, always prioritize data minimization and avoid storing or processing confidential or otherwise sensitive information (like personal data - PII) unless absolutely necessary. For that case, suggest strong encryption at rest and in transit, and recommend techniques like anonymization. For example: *"Generate a function that securely handles user input for a registration form, asking for necessary fields to avoid logging sensitive information of PII. Ensure that no sensitive or PII is stored in plaintext"*. [[21]](#21) --- @@ -35,22 +35,22 @@ One of the first sections in your instructions should reinforce general secure c Modern software heavily relies on third-party libraries and dependencies. It's crucial that your AI assistant's instructions cover supply chain security, ensuring that suggested dependencies and build processes are secure: -* **Safe Dependency Selection:** Instruct the AI to prefer well-vetted, reputable libraries when suggesting code that pulls in external packages. For example: *"Use popular, community-trusted libraries for common tasks (and avoid adding obscure dependencies if a standard library or well-known package can do the same job)".* Emphasize evaluating packages before use – as a developer would manually. [[19]](#19) -* **Use Package Managers & Lock Versions:** Your instructions should tell the AI to use proper package management. For instance: *"Always use the official package manager for the given language (npm, pip, Maven, etc.) to install libraries, rather than copying code snippets".* Also, instruct it to specify version ranges or exact versions that are known to be secure. By doing so, the AI will generate code that, for example, uses a `requirements.txt` or `package.json` entry, which aids in maintaining supply chain integrity. [[20]](#20) -* **Stay Updated & Monitor Vulnerabilities:** Include guidance for keeping dependencies up-to-date. For example: *"When suggesting dependency versions, prefer the latest stable release and mention updating dependencies regularly to patch vulnerabilities"*. [[21]](#21) [[22]](#22) -* **Generate Software Bill of Materials (SBOM):** Instruct the AI to create and maintaining SBOMs for better visibility into your software supply chain. For example: *"Generate a Software Bill of Materials (SBOM) by using tools that support standard formats like SPDX or CycloneDX".* You can also mention provenance tracking: *"Where applicable, use in-toto attestations or similar frameworks to create verifiable records of your build and deployment processes".* This ensures comprehensive tracking of what goes into your software and provides the foundation for ongoing vulnerability monitoring and incident response. [[23]](#23) -* **Integrity Verification:** To further secure the supply chain, you can instruct the assistant to show how to verify what it uses. For instance: *"When adding important external resources (scripts, containers, etc.), include steps to verify integrity (like checksum verification or signature validation) if applicable".* [[24]](#24) +* **Safe Dependency Selection:** Instruct the AI to prefer well-vetted, reputable libraries when suggesting code that pulls in external packages. For example: *"Use popular, community-trusted libraries for common tasks (and avoid adding obscure dependencies if a standard library or well-known package can do the same job)".* Emphasize evaluating packages before use – as a developer would manually. [[22]](#22) +* **Use Package Managers & Lock Versions:** Your instructions should tell the AI to use proper package management. For instance: *"Always use the official package manager for the given language (npm, pip, Maven, etc.) to install libraries, rather than copying code snippets".* Also, instruct it to specify version ranges or exact versions that are known to be secure. By doing so, the AI will generate code that, for example, uses a `requirements.txt` or `package.json` entry, which aids in maintaining supply chain integrity. [[23]](#23) +* **Stay Updated & Monitor Vulnerabilities:** Include guidance for keeping dependencies up-to-date. For example: *"When suggesting dependency versions, prefer the latest stable release and mention updating dependencies regularly to patch vulnerabilities"*. [[24]](#24) [[25]](#25) +* **Generate Software Bill of Materials (SBOM):** Instruct the AI to create and maintaining SBOMs for better visibility into your software supply chain. For example: *"Generate a Software Bill of Materials (SBOM) by using tools that support standard formats like SPDX or CycloneDX".* You can also mention provenance tracking: *"Where applicable, use in-toto attestations or similar frameworks to create verifiable records of your build and deployment processes".* This ensures comprehensive tracking of what goes into your software and provides the foundation for ongoing vulnerability monitoring and incident response. [[26]](#26) +* **Integrity Verification:** To further secure the supply chain, you can instruct the assistant to show how to verify what it uses. For instance: *"When adding important external resources (scripts, containers, etc.), include steps to verify integrity (like checksum verification or signature validation) if applicable".* [[27]](#27) --- ## **Platform and Runtime Security Considerations** Every software runs somewhere – be it a web browser, a cloud platform, an IoT device, or an OS. Your guide for the AI assistant should have a section addressing platform-specific and runtime security concerns: -* **Operating System and Environment:** If your code will run on a server or user machine, instruct the AI on safe system interactions. *"When writing file or OS-level operations, use safe functions and check for errors (e.g., use secure file modes, avoid temp files without proper randomness, etc.). If running as a service, drop privileges when possible".* This ensures the AI's output respects security at the OS level (for example, not recommending running as root unless absolutely necessary). Emphasize least privilege: for instance, code for a service should use the minimal privileges needed. The assistant could then suggest using system APIs that limit permissions or show how to configure a Linux capability instead of full root access. [[25]](#25) [[26]](#26) -* **Web and Cloud Platforms:** For web applications, direct the AI to include web security best practices. *"Always include appropriate security headers (Content Security Policy, X-Frame-Options, etc.) in web responses, and use frameworks' built-in protections for cookies and sessions".* You can also mention using HTTPS for any client-server communication. On cloud platforms, instruct it on secure use of cloud services: *"When generating code for cloud services (AWS/Azure/GCP), follow the provider's security guidelines (e.g., use parameterized queries for cloud databases, encrypt data at rest and in transit, handle keys via cloud KMS)".* [[27]](#27) [[28]](#28) -* **Container and Deployment Considerations:** If your project uses containers or orchestrators (like Docker, Kubernetes), include guidance like: *"Use minimal base images and avoid running containers with the root user. Use official images from trusted sources, and pin image versions using immutable digests (e.g., SHA256 hashes) instead of mutable tags like latest".* This will cause the AI to generate Dockerfiles or deployment scripts that adhere to container security best practices and ties into supply chain concerns as well – using only trusted images and specific versions that ensure consistency and prevent supply chain attacks. Additionally, instruct the AI on container signature verification: *"When working with container images, verify both the integrity and authenticity of images using container signing tools like cosign or notation. Include steps to verify signatures from trusted publishers and implement admission controllers in Kubernetes to enforce signature verification policies".* Instruct the AI to further control deployment artifacts. For instance, by ensuring production websites only load resources from trusted domains to avoid subversion: *"When generating HTML/JS, do not include direct links to untrusted third-party hosts for critical libraries; use our locally hosted or CDN with integrity checks".* [[29]](#29) [[30]](#30) -* **Mobile and Desktop App Security:** If relevant, instruct on platform-specific secure coding. For mobile apps, you might say: *"Do not suggest storing sensitive data in plaintext on the device; use the platform's secure storage APIs".* For desktop apps, *"Prefer high-level libraries for cryptography rather than rolling your own".* These platform notes ensure the AI isn't blind to the context in which code runs. It will include, for example, usage of Android's `SharedPreferences` with encryption or Apple's Keychain for iOS if dealing with credentials, reflecting platform best practices. [[31]](#31) -* **Operational Security:** Instruct the AI to consider devops security aspects. For example: *"When generating github actions or CI/CD pipelines, ensure secrets are stored securely (e.g., using GitHub Secrets or environment variables) and not hard-coded in the workflow files"*. This will help the AI generate secure CI/CD configurations that prevent accidental exposure of sensitive information. You can also instruct it to include security checks in the pipeline: *"Include steps to run security scans (SAST/DAST) and dependency checks in the CI/CD pipeline to catch vulnerabilities early"*. This ensures that the AI's generated code integrates security into the development lifecycle, rather than treating it as an afterthought. Treat devops tools with the same security rigor as application code. For example, instruct the AI to use secure coding practices in infrastructure-as-code (IaC) scripts, such as Terraform or CloudFormation: *"When generating IaC scripts, ensure they follow security best practices (e.g., restrict access to resources, use secure storage for secrets, and validate inputs)"* and *"use the latest versions of devops dependencies such as GitHub actions and lock the version to specific SHA"*. [[32]](#32) [[33]](#33) +* **Operating System and Environment:** If your code will run on a server or user machine, instruct the AI on safe system interactions. *"When writing file or OS-level operations, use safe functions and check for errors (e.g., use secure file modes, avoid temp files without proper randomness, etc.). If running as a service, drop privileges when possible".* This ensures the AI's output respects security at the OS level (for example, not recommending running as root unless absolutely necessary). Emphasize least privilege: for instance, code for a service should use the minimal privileges needed. The assistant could then suggest using system APIs that limit permissions or show how to configure a Linux capability instead of full root access. [[28]](#28) [[29]](#29) +* **Web and Cloud Platforms:** For web applications, direct the AI to include web security best practices. *"Always include appropriate security headers (Content Security Policy, X-Frame-Options, etc.) in web responses, and use frameworks' built-in protections for cookies and sessions".* You can also mention using HTTPS for any client-server communication. On cloud platforms, instruct it on secure use of cloud services: *"When generating code for cloud services (AWS/Azure/GCP), follow the provider's security guidelines (e.g., use parameterized queries for cloud databases, encrypt data at rest and in transit, handle keys via cloud KMS)".* [[30]](#30) [[31]](#31) +* **Container and Deployment Considerations:** If your project uses containers or orchestrators (like Docker, Kubernetes), include guidance like: *"Use minimal base images and avoid running containers with the root user. Use official images from trusted sources, and pin image versions using immutable digests (e.g., SHA256 hashes) instead of mutable tags like latest".* This will cause the AI to generate Dockerfiles or deployment scripts that adhere to container security best practices and ties into supply chain concerns as well – using only trusted images and specific versions that ensure consistency and prevent supply chain attacks. Additionally, instruct the AI on container signature verification: *"When working with container images, verify both the integrity and authenticity of images using container signing tools like cosign or notation. Include steps to verify signatures from trusted publishers and implement admission controllers in Kubernetes to enforce signature verification policies".* Instruct the AI to further control deployment artifacts. For instance, by ensuring production websites only load resources from trusted domains to avoid subversion: *"When generating HTML/JS, do not include direct links to untrusted third-party hosts for critical libraries; use our locally hosted or CDN with integrity checks".* [[32]](#32) [[33]](#33) +* **Mobile and Desktop App Security:** If relevant, instruct on platform-specific secure coding. For mobile apps, you might say: *"Do not suggest storing sensitive data in plaintext on the device; use the platform's secure storage APIs".* For desktop apps, *"Prefer high-level libraries for cryptography rather than rolling your own".* These platform notes ensure the AI isn't blind to the context in which code runs. It will include, for example, usage of Android's `SharedPreferences` with encryption or Apple's Keychain for iOS if dealing with credentials, reflecting platform best practices. [[34]](#34) +* **Operational Security:** Instruct the AI to consider devops security aspects. For example: *"When generating github actions or CI/CD pipelines, ensure secrets are stored securely (e.g., using GitHub Secrets or environment variables) and not hard-coded in the workflow files"*. This will help the AI generate secure CI/CD configurations that prevent accidental exposure of sensitive information. You can also instruct it to include security checks in the pipeline: *"Include steps to run security scans (SAST/DAST) and dependency checks in the CI/CD pipeline to catch vulnerabilities early"*. This ensures that the AI's generated code integrates security into the development lifecycle, rather than treating it as an afterthought. Treat devops tools with the same security rigor as application code. For example, instruct the AI to use secure coding practices in infrastructure-as-code (IaC) scripts, such as Terraform or CloudFormation: *"When generating IaC scripts, ensure they follow security best practices (e.g., restrict access to resources, use secure storage for secrets, and validate inputs)"* and *"use the latest versions of devops dependencies such as GitHub actions and lock the version to specific SHA"*. [[35]](#35) [[36]](#36) --- @@ -58,10 +58,10 @@ Every software runs somewhere – be it a web browser, a cloud platform, an IoT It's valuable to dedicate individual paragraphs to language-specific security considerations that are relevant to your project. The following section serves as examples, or special cases the AI should handle to highlight unique vulnerabilities or features of that language and how the AI should approach them: -* **C/C++ (Memory-Unsafe Languages):** For languages without automatic memory safety, instruct the AI to be extra cautious with memory management. *"In C or C++ code, always use bounds-checked functions (e.g., `strncpy` over `strcpy`), avoid dangerous functions like `gets`, and include buffer size constants to prevent overflow. Enable compiler defenses (stack canaries, fortify source, DEP/NX) in any build configurations you suggest"*. By giving such instructions, the assistant might prefer safer standard library calls or even suggest modern C++ classes (`std::vector` instead of raw arrays) to reduce manual memory handling. It will also acknowledge when an operation is risky, possibly inserting comments like "// ensure no buffer overflow". [[34]](#34) [[35]](#35) -* **Rust, Go, and Memory-Safe Languages:** If the project involves memory-safe languages (Rust, Go, Java, C\#, etc.), you can note that the AI should leverage their safety features. *"In Rust code, avoid using `unsafe` blocks unless absolutely necessary and document any `unsafe` usage with justification".* Memory-safe-by-default languages enforce a lot at compile time, but you should still have the AI follow best practices of those ecosystems. For example, instruct: *"In any memory-safe language, prefer using safe library functions and types; don't circumvent their safety without cause".* If a language offers any tools to verify memory access, direct the AI assistant to use them while building or testing your code. For Example "*In go code, use the data race detector when building the application*". [[36]](#36) -* **Python and Dynamic Languages:** Python, JavaScript, and other high-level languages manage memory for you, but come with their own security pitfalls. In your instructions, emphasize things like avoiding exec/eval with untrusted input in Python and being careful with command execution. *"For Python, do not use `exec`/`eval` on user input and prefer safe APIs (e.g., use the `subprocess` module with `shell=False` to avoid shell injection)".* Additionally, mention type checking or the use of linters: *"Follow PEP 8 and use type hints, as this can catch misuse early".* For JavaScript/TypeScript, you might add: *"When generating Node.js code, use prepared statements for database queries (just like any other language) and encode any data that goes into HTML to prevent XSS".* These instructions incorporate known best practices (like those from OWASP cheat sheets) directly into the AI's behavior. [[37]](#37) -* **Java/C\# and Enterprise Languages:** In languages often used for large applications, you might focus on frameworks and configurations. *"For Java, when suggesting web code (e.g., using Spring), ensure to use built-in security annotations and avoid old, vulnerable libraries (e.g., use `BCryptPasswordEncoder` rather than writing a custom password hash)".* For C\#, similarly: *"Use .NET's cryptography and identity libraries instead of custom solutions".* Also instruct about managing object deserialization (both Java and C\# have had vulnerabilities in this area): *"Never suggest turning off security features like XML entity security or type checking during deserialization".* These language-specific notes guide the AI to incorporate the well-known secure patterns of each ecosystem. [[38]](#38) +* **C/C++ (Memory-Unsafe Languages):** For languages without automatic memory safety, instruct the AI to be extra cautious with memory management. *"In C or C++ code, always use bounds-checked functions (e.g., `strncpy` over `strcpy`), avoid dangerous functions like `gets`, and include buffer size constants to prevent overflow. Enable compiler defenses (stack canaries, fortify source, DEP/NX) in any build configurations you suggest"*. By giving such instructions, the assistant might prefer safer standard library calls or even suggest modern C++ classes (`std::vector` instead of raw arrays) to reduce manual memory handling. It will also acknowledge when an operation is risky, possibly inserting comments like "// ensure no buffer overflow". [[37]](#37) [[38]](#38) +* **Rust, Go, and Memory-Safe Languages:** If the project involves memory-safe languages (Rust, Go, Java, C\#, etc.), you can note that the AI should leverage their safety features. *"In Rust code, avoid using `unsafe` blocks unless absolutely necessary and document any `unsafe` usage with justification".* Memory-safe-by-default languages enforce a lot at compile time, but you should still have the AI follow best practices of those ecosystems. For example, instruct: *"In any memory-safe language, prefer using safe library functions and types; don't circumvent their safety without cause".* If a language offers any tools to verify memory access, direct the AI assistant to use them while building or testing your code. For Example "*In go code, use the data race detector when building the application*". [[39]](#39) +* **Python and Dynamic Languages:** Python, JavaScript, and other high-level languages manage memory for you, but come with their own security pitfalls. In your instructions, emphasize things like avoiding exec/eval with untrusted input in Python and being careful with command execution. *"For Python, do not use `exec`/`eval` on user input and prefer safe APIs (e.g., use the `subprocess` module with `shell=False` to avoid shell injection)".* Additionally, mention type checking or the use of linters: *"Follow PEP 8 and use type hints, as this can catch misuse early".* For JavaScript/TypeScript, you might add: *"When generating Node.js code, use prepared statements for database queries (just like any other language) and encode any data that goes into HTML to prevent XSS".* These instructions incorporate known best practices (like those from OWASP cheat sheets) directly into the AI's behavior. [[40]](#40) +* **Java/C\# and Enterprise Languages:** In languages often used for large applications, you might focus on frameworks and configurations. *"For Java, when suggesting web code (e.g., using Spring), ensure to use built-in security annotations and avoid old, vulnerable libraries (e.g., use `BCryptPasswordEncoder` rather than writing a custom password hash)".* For C\#, similarly: *"Use .NET's cryptography and identity libraries instead of custom solutions".* Also instruct about managing object deserialization (both Java and C\# have had vulnerabilities in this area): *"Never suggest turning off security features like XML entity security or type checking during deserialization".* These language-specific notes guide the AI to incorporate the well-known secure patterns of each ecosystem. [[41]](#41) --- @@ -69,11 +69,11 @@ It's valuable to dedicate individual paragraphs to language-specific security co To strengthen your AI assistant's guidance, you should point it toward established security standards and frameworks. By referencing these in the instructions, the AI will be biased toward solutions that comply with widely accepted practices: -* **Use Industry Standards as Keywords:** Mention standards like **OWASP Top 10**, **OWASP ASVS**, **CWE/SANS Top 25**, or language-specific guidelines (Java's SEI CERT secure coding standards, for instance). For example: *"Code suggestions should adhere to OWASP Top 10 principles (e.g., avoid injection, enforce access control) and follow the OWASP ASVS requirements where applicable"*. [[39]](#39) -* **Reference SAFECode and Other Guides:** SAFECode (Software Assurance Forum for Excellence in Code) publishes **Fundamental Practices for Secure Software Development**, and including a note about it can be beneficial. *"Our project follows SAFECode's secure development practices – the AI should prioritize those (e.g., proper validation, authentication, cryptography usage per SAFECode guidance)"*. This ties the assistant's suggestions to a comprehensive set of secure development principles. Similarly, you can reference the **Secure Software Development Lifecycle (SSDLC)** or standards like **ISO 27034** if your organization uses them, to align AI outputs with those processes. [[40]](#40) -* **Compliance and Regulatory Security:** If you operate in an industry with specific regulations (like healthcare or finance), instruct the AI with those in mind. For instance: *"When generating code, consider compliance requirements (e.g., HIPAA privacy rules for medical data, PCI-DSS for credit card info) – do not output code that logs or transmits sensitive data in insecure ways"*. This ensures the AI's suggestions are compliant with external standards, for example, it might suggest encryption for personal data fields by default. [[41]](#41) -* **Continuous Improvement Hooks:** Encourage practices that integrate with ongoing security processes. For example: *"Include comments or TODOs in code suggesting security reviews for complex logic, and note if any third-party component might need a future update or audit".* This way, the AI not only writes code but also flags areas for human scrutiny or future action. If the AI outputs a piece of cryptography code, for instance, it might add a comment "// TODO: security review for cryptographic implementation" because you instructed it that critical code should not go un-reviewed. [[42]](#42) -* **Integrate and Act on Automated Security Tooling:** Instruct the AI to proactively run automated security tools (e.g., SAST, DAST, SCA) during code generation and refinement. Example: "When writing or reviewing code, run or simulate the use of tools like CodeQL, Bandit, Semgrep, or OWASP Dependency-Check. Identify any flagged vulnerabilities or outdated dependencies and revise the code accordingly. Repeat this process until the code passes all simulated scans". This positions the AI not just as a code generator but as a security-aware assistant capable of iterating based on tool feedback. [[43]](#43) [[44]](#44) +* **Use Industry Standards as Keywords:** Mention standards like **OWASP Top 10**, **OWASP ASVS**, **CWE/SANS Top 25**, or language-specific guidelines (Java's SEI CERT secure coding standards, for instance). For example: *"Code suggestions should adhere to OWASP Top 10 principles (e.g., avoid injection, enforce access control) and follow the OWASP ASVS requirements where applicable"*. [[42]](#42) +* **Reference SAFECode and Other Guides:** SAFECode (Software Assurance Forum for Excellence in Code) publishes **Fundamental Practices for Secure Software Development**, and including a note about it can be beneficial. *"Our project follows SAFECode's secure development practices – the AI should prioritize those (e.g., proper validation, authentication, cryptography usage per SAFECode guidance)"*. This ties the assistant's suggestions to a comprehensive set of secure development principles. Similarly, you can reference the **Secure Software Development Lifecycle (SSDLC)** or standards like **ISO 27034** if your organization uses them, to align AI outputs with those processes. [[43]](#43) +* **Compliance and Regulatory Security:** If you operate in an industry with specific regulations (like healthcare or finance), instruct the AI with those in mind. For instance: *"When generating code, consider compliance requirements (e.g., HIPAA privacy rules for medical data, PCI-DSS for credit card info) – do not output code that logs or transmits sensitive data in insecure ways"*. This ensures the AI's suggestions are compliant with external standards, for example, it might suggest encryption for personal data fields by default. [[44]](#44) +* **Continuous Improvement Hooks:** Encourage practices that integrate with ongoing security processes. For example: *"Include comments or TODOs in code suggesting security reviews for complex logic, and note if any third-party component might need a future update or audit".* This way, the AI not only writes code but also flags areas for human scrutiny or future action. If the AI outputs a piece of cryptography code, for instance, it might add a comment "// TODO: security review for cryptographic implementation" because you instructed it that critical code should not go un-reviewed. [[45]](#45) +* **Integrate and Act on Automated Security Tooling:** Instruct the AI to proactively run automated security tools (e.g., SAST, DAST, SCA) during code generation and refinement. Example: "When writing or reviewing code, run or simulate the use of tools like CodeQL, Bandit, Semgrep, or OWASP Dependency-Check. Identify any flagged vulnerabilities or outdated dependencies and revise the code accordingly. Repeat this process until the code passes all simulated scans". This positions the AI not just as a code generator but as a security-aware assistant capable of iterating based on tool feedback. [[46]](#46) [[47]](#47) --- @@ -83,92 +83,101 @@ To strengthen your AI assistant's guidance, you should point it toward establish [2] "AI coding assistants are no substitute for experienced developers. An unrestrained use of the tools can have severe security implications." (ANSSI, BSI - [AI Coding Assistants](https://www.bsi.bund.de/SharedDocs/Downloads/EN/BSI/KI/ANSSI_BSI_AI_Coding_Assistants.pdf?__blob=publicationFile&v=7)) -[3] "One of the major concerns regarding LLM-generated code is its security. As code snippets generated by LLMs are increasingly incorporated into industrial-level software and systems, it is critical to ensure that LLM-generated code is free of vulnerabilities that could be exploited by attackers" (Shih-Chieh Dai, Jun Xu, Guanhong Tao - [A Comprehensive Study of LLM Secure Code Generation](https://arxiv.org/abs/2503.15554)) +[3] "on average, we could successfully execute security exploits on around half of the correct programs generated by each LLM; and in less popular backend frameworks, models further struggle to generate correct and secure applications (Mark Vero, Niels Mündler, Victor Chibotaru, Veselin Raychev, Maximilian Baader, Nikola Jovanović, Jingxuan He, Martin Vechev - [Can LLMs Generate Correct and Secure Backends?](https://arxiv.org/abs/2502.11844))" -[4] "About 40% of the programs had security vulnerabilities... One cause of these security flaws is the use of outdated programs in the training data of the AI models, leading to the suggestion of outdated and insecure best practices. This is evident, for example, in password encryption, where insecure methods such as MD5 or a single iteration of SHA-256 are still often used" (ANSSI, BSI - [AI Coding Assistants](https://www.bsi.bund.de/SharedDocs/Downloads/EN/BSI/KI/ANSSI_BSI_AI_Coding_Assistants.pdf?__blob=publicationFile&v=7)) +[4] "One of the major concerns regarding LLM-generated code is its security. As code snippets generated by LLMs are increasingly incorporated into industrial-level software and systems, it is critical to ensure that LLM-generated code is free of vulnerabilities that could be exploited by attackers" (Shih-Chieh Dai, Jun Xu, Guanhong Tao - [A Comprehensive Study of LLM Secure Code Generation](https://arxiv.org/abs/2503.15554)) -[5] "Improve the prompt: The user should improve the prompt by indicating each and every aspect of the security parameters to derive the secure web application code from the LLMs" (Swaroop Dora, Deven Lunkad, Naziya Aslam, S. Venkatesan, Sandeep Kumar Shukla - [The Hidden Risks of LLM-Generated Web Application Code: A Security-Centric Evaluation of Code Generation Capabilities in Large Language Models](https://arxiv.org/abs/2504.20612)) +[5] "About 40% of the programs had security vulnerabilities... One cause of these security flaws is the use of outdated programs in the training data of the AI models, leading to the suggestion of outdated and insecure best practices. This is evident, for example, in password encryption, where insecure methods such as MD5 or a single iteration of SHA-256 are still often used" (ANSSI, BSI - [AI Coding Assistants](https://www.bsi.bund.de/SharedDocs/Downloads/EN/BSI/KI/ANSSI_BSI_AI_Coding_Assistants.pdf?__blob=publicationFile&v=7)) -[6] "These reductions suggest that newer models can effectively leverage security hints to improve code safety" (Hao Yan, Swapneel Suhas Vaidya, Xiaokuan Zhang, Ziyu Yao - [Guiding AI to Fix Its Own Flaws: An Empirical Study on LLM-Driven Secure Code Generation](https://arxiv.org/abs/2506.23034)]) +[6] "Improve the prompt: The user should improve the prompt by indicating each and every aspect of the security parameters to derive the secure web application code from the LLMs" (Swaroop Dora, Deven Lunkad, Naziya Aslam, S. Venkatesan, Sandeep Kumar Shukla - [The Hidden Risks of LLM-Generated Web Application Code: A Security-Centric Evaluation of Code Generation Capabilities in Large Language Models](https://arxiv.org/abs/2504.20612)) -[7] "Injection attacks remain one of the most critical vulnerabilities in web applications ([OWASP Top 10](https://owasp.org/Top10/A03_2021-Injection/))... LLM- +[7] "These reductions suggest that newer models can effectively leverage security hints to improve code safety" (Hao Yan, Swapneel Suhas Vaidya, Xiaokuan Zhang, Ziyu Yao - [Guiding AI to Fix Its Own Flaws: An Empirical Study on LLM-Driven Secure Code Generation](https://arxiv.org/abs/2506.23034)]) + +[8] "Injection attacks remain one of the most critical vulnerabilities in web applications ([OWASP Top 10](https://owasp.org/Top10/A03_2021-Injection/))... LLM- generated code must handle user input securely to prevent exploitation"(Swaroop Dora, Deven Lunkad, Naziya Aslam, S. Venkatesan, Sandeep Kumar Shukla - [The Hidden Risks of LLM-Generated Web Application Code](https://arxiv.org/abs/2504.20612)) -[8]"... systematic review shows that code injection (CWE-94) and OS command injection (CWE-78) are common in AI-generated code." (Wiz Research - [Rules Files for Safer Vibe Coding](https://www.wiz.io/blog/safer-vibe-coding-rules-files)) +[9]"... systematic review shows that code injection (CWE-94) and OS command injection (CWE-78) are common in AI-generated code." (Wiz Research - [Rules Files for Safer Vibe Coding](https://www.wiz.io/blog/safer-vibe-coding-rules-files)) + +[10]"All models frequently generate vulnerable code across diverse vulnerability types... All models tend to generate vulnerabilities for both common (top-25) and less common types" (Hao Yan, Swapneel Suhas Vaidya, Xiaokuan Zhang, Ziyu Yao - [Guiding AI to Fix Its Own Flaws: An Empirical Study on LLM-Driven Secure Code Generation](https://arxiv.org/abs/2506.23034)) -[9]"Overall, Copilot performed poorly in this CWE (CWE-522: Insufficiently Protected Credentials)... it frequently tried to generate code which was based on the insecure 'MD5' hashing algorithm" (Hammond Pearce, Baleegh Ahmad, Benjamin Tan, Brendan Dolan-Gavitt, Ramesh Karri - [Asleep at the Keyboard? Assessing the +[11]"Overall, Copilot performed poorly in this CWE (CWE-522: Insufficiently Protected Credentials)... it frequently tried to generate code which was based on the insecure 'MD5' hashing algorithm" (Hammond Pearce, Baleegh Ahmad, Benjamin Tan, Brendan Dolan-Gavitt, Ramesh Karri - [Asleep at the Keyboard? Assessing the Security of GitHub Copilot's Code Contributions ](https://arxiv.org/abs/2108.09293)) -[10] "Participants with access to an AI assistant were ... significantly more likely to write an insecure solution ... also less likely to authenticate the +[12] "Participants with access to an AI assistant were ... significantly more likely to write an insecure solution ... also less likely to authenticate the final returned value" (Neil Perry, Megha Srivastava, Deepak Kumar, Dan Boneh - [Do Users Write More Insecure Code with AI Assistants?](https://arxiv.org/abs/2211.03622)) -[11] "The analysis reveals critical vulnerabilities in authentication +[13] "The analysis reveals critical vulnerabilities in authentication mechanisms, session management..." (Swaroop Dora, Deven Lunkad, Naziya Aslam, S. Venkatesan, Sandeep Kumar Shukla - [The Hidden Risks of LLM-Generated Web Application Code](https://arxiv.org/abs/2504.20612)) -[12] "Generic Error Messages: Ensuring error messages do not disclose username existence or password policies prevents attackers from gaining insights during brute-force attempts... Leaking system information through verbose error messages can provide attackers valuable insights into potential vulnerabilities within authentication systems" (Swaroop Dora, Deven Lunkad, Naziya Aslam, S. Venkatesan, Sandeep Kumar Shukla - [The Hidden Risks of LLM-Generated Web Application Code](https://arxiv.org/abs/2504.20612)) +[14] "Generic Error Messages: Ensuring error messages do not disclose username existence or password policies prevents attackers from gaining insights during brute-force attempts... Leaking system information through verbose error messages can provide attackers valuable insights into potential vulnerabilities within authentication systems" (Swaroop Dora, Deven Lunkad, Naziya Aslam, S. Venkatesan, Sandeep Kumar Shukla - [The Hidden Risks of LLM-Generated Web Application Code](https://arxiv.org/abs/2504.20612)) + +[15] "It's clear that currently AI-Generated code is not secure by default... (Wiz Research - [Rules Files for Safer Vibe Coding](https://www.wiz.io/blog/safer-vibe-coding-rules-files)) -[13] "It's clear that currently AI-Generated code is not secure by default... (Wiz Research - [Rules Files for Safer Vibe Coding](https://www.wiz.io/blog/safer-vibe-coding-rules-files)) +[16] "All models require substantial improvements in authentication security, session management, error handling and HTTP security headers to align with current industry best practices and established frameworks, such as the NIST cybersecurity guidelines " (Swaroop Dora, Deven Lunkad, Naziya Aslam, S. Venkatesan, Sandeep Kumar Shukla - [The Hidden Risks of LLM-Generated Web Application Code](https://arxiv.org/abs/2504.20612)) -[14] "All models require substantial improvements in authentication security, session management, error handling and HTTP security headers to align with current industry best practices and established frameworks, such as the NIST cybersecurity guidelines " (Swaroop Dora, Deven Lunkad, Naziya Aslam, S. Venkatesan, Sandeep Kumar Shukla - [The Hidden Risks of LLM-Generated Web Application Code](https://arxiv.org/abs/2504.20612)) +[17] "The generation of test cases is another area where coding assistants can offer support. They can automatically propose test cases and unit tests that also cover edge cases" (ANSSI, BSI - [AI Coding Assistants](https://www.bsi.bund.de/SharedDocs/Downloads/EN/BSI/KI/ANSSI_BSI_AI_Coding_Assistants.pdf?__blob=publicationFile&v=7)) -[15] "The generation of test cases is another area where coding assistants can offer support. They can automatically propose test cases and unit tests that also cover edge cases" (ANSSI, BSI - [AI Coding Assistants](https://www.bsi.bund.de/SharedDocs/Downloads/EN/BSI/KI/ANSSI_BSI_AI_Coding_Assistants.pdf?__blob=publicationFile&v=7)) +[18] "Even flagship models struggle to generate correct and secure application backends" (Mark Vero, Niels Mündler, Victor Chibotaru, Veselin Raychev, Maximilian Baader, Nikola Jovanović, Jingxuan He, Martin Vechev - [Can LLMs Generate Correct and Secure Backends?](https://arxiv.org/abs/2502.11844)) -[16] "It's important to pay particularly close attention to auto-generated comments... make sure you understand and agree with them... It's a good practice to consider implementing a code-tagging system to differentiate between AI- and human-created code." (FOSSA - [5 Ways to Reduce GitHub Copilot Security and Legal Risks](https://fossa.com/blog/5-ways-to-reduce-github-copilot-security-and-legal-risks/)) +[19] "It's important to pay particularly close attention to auto-generated comments... make sure you understand and agree with them... It's a good practice to consider implementing a code-tagging system to differentiate between AI- and human-created code." (FOSSA - [5 Ways to Reduce GitHub Copilot Security and Legal Risks](https://fossa.com/blog/5-ways-to-reduce-github-copilot-security-and-legal-risks/)) -[17] "It might be beneficial to flag AI generated code blocks and to document the used AI tools. This might help security testing and it can also be a useful information for third-party auditors, e.g. in the context of a security certification process" (ANSSI, BSI - [AI Coding Assistants](https://www.bsi.bund.de/SharedDocs/Downloads/EN/BSI/KI/ANSSI_BSI_AI_Coding_Assistants.pdf?__blob=publicationFile&v=7)) +[20] "It might be beneficial to flag AI generated code blocks and to document the used AI tools. This might help security testing and it can also be a useful information for third-party auditors, e.g. in the context of a security certification process" (ANSSI, BSI - [AI Coding Assistants](https://www.bsi.bund.de/SharedDocs/Downloads/EN/BSI/KI/ANSSI_BSI_AI_Coding_Assistants.pdf?__blob=publicationFile&v=7)) -[18] "Encryption safeguards sensitive data at rest and in transit. Weak encryption methods or lack of +[21] "Encryption safeguards sensitive data at rest and in transit. Weak encryption methods or lack of encryption can expose passwords, personal information, and financial data... If passwords are stored in plain text or hashed without salting, attackers with database access can easily decrypt credentials, leading to mass account breaches." (Swaroop Dora, Deven Lunkad, Naziya Aslam, S. Venkatesan, Sandeep Kumar Shukla - [The Hidden Risks of LLM-Generated Web Application Code](https://arxiv.org/abs/2504.20612)) -[19] "Package hallucination occurs when an LLM generates code that recommends or contains a reference to a package that does not actually exist. An adversary can exploit package hallucinations... by publishing a package to an open-source repository with the same name as the hallucinated or fictitious package and containing some malicious code/functionality" (Joseph Spracklen, Raveen Wijewickrama, A H M Nazmus Sakib, Anindya Maiti, Bimal Viswanath, Murtuza Jadliwala - [We Have a Package for You! A Comprehensive Analysis of Package Hallucinations by Code Generating LLMs](https://arxiv.org/abs/2406.10279)) +[22] "Package hallucination occurs when an LLM generates code that recommends or contains a reference to a package that does not actually exist. An adversary can exploit package hallucinations... by publishing a package to an open-source repository with the same name as the hallucinated or fictitious package and containing some malicious code/functionality" (Joseph Spracklen, Raveen Wijewickrama, A H M Nazmus Sakib, Anindya Maiti, Bimal Viswanath, Murtuza Jadliwala - [We Have a Package for You! A Comprehensive Analysis of Package Hallucinations by Code Generating LLMs](https://arxiv.org/abs/2406.10279)) + +[23] "Avoid... copy reused code from other packages... instead of using a package manager to automate identifying and updating reused packages... Use a package manager to manage it, one that records the specific version numbers" (OpenSSF - [Secure Software Development Fundamentals](https://github.com/ossf/secure-sw-dev-fundamentals/blob/main/docs/lfd121.md#preparing-for-the-inevitable-vulnerabilities-in-your-dependencies)) + +[24] "LLM sometimes produces code that uses older ... libraries, which are not compliant with modern security. This increases the security risk... they are not updated with the latest security vulnerabilities and threats, leaving the generated code open to new attacks." (Ariful Haque, Sunzida Siddique, Md. Mahfuzur Rahman, Ahmed Rafi Hasan, Laxmi Rani Das, Marufa Kamal, Tasnim Masura, Kishor Datta Gupta - [Exploring Hallucinations and Security Risks in AI-Assisted Software Development with Insights for LLM Deployment](https://arxiv.org/abs/2502.18468v1)) + +[25] "LLMs cannot update themselves with new information after release and have no knowledge of the world beyond their training data cutoff date." (Joseph Spracklen, Raveen Wijewickrama, A H M Nazmus Sakib, Anindya Maiti, Bimal Viswanath, Murtuza Jadliwala - [We Have a Package for You! A Comprehensive Analysis of Package Hallucinations](https://arxiv.org/abs/2406.10279)) -[20] "Avoid... copy reused code from other packages... instead of using a package manager to automate identifying and updating reused packages... Use a package manager to manage it, one that records the specific version numbers" (OpenSSF - [Secure Software Development Fundamentals](https://github.com/ossf/secure-sw-dev-fundamentals/blob/main/docs/lfd121.md#preparing-for-the-inevitable-vulnerabilities-in-your-dependencies)) +[26] "The creation of a Software Bill of Materials (SBOM) allows you to retrospectively understand whether vulnerable libraries were used and enables a targeted response if a vulnerability of certain components becomes known." (ANSSI, BSI - [AI Coding Assistants](https://www.bsi.bund.de/SharedDocs/Downloads/EN/BSI/KI/ANSSI_BSI_AI_Coding_Assistants.pdf?__blob=publicationFile&v=7)) -[21] "LLM sometimes produces code that uses older ... libraries, which are not compliant with modern security. This increases the security risk... they are not updated with the latest security vulnerabilities and threats, leaving the generated code open to new attacks." (Ariful Haque, Sunzida Siddique, Md. Mahfuzur Rahman, Ahmed Rafi Hasan, Laxmi Rani Das, Marufa Kamal, Tasnim Masura, Kishor Datta Gupta - [Exploring Hallucinations and Security Risks in AI-Assisted Software Development with Insights for LLM Deployment](https://arxiv.org/abs/2502.18468v1)) +[27] "Public OSS repositories ... have implemented various measures, including ... software signing to mitigate the distribution of malicious packages... Self-refinement strategies... utilize the model itself to detect and refine potential hallucinations" (Joseph Spracklen, Raveen Wijewickrama, A H M Nazmus Sakib, Anindya Maiti, Bimal Viswanath, Murtuza Jadliwala - [We Have a Package for You! A Comprehensive Analysis of Package Hallucinations](https://arxiv.org/abs/2406.10279)) -[22] "LLMs cannot update themselves with new information after release and have no knowledge of the world beyond their training data cutoff date." (Joseph Spracklen, Raveen Wijewickrama, A H M Nazmus Sakib, Anindya Maiti, Bimal Viswanath, Murtuza Jadliwala - [We Have a Package for You! A Comprehensive Analysis of Package Hallucinations](https://arxiv.org/abs/2406.10279)) +[28] "LLM often lacks the deep contextual understanding needed to create secure code, resulting in sometimes irrelevant advice for specific security contexts" (Ariful Haque, Sunzida Siddique, Md. Mahfuzur Rahman, Ahmed Rafi Hasan, Laxmi Rani Das, Marufa Kamal, Tasnim Masura, Kishor Datta Gupta - [Exploring Hallucinations and Security Risks in AI-Assisted Software Development with Insights for LLM Deployment](https://arxiv.org/abs/2502.18468v1)) -[23] "The creation of a Software Bill of Materials (SBOM) allows you to retrospectively understand whether vulnerable libraries were used and enables a targeted response if a vulnerability of certain components becomes known." (ANSSI, BSI - [AI Coding Assistants](https://www.bsi.bund.de/SharedDocs/Downloads/EN/BSI/KI/ANSSI_BSI_AI_Coding_Assistants.pdf?__blob=publicationFile&v=7)) +[29] "... code directly executes the user-provided shell script without checking for malicious commands... It uses the system call os.system() to execute the copy command, which is passed as a string. If the original_location or new_location variables contain malicious input, this could lead to command injection" (Shih-Chieh Dai, Jun Xu, Guanhong Tao - [A Comprehensive Study of LLM Secure Code Generation](https://arxiv.org/abs/2503.15554)) -[24] "Public OSS repositories ... have implemented various measures, including ... software signing to mitigate the distribution of malicious packages... Self-refinement strategies... utilize the model itself to detect and refine potential hallucinations" (Joseph Spracklen, Raveen Wijewickrama, A H M Nazmus Sakib, Anindya Maiti, Bimal Viswanath, Murtuza Jadliwala - [We Have a Package for You! A Comprehensive Analysis of Package Hallucinations](https://arxiv.org/abs/2406.10279)) +[30] "Although some models implement security measures to a limited extent, none fully align with industry best practices" (Swaroop Dora, Deven Lunkad, Naziya Aslam, S. Venkatesan, Sandeep Kumar Shukla - [The Hidden Risks of LLM-Generated Web Application Code: A Security-Centric Evaluation of Code Generation Capabilities in Large Language Models](https://arxiv.org/abs/2504.20612)) -[25] "LLM often lacks the deep contextual understanding needed to create secure code, resulting in sometimes irrelevant advice for specific security contexts" (Ariful Haque, Sunzida Siddique, Md. Mahfuzur Rahman, Ahmed Rafi Hasan, Laxmi Rani Das, Marufa Kamal, Tasnim Masura, Kishor Datta Gupta - [Exploring Hallucinations and Security Risks in AI-Assisted Software Development with Insights for LLM Deployment](https://arxiv.org/abs/2502.18468v1)) +[31] "36% of participants with access to the AI assistant writing solutions that are vulnerable to SQL injections compared to 7% of the control group" (Neil Perry, Megha Srivastava, Deepak Kumar, Dan Boneh - [Do Users Write More Insecure Code with AI Assistants?](https://arxiv.org/abs/2211.03622)) -[26] "... code directly executes the user-provided shell script without checking for malicious commands... It uses the system call os.system() to execute the copy command, which is passed as a string. If the original_location or new_location variables contain malicious input, this could lead to command injection" (Shih-Chieh Dai, Jun Xu, Guanhong Tao - [A Comprehensive Study of LLM Secure Code Generation](https://arxiv.org/abs/2503.15554)) +[32] "... detailed information about the infrastructure and resource configuration can be sensitive. The last part is particularly relevant if approaches like Infrastructure as Code (IaC) are used" (ANSSI, BSI - [AI Coding Assistants](https://www.bsi.bund.de/SharedDocs/Downloads/EN/BSI/KI/ANSSI_BSI_AI_Coding_Assistants.pdf?__blob=publicationFile&v=7)) -[27] "Although some models implement security measures to a limited extent, none fully align with industry best practices" (Swaroop Dora, Deven Lunkad, Naziya Aslam, S. Venkatesan, Sandeep Kumar Shukla - [The Hidden Risks of LLM-Generated Web Application Code: A Security-Centric Evaluation of Code Generation Capabilities in Large Language Models](https://arxiv.org/abs/2504.20612)) +[33] "The open nature of these repositories... makes them an attractive platform for malware distribution. For instance, a total of 245,000 malicious code packages were discovered in open-source software repositories in 2023 alone (Joseph Spracklen, Raveen Wijewickrama, A H M Nazmus Sakib, Anindya Maiti, Bimal Viswanath, Murtuza Jadliwala - [We Have a Package for You! A Comprehensive Analysis of Package Hallucinations](https://arxiv.org/abs/2406.10279)) -[28] "36% of participants with access to the AI assistant writing solutions that are vulnerable to SQL injections compared to 7% of the control group" (Neil Perry, Megha Srivastava, Deepak Kumar, Dan Boneh - [Do Users Write More Insecure Code with AI Assistants?](https://arxiv.org/abs/2211.03622)) +[34] "... we produce 89 different scenarios for Copilot to complete, producing 1,689 programs. Of these, we found approximately 40% to be vulnerable" (Hammond Pearce, Baleegh Ahmad, Benjamin Tan, Brendan Dolan-Gavitt, Ramesh Karri - [Asleep at the Keyboard? Assessing the Security of GitHub Copilot's Code Contributions](https://arxiv.org/abs/2108.09293)) -[29] "... detailed information about the infrastructure and resource configuration can be sensitive. The last part is particularly relevant if approaches like Infrastructure as Code (IaC) are used" (ANSSI, BSI - [AI Coding Assistants](https://www.bsi.bund.de/SharedDocs/Downloads/EN/BSI/KI/ANSSI_BSI_AI_Coding_Assistants.pdf?__blob=publicationFile&v=7)) +[35] "The use of uncontrolled access to coding assistants ... in the cloud should be prohibited for business software development." (ANSSI, BSI - [AI Coding Assistants](https://www.bsi.bund.de/SharedDocs/Downloads/EN/BSI/KI/ANSSI_BSI_AI_Coding_Assistants.pdf?__blob=publicationFile&v=7)) -[30] "The open nature of these repositories... makes them an attractive platform for malware distribution. For instance, a total of 245,000 malicious code packages were discovered in open-source software repositories in 2023 alone (Joseph Spracklen, Raveen Wijewickrama, A H M Nazmus Sakib, Anindya Maiti, Bimal Viswanath, Murtuza Jadliwala - [We Have a Package for You! A Comprehensive Analysis of Package Hallucinations](https://arxiv.org/abs/2406.10279)) +[36] "Many projects apply a ... DevSecOps approach, which intentionally blend these processes together with development... having secure distribution, fielding, operations, and disposal is critical for software to be secure in the real world." (OpenSSF - [Secure Software Development Fundamentals](https://github.com/ossf/secure-sw-dev-fundamentals/blob/main/docs/lfd121.md#preparing-for-the-inevitable-vulnerabilities-in-your-dependencies)) -[31] "... we produce 89 different scenarios for Copilot to complete, producing 1,689 programs. Of these, we found approximately 40% to be vulnerable" (Hammond Pearce, Baleegh Ahmad, Benjamin Tan, Brendan Dolan-Gavitt, Ramesh Karri - [Asleep at the Keyboard? Assessing the Security of GitHub Copilot's Code Contributions](https://arxiv.org/abs/2108.09293)) +[37] "The security vulnerabilities in LLM-generated correct code generally stem from missing conditional checks, incorrect memory allocations, or incorrect conditional checks" (Connor Dilgren, Purva Chiniya, Luke Griffith, Yu Ding, Yizheng Chen - [Benchmarking LLMs for Secure Code Generation in Real-World Repositories](https://arxiv.org/abs/2504.21205)) -[32] "The use of uncontrolled access to coding assistants ... in the cloud should be prohibited for business software development." (ANSSI, BSI - [AI Coding Assistants](https://www.bsi.bund.de/SharedDocs/Downloads/EN/BSI/KI/ANSSI_BSI_AI_Coding_Assistants.pdf?__blob=publicationFile&v=7)) +[38] "... we observe that participants in the experiment group were significantly more likely to introduce integer overflow mistakes in their solutions" (Neil Perry, Megha Srivastava, Deepak Kumar, Dan Boneh - [Do Users Write More Insecure Code with AI Assistants?](https://arxiv.org/abs/2211.03622)) -[33] "Many projects apply a ... DevSecOps approach, which intentionally blend these processes together with development... having secure distribution, fielding, operations, and disposal is critical for software to be secure in the real world." (OpenSSF - [Secure Software Development Fundamentals](https://github.com/ossf/secure-sw-dev-fundamentals/blob/main/docs/lfd121.md#preparing-for-the-inevitable-vulnerabilities-in-your-dependencies)) +[39] "Even when a language is memory safe by default, it is still vital to follow that language ecosystem's best practices and to use external tools to ensure safety not only within your code" (OpenSSF - [The Memory Safety Continuum](https://memorysafety.openssf.org/memory-safety-continuum/#writing-software-in-memory-safe-by-default-languages)) -[34] "The security vulnerabilities in LLM-generated correct code generally stem from missing conditional checks, incorrect memory allocations, or incorrect conditional checks" (Connor Dilgren, Purva Chiniya, Luke Griffith, Yu Ding, Yizheng Chen - [Benchmarking LLMs for Secure Code Generation in Real-World Repositories](https://arxiv.org/abs/2504.21205)) +[40] "... we reveal that although LLMs are prone to generating insecure code, advanced models can benefit from vulnerability hints and fine-grained feedback to avoid or fix vulnerabilities." (Hao Yan, Swapneel Suhas Vaidya, Xiaokuan Zhang, Ziyu Yao - [Guiding AI to Fix Its Own Flaws: An Empirical Study on LLM-Driven Secure Code Generation](https://arxiv.org/abs/2506.23034)) -[35] "... we observe that participants in the experiment group were significantly more likely to introduce integer overflow mistakes in their solutions" (Neil Perry, Megha Srivastava, Deepak Kumar, Dan Boneh - [Do Users Write More Insecure Code with AI Assistants?](https://arxiv.org/abs/2211.03622)) +[41] "Often times, responses from the AI assistant use libraries that explicitly flag that they are insecure in the documentation for the library" (Neil Perry, Megha Srivastava, Deepak Kumar, Dan Boneh - [Do Users Write More Insecure Code with AI Assistants?](https://arxiv.org/abs/2211.03622)) -[36] "Even when a language is memory safe by default, it is still vital to follow that language ecosystem's best practices and to use external tools to ensure safety not only within your code" (OpenSSF - [The Memory Safety Continuum](https://memorysafety.openssf.org/memory-safety-continuum/#writing-software-in-memory-safe-by-default-languages)) +[42] "Additionally, we can reduce security risks by using OWASP guidelines and using trusted libraries" (Ariful Haque, Sunzida Siddique, Md. Mahfuzur Rahman, Ahmed Rafi Hasan, Laxmi Rani Das, Marufa Kamal, Tasnim Masura, Kishor Datta Gupta - [Exploring Hallucinations and Security Risks in AI-Assisted Software Development with Insights for LLM Deployment](https://arxiv.org/abs/2502.18468v1)) -[37] "... we reveal that although LLMs are prone to generating insecure code, advanced models can benefit from vulnerability hints and fine-grained feedback to avoid or fix vulnerabilities." (Hao Yan, Swapneel Suhas Vaidya, Xiaokuan Zhang, Ziyu Yao - [Guiding AI to Fix Its Own Flaws: An Empirical Study on LLM-Driven Secure Code Generation](https://arxiv.org/abs/2506.23034)) +[43] "We found that those who specified task instructions... generated more secure code." (Neil Perry, Megha Srivastava, Deepak Kumar, Dan Boneh - [Do Users Write More Insecure Code with AI Assistants?](https://arxiv.org/abs/2211.03622)) -[38] "Often times, responses from the AI assistant use libraries that explicitly flag that they are insecure in the documentation for the library" (Neil Perry, Megha Srivastava, Deepak Kumar, Dan Boneh - [Do Users Write More Insecure Code with AI Assistants?](https://arxiv.org/abs/2211.03622)) +[44] "Compliance with regulations like GDPR and HIPAA is essential to minimize legal risks and uphold organizational reputation in sensitive environments" (Ariful Haque, Sunzida Siddique, Md. Mahfuzur Rahman, Ahmed Rafi Hasan, Laxmi Rani Das, Marufa Kamal, Tasnim Masura, Kishor Datta Gupta - [Exploring Hallucinations and Security Risks in AI-Assisted Software Development with Insights for LLM Deployment](https://arxiv.org/abs/2502.18468v1)) -[39] "Additionally, we can reduce security risks by using OWASP guidelines and using trusted libraries" (Ariful Haque, Sunzida Siddique, Md. Mahfuzur Rahman, Ahmed Rafi Hasan, Laxmi Rani Das, Marufa Kamal, Tasnim Masura, Kishor Datta Gupta - [Exploring Hallucinations and Security Risks in AI-Assisted Software Development with Insights for LLM Deployment](https://arxiv.org/abs/2502.18468v1)) +[45] "Generated content, in particular source code, should generally be reviewed and understood by the developers... It might be beneficial to 'deconstruct' AI-generated code and the used prompts in public code reviews within the company. This allows knowledge and experiences to be shared across teams." (ANSSI, BSI - [AI Coding Assistants](https://www.bsi.bund.de/SharedDocs/Downloads/EN/BSI/KI/ANSSI_BSI_AI_Coding_Assistants.pdf?__blob=publicationFile&v=7)) -[40] "We found that those who specified task instructions... generated more secure code." (Neil Perry, Megha Srivastava, Deepak Kumar, Dan Boneh - [Do Users Write More Insecure Code with AI Assistants?](https://arxiv.org/abs/2211.03622)) +[46] "Automated vulnerability scanners or approaches like chatbots that critically question the generated source code ('source code critics') can reduce the risk" (ANSSI, BSI - [AI Coding Assistants](https://www.bsi.bund.de/SharedDocs/Downloads/EN/BSI/KI/ANSSI_BSI_AI_Coding_Assistants.pdf?__blob=publicationFile&v=7)) -[41] "Compliance with regulations like GDPR and HIPAA is essential to minimize legal risks and uphold organizational reputation in sensitive environments" (Ariful Haque, Sunzida Siddique, Md. Mahfuzur Rahman, Ahmed Rafi Hasan, Laxmi Rani Das, Marufa Kamal, Tasnim Masura, Kishor Datta Gupta - [Exploring Hallucinations and Security Risks in AI-Assisted Software Development with Insights for LLM Deployment](https://arxiv.org/abs/2502.18468v1)) +[47] "... post-processing the output ... has a measurable impact on code quality, and is LLM-agnostic... Presumably, non-LLM static analyzers or linters may be integrated as part of the code generation procedure to provide checks along the way and avoid producing code that is visibly incorrect or dangerous" (Frontiers - [A systematic literature review on the impact of AI models on the security of code generation](https://www.frontiersin.org/journals/big-data/articles/10.3389/fdata.2024.1386720/full)) -[42] "Generated content, in particular source code, should generally be reviewed and understood by the developers... It might be beneficial to 'deconstruct' AI-generated code and the used prompts in public code reviews within the company. This allows knowledge and experiences to be shared across teams." (ANSSI, BSI - [AI Coding Assistants](https://www.bsi.bund.de/SharedDocs/Downloads/EN/BSI/KI/ANSSI_BSI_AI_Coding_Assistants.pdf?__blob=publicationFile&v=7)) -[43] "Automated vulnerability scanners or approaches like chatbots that critically question the generated source code ('source code critics') can reduce the risk" (ANSSI, BSI - [AI Coding Assistants](https://www.bsi.bund.de/SharedDocs/Downloads/EN/BSI/KI/ANSSI_BSI_AI_Coding_Assistants.pdf?__blob=publicationFile&v=7)) -[44] "... post-processing the output ... has a measurable impact on code quality, and is LLM-agnostic... Presumably, non-LLM static analyzers or linters may be integrated as part of the code generation procedure to provide checks along the way and avoid producing code that is visibly incorrect or dangerous" (Frontiers - [A systematic literature review on the impact of AI models on the security of code generation](https://www.frontiersin.org/journals/big-data/articles/10.3389/fdata.2024.1386720/full)) From 79d0686d970397fa8311c843303a62e2eaaa1221 Mon Sep 17 00:00:00 2001 From: balteravishay Date: Wed, 30 Jul 2025 11:29:36 +0100 Subject: [PATCH 04/31] fix lint Signed-off-by: balteravishay --- ...Focused-Guide-for-AI-Code-Assistant-Instructions.md | 10 ++++------ 1 file changed, 4 insertions(+), 6 deletions(-) diff --git a/docs/Security-Focused-Guide-for-AI-Code-Assistant-Instructions.md b/docs/Security-Focused-Guide-for-AI-Code-Assistant-Instructions.md index b6b7214c..70a06ae9 100644 --- a/docs/Security-Focused-Guide-for-AI-Code-Assistant-Instructions.md +++ b/docs/Security-Focused-Guide-for-AI-Code-Assistant-Instructions.md @@ -4,7 +4,7 @@ AI code assistants can significantly speed up development, but they need guidanc --- -## TL;DR +## TL;DR Short on time? Here's what really matters: @@ -35,11 +35,12 @@ One of the first sections in your instructions should reinforce general secure c Modern software heavily relies on third-party libraries and dependencies. It's crucial that your AI assistant's instructions cover supply chain security, ensuring that suggested dependencies and build processes are secure: -* **Safe Dependency Selection:** Instruct the AI to prefer well-vetted, reputable libraries when suggesting code that pulls in external packages. For example: *"Use popular, community-trusted libraries for common tasks (and avoid adding obscure dependencies if a standard library or well-known package can do the same job)".* Emphasize evaluating packages before use – as a developer would manually. [[22]](#22) +* **Safe Dependency Selection:** Instruct the AI to prefer well-vetted, reputable libraries when suggesting code that pulls in external packages. For example: *"Use popular, community-trusted libraries for common tasks (and avoid adding obscure dependencies if a standard library or well-known package can do the same job)".* Emphasize evaluating packages before use – as a developer would manually. [[22]](#22) * **Use Package Managers & Lock Versions:** Your instructions should tell the AI to use proper package management. For instance: *"Always use the official package manager for the given language (npm, pip, Maven, etc.) to install libraries, rather than copying code snippets".* Also, instruct it to specify version ranges or exact versions that are known to be secure. By doing so, the AI will generate code that, for example, uses a `requirements.txt` or `package.json` entry, which aids in maintaining supply chain integrity. [[23]](#23) * **Stay Updated & Monitor Vulnerabilities:** Include guidance for keeping dependencies up-to-date. For example: *"When suggesting dependency versions, prefer the latest stable release and mention updating dependencies regularly to patch vulnerabilities"*. [[24]](#24) [[25]](#25) * **Generate Software Bill of Materials (SBOM):** Instruct the AI to create and maintaining SBOMs for better visibility into your software supply chain. For example: *"Generate a Software Bill of Materials (SBOM) by using tools that support standard formats like SPDX or CycloneDX".* You can also mention provenance tracking: *"Where applicable, use in-toto attestations or similar frameworks to create verifiable records of your build and deployment processes".* This ensures comprehensive tracking of what goes into your software and provides the foundation for ongoing vulnerability monitoring and incident response. [[26]](#26) * **Integrity Verification:** To further secure the supply chain, you can instruct the assistant to show how to verify what it uses. For instance: *"When adding important external resources (scripts, containers, etc.), include steps to verify integrity (like checksum verification or signature validation) if applicable".* [[27]](#27) + --- ## **Platform and Runtime Security Considerations** @@ -61,7 +62,7 @@ It's valuable to dedicate individual paragraphs to language-specific security co * **C/C++ (Memory-Unsafe Languages):** For languages without automatic memory safety, instruct the AI to be extra cautious with memory management. *"In C or C++ code, always use bounds-checked functions (e.g., `strncpy` over `strcpy`), avoid dangerous functions like `gets`, and include buffer size constants to prevent overflow. Enable compiler defenses (stack canaries, fortify source, DEP/NX) in any build configurations you suggest"*. By giving such instructions, the assistant might prefer safer standard library calls or even suggest modern C++ classes (`std::vector` instead of raw arrays) to reduce manual memory handling. It will also acknowledge when an operation is risky, possibly inserting comments like "// ensure no buffer overflow". [[37]](#37) [[38]](#38) * **Rust, Go, and Memory-Safe Languages:** If the project involves memory-safe languages (Rust, Go, Java, C\#, etc.), you can note that the AI should leverage their safety features. *"In Rust code, avoid using `unsafe` blocks unless absolutely necessary and document any `unsafe` usage with justification".* Memory-safe-by-default languages enforce a lot at compile time, but you should still have the AI follow best practices of those ecosystems. For example, instruct: *"In any memory-safe language, prefer using safe library functions and types; don't circumvent their safety without cause".* If a language offers any tools to verify memory access, direct the AI assistant to use them while building or testing your code. For Example "*In go code, use the data race detector when building the application*". [[39]](#39) * **Python and Dynamic Languages:** Python, JavaScript, and other high-level languages manage memory for you, but come with their own security pitfalls. In your instructions, emphasize things like avoiding exec/eval with untrusted input in Python and being careful with command execution. *"For Python, do not use `exec`/`eval` on user input and prefer safe APIs (e.g., use the `subprocess` module with `shell=False` to avoid shell injection)".* Additionally, mention type checking or the use of linters: *"Follow PEP 8 and use type hints, as this can catch misuse early".* For JavaScript/TypeScript, you might add: *"When generating Node.js code, use prepared statements for database queries (just like any other language) and encode any data that goes into HTML to prevent XSS".* These instructions incorporate known best practices (like those from OWASP cheat sheets) directly into the AI's behavior. [[40]](#40) -* **Java/C\# and Enterprise Languages:** In languages often used for large applications, you might focus on frameworks and configurations. *"For Java, when suggesting web code (e.g., using Spring), ensure to use built-in security annotations and avoid old, vulnerable libraries (e.g., use `BCryptPasswordEncoder` rather than writing a custom password hash)".* For C\#, similarly: *"Use .NET's cryptography and identity libraries instead of custom solutions".* Also instruct about managing object deserialization (both Java and C\# have had vulnerabilities in this area): *"Never suggest turning off security features like XML entity security or type checking during deserialization".* These language-specific notes guide the AI to incorporate the well-known secure patterns of each ecosystem. [[41]](#41) +* **Java/C\# and Enterprise Languages:** In languages often used for large applications, you might focus on frameworks and configurations. *"For Java, when suggesting web code (e.g., using Spring), ensure to use built-in security annotations and avoid old, vulnerable libraries (e.g., use `BCryptPasswordEncoder` rather than writing a custom password hash)".* For C\#, similarly: *"Use .NET's cryptography and identity libraries instead of custom solutions".* Also instruct about managing object deserialization (both Java and C\# have had vulnerabilities in this area): *"Never suggest turning off security features like XML entity security or type checking during deserialization".* These language-specific notes guide the AI to incorporate the well-known secure patterns of each ecosystem. [[41]](#41) --- @@ -178,6 +179,3 @@ encryption can expose passwords, personal information, and financial data... If [46] "Automated vulnerability scanners or approaches like chatbots that critically question the generated source code ('source code critics') can reduce the risk" (ANSSI, BSI - [AI Coding Assistants](https://www.bsi.bund.de/SharedDocs/Downloads/EN/BSI/KI/ANSSI_BSI_AI_Coding_Assistants.pdf?__blob=publicationFile&v=7)) [47] "... post-processing the output ... has a measurable impact on code quality, and is LLM-agnostic... Presumably, non-LLM static analyzers or linters may be integrated as part of the code generation procedure to provide checks along the way and avoid producing code that is visibly incorrect or dangerous" (Frontiers - [A systematic literature review on the impact of AI models on the security of code generation](https://www.frontiersin.org/journals/big-data/articles/10.3389/fdata.2024.1386720/full)) - - - From 9aed4c6155c7ce7aadabc726b5a2ea8562681256 Mon Sep 17 00:00:00 2001 From: Avishay Balter Date: Wed, 30 Jul 2025 11:58:26 +0100 Subject: [PATCH 05/31] Update docs/developers.md Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> Signed-off-by: Avishay Balter Signed-off-by: balteravishay --- docs/developers.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/developers.md b/docs/developers.md index ac537c23..dbbd7642 100644 --- a/docs/developers.md +++ b/docs/developers.md @@ -12,7 +12,7 @@ If you develop or build software, here are some ready-to-go resources from the O * [Concise Guide for Developing More Secure Software](https://best.openssf.org/Concise-Guide-for-Developing-More-Secure-Software) - a short "start here" page for how to develop secure software. * [Guide to Implementing a Coordinated Vulnerability Disclosure Process for Open Source Projects](https://github.com/ossf/oss-vulnerability-guide/blob/main/maintainer-guide.md#readme) - we recommend OSS projects use this _before_ you get a vulnerability report! -* [Source Code Management Best Practices Guide](https://best.openssf.org/SCM-BestPractices/) - Guide for securing and implementing best practices for SCM platforms, including GitHub and GitL +* [Source Code Management Best Practices Guide](https://best.openssf.org/SCM-BestPractices/) - Guide for securing and implementing best practices for SCM platforms, including GitHub and GitLab * [Secure AI Assistant Instructions](https://best.openssf.org/Security-Focused-Guide-for-AI-Code-Assistant-Instructions) - a guide for how to write instructions for AI code assistants so that they generate more secure code. You can also see the full list of [Guides released by the OpenSSF](https://openssf.org/resources/guides/). From 4154ea30756871b5e775b212cdd876062212c44d Mon Sep 17 00:00:00 2001 From: Avishay Balter Date: Wed, 30 Jul 2025 11:59:04 +0100 Subject: [PATCH 06/31] Update docs/Security-Focused-Guide-for-AI-Code-Assistant-Instructions.md Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> Signed-off-by: Avishay Balter Signed-off-by: balteravishay --- ...Security-Focused-Guide-for-AI-Code-Assistant-Instructions.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/Security-Focused-Guide-for-AI-Code-Assistant-Instructions.md b/docs/Security-Focused-Guide-for-AI-Code-Assistant-Instructions.md index 70a06ae9..c6ba6df1 100644 --- a/docs/Security-Focused-Guide-for-AI-Code-Assistant-Instructions.md +++ b/docs/Security-Focused-Guide-for-AI-Code-Assistant-Instructions.md @@ -24,7 +24,7 @@ One of the first sections in your instructions should reinforce general secure c * **Input Validation & Output Encoding:** Instruct the AI to treat all external inputs as untrusted and to validate them. *Example: "user inputs should be checked for expected format and length*". Any output should be properly encoded to prevent injection attacks such as SQL injection or cross-site scripting (XSS). *Example: "Always validate function arguments and use parameterized queries for database access"* and *"Escape special characters in user-generated content before rendering it in HTML"*. Similarly, specify that when generating output contexts such as HTML or SQL, the assistant should use safe frameworks or encoding functions to avoid vulnerabilities. [[8]](#8) [[9]](#9) [[10]](#10) * **Authentication, Authorization & Secrets Management:** Emphasize that credentials and sensitive tokens must never be hard-coded or exposed. Your instructions can say: *"Never include API keys, passwords, or secrets in code output, and use environment variables or secure vault references instead"*. Also instruct the AI to use secure authentication flows (for instance, using industry-standard libraries for handling passwords or tokens) and to enforce role-based access checks where appropriate. [[11]](#11) [[12]](#12) [[13]](#13) * **Error Handling & Logging:** Guide the AI to implement errors securely by catching exceptions and failures without revealing sensitive info (stack traces, server paths, etc.) to the end-user. In your instructions, you might include: *"When generating code, handle errors gracefully and log them, but do not expose internal details or secrets in error messages".* This ensures the assistant's suggestions include secure error-handling patterns (like generic user-facing messages and detailed logs only on the server side). Additionally, instruct the AI to use logging frameworks that can be configured for security (e.g. avoiding logging of personal data or secrets). [[14]](#14) -* **Secure Defaults & Configurations:** Include guidance such as: *"Prefer safe defaults in configurations – for example, use HTTPS by default, require strong encryption algorithms, and disable insecure protocols or options".* By specifying this, the AI will be more likely to generate code that opts-in to security features. Always instruct the AI to follow the principle of least privilege (e.g. minimal file system permissions, least-privileged user accounts for services, etc.) in any configuration or code it proposes. [[13]](#15) [[14]](#16) +* **Secure Defaults & Configurations:** Include guidance such as: *"Prefer safe defaults in configurations – for example, use HTTPS by default, require strong encryption algorithms, and disable insecure protocols or options".* By specifying this, the AI will be more likely to generate code that opts-in to security features. Always instruct the AI to follow the principle of least privilege (e.g. minimal file system permissions, least-privileged user accounts for services, etc.) in any configuration or code it proposes. [[15]](#15) [[16]](#16) * **Testing for Security:** Encourage the AI to produce or suggest tests for critical code paths including negative tests that verify that what shouldn't happen, doesn't happen. In your instructions, add: *"When applicable, generate unit tests for security-critical functions (including negative tests to ensure the code fails safely)"*. [[17]](#17) [[18]](#18) * **Call out and review stubbed code:** If the AI generates code that is stubbed or incomplete, instruct it to flag these areas for review. For example: *"If you generate placeholder code (e.g., `TODO` comments), ensure it is marked for security review before deployment".* This will help ensure that any incomplete code does not inadvertently introduce vulnerabilities. [[19]](#19) [[20]](#20) * **Data Protection:** When generating code, always prioritize data minimization and avoid storing or processing confidential or otherwise sensitive information (like personal data - PII) unless absolutely necessary. For that case, suggest strong encryption at rest and in transit, and recommend techniques like anonymization. For example: *"Generate a function that securely handles user input for a registration form, asking for necessary fields to avoid logging sensitive information of PII. Ensure that no sensitive or PII is stored in plaintext"*. [[21]](#21) From 1c7027644045932550590e97580562fbcdd8ad86 Mon Sep 17 00:00:00 2001 From: Avishay Balter Date: Wed, 30 Jul 2025 14:34:25 +0100 Subject: [PATCH 07/31] Update docs/developers.md Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> Signed-off-by: Avishay Balter Signed-off-by: balteravishay --- docs/developers.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/developers.md b/docs/developers.md index dbbd7642..cea51ee0 100644 --- a/docs/developers.md +++ b/docs/developers.md @@ -12,7 +12,7 @@ If you develop or build software, here are some ready-to-go resources from the O * [Concise Guide for Developing More Secure Software](https://best.openssf.org/Concise-Guide-for-Developing-More-Secure-Software) - a short "start here" page for how to develop secure software. * [Guide to Implementing a Coordinated Vulnerability Disclosure Process for Open Source Projects](https://github.com/ossf/oss-vulnerability-guide/blob/main/maintainer-guide.md#readme) - we recommend OSS projects use this _before_ you get a vulnerability report! -* [Source Code Management Best Practices Guide](https://best.openssf.org/SCM-BestPractices/) - Guide for securing and implementing best practices for SCM platforms, including GitHub and GitLab +* [Source Code Management Best Practices Guide](https://best.openssf.org/SCM-BestPractices/) - Guide for securing and implementing best practices for SCM platforms, including GitHub and GitLab. * [Secure AI Assistant Instructions](https://best.openssf.org/Security-Focused-Guide-for-AI-Code-Assistant-Instructions) - a guide for how to write instructions for AI code assistants so that they generate more secure code. You can also see the full list of [Guides released by the OpenSSF](https://openssf.org/resources/guides/). From b85b1b3808e84091a000e0ca818067484ea40c68 Mon Sep 17 00:00:00 2001 From: balteravishay Date: Wed, 30 Jul 2025 14:37:54 +0100 Subject: [PATCH 08/31] fix comments Signed-off-by: balteravishay --- ...Security-Focused-Guide-for-AI-Code-Assistant-Instructions.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/Security-Focused-Guide-for-AI-Code-Assistant-Instructions.md b/docs/Security-Focused-Guide-for-AI-Code-Assistant-Instructions.md index c6ba6df1..5032a477 100644 --- a/docs/Security-Focused-Guide-for-AI-Code-Assistant-Instructions.md +++ b/docs/Security-Focused-Guide-for-AI-Code-Assistant-Instructions.md @@ -21,7 +21,7 @@ By keeping these points in mind, you can harness AI code assistants effectively One of the first sections in your instructions should reinforce general secure coding best practices. These principles apply to all languages and frameworks, and you want the AI to **always** keep them in mind when generating code: -* **Input Validation & Output Encoding:** Instruct the AI to treat all external inputs as untrusted and to validate them. *Example: "user inputs should be checked for expected format and length*". Any output should be properly encoded to prevent injection attacks such as SQL injection or cross-site scripting (XSS). *Example: "Always validate function arguments and use parameterized queries for database access"* and *"Escape special characters in user-generated content before rendering it in HTML"*. Similarly, specify that when generating output contexts such as HTML or SQL, the assistant should use safe frameworks or encoding functions to avoid vulnerabilities. [[8]](#8) [[9]](#9) [[10]](#10) +* **Input Validation & Output Encoding:** Instruct the AI to treat all external inputs as untrusted and to validate them. *Example: "user inputs should be checked for expected format and length"*. Any output should be properly encoded to prevent injection attacks such as SQL injection or cross-site scripting (XSS). *Example: "Always validate function arguments and use parameterized queries for database access"* and *"Escape special characters in user-generated content before rendering it in HTML"*. Similarly, specify that when generating output contexts such as HTML or SQL, the assistant should use safe frameworks or encoding functions to avoid vulnerabilities. [[8]](#8) [[9]](#9) [[10]](#10) * **Authentication, Authorization & Secrets Management:** Emphasize that credentials and sensitive tokens must never be hard-coded or exposed. Your instructions can say: *"Never include API keys, passwords, or secrets in code output, and use environment variables or secure vault references instead"*. Also instruct the AI to use secure authentication flows (for instance, using industry-standard libraries for handling passwords or tokens) and to enforce role-based access checks where appropriate. [[11]](#11) [[12]](#12) [[13]](#13) * **Error Handling & Logging:** Guide the AI to implement errors securely by catching exceptions and failures without revealing sensitive info (stack traces, server paths, etc.) to the end-user. In your instructions, you might include: *"When generating code, handle errors gracefully and log them, but do not expose internal details or secrets in error messages".* This ensures the assistant's suggestions include secure error-handling patterns (like generic user-facing messages and detailed logs only on the server side). Additionally, instruct the AI to use logging frameworks that can be configured for security (e.g. avoiding logging of personal data or secrets). [[14]](#14) * **Secure Defaults & Configurations:** Include guidance such as: *"Prefer safe defaults in configurations – for example, use HTTPS by default, require strong encryption algorithms, and disable insecure protocols or options".* By specifying this, the AI will be more likely to generate code that opts-in to security features. Always instruct the AI to follow the principle of least privilege (e.g. minimal file system permissions, least-privileged user accounts for services, etc.) in any configuration or code it proposes. [[15]](#15) [[16]](#16) From 4186845693b0d80a284a469b0fc9bf1227bf16c4 Mon Sep 17 00:00:00 2001 From: Avishay Balter Date: Fri, 1 Aug 2025 12:55:00 +0100 Subject: [PATCH 09/31] Update docs/Security-Focused-Guide-for-AI-Code-Assistant-Instructions.md Co-authored-by: David A. Wheeler Signed-off-by: Avishay Balter Signed-off-by: balteravishay --- ...Security-Focused-Guide-for-AI-Code-Assistant-Instructions.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/Security-Focused-Guide-for-AI-Code-Assistant-Instructions.md b/docs/Security-Focused-Guide-for-AI-Code-Assistant-Instructions.md index 5032a477..4810cbeb 100644 --- a/docs/Security-Focused-Guide-for-AI-Code-Assistant-Instructions.md +++ b/docs/Security-Focused-Guide-for-AI-Code-Assistant-Instructions.md @@ -35,7 +35,7 @@ One of the first sections in your instructions should reinforce general secure c Modern software heavily relies on third-party libraries and dependencies. It's crucial that your AI assistant's instructions cover supply chain security, ensuring that suggested dependencies and build processes are secure: -* **Safe Dependency Selection:** Instruct the AI to prefer well-vetted, reputable libraries when suggesting code that pulls in external packages. For example: *"Use popular, community-trusted libraries for common tasks (and avoid adding obscure dependencies if a standard library or well-known package can do the same job)".* Emphasize evaluating packages before use – as a developer would manually. [[22]](#22) +* **Safe Dependency Selection:** Instruct the AI to prefer well-vetted, reputable libraries when suggesting code that pulls in external packages. This is especially important to counter hallucinated package names; one study found 19.7% proposed packages did not exist [[48]](#48). These hallucinations enable "slopsquatting" attacks, where attackers create malicious packages with names commonly hallucinated by AI models [[49]](#49). While it's important to independently check new dependencies, AI tools can often self-identify these names if asked to do so [[50]](#50). For example: *"Use popular, community-trusted libraries for common tasks (and avoid adding obscure dependencies if a standard library or well-known package can do the same job). Avoid adding dependencies that may be malicious or hallucinated.".* Emphasize evaluating packages before use – as a developer would manually. [[22]](#22) * **Use Package Managers & Lock Versions:** Your instructions should tell the AI to use proper package management. For instance: *"Always use the official package manager for the given language (npm, pip, Maven, etc.) to install libraries, rather than copying code snippets".* Also, instruct it to specify version ranges or exact versions that are known to be secure. By doing so, the AI will generate code that, for example, uses a `requirements.txt` or `package.json` entry, which aids in maintaining supply chain integrity. [[23]](#23) * **Stay Updated & Monitor Vulnerabilities:** Include guidance for keeping dependencies up-to-date. For example: *"When suggesting dependency versions, prefer the latest stable release and mention updating dependencies regularly to patch vulnerabilities"*. [[24]](#24) [[25]](#25) * **Generate Software Bill of Materials (SBOM):** Instruct the AI to create and maintaining SBOMs for better visibility into your software supply chain. For example: *"Generate a Software Bill of Materials (SBOM) by using tools that support standard formats like SPDX or CycloneDX".* You can also mention provenance tracking: *"Where applicable, use in-toto attestations or similar frameworks to create verifiable records of your build and deployment processes".* This ensures comprehensive tracking of what goes into your software and provides the foundation for ongoing vulnerability monitoring and incident response. [[26]](#26) From 35b058e41f65cad59d084e385149cc50f6db4bea Mon Sep 17 00:00:00 2001 From: Avishay Balter Date: Fri, 1 Aug 2025 12:55:10 +0100 Subject: [PATCH 10/31] Update docs/Security-Focused-Guide-for-AI-Code-Assistant-Instructions.md Co-authored-by: David A. Wheeler Signed-off-by: Avishay Balter Signed-off-by: balteravishay --- ...ty-Focused-Guide-for-AI-Code-Assistant-Instructions.md | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/docs/Security-Focused-Guide-for-AI-Code-Assistant-Instructions.md b/docs/Security-Focused-Guide-for-AI-Code-Assistant-Instructions.md index 4810cbeb..8f3884f0 100644 --- a/docs/Security-Focused-Guide-for-AI-Code-Assistant-Instructions.md +++ b/docs/Security-Focused-Guide-for-AI-Code-Assistant-Instructions.md @@ -179,3 +179,11 @@ encryption can expose passwords, personal information, and financial data... If [46] "Automated vulnerability scanners or approaches like chatbots that critically question the generated source code ('source code critics') can reduce the risk" (ANSSI, BSI - [AI Coding Assistants](https://www.bsi.bund.de/SharedDocs/Downloads/EN/BSI/KI/ANSSI_BSI_AI_Coding_Assistants.pdf?__blob=publicationFile&v=7)) [47] "... post-processing the output ... has a measurable impact on code quality, and is LLM-agnostic... Presumably, non-LLM static analyzers or linters may be integrated as part of the code generation procedure to provide checks along the way and avoid producing code that is visibly incorrect or dangerous" (Frontiers - [A systematic literature review on the impact of AI models on the security of code generation](https://www.frontiersin.org/journals/big-data/articles/10.3389/fdata.2024.1386720/full)) + +[48] "These 30 tests generated a total of 2.23 million packages in response to our prompts, of which 440,445 (19.7%) were determined to be hallucinations, including 205,474 unique non-existent packages (i.e. packages that do not exist in PyPI or npm repositories and were distinct entries in the hallucination count, irrespective of their multiple occurrences) (Spracklen - [We Have a Package for You! A Comprehensive Analysis of Package Hallucinations by Code Generating LLMs](https://arxiv.org/abs/2406.10279)) + +[49] "A new class of supply chain attacks named 'slopsquatting' has emerged from the increased use of generative AI tools for coding and the model's tendency to "hallucinate" non-existent package names. The term slopsquatting was coined by security researcher [Seth Larson](https://mastodon.social/@andrewnez/114302875075999244) as a spin on typosquatting, an attack method that tricks developers into installing malicious packages by using names that closely resemble popular libraries. Unlike typosquatting, slopsquatting doesn't rely on misspellings. Instead, threat actors could create malicious packages on indexes like PyPI and npm named after ones commonly made up by AI models in coding examples" (Toulas - [AI-hallucinated code dependencies become new supply chain risk](https://www.bleepingcomputer.com/news/security/ai-hallucinated-code-dependencies-become-new-supply-chain-risk/) + +[50] "3 of the 4 models ... proved to be highly adept in detecting their own hallucinations with detection accuracy above 75%. Table 2 displays the recall and precision values for this test, with similarly strong performance across the 3 proficient models. This phenomenon implies that each model’s specific error patterns are detectable by the same mechanisms that generate them, suggesting an inherent self-regulatory capability. The indication that these models have an implicit understanding of their own generative patterns that could be leveraged for self-improvement is an important finding for developing mitigation strategies." (Spracklen - [We Have a Package for You! A Comprehensive Analysis of Package Hallucinations by Code Generating LLMs](https://arxiv.org/abs/2406.10279)) + +[51] "). Across all the examined LLMs, the persona/memetic proxy approach has led to the highest average number of security weaknesses among all the evaluated prompting techniques excluding the baseline prompt that does not include any security specifications." (Tony - [Prompting Techniques for Secure Code Generation: A Systematic Investigation](https://arxiv.org/abs/2407.07064v2)) From 02b8f333ca6e74017c4ec71c7b2ee9f920dc3a66 Mon Sep 17 00:00:00 2001 From: Avishay Balter Date: Fri, 1 Aug 2025 12:55:20 +0100 Subject: [PATCH 11/31] Update docs/Security-Focused-Guide-for-AI-Code-Assistant-Instructions.md Co-authored-by: David A. Wheeler Signed-off-by: Avishay Balter Signed-off-by: balteravishay --- ...Security-Focused-Guide-for-AI-Code-Assistant-Instructions.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/Security-Focused-Guide-for-AI-Code-Assistant-Instructions.md b/docs/Security-Focused-Guide-for-AI-Code-Assistant-Instructions.md index 8f3884f0..3075c103 100644 --- a/docs/Security-Focused-Guide-for-AI-Code-Assistant-Instructions.md +++ b/docs/Security-Focused-Guide-for-AI-Code-Assistant-Instructions.md @@ -1,7 +1,7 @@ # Security-Focused Guide for AI Code Assistant Instructions AI code assistants can significantly speed up development, but they need guidance to produce **secure** and robust code. This guide explains how to create custom instructions (e.g. [GitHub Copilot instructions file](https://docs.github.com/en/copilot/how-tos/custom-instructions/adding-repository-custom-instructions-for-github-copilot), [Cline instructions file](https://docs.cline.bot/enterprise-solutions/custom-instructions), [Cursor rules](https://docs.cursor.com/context/rules), [Claude markdown](https://docs.anthropic.com/en/docs/claude-code/common-workflows#create-an-effective-claude-md-file), etc.) that ensure the AI assistant accounts for application code security, supply chain safety, and platform or language-specific considerations and to embed a "security conscience" into the tool. In practice, this means fewer vulnerabilities making it into your codebase. Remember that these instructions should be kept concise, specific, and actionable – the goal is to influence the AI's behaviour without overwhelming it. [[1]](#1) - +These recommendations are based on expert opinion and various recommendations in the literature. We encourage experimentation and feedback to improve these recommendations. We, as an industry, are together learning how to best use these tools. --- ## TL;DR From 36a3b6e9e8fb7b38ee06b7b062f4634791501503 Mon Sep 17 00:00:00 2001 From: Avishay Balter Date: Fri, 1 Aug 2025 12:56:01 +0100 Subject: [PATCH 12/31] Update docs/Security-Focused-Guide-for-AI-Code-Assistant-Instructions.md Co-authored-by: David A. Wheeler Signed-off-by: Avishay Balter Signed-off-by: balteravishay --- .../Security-Focused-Guide-for-AI-Code-Assistant-Instructions.md | 1 + 1 file changed, 1 insertion(+) diff --git a/docs/Security-Focused-Guide-for-AI-Code-Assistant-Instructions.md b/docs/Security-Focused-Guide-for-AI-Code-Assistant-Instructions.md index 3075c103..3369d638 100644 --- a/docs/Security-Focused-Guide-for-AI-Code-Assistant-Instructions.md +++ b/docs/Security-Focused-Guide-for-AI-Code-Assistant-Instructions.md @@ -29,6 +29,7 @@ One of the first sections in your instructions should reinforce general secure c * **Call out and review stubbed code:** If the AI generates code that is stubbed or incomplete, instruct it to flag these areas for review. For example: *"If you generate placeholder code (e.g., `TODO` comments), ensure it is marked for security review before deployment".* This will help ensure that any incomplete code does not inadvertently introduce vulnerabilities. [[19]](#19) [[20]](#20) * **Data Protection:** When generating code, always prioritize data minimization and avoid storing or processing confidential or otherwise sensitive information (like personal data - PII) unless absolutely necessary. For that case, suggest strong encryption at rest and in transit, and recommend techniques like anonymization. For example: *"Generate a function that securely handles user input for a registration form, asking for necessary fields to avoid logging sensitive information of PII. Ensure that no sensitive or PII is stored in plaintext"*. [[21]](#21) +Note that we are *not* currently recommending in the general case that the AI be told to respond from a particular viewpoint (e.g., a role or persona) or character aka "persona pattern/memetic proxy". An example of this approach would be the instruction "Act as a software security expert. Provide outputs that a security expert would give." One set of experiments found that this approach performs poorly, producing the worst number of security weaknesses compared to other approaches. [[51]](#51) However, we encourage continued experimentation, and may change our recommendations based on future information. --- ## **Addressing Software Supply Chain Security** From d118e9bcd1768a49e6b0930a721fdbbe075077c2 Mon Sep 17 00:00:00 2001 From: balteravishay Date: Fri, 1 Aug 2025 13:56:30 +0100 Subject: [PATCH 13/31] pr comments Signed-off-by: balteravishay --- ...uide-for-AI-Code-Assistant-Instructions.md | 167 +++++++++--------- docs/index.md | 2 +- 2 files changed, 85 insertions(+), 84 deletions(-) diff --git a/docs/Security-Focused-Guide-for-AI-Code-Assistant-Instructions.md b/docs/Security-Focused-Guide-for-AI-Code-Assistant-Instructions.md index 3369d638..37af8838 100644 --- a/docs/Security-Focused-Guide-for-AI-Code-Assistant-Instructions.md +++ b/docs/Security-Focused-Guide-for-AI-Code-Assistant-Instructions.md @@ -1,17 +1,18 @@ # Security-Focused Guide for AI Code Assistant Instructions -AI code assistants can significantly speed up development, but they need guidance to produce **secure** and robust code. This guide explains how to create custom instructions (e.g. [GitHub Copilot instructions file](https://docs.github.com/en/copilot/how-tos/custom-instructions/adding-repository-custom-instructions-for-github-copilot), [Cline instructions file](https://docs.cline.bot/enterprise-solutions/custom-instructions), [Cursor rules](https://docs.cursor.com/context/rules), [Claude markdown](https://docs.anthropic.com/en/docs/claude-code/common-workflows#create-an-effective-claude-md-file), etc.) that ensure the AI assistant accounts for application code security, supply chain safety, and platform or language-specific considerations and to embed a "security conscience" into the tool. In practice, this means fewer vulnerabilities making it into your codebase. Remember that these instructions should be kept concise, specific, and actionable – the goal is to influence the AI's behaviour without overwhelming it. [[1]](#1) +AI code assistants can significantly speed up development, but they need guidance to produce **secure** and robust code. This guide explains how to create custom instructions (e.g. [GitHub Copilot instructions file](https://docs.github.com/en/copilot/how-tos/custom-instructions/adding-repository-custom-instructions-for-github-copilot), [Cline instructions file](https://docs.cline.bot/enterprise-solutions/custom-instructions), [Cursor rules](https://docs.cursor.com/context/rules), [Claude markdown](https://docs.anthropic.com/en/docs/claude-code/common-workflows#create-an-effective-claude-md-file), etc.) that ensure the AI assistant accounts for application code security, supply chain safety, and platform or language-specific considerations and to embed a "security conscience" into the tool. In practice, this means fewer vulnerabilities making it into your codebase. Remember that these instructions should be kept concise, specific, and actionable – the goal is to influence the AI's behaviour without overwhelming it. [[wiz2025a]](#wiz2025a) These recommendations are based on expert opinion and various recommendations in the literature. We encourage experimentation and feedback to improve these recommendations. We, as an industry, are together learning how to best use these tools. + --- ## TL;DR Short on time? Here's what really matters: -* **You Are the Pilot – AI is the Co-pilot:** The developer (you) remains in full control of the code. Critically evaluate and edit AI-generated code just as you would code written by a human colleague and never blindly accept suggestions. [[2]](#2) -* **Apply Engineering Best Practices Always:** AI-generated code isn't a shortcut around engineering processes such as code reviews, testing, static analysis, documentation, and version control discipline. [[3]](#3) -* **Be Security-Conscious:** Assume AI-written code can have bugs or vulnerabilities, because it often does. AI coding assistants can introduce security issues like using outdated cryptography or outdated dependencies, ignoring error handling, or leaking secrets. Check for any secrets or sensitive data in the suggested code. Make sure dependency suggestions are safe and not pulling in known vulnerable packages. [[4]](#4), [[5]](#5) -* **Guide the AI:** AI is a powerful assistant, but it works best with your guidance. Write clear prompts that specify security requirements. Don't hesitate to modify or reject AI outputs. Direct your AI tool to build its own instructions file based on this guide. [[6]](#6) [[7]](#7) +* **You Are the Pilot – AI is the Co-pilot:** The developer (you) remains in full control of the code. Critically evaluate and edit AI-generated code just as you would code written by a human colleague and never blindly accept suggestions. [[anssibsi2024a]](#anssibsi2024a) +* **Apply Engineering Best Practices Always:** AI-generated code isn't a shortcut around engineering processes such as code reviews, testing, static analysis, documentation, and version control discipline. [[markvero2025a]](#markvero2025a) +* **Be Security-Conscious:** Assume AI-written code can have bugs or vulnerabilities, because it often does. AI coding assistants can introduce security issues like using outdated cryptography or outdated dependencies, ignoring error handling, or leaking secrets. Check for any secrets or sensitive data in the suggested code. Make sure dependency suggestions are safe and not pulling in known vulnerable packages. [[shihchiehdai2025a]](#shihchiehdai2025a), [[anssibsi2024b]](#anssibsi2024b) +* **Guide the AI:** AI is a powerful assistant, but it works best with your guidance. Write clear prompts that specify security requirements. Don't hesitate to modify or reject AI outputs. Direct your AI tool to build its own instructions file based on this guide. [[swaroopdora2025a]](#swaroopdora2025a) [[haoyan2025a]](#haoyan2025a) By keeping these points in mind, you can harness AI code assistants effectively without sacrificing quality or security. @@ -21,26 +22,26 @@ By keeping these points in mind, you can harness AI code assistants effectively One of the first sections in your instructions should reinforce general secure coding best practices. These principles apply to all languages and frameworks, and you want the AI to **always** keep them in mind when generating code: -* **Input Validation & Output Encoding:** Instruct the AI to treat all external inputs as untrusted and to validate them. *Example: "user inputs should be checked for expected format and length"*. Any output should be properly encoded to prevent injection attacks such as SQL injection or cross-site scripting (XSS). *Example: "Always validate function arguments and use parameterized queries for database access"* and *"Escape special characters in user-generated content before rendering it in HTML"*. Similarly, specify that when generating output contexts such as HTML or SQL, the assistant should use safe frameworks or encoding functions to avoid vulnerabilities. [[8]](#8) [[9]](#9) [[10]](#10) -* **Authentication, Authorization & Secrets Management:** Emphasize that credentials and sensitive tokens must never be hard-coded or exposed. Your instructions can say: *"Never include API keys, passwords, or secrets in code output, and use environment variables or secure vault references instead"*. Also instruct the AI to use secure authentication flows (for instance, using industry-standard libraries for handling passwords or tokens) and to enforce role-based access checks where appropriate. [[11]](#11) [[12]](#12) [[13]](#13) -* **Error Handling & Logging:** Guide the AI to implement errors securely by catching exceptions and failures without revealing sensitive info (stack traces, server paths, etc.) to the end-user. In your instructions, you might include: *"When generating code, handle errors gracefully and log them, but do not expose internal details or secrets in error messages".* This ensures the assistant's suggestions include secure error-handling patterns (like generic user-facing messages and detailed logs only on the server side). Additionally, instruct the AI to use logging frameworks that can be configured for security (e.g. avoiding logging of personal data or secrets). [[14]](#14) -* **Secure Defaults & Configurations:** Include guidance such as: *"Prefer safe defaults in configurations – for example, use HTTPS by default, require strong encryption algorithms, and disable insecure protocols or options".* By specifying this, the AI will be more likely to generate code that opts-in to security features. Always instruct the AI to follow the principle of least privilege (e.g. minimal file system permissions, least-privileged user accounts for services, etc.) in any configuration or code it proposes. [[15]](#15) [[16]](#16) -* **Testing for Security:** Encourage the AI to produce or suggest tests for critical code paths including negative tests that verify that what shouldn't happen, doesn't happen. In your instructions, add: *"When applicable, generate unit tests for security-critical functions (including negative tests to ensure the code fails safely)"*. [[17]](#17) [[18]](#18) -* **Call out and review stubbed code:** If the AI generates code that is stubbed or incomplete, instruct it to flag these areas for review. For example: *"If you generate placeholder code (e.g., `TODO` comments), ensure it is marked for security review before deployment".* This will help ensure that any incomplete code does not inadvertently introduce vulnerabilities. [[19]](#19) [[20]](#20) -* **Data Protection:** When generating code, always prioritize data minimization and avoid storing or processing confidential or otherwise sensitive information (like personal data - PII) unless absolutely necessary. For that case, suggest strong encryption at rest and in transit, and recommend techniques like anonymization. For example: *"Generate a function that securely handles user input for a registration form, asking for necessary fields to avoid logging sensitive information of PII. Ensure that no sensitive or PII is stored in plaintext"*. [[21]](#21) +* **Input Validation & Output Encoding:** Instruct the AI to treat all external inputs as untrusted and to validate them. *Example: "user inputs should be checked for expected format and length"*. Any output should be properly encoded to prevent injection attacks such as SQL injection or cross-site scripting (XSS). *Example: "Always validate function arguments and use parameterized queries for database access"* and *"Escape special characters in user-generated content before rendering it in HTML"*. Similarly, specify that when generating output contexts such as HTML or SQL, the assistant should use safe frameworks or encoding functions to avoid vulnerabilities. [[swaroopdora2025b]](#swaroopdora2025b) [[wiz2025b]](#wiz2025b) [[haoyan2025b]](#haoyan2025b) +* **Authentication, Authorization & Secrets Management:** Emphasize that credentials and sensitive tokens must never be hard-coded or exposed. Your instructions can say: *"Never include API keys, passwords, or secrets in code output, and use environment variables or secure vault references instead"*. Also instruct the AI to use secure authentication flows (for instance, using industry-standard libraries for handling passwords or tokens) and to enforce role-based access checks where appropriate. [[hammondpearce2021a]](#hammondpearce2021a) [[neilperry2022a]](#neilperry2022a) [[swaroopdora2025c]](#swaroopdora2025c) +* **Error Handling & Logging:** Guide the AI to implement errors securely by catching exceptions and failures without revealing sensitive info (stack traces, server paths, etc.) to the end-user. In your instructions, you might include: *"When generating code, handle errors gracefully and log them, but do not expose internal details or secrets in error messages".* This ensures the assistant's suggestions include secure error-handling patterns (like generic user-facing messages and detailed logs only on the server side). Additionally, instruct the AI to use logging frameworks that can be configured for security (e.g. avoiding logging of personal data or secrets). [[swaroopdora2025d]](#swaroopdora2025d) +* **Secure Defaults & Configurations:** Include guidance such as: *"Prefer safe defaults in configurations – for example, use HTTPS by default, require strong encryption algorithms, and disable insecure protocols or options".* By specifying this, the AI will be more likely to generate code that opts-in to security features. Always instruct the AI to follow the principle of least privilege (e.g. minimal file system permissions, least-privileged user accounts for services, etc.) in any configuration or code it proposes. [[wiz2025c]](#wiz2025c) [[swaroopdora2025e]](#swaroopdora2025e) +* **Testing for Security:** Encourage the AI to produce or suggest tests for critical code paths including negative tests that verify that what shouldn't happen, doesn't happen. In your instructions, add: *"When applicable, generate unit tests for security-critical functions (including negative tests to ensure the code fails safely)"*. [[anssibsi2024c]](#anssibsi2024c) [[markvero2025b]](#markvero2025b) +* **Call out and review stubbed code:** If the AI generates code that is stubbed or incomplete, instruct it to flag these areas for review. For example: *"If you generate placeholder code (e.g., `TODO` comments), ensure it is marked for security review before deployment".* This will help ensure that any incomplete code does not inadvertently introduce vulnerabilities. [[fossa2023]](#fossa2023) [[anssibsi2024d]](#anssibsi2024d) +* **Data Protection:** When generating code, always prioritize data minimization and avoid storing or processing confidential or otherwise sensitive information (like personal data - PII) unless absolutely necessary. For that case, suggest strong encryption at rest and in transit, and recommend techniques like anonymization. For example: *"Generate a function that securely handles user input for a registration form, asking for necessary fields to avoid logging sensitive information of PII. Ensure that no sensitive or PII is stored in plaintext"*. [[swaroopdora2025f]](#swaroopdora2025f) -Note that we are *not* currently recommending in the general case that the AI be told to respond from a particular viewpoint (e.g., a role or persona) or character aka "persona pattern/memetic proxy". An example of this approach would be the instruction "Act as a software security expert. Provide outputs that a security expert would give." One set of experiments found that this approach performs poorly, producing the worst number of security weaknesses compared to other approaches. [[51]](#51) However, we encourage continued experimentation, and may change our recommendations based on future information. +Note that we are *not* currently recommending in the general case that the AI be told to respond from a particular viewpoint (e.g., a role or persona) or character aka "persona pattern/memetic proxy". An example of this approach would be the instruction "Act as a software security expert. Provide outputs that a security expert would give." One set of experiments found that this approach performs poorly, producing the worst number of security weaknesses compared to other approaches. [[catherinetony2024]](#catherinetony2024) However, we encourage continued experimentation, and may change our recommendations based on future information. --- ## **Addressing Software Supply Chain Security** Modern software heavily relies on third-party libraries and dependencies. It's crucial that your AI assistant's instructions cover supply chain security, ensuring that suggested dependencies and build processes are secure: -* **Safe Dependency Selection:** Instruct the AI to prefer well-vetted, reputable libraries when suggesting code that pulls in external packages. This is especially important to counter hallucinated package names; one study found 19.7% proposed packages did not exist [[48]](#48). These hallucinations enable "slopsquatting" attacks, where attackers create malicious packages with names commonly hallucinated by AI models [[49]](#49). While it's important to independently check new dependencies, AI tools can often self-identify these names if asked to do so [[50]](#50). For example: *"Use popular, community-trusted libraries for common tasks (and avoid adding obscure dependencies if a standard library or well-known package can do the same job). Avoid adding dependencies that may be malicious or hallucinated.".* Emphasize evaluating packages before use – as a developer would manually. [[22]](#22) -* **Use Package Managers & Lock Versions:** Your instructions should tell the AI to use proper package management. For instance: *"Always use the official package manager for the given language (npm, pip, Maven, etc.) to install libraries, rather than copying code snippets".* Also, instruct it to specify version ranges or exact versions that are known to be secure. By doing so, the AI will generate code that, for example, uses a `requirements.txt` or `package.json` entry, which aids in maintaining supply chain integrity. [[23]](#23) -* **Stay Updated & Monitor Vulnerabilities:** Include guidance for keeping dependencies up-to-date. For example: *"When suggesting dependency versions, prefer the latest stable release and mention updating dependencies regularly to patch vulnerabilities"*. [[24]](#24) [[25]](#25) -* **Generate Software Bill of Materials (SBOM):** Instruct the AI to create and maintaining SBOMs for better visibility into your software supply chain. For example: *"Generate a Software Bill of Materials (SBOM) by using tools that support standard formats like SPDX or CycloneDX".* You can also mention provenance tracking: *"Where applicable, use in-toto attestations or similar frameworks to create verifiable records of your build and deployment processes".* This ensures comprehensive tracking of what goes into your software and provides the foundation for ongoing vulnerability monitoring and incident response. [[26]](#26) -* **Integrity Verification:** To further secure the supply chain, you can instruct the assistant to show how to verify what it uses. For instance: *"When adding important external resources (scripts, containers, etc.), include steps to verify integrity (like checksum verification or signature validation) if applicable".* [[27]](#27) +* **Safe Dependency Selection:** Instruct the AI to prefer well-vetted, reputable libraries when suggesting code that pulls in external packages. This is especially important to counter hallucinated package names; one study found 19.7% proposed packages did not exist [[josephspracklen2024e]](#josephspracklen2024e). These hallucinations enable "slopsquatting" attacks, where attackers create malicious packages with names commonly hallucinated by AI models [[billtoulas2025a]](#billtoulas2025a). While it's important to independently check new dependencies, AI tools can often self-identify these names if asked to do so [[josephspracklen2024f]](#josephspracklen2024f). For example: *"Use popular, community-trusted libraries for common tasks (and avoid adding obscure dependencies if a standard library or well-known package can do the same job). Avoid adding dependencies that may be malicious or hallucinated.".* Emphasize evaluating packages before use – as a developer would manually. [[josephspracklen2024a]](#josephspracklen2024a) +* **Use Package Managers & Lock Versions:** Your instructions should tell the AI to use proper package management. For instance: *"Always use the official package manager for the given language (npm, pip, Maven, etc.) to install libraries, rather than copying code snippets".* Also, instruct it to specify version ranges or exact versions that are known to be secure. By doing so, the AI will generate code that, for example, uses a `requirements.txt` or `package.json` entry, which aids in maintaining supply chain integrity. [[openssf2023a]](#openssf2023a) +* **Stay Updated & Monitor Vulnerabilities:** Include guidance for keeping dependencies up-to-date. For example: *"When suggesting dependency versions, prefer the latest stable release and mention updating dependencies regularly to patch vulnerabilities"*. [[arifulhaque2025a]](#arifulhaque2025a) [[josephspracklen2024b]](#josephspracklen2024b) +* **Generate Software Bill of Materials (SBOM):** Instruct the AI to create and maintaining SBOMs for better visibility into your software supply chain. For example: *"Generate a Software Bill of Materials (SBOM) by using tools that support standard formats like SPDX or CycloneDX".* You can also mention provenance tracking: *"Where applicable, use in-toto attestations or similar frameworks to create verifiable records of your build and deployment processes".* This ensures comprehensive tracking of what goes into your software and provides the foundation for ongoing vulnerability monitoring and incident response. [[anssibsi2024e]](#anssibsi2024e) +* **Integrity Verification:** To further secure the supply chain, you can instruct the assistant to show how to verify what it uses. For instance: *"When adding important external resources (scripts, containers, etc.), include steps to verify integrity (like checksum verification or signature validation) if applicable".* [[josephspracklen2024c]](#josephspracklen2024c) --- @@ -48,11 +49,11 @@ Modern software heavily relies on third-party libraries and dependencies. It's c Every software runs somewhere – be it a web browser, a cloud platform, an IoT device, or an OS. Your guide for the AI assistant should have a section addressing platform-specific and runtime security concerns: -* **Operating System and Environment:** If your code will run on a server or user machine, instruct the AI on safe system interactions. *"When writing file or OS-level operations, use safe functions and check for errors (e.g., use secure file modes, avoid temp files without proper randomness, etc.). If running as a service, drop privileges when possible".* This ensures the AI's output respects security at the OS level (for example, not recommending running as root unless absolutely necessary). Emphasize least privilege: for instance, code for a service should use the minimal privileges needed. The assistant could then suggest using system APIs that limit permissions or show how to configure a Linux capability instead of full root access. [[28]](#28) [[29]](#29) -* **Web and Cloud Platforms:** For web applications, direct the AI to include web security best practices. *"Always include appropriate security headers (Content Security Policy, X-Frame-Options, etc.) in web responses, and use frameworks' built-in protections for cookies and sessions".* You can also mention using HTTPS for any client-server communication. On cloud platforms, instruct it on secure use of cloud services: *"When generating code for cloud services (AWS/Azure/GCP), follow the provider's security guidelines (e.g., use parameterized queries for cloud databases, encrypt data at rest and in transit, handle keys via cloud KMS)".* [[30]](#30) [[31]](#31) -* **Container and Deployment Considerations:** If your project uses containers or orchestrators (like Docker, Kubernetes), include guidance like: *"Use minimal base images and avoid running containers with the root user. Use official images from trusted sources, and pin image versions using immutable digests (e.g., SHA256 hashes) instead of mutable tags like latest".* This will cause the AI to generate Dockerfiles or deployment scripts that adhere to container security best practices and ties into supply chain concerns as well – using only trusted images and specific versions that ensure consistency and prevent supply chain attacks. Additionally, instruct the AI on container signature verification: *"When working with container images, verify both the integrity and authenticity of images using container signing tools like cosign or notation. Include steps to verify signatures from trusted publishers and implement admission controllers in Kubernetes to enforce signature verification policies".* Instruct the AI to further control deployment artifacts. For instance, by ensuring production websites only load resources from trusted domains to avoid subversion: *"When generating HTML/JS, do not include direct links to untrusted third-party hosts for critical libraries; use our locally hosted or CDN with integrity checks".* [[32]](#32) [[33]](#33) -* **Mobile and Desktop App Security:** If relevant, instruct on platform-specific secure coding. For mobile apps, you might say: *"Do not suggest storing sensitive data in plaintext on the device; use the platform's secure storage APIs".* For desktop apps, *"Prefer high-level libraries for cryptography rather than rolling your own".* These platform notes ensure the AI isn't blind to the context in which code runs. It will include, for example, usage of Android's `SharedPreferences` with encryption or Apple's Keychain for iOS if dealing with credentials, reflecting platform best practices. [[34]](#34) -* **Operational Security:** Instruct the AI to consider devops security aspects. For example: *"When generating github actions or CI/CD pipelines, ensure secrets are stored securely (e.g., using GitHub Secrets or environment variables) and not hard-coded in the workflow files"*. This will help the AI generate secure CI/CD configurations that prevent accidental exposure of sensitive information. You can also instruct it to include security checks in the pipeline: *"Include steps to run security scans (SAST/DAST) and dependency checks in the CI/CD pipeline to catch vulnerabilities early"*. This ensures that the AI's generated code integrates security into the development lifecycle, rather than treating it as an afterthought. Treat devops tools with the same security rigor as application code. For example, instruct the AI to use secure coding practices in infrastructure-as-code (IaC) scripts, such as Terraform or CloudFormation: *"When generating IaC scripts, ensure they follow security best practices (e.g., restrict access to resources, use secure storage for secrets, and validate inputs)"* and *"use the latest versions of devops dependencies such as GitHub actions and lock the version to specific SHA"*. [[35]](#35) [[36]](#36) +* **Operating System and Environment:** If your code will run on a server or user machine, instruct the AI on safe system interactions. *"When writing file or OS-level operations, use safe functions and check for errors (e.g., use secure file modes, avoid temp files without proper randomness, etc.). If running as a service, drop privileges when possible".* This ensures the AI's output respects security at the OS level (for example, not recommending running as root unless absolutely necessary). Emphasize least privilege: for instance, code for a service should use the minimal privileges needed. The assistant could then suggest using system APIs that limit permissions or show how to configure a Linux capability instead of full root access. [[arifulhaque2025b]](#arifulhaque2025b) [[shihchiehdai2025b]](#shihchiehdai2025b) +* **Web and Cloud Platforms:** For web applications, direct the AI to include web security best practices. *"Always include appropriate security headers (Content Security Policy, X-Frame-Options, etc.) in web responses, and use frameworks' built-in protections for cookies and sessions".* You can also mention using HTTPS for any client-server communication. On cloud platforms, instruct it on secure use of cloud services: *"When generating code for cloud services (AWS/Azure/GCP), follow the provider's security guidelines (e.g., use parameterized queries for cloud databases, encrypt data at rest and in transit, handle keys via cloud KMS)".* [[swaroopdora2025g]](#swaroopdora2025g) [[neilperry2022b]](#neilperry2022b) +* **Container and Deployment Considerations:** If your project uses containers or orchestrators (like Docker, Kubernetes), include guidance like: *"Use minimal base images and avoid running containers with the root user. Use official images from trusted sources, and pin image versions using immutable digests (e.g., SHA256 hashes) instead of mutable tags like latest".* This will cause the AI to generate Dockerfiles or deployment scripts that adhere to container security best practices and ties into supply chain concerns as well – using only trusted images and specific versions that ensure consistency and prevent supply chain attacks. Additionally, instruct the AI on container signature verification: *"When working with container images, verify both the integrity and authenticity of images using container signing tools like cosign or notation. Include steps to verify signatures from trusted publishers and implement admission controllers in Kubernetes to enforce signature verification policies".* Instruct the AI to further control deployment artifacts. For instance, by ensuring production websites only load resources from trusted domains to avoid subversion: *"When generating HTML/JS, do not include direct links to untrusted third-party hosts for critical libraries; use our locally hosted or CDN with integrity checks".* [[anssibsi2024f]](#anssibsi2024f) [[josephspracklen2024d]](#josephspracklen2024d) +* **Mobile and Desktop App Security:** If relevant, instruct on platform-specific secure coding. For mobile apps, you might say: *"Do not suggest storing sensitive data in plaintext on the device; use the platform's secure storage APIs".* For desktop apps, *"Prefer high-level libraries for cryptography rather than rolling your own".* These platform notes ensure the AI isn't blind to the context in which code runs. It will include, for example, usage of Android's `SharedPreferences` with encryption or Apple's Keychain for iOS if dealing with credentials, reflecting platform best practices. [[hammondpearce2021b]](#hammondpearce2021b) +* **Operational Security:** Instruct the AI to consider devops security aspects. For example: *"When generating github actions or CI/CD pipelines, ensure secrets are stored securely (e.g., using GitHub Secrets or environment variables) and not hard-coded in the workflow files"*. This will help the AI generate secure CI/CD configurations that prevent accidental exposure of sensitive information. You can also instruct it to include security checks in the pipeline: *"Include steps to run security scans (SAST/DAST) and dependency checks in the CI/CD pipeline to catch vulnerabilities early"*. This ensures that the AI's generated code integrates security into the development lifecycle, rather than treating it as an afterthought. Treat devops tools with the same security rigor as application code. For example, instruct the AI to use secure coding practices in infrastructure-as-code (IaC) scripts, such as Terraform or CloudFormation: *"When generating IaC scripts, ensure they follow security best practices (e.g., restrict access to resources, use secure storage for secrets, and validate inputs)"* and *"use the latest versions of devops dependencies such as GitHub actions and lock the version to specific SHA"*. [[anssibsi2024g]](#anssibsi2024g) [[openssf2023b]](#openssf2023b) --- @@ -60,10 +61,10 @@ Every software runs somewhere – be it a web browser, a cloud platform, an IoT It's valuable to dedicate individual paragraphs to language-specific security considerations that are relevant to your project. The following section serves as examples, or special cases the AI should handle to highlight unique vulnerabilities or features of that language and how the AI should approach them: -* **C/C++ (Memory-Unsafe Languages):** For languages without automatic memory safety, instruct the AI to be extra cautious with memory management. *"In C or C++ code, always use bounds-checked functions (e.g., `strncpy` over `strcpy`), avoid dangerous functions like `gets`, and include buffer size constants to prevent overflow. Enable compiler defenses (stack canaries, fortify source, DEP/NX) in any build configurations you suggest"*. By giving such instructions, the assistant might prefer safer standard library calls or even suggest modern C++ classes (`std::vector` instead of raw arrays) to reduce manual memory handling. It will also acknowledge when an operation is risky, possibly inserting comments like "// ensure no buffer overflow". [[37]](#37) [[38]](#38) -* **Rust, Go, and Memory-Safe Languages:** If the project involves memory-safe languages (Rust, Go, Java, C\#, etc.), you can note that the AI should leverage their safety features. *"In Rust code, avoid using `unsafe` blocks unless absolutely necessary and document any `unsafe` usage with justification".* Memory-safe-by-default languages enforce a lot at compile time, but you should still have the AI follow best practices of those ecosystems. For example, instruct: *"In any memory-safe language, prefer using safe library functions and types; don't circumvent their safety without cause".* If a language offers any tools to verify memory access, direct the AI assistant to use them while building or testing your code. For Example "*In go code, use the data race detector when building the application*". [[39]](#39) -* **Python and Dynamic Languages:** Python, JavaScript, and other high-level languages manage memory for you, but come with their own security pitfalls. In your instructions, emphasize things like avoiding exec/eval with untrusted input in Python and being careful with command execution. *"For Python, do not use `exec`/`eval` on user input and prefer safe APIs (e.g., use the `subprocess` module with `shell=False` to avoid shell injection)".* Additionally, mention type checking or the use of linters: *"Follow PEP 8 and use type hints, as this can catch misuse early".* For JavaScript/TypeScript, you might add: *"When generating Node.js code, use prepared statements for database queries (just like any other language) and encode any data that goes into HTML to prevent XSS".* These instructions incorporate known best practices (like those from OWASP cheat sheets) directly into the AI's behavior. [[40]](#40) -* **Java/C\# and Enterprise Languages:** In languages often used for large applications, you might focus on frameworks and configurations. *"For Java, when suggesting web code (e.g., using Spring), ensure to use built-in security annotations and avoid old, vulnerable libraries (e.g., use `BCryptPasswordEncoder` rather than writing a custom password hash)".* For C\#, similarly: *"Use .NET's cryptography and identity libraries instead of custom solutions".* Also instruct about managing object deserialization (both Java and C\# have had vulnerabilities in this area): *"Never suggest turning off security features like XML entity security or type checking during deserialization".* These language-specific notes guide the AI to incorporate the well-known secure patterns of each ecosystem. [[41]](#41) +* **C/C++ (Memory-Unsafe Languages):** For languages without automatic memory safety, instruct the AI to be extra cautious with memory management. *"In C or C++ code, always use bounds-checked functions (e.g., `strncpy` over `strcpy`), avoid dangerous functions like `gets`, and include buffer size constants to prevent overflow. Enable compiler defenses (stack canaries, fortify source, DEP/NX) in any build configurations you suggest"*. By giving such instructions, the assistant might prefer safer standard library calls or even suggest modern C++ classes (`std::vector` instead of raw arrays) to reduce manual memory handling. It will also acknowledge when an operation is risky, possibly inserting comments like "// ensure no buffer overflow". [[connordilgren2025a]](#connordilgren2025a) [[neilperry2022c]](#neilperry2022c) +* **Rust, Go, and Memory-Safe Languages:** If the project involves memory-safe languages (Rust, Go, Java, C\#, etc.), you can note that the AI should leverage their safety features. *"In Rust code, avoid using `unsafe` blocks unless absolutely necessary and document any `unsafe` usage with justification".* Memory-safe-by-default languages enforce a lot at compile time, but you should still have the AI follow best practices of those ecosystems. For example, instruct: *"In any memory-safe language, prefer using safe library functions and types; don't circumvent their safety without cause".* If a language offers any tools to verify memory access, direct the AI assistant to use them while building or testing your code. For Example "*In go code, use the data race detector when building the application*". [[openssf2023c]](#openssf2023c) +* **Python and Dynamic Languages:** Python, JavaScript, and other high-level languages manage memory for you, but come with their own security pitfalls. In your instructions, emphasize things like avoiding exec/eval with untrusted input in Python and being careful with command execution. *"For Python, do not use `exec`/`eval` on user input and prefer safe APIs (e.g., use the `subprocess` module with `shell=False` to avoid shell injection)".* Additionally, mention type checking or the use of linters: *"Follow PEP 8 and use type hints, as this can catch misuse early".* For JavaScript/TypeScript, you might add: *"When generating Node.js code, use prepared statements for database queries (just like any other language) and encode any data that goes into HTML to prevent XSS".* These instructions incorporate known best practices (like those from OWASP cheat sheets) directly into the AI's behavior. [[haoyan2025c]](#haoyan2025c) +* **Java/C\# and Enterprise Languages:** In languages often used for large applications, you might focus on frameworks and configurations. *"For Java, when suggesting web code (e.g., using Spring), ensure to use built-in security annotations and avoid old, vulnerable libraries (e.g., use `BCryptPasswordEncoder` rather than writing a custom password hash)".* For C\#, similarly: *"Use .NET's cryptography and identity libraries instead of custom solutions".* Also instruct about managing object deserialization (both Java and C\# have had vulnerabilities in this area): *"Never suggest turning off security features like XML entity security or type checking during deserialization".* These language-specific notes guide the AI to incorporate the well-known secure patterns of each ecosystem. [[neilperry2022d]](#neilperry2022d) --- @@ -71,120 +72,120 @@ It's valuable to dedicate individual paragraphs to language-specific security co To strengthen your AI assistant's guidance, you should point it toward established security standards and frameworks. By referencing these in the instructions, the AI will be biased toward solutions that comply with widely accepted practices: -* **Use Industry Standards as Keywords:** Mention standards like **OWASP Top 10**, **OWASP ASVS**, **CWE/SANS Top 25**, or language-specific guidelines (Java's SEI CERT secure coding standards, for instance). For example: *"Code suggestions should adhere to OWASP Top 10 principles (e.g., avoid injection, enforce access control) and follow the OWASP ASVS requirements where applicable"*. [[42]](#42) -* **Reference SAFECode and Other Guides:** SAFECode (Software Assurance Forum for Excellence in Code) publishes **Fundamental Practices for Secure Software Development**, and including a note about it can be beneficial. *"Our project follows SAFECode's secure development practices – the AI should prioritize those (e.g., proper validation, authentication, cryptography usage per SAFECode guidance)"*. This ties the assistant's suggestions to a comprehensive set of secure development principles. Similarly, you can reference the **Secure Software Development Lifecycle (SSDLC)** or standards like **ISO 27034** if your organization uses them, to align AI outputs with those processes. [[43]](#43) -* **Compliance and Regulatory Security:** If you operate in an industry with specific regulations (like healthcare or finance), instruct the AI with those in mind. For instance: *"When generating code, consider compliance requirements (e.g., HIPAA privacy rules for medical data, PCI-DSS for credit card info) – do not output code that logs or transmits sensitive data in insecure ways"*. This ensures the AI's suggestions are compliant with external standards, for example, it might suggest encryption for personal data fields by default. [[44]](#44) -* **Continuous Improvement Hooks:** Encourage practices that integrate with ongoing security processes. For example: *"Include comments or TODOs in code suggesting security reviews for complex logic, and note if any third-party component might need a future update or audit".* This way, the AI not only writes code but also flags areas for human scrutiny or future action. If the AI outputs a piece of cryptography code, for instance, it might add a comment "// TODO: security review for cryptographic implementation" because you instructed it that critical code should not go un-reviewed. [[45]](#45) -* **Integrate and Act on Automated Security Tooling:** Instruct the AI to proactively run automated security tools (e.g., SAST, DAST, SCA) during code generation and refinement. Example: "When writing or reviewing code, run or simulate the use of tools like CodeQL, Bandit, Semgrep, or OWASP Dependency-Check. Identify any flagged vulnerabilities or outdated dependencies and revise the code accordingly. Repeat this process until the code passes all simulated scans". This positions the AI not just as a code generator but as a security-aware assistant capable of iterating based on tool feedback. [[46]](#46) [[47]](#47) +* **Use Industry Standards as Keywords:** Mention standards like **OWASP Top 10**, **OWASP ASVS**, **CWE/SANS Top 25**, or language-specific guidelines (Java's SEI CERT secure coding standards, for instance). For example: *"Code suggestions should adhere to OWASP Top 10 principles (e.g., avoid injection, enforce access control) and follow the OWASP ASVS requirements where applicable"*. [[arifulhaque2025c]](#arifulhaque2025c) +* **Reference SAFECode and Other Guides:** SAFECode (Software Assurance Forum for Excellence in Code) publishes **Fundamental Practices for Secure Software Development**, and including a note about it can be beneficial. *"Our project follows SAFECode's secure development practices – the AI should prioritize those (e.g., proper validation, authentication, cryptography usage per SAFECode guidance)"*. This ties the assistant's suggestions to a comprehensive set of secure development principles. Similarly, you can reference the **Secure Software Development Lifecycle (SSDLC)** or standards like **ISO 27034** if your organization uses them, to align AI outputs with those processes. [[neilperry2022e]](#neilperry2022e) +* **Compliance and Regulatory Security:** If you operate in an industry with specific regulations (like healthcare or finance), instruct the AI with those in mind. For instance: *"When generating code, consider compliance requirements (e.g., HIPAA privacy rules for medical data, PCI-DSS for credit card info) – do not output code that logs or transmits sensitive data in insecure ways"*. This ensures the AI's suggestions are compliant with external standards, for example, it might suggest encryption for personal data fields by default. [[arifulhaque2025d]](#arifulhaque2025d) +* **Continuous Improvement Hooks:** Encourage practices that integrate with ongoing security processes. For example: *"Include comments or TODOs in code suggesting security reviews for complex logic, and note if any third-party component might need a future update or audit".* This way, the AI not only writes code but also flags areas for human scrutiny or future action. If the AI outputs a piece of cryptography code, for instance, it might add a comment "// TODO: security review for cryptographic implementation" because you instructed it that critical code should not go un-reviewed. [[anssibsi2024h]](#anssibsi2024h) +* **Integrate and Act on Automated Security Tooling:** Instruct the AI to proactively run automated security tools (e.g., SAST, DAST, SCA) during code generation and refinement. Example: "When writing or reviewing code, run or simulate the use of tools like CodeQL, Bandit, Semgrep, or OWASP Dependency-Check. Identify any flagged vulnerabilities or outdated dependencies and revise the code accordingly. Repeat this process until the code passes all simulated scans". This positions the AI not just as a code generator but as a security-aware assistant capable of iterating based on tool feedback. [[anssibsi2024i]](#anssibsi2024i) [[frontiers2024a]](#frontiers2024a) --- ## Appendix: Citations and References -[1] "Rules files are an emerging pattern to allow you to provide standard guidance to AI coding assistants. You can use these rules to establish project, company, or developer specific context, preferences, or workflows" (Wiz Research - [Rules Files for Safer Vibe Coding](https://www.wiz.io/blog/safer-vibe-coding-rules-files)) +[wiz2025a] "Rules files are an emerging pattern to allow you to provide standard guidance to AI coding assistants. You can use these rules to establish project, company, or developer specific context, preferences, or workflows" (Wiz Research - [Rules Files for Safer Vibe Coding](https://www.wiz.io/blog/safer-vibe-coding-rules-files)) -[2] "AI coding assistants are no substitute for experienced developers. An unrestrained use of the tools can have severe security implications." (ANSSI, BSI - [AI Coding Assistants](https://www.bsi.bund.de/SharedDocs/Downloads/EN/BSI/KI/ANSSI_BSI_AI_Coding_Assistants.pdf?__blob=publicationFile&v=7)) +[anssibsi2024a] "AI coding assistants are no substitute for experienced developers. An unrestrained use of the tools can have severe security implications." (ANSSI, BSI - [AI Coding Assistants](https://www.bsi.bund.de/SharedDocs/Downloads/EN/BSI/KI/ANSSI_BSI_AI_Coding_Assistants.pdf?__blob=publicationFile&v=7)) -[3] "on average, we could successfully execute security exploits on around half of the correct programs generated by each LLM; and in less popular backend frameworks, models further struggle to generate correct and secure applications (Mark Vero, Niels Mündler, Victor Chibotaru, Veselin Raychev, Maximilian Baader, Nikola Jovanović, Jingxuan He, Martin Vechev - [Can LLMs Generate Correct and Secure Backends?](https://arxiv.org/abs/2502.11844))" +[markvero2025a] "on average, we could successfully execute security exploits on around half of the correct programs generated by each LLM; and in less popular backend frameworks, models further struggle to generate correct and secure applications (Mark Vero, Niels Mündler, Victor Chibotaru, Veselin Raychev, Maximilian Baader, Nikola Jovanović, Jingxuan He, Martin Vechev - [Can LLMs Generate Correct and Secure Backends?](https://arxiv.org/abs/2502.11844))" -[4] "One of the major concerns regarding LLM-generated code is its security. As code snippets generated by LLMs are increasingly incorporated into industrial-level software and systems, it is critical to ensure that LLM-generated code is free of vulnerabilities that could be exploited by attackers" (Shih-Chieh Dai, Jun Xu, Guanhong Tao - [A Comprehensive Study of LLM Secure Code Generation](https://arxiv.org/abs/2503.15554)) +[shihchiehdai2025a] "One of the major concerns regarding LLM-generated code is its security. As code snippets generated by LLMs are increasingly incorporated into industrial-level software and systems, it is critical to ensure that LLM-generated code is free of vulnerabilities that could be exploited by attackers" (Shih-Chieh Dai, Jun Xu, Guanhong Tao - [A Comprehensive Study of LLM Secure Code Generation](https://arxiv.org/abs/2503.15554)) -[5] "About 40% of the programs had security vulnerabilities... One cause of these security flaws is the use of outdated programs in the training data of the AI models, leading to the suggestion of outdated and insecure best practices. This is evident, for example, in password encryption, where insecure methods such as MD5 or a single iteration of SHA-256 are still often used" (ANSSI, BSI - [AI Coding Assistants](https://www.bsi.bund.de/SharedDocs/Downloads/EN/BSI/KI/ANSSI_BSI_AI_Coding_Assistants.pdf?__blob=publicationFile&v=7)) +[anssibsi2024b] "About 40% of the programs had security vulnerabilities... One cause of these security flaws is the use of outdated programs in the training data of the AI models, leading to the suggestion of outdated and insecure best practices. This is evident, for example, in password encryption, where insecure methods such as MD5 or a single iteration of SHA-256 are still often used" (ANSSI, BSI - [AI Coding Assistants](https://www.bsi.bund.de/SharedDocs/Downloads/EN/BSI/KI/ANSSI_BSI_AI_Coding_Assistants.pdf?__blob=publicationFile&v=7)) -[6] "Improve the prompt: The user should improve the prompt by indicating each and every aspect of the security parameters to derive the secure web application code from the LLMs" (Swaroop Dora, Deven Lunkad, Naziya Aslam, S. Venkatesan, Sandeep Kumar Shukla - [The Hidden Risks of LLM-Generated Web Application Code: A Security-Centric Evaluation of Code Generation Capabilities in Large Language Models](https://arxiv.org/abs/2504.20612)) +[swaroopdora2025a] "Improve the prompt: The user should improve the prompt by indicating each and every aspect of the security parameters to derive the secure web application code from the LLMs" (Swaroop Dora, Deven Lunkad, Naziya Aslam, S. Venkatesan, Sandeep Kumar Shukla - [The Hidden Risks of LLM-Generated Web Application Code: A Security-Centric Evaluation of Code Generation Capabilities in Large Language Models](https://arxiv.org/abs/2504.20612)) -[7] "These reductions suggest that newer models can effectively leverage security hints to improve code safety" (Hao Yan, Swapneel Suhas Vaidya, Xiaokuan Zhang, Ziyu Yao - [Guiding AI to Fix Its Own Flaws: An Empirical Study on LLM-Driven Secure Code Generation](https://arxiv.org/abs/2506.23034)]) +[haoyan2025a] "These reductions suggest that newer models can effectively leverage security hints to improve code safety" (Hao Yan, Swapneel Suhas Vaidya, Xiaokuan Zhang, Ziyu Yao - [Guiding AI to Fix Its Own Flaws: An Empirical Study on LLM-Driven Secure Code Generation](https://arxiv.org/abs/2506.23034)]) -[8] "Injection attacks remain one of the most critical vulnerabilities in web applications ([OWASP Top 10](https://owasp.org/Top10/A03_2021-Injection/))... LLM- +[swaroopdora2025b] "Injection attacks remain one of the most critical vulnerabilities in web applications ([OWASP Top 10](https://owasp.org/Top10/A03_2021-Injection/))... LLM- generated code must handle user input securely to prevent exploitation"(Swaroop Dora, Deven Lunkad, Naziya Aslam, S. Venkatesan, Sandeep Kumar Shukla - [The Hidden Risks of LLM-Generated Web Application Code](https://arxiv.org/abs/2504.20612)) -[9]"... systematic review shows that code injection (CWE-94) and OS command injection (CWE-78) are common in AI-generated code." (Wiz Research - [Rules Files for Safer Vibe Coding](https://www.wiz.io/blog/safer-vibe-coding-rules-files)) +[wiz2025b]"... systematic review shows that code injection (CWE-94) and OS command injection (CWE-78) are common in AI-generated code." (Wiz Research - [Rules Files for Safer Vibe Coding](https://www.wiz.io/blog/safer-vibe-coding-rules-files)) -[10]"All models frequently generate vulnerable code across diverse vulnerability types... All models tend to generate vulnerabilities for both common (top-25) and less common types" (Hao Yan, Swapneel Suhas Vaidya, Xiaokuan Zhang, Ziyu Yao - [Guiding AI to Fix Its Own Flaws: An Empirical Study on LLM-Driven Secure Code Generation](https://arxiv.org/abs/2506.23034)) +[haoyan2025b]"All models frequently generate vulnerable code across diverse vulnerability types... All models tend to generate vulnerabilities for both common (top-25) and less common types" (Hao Yan, Swapneel Suhas Vaidya, Xiaokuan Zhang, Ziyu Yao - [Guiding AI to Fix Its Own Flaws: An Empirical Study on LLM-Driven Secure Code Generation](https://arxiv.org/abs/2506.23034)) -[11]"Overall, Copilot performed poorly in this CWE (CWE-522: Insufficiently Protected Credentials)... it frequently tried to generate code which was based on the insecure 'MD5' hashing algorithm" (Hammond Pearce, Baleegh Ahmad, Benjamin Tan, Brendan Dolan-Gavitt, Ramesh Karri - [Asleep at the Keyboard? Assessing the +[hammondpearce2021a]"Overall, Copilot performed poorly in this CWE (CWE-522: Insufficiently Protected Credentials)... it frequently tried to generate code which was based on the insecure 'MD5' hashing algorithm" (Hammond Pearce, Baleegh Ahmad, Benjamin Tan, Brendan Dolan-Gavitt, Ramesh Karri - [Asleep at the Keyboard? Assessing the Security of GitHub Copilot's Code Contributions ](https://arxiv.org/abs/2108.09293)) -[12] "Participants with access to an AI assistant were ... significantly more likely to write an insecure solution ... also less likely to authenticate the +[neilperry2022a] "Participants with access to an AI assistant were ... significantly more likely to write an insecure solution ... also less likely to authenticate the final returned value" (Neil Perry, Megha Srivastava, Deepak Kumar, Dan Boneh - [Do Users Write More Insecure Code with AI Assistants?](https://arxiv.org/abs/2211.03622)) -[13] "The analysis reveals critical vulnerabilities in authentication +[swaroopdora2025c] "The analysis reveals critical vulnerabilities in authentication mechanisms, session management..." (Swaroop Dora, Deven Lunkad, Naziya Aslam, S. Venkatesan, Sandeep Kumar Shukla - [The Hidden Risks of LLM-Generated Web Application Code](https://arxiv.org/abs/2504.20612)) -[14] "Generic Error Messages: Ensuring error messages do not disclose username existence or password policies prevents attackers from gaining insights during brute-force attempts... Leaking system information through verbose error messages can provide attackers valuable insights into potential vulnerabilities within authentication systems" (Swaroop Dora, Deven Lunkad, Naziya Aslam, S. Venkatesan, Sandeep Kumar Shukla - [The Hidden Risks of LLM-Generated Web Application Code](https://arxiv.org/abs/2504.20612)) +[swaroopdora2025d] "Generic Error Messages: Ensuring error messages do not disclose username existence or password policies prevents attackers from gaining insights during brute-force attempts... Leaking system information through verbose error messages can provide attackers valuable insights into potential vulnerabilities within authentication systems" (Swaroop Dora, Deven Lunkad, Naziya Aslam, S. Venkatesan, Sandeep Kumar Shukla - [The Hidden Risks of LLM-Generated Web Application Code](https://arxiv.org/abs/2504.20612)) -[15] "It's clear that currently AI-Generated code is not secure by default... (Wiz Research - [Rules Files for Safer Vibe Coding](https://www.wiz.io/blog/safer-vibe-coding-rules-files)) +[wiz2025c] "It's clear that currently AI-Generated code is not secure by default... (Wiz Research - [Rules Files for Safer Vibe Coding](https://www.wiz.io/blog/safer-vibe-coding-rules-files)) -[16] "All models require substantial improvements in authentication security, session management, error handling and HTTP security headers to align with current industry best practices and established frameworks, such as the NIST cybersecurity guidelines " (Swaroop Dora, Deven Lunkad, Naziya Aslam, S. Venkatesan, Sandeep Kumar Shukla - [The Hidden Risks of LLM-Generated Web Application Code](https://arxiv.org/abs/2504.20612)) +[swaroopdora2025e] "All models require substantial improvements in authentication security, session management, error handling and HTTP security headers to align with current industry best practices and established frameworks, such as the NIST cybersecurity guidelines " (Swaroop Dora, Deven Lunkad, Naziya Aslam, S. Venkatesan, Sandeep Kumar Shukla - [The Hidden Risks of LLM-Generated Web Application Code](https://arxiv.org/abs/2504.20612)) -[17] "The generation of test cases is another area where coding assistants can offer support. They can automatically propose test cases and unit tests that also cover edge cases" (ANSSI, BSI - [AI Coding Assistants](https://www.bsi.bund.de/SharedDocs/Downloads/EN/BSI/KI/ANSSI_BSI_AI_Coding_Assistants.pdf?__blob=publicationFile&v=7)) +[anssibsi2024c] "The generation of test cases is another area where coding assistants can offer support. They can automatically propose test cases and unit tests that also cover edge cases" (ANSSI, BSI - [AI Coding Assistants](https://www.bsi.bund.de/SharedDocs/Downloads/EN/BSI/KI/ANSSI_BSI_AI_Coding_Assistants.pdf?__blob=publicationFile&v=7)) -[18] "Even flagship models struggle to generate correct and secure application backends" (Mark Vero, Niels Mündler, Victor Chibotaru, Veselin Raychev, Maximilian Baader, Nikola Jovanović, Jingxuan He, Martin Vechev - [Can LLMs Generate Correct and Secure Backends?](https://arxiv.org/abs/2502.11844)) +[markvero2025b] "Even flagship models struggle to generate correct and secure application backends" (Mark Vero, Niels Mündler, Victor Chibotaru, Veselin Raychev, Maximilian Baader, Nikola Jovanović, Jingxuan He, Martin Vechev - [Can LLMs Generate Correct and Secure Backends?](https://arxiv.org/abs/2502.11844)) -[19] "It's important to pay particularly close attention to auto-generated comments... make sure you understand and agree with them... It's a good practice to consider implementing a code-tagging system to differentiate between AI- and human-created code." (FOSSA - [5 Ways to Reduce GitHub Copilot Security and Legal Risks](https://fossa.com/blog/5-ways-to-reduce-github-copilot-security-and-legal-risks/)) +[fossa2023] "It's important to pay particularly close attention to auto-generated comments... make sure you understand and agree with them... It's a good practice to consider implementing a code-tagging system to differentiate between AI- and human-created code." (FOSSA - [5 Ways to Reduce GitHub Copilot Security and Legal Risks](https://fossa.com/blog/5-ways-to-reduce-github-copilot-security-and-legal-risks/)) -[20] "It might be beneficial to flag AI generated code blocks and to document the used AI tools. This might help security testing and it can also be a useful information for third-party auditors, e.g. in the context of a security certification process" (ANSSI, BSI - [AI Coding Assistants](https://www.bsi.bund.de/SharedDocs/Downloads/EN/BSI/KI/ANSSI_BSI_AI_Coding_Assistants.pdf?__blob=publicationFile&v=7)) +[anssibsi2024d] "It might be beneficial to flag AI generated code blocks and to document the used AI tools. This might help security testing and it can also be a useful information for third-party auditors, e.g. in the context of a security certification process" (ANSSI, BSI - [AI Coding Assistants](https://www.bsi.bund.de/SharedDocs/Downloads/EN/BSI/KI/ANSSI_BSI_AI_Coding_Assistants.pdf?__blob=publicationFile&v=7)) -[21] "Encryption safeguards sensitive data at rest and in transit. Weak encryption methods or lack of +[swaroopdora2025f] "Encryption safeguards sensitive data at rest and in transit. Weak encryption methods or lack of encryption can expose passwords, personal information, and financial data... If passwords are stored in plain text or hashed without salting, attackers with database access can easily decrypt credentials, leading to mass account breaches." (Swaroop Dora, Deven Lunkad, Naziya Aslam, S. Venkatesan, Sandeep Kumar Shukla - [The Hidden Risks of LLM-Generated Web Application Code](https://arxiv.org/abs/2504.20612)) -[22] "Package hallucination occurs when an LLM generates code that recommends or contains a reference to a package that does not actually exist. An adversary can exploit package hallucinations... by publishing a package to an open-source repository with the same name as the hallucinated or fictitious package and containing some malicious code/functionality" (Joseph Spracklen, Raveen Wijewickrama, A H M Nazmus Sakib, Anindya Maiti, Bimal Viswanath, Murtuza Jadliwala - [We Have a Package for You! A Comprehensive Analysis of Package Hallucinations by Code Generating LLMs](https://arxiv.org/abs/2406.10279)) +[josephspracklen2024a] "Package hallucination occurs when an LLM generates code that recommends or contains a reference to a package that does not actually exist. An adversary can exploit package hallucinations... by publishing a package to an open-source repository with the same name as the hallucinated or fictitious package and containing some malicious code/functionality" (Joseph Spracklen, Raveen Wijewickrama, A H M Nazmus Sakib, Anindya Maiti, Bimal Viswanath, Murtuza Jadliwala - [We Have a Package for You! A Comprehensive Analysis of Package Hallucinations by Code Generating LLMs](https://arxiv.org/abs/2406.10279)) -[23] "Avoid... copy reused code from other packages... instead of using a package manager to automate identifying and updating reused packages... Use a package manager to manage it, one that records the specific version numbers" (OpenSSF - [Secure Software Development Fundamentals](https://github.com/ossf/secure-sw-dev-fundamentals/blob/main/docs/lfd121.md#preparing-for-the-inevitable-vulnerabilities-in-your-dependencies)) +[openssf2023a] "Avoid... copy reused code from other packages... instead of using a package manager to automate identifying and updating reused packages... Use a package manager to manage it, one that records the specific version numbers" (OpenSSF - [Secure Software Development Fundamentals](https://github.com/ossf/secure-sw-dev-fundamentals/blob/main/docs/lfd121.md#preparing-for-the-inevitable-vulnerabilities-in-your-dependencies)) -[24] "LLM sometimes produces code that uses older ... libraries, which are not compliant with modern security. This increases the security risk... they are not updated with the latest security vulnerabilities and threats, leaving the generated code open to new attacks." (Ariful Haque, Sunzida Siddique, Md. Mahfuzur Rahman, Ahmed Rafi Hasan, Laxmi Rani Das, Marufa Kamal, Tasnim Masura, Kishor Datta Gupta - [Exploring Hallucinations and Security Risks in AI-Assisted Software Development with Insights for LLM Deployment](https://arxiv.org/abs/2502.18468v1)) +[arifulhaque2025a] "LLM sometimes produces code that uses older ... libraries, which are not compliant with modern security. This increases the security risk... they are not updated with the latest security vulnerabilities and threats, leaving the generated code open to new attacks." (Ariful Haque, Sunzida Siddique, Md. Mahfuzur Rahman, Ahmed Rafi Hasan, Laxmi Rani Das, Marufa Kamal, Tasnim Masura, Kishor Datta Gupta - [Exploring Hallucinations and Security Risks in AI-Assisted Software Development with Insights for LLM Deployment](https://arxiv.org/abs/2502.18468v1)) -[25] "LLMs cannot update themselves with new information after release and have no knowledge of the world beyond their training data cutoff date." (Joseph Spracklen, Raveen Wijewickrama, A H M Nazmus Sakib, Anindya Maiti, Bimal Viswanath, Murtuza Jadliwala - [We Have a Package for You! A Comprehensive Analysis of Package Hallucinations](https://arxiv.org/abs/2406.10279)) +[josephspracklen2024b] "LLMs cannot update themselves with new information after release and have no knowledge of the world beyond their training data cutoff date." (Joseph Spracklen, Raveen Wijewickrama, A H M Nazmus Sakib, Anindya Maiti, Bimal Viswanath, Murtuza Jadliwala - [We Have a Package for You! A Comprehensive Analysis of Package Hallucinations](https://arxiv.org/abs/2406.10279)) -[26] "The creation of a Software Bill of Materials (SBOM) allows you to retrospectively understand whether vulnerable libraries were used and enables a targeted response if a vulnerability of certain components becomes known." (ANSSI, BSI - [AI Coding Assistants](https://www.bsi.bund.de/SharedDocs/Downloads/EN/BSI/KI/ANSSI_BSI_AI_Coding_Assistants.pdf?__blob=publicationFile&v=7)) +[anssibsi2024e] "The creation of a Software Bill of Materials (SBOM) allows you to retrospectively understand whether vulnerable libraries were used and enables a targeted response if a vulnerability of certain components becomes known." (ANSSI, BSI - [AI Coding Assistants](https://www.bsi.bund.de/SharedDocs/Downloads/EN/BSI/KI/ANSSI_BSI_AI_Coding_Assistants.pdf?__blob=publicationFile&v=7)) -[27] "Public OSS repositories ... have implemented various measures, including ... software signing to mitigate the distribution of malicious packages... Self-refinement strategies... utilize the model itself to detect and refine potential hallucinations" (Joseph Spracklen, Raveen Wijewickrama, A H M Nazmus Sakib, Anindya Maiti, Bimal Viswanath, Murtuza Jadliwala - [We Have a Package for You! A Comprehensive Analysis of Package Hallucinations](https://arxiv.org/abs/2406.10279)) +[josephspracklen2024c] "Public OSS repositories ... have implemented various measures, including ... software signing to mitigate the distribution of malicious packages... Self-refinement strategies... utilize the model itself to detect and refine potential hallucinations" (Joseph Spracklen, Raveen Wijewickrama, A H M Nazmus Sakib, Anindya Maiti, Bimal Viswanath, Murtuza Jadliwala - [We Have a Package for You! A Comprehensive Analysis of Package Hallucinations](https://arxiv.org/abs/2406.10279)) -[28] "LLM often lacks the deep contextual understanding needed to create secure code, resulting in sometimes irrelevant advice for specific security contexts" (Ariful Haque, Sunzida Siddique, Md. Mahfuzur Rahman, Ahmed Rafi Hasan, Laxmi Rani Das, Marufa Kamal, Tasnim Masura, Kishor Datta Gupta - [Exploring Hallucinations and Security Risks in AI-Assisted Software Development with Insights for LLM Deployment](https://arxiv.org/abs/2502.18468v1)) +[arifulhaque2025b] "LLM often lacks the deep contextual understanding needed to create secure code, resulting in sometimes irrelevant advice for specific security contexts" (Ariful Haque, Sunzida Siddique, Md. Mahfuzur Rahman, Ahmed Rafi Hasan, Laxmi Rani Das, Marufa Kamal, Tasnim Masura, Kishor Datta Gupta - [Exploring Hallucinations and Security Risks in AI-Assisted Software Development with Insights for LLM Deployment](https://arxiv.org/abs/2502.18468v1)) -[29] "... code directly executes the user-provided shell script without checking for malicious commands... It uses the system call os.system() to execute the copy command, which is passed as a string. If the original_location or new_location variables contain malicious input, this could lead to command injection" (Shih-Chieh Dai, Jun Xu, Guanhong Tao - [A Comprehensive Study of LLM Secure Code Generation](https://arxiv.org/abs/2503.15554)) +[shihchiehdai2025b] "... code directly executes the user-provided shell script without checking for malicious commands... It uses the system call os.system() to execute the copy command, which is passed as a string. If the original_location or new_location variables contain malicious input, this could lead to command injection" (Shih-Chieh Dai, Jun Xu, Guanhong Tao - [A Comprehensive Study of LLM Secure Code Generation](https://arxiv.org/abs/2503.15554)) -[30] "Although some models implement security measures to a limited extent, none fully align with industry best practices" (Swaroop Dora, Deven Lunkad, Naziya Aslam, S. Venkatesan, Sandeep Kumar Shukla - [The Hidden Risks of LLM-Generated Web Application Code: A Security-Centric Evaluation of Code Generation Capabilities in Large Language Models](https://arxiv.org/abs/2504.20612)) +[swaroopdora2025g] "Although some models implement security measures to a limited extent, none fully align with industry best practices" (Swaroop Dora, Deven Lunkad, Naziya Aslam, S. Venkatesan, Sandeep Kumar Shukla - [The Hidden Risks of LLM-Generated Web Application Code: A Security-Centric Evaluation of Code Generation Capabilities in Large Language Models](https://arxiv.org/abs/2504.20612)) -[31] "36% of participants with access to the AI assistant writing solutions that are vulnerable to SQL injections compared to 7% of the control group" (Neil Perry, Megha Srivastava, Deepak Kumar, Dan Boneh - [Do Users Write More Insecure Code with AI Assistants?](https://arxiv.org/abs/2211.03622)) +[neilperry2022b] "36% of participants with access to the AI assistant writing solutions that are vulnerable to SQL injections compared to 7% of the control group" (Neil Perry, Megha Srivastava, Deepak Kumar, Dan Boneh - [Do Users Write More Insecure Code with AI Assistants?](https://arxiv.org/abs/2211.03622)) -[32] "... detailed information about the infrastructure and resource configuration can be sensitive. The last part is particularly relevant if approaches like Infrastructure as Code (IaC) are used" (ANSSI, BSI - [AI Coding Assistants](https://www.bsi.bund.de/SharedDocs/Downloads/EN/BSI/KI/ANSSI_BSI_AI_Coding_Assistants.pdf?__blob=publicationFile&v=7)) +[anssibsi2024f] "... detailed information about the infrastructure and resource configuration can be sensitive. The last part is particularly relevant if approaches like Infrastructure as Code (IaC) are used" (ANSSI, BSI - [AI Coding Assistants](https://www.bsi.bund.de/SharedDocs/Downloads/EN/BSI/KI/ANSSI_BSI_AI_Coding_Assistants.pdf?__blob=publicationFile&v=7)) -[33] "The open nature of these repositories... makes them an attractive platform for malware distribution. For instance, a total of 245,000 malicious code packages were discovered in open-source software repositories in 2023 alone (Joseph Spracklen, Raveen Wijewickrama, A H M Nazmus Sakib, Anindya Maiti, Bimal Viswanath, Murtuza Jadliwala - [We Have a Package for You! A Comprehensive Analysis of Package Hallucinations](https://arxiv.org/abs/2406.10279)) +[josephspracklen2024d] "The open nature of these repositories... makes them an attractive platform for malware distribution. For instance, a total of 245,000 malicious code packages were discovered in open-source software repositories in 2023 alone (Joseph Spracklen, Raveen Wijewickrama, A H M Nazmus Sakib, Anindya Maiti, Bimal Viswanath, Murtuza Jadliwala - [We Have a Package for You! A Comprehensive Analysis of Package Hallucinations](https://arxiv.org/abs/2406.10279)) -[34] "... we produce 89 different scenarios for Copilot to complete, producing 1,689 programs. Of these, we found approximately 40% to be vulnerable" (Hammond Pearce, Baleegh Ahmad, Benjamin Tan, Brendan Dolan-Gavitt, Ramesh Karri - [Asleep at the Keyboard? Assessing the Security of GitHub Copilot's Code Contributions](https://arxiv.org/abs/2108.09293)) +[hammondpearce2021b] "... we produce 89 different scenarios for Copilot to complete, producing 1,689 programs. Of these, we found approximately 40% to be vulnerable" (Hammond Pearce, Baleegh Ahmad, Benjamin Tan, Brendan Dolan-Gavitt, Ramesh Karri - [Asleep at the Keyboard? Assessing the Security of GitHub Copilot's Code Contributions](https://arxiv.org/abs/2108.09293)) -[35] "The use of uncontrolled access to coding assistants ... in the cloud should be prohibited for business software development." (ANSSI, BSI - [AI Coding Assistants](https://www.bsi.bund.de/SharedDocs/Downloads/EN/BSI/KI/ANSSI_BSI_AI_Coding_Assistants.pdf?__blob=publicationFile&v=7)) +[anssibsi2024g] "The use of uncontrolled access to coding assistants ... in the cloud should be prohibited for business software development." (ANSSI, BSI - [AI Coding Assistants](https://www.bsi.bund.de/SharedDocs/Downloads/EN/BSI/KI/ANSSI_BSI_AI_Coding_Assistants.pdf?__blob=publicationFile&v=7)) -[36] "Many projects apply a ... DevSecOps approach, which intentionally blend these processes together with development... having secure distribution, fielding, operations, and disposal is critical for software to be secure in the real world." (OpenSSF - [Secure Software Development Fundamentals](https://github.com/ossf/secure-sw-dev-fundamentals/blob/main/docs/lfd121.md#preparing-for-the-inevitable-vulnerabilities-in-your-dependencies)) +[openssf2023b] "Many projects apply a ... DevSecOps approach, which intentionally blend these processes together with development... having secure distribution, fielding, operations, and disposal is critical for software to be secure in the real world." (OpenSSF - [Secure Software Development Fundamentals](https://github.com/ossf/secure-sw-dev-fundamentals/blob/main/docs/lfd121.md#preparing-for-the-inevitable-vulnerabilities-in-your-dependencies)) -[37] "The security vulnerabilities in LLM-generated correct code generally stem from missing conditional checks, incorrect memory allocations, or incorrect conditional checks" (Connor Dilgren, Purva Chiniya, Luke Griffith, Yu Ding, Yizheng Chen - [Benchmarking LLMs for Secure Code Generation in Real-World Repositories](https://arxiv.org/abs/2504.21205)) +[connordilgren2025a] "The security vulnerabilities in LLM-generated correct code generally stem from missing conditional checks, incorrect memory allocations, or incorrect conditional checks" (Connor Dilgren, Purva Chiniya, Luke Griffith, Yu Ding, Yizheng Chen - [Benchmarking LLMs for Secure Code Generation in Real-World Repositories](https://arxiv.org/abs/2504.21205)) -[38] "... we observe that participants in the experiment group were significantly more likely to introduce integer overflow mistakes in their solutions" (Neil Perry, Megha Srivastava, Deepak Kumar, Dan Boneh - [Do Users Write More Insecure Code with AI Assistants?](https://arxiv.org/abs/2211.03622)) +[neilperry2022c] "... we observe that participants in the experiment group were significantly more likely to introduce integer overflow mistakes in their solutions" (Neil Perry, Megha Srivastava, Deepak Kumar, Dan Boneh - [Do Users Write More Insecure Code with AI Assistants?](https://arxiv.org/abs/2211.03622)) -[39] "Even when a language is memory safe by default, it is still vital to follow that language ecosystem's best practices and to use external tools to ensure safety not only within your code" (OpenSSF - [The Memory Safety Continuum](https://memorysafety.openssf.org/memory-safety-continuum/#writing-software-in-memory-safe-by-default-languages)) +[openssf2023c] "Even when a language is memory safe by default, it is still vital to follow that language ecosystem's best practices and to use external tools to ensure safety not only within your code" (OpenSSF - [The Memory Safety Continuum](https://memorysafety.openssf.org/memory-safety-continuum/#writing-software-in-memory-safe-by-default-languages)) -[40] "... we reveal that although LLMs are prone to generating insecure code, advanced models can benefit from vulnerability hints and fine-grained feedback to avoid or fix vulnerabilities." (Hao Yan, Swapneel Suhas Vaidya, Xiaokuan Zhang, Ziyu Yao - [Guiding AI to Fix Its Own Flaws: An Empirical Study on LLM-Driven Secure Code Generation](https://arxiv.org/abs/2506.23034)) +[haoyan2025c] "... we reveal that although LLMs are prone to generating insecure code, advanced models can benefit from vulnerability hints and fine-grained feedback to avoid or fix vulnerabilities." (Hao Yan, Swapneel Suhas Vaidya, Xiaokuan Zhang, Ziyu Yao - [Guiding AI to Fix Its Own Flaws: An Empirical Study on LLM-Driven Secure Code Generation](https://arxiv.org/abs/2506.23034)) -[41] "Often times, responses from the AI assistant use libraries that explicitly flag that they are insecure in the documentation for the library" (Neil Perry, Megha Srivastava, Deepak Kumar, Dan Boneh - [Do Users Write More Insecure Code with AI Assistants?](https://arxiv.org/abs/2211.03622)) +[neilperry2022d] "Often times, responses from the AI assistant use libraries that explicitly flag that they are insecure in the documentation for the library" (Neil Perry, Megha Srivastava, Deepak Kumar, Dan Boneh - [Do Users Write More Insecure Code with AI Assistants?](https://arxiv.org/abs/2211.03622)) -[42] "Additionally, we can reduce security risks by using OWASP guidelines and using trusted libraries" (Ariful Haque, Sunzida Siddique, Md. Mahfuzur Rahman, Ahmed Rafi Hasan, Laxmi Rani Das, Marufa Kamal, Tasnim Masura, Kishor Datta Gupta - [Exploring Hallucinations and Security Risks in AI-Assisted Software Development with Insights for LLM Deployment](https://arxiv.org/abs/2502.18468v1)) +[arifulhaque2025c] "Additionally, we can reduce security risks by using OWASP guidelines and using trusted libraries" (Ariful Haque, Sunzida Siddique, Md. Mahfuzur Rahman, Ahmed Rafi Hasan, Laxmi Rani Das, Marufa Kamal, Tasnim Masura, Kishor Datta Gupta - [Exploring Hallucinations and Security Risks in AI-Assisted Software Development with Insights for LLM Deployment](https://arxiv.org/abs/2502.18468v1)) -[43] "We found that those who specified task instructions... generated more secure code." (Neil Perry, Megha Srivastava, Deepak Kumar, Dan Boneh - [Do Users Write More Insecure Code with AI Assistants?](https://arxiv.org/abs/2211.03622)) +[neilperry2022e] "We found that those who specified task instructions... generated more secure code." (Neil Perry, Megha Srivastava, Deepak Kumar, Dan Boneh - [Do Users Write More Insecure Code with AI Assistants?](https://arxiv.org/abs/2211.03622)) -[44] "Compliance with regulations like GDPR and HIPAA is essential to minimize legal risks and uphold organizational reputation in sensitive environments" (Ariful Haque, Sunzida Siddique, Md. Mahfuzur Rahman, Ahmed Rafi Hasan, Laxmi Rani Das, Marufa Kamal, Tasnim Masura, Kishor Datta Gupta - [Exploring Hallucinations and Security Risks in AI-Assisted Software Development with Insights for LLM Deployment](https://arxiv.org/abs/2502.18468v1)) +[arifulhaque2025d] "Compliance with regulations like GDPR and HIPAA is essential to minimize legal risks and uphold organizational reputation in sensitive environments" (Ariful Haque, Sunzida Siddique, Md. Mahfuzur Rahman, Ahmed Rafi Hasan, Laxmi Rani Das, Marufa Kamal, Tasnim Masura, Kishor Datta Gupta - [Exploring Hallucinations and Security Risks in AI-Assisted Software Development with Insights for LLM Deployment](https://arxiv.org/abs/2502.18468v1)) -[45] "Generated content, in particular source code, should generally be reviewed and understood by the developers... It might be beneficial to 'deconstruct' AI-generated code and the used prompts in public code reviews within the company. This allows knowledge and experiences to be shared across teams." (ANSSI, BSI - [AI Coding Assistants](https://www.bsi.bund.de/SharedDocs/Downloads/EN/BSI/KI/ANSSI_BSI_AI_Coding_Assistants.pdf?__blob=publicationFile&v=7)) +[anssibsi2024h] "Generated content, in particular source code, should generally be reviewed and understood by the developers... It might be beneficial to 'deconstruct' AI-generated code and the used prompts in public code reviews within the company. This allows knowledge and experiences to be shared across teams." (ANSSI, BSI - [AI Coding Assistants](https://www.bsi.bund.de/SharedDocs/Downloads/EN/BSI/KI/ANSSI_BSI_AI_Coding_Assistants.pdf?__blob=publicationFile&v=7)) -[46] "Automated vulnerability scanners or approaches like chatbots that critically question the generated source code ('source code critics') can reduce the risk" (ANSSI, BSI - [AI Coding Assistants](https://www.bsi.bund.de/SharedDocs/Downloads/EN/BSI/KI/ANSSI_BSI_AI_Coding_Assistants.pdf?__blob=publicationFile&v=7)) +[anssibsi2024i] "Automated vulnerability scanners or approaches like chatbots that critically question the generated source code ('source code critics') can reduce the risk" (ANSSI, BSI - [AI Coding Assistants](https://www.bsi.bund.de/SharedDocs/Downloads/EN/BSI/KI/ANSSI_BSI_AI_Coding_Assistants.pdf?__blob=publicationFile&v=7)) -[47] "... post-processing the output ... has a measurable impact on code quality, and is LLM-agnostic... Presumably, non-LLM static analyzers or linters may be integrated as part of the code generation procedure to provide checks along the way and avoid producing code that is visibly incorrect or dangerous" (Frontiers - [A systematic literature review on the impact of AI models on the security of code generation](https://www.frontiersin.org/journals/big-data/articles/10.3389/fdata.2024.1386720/full)) +[frontiers2024a] "... post-processing the output ... has a measurable impact on code quality, and is LLM-agnostic... Presumably, non-LLM static analyzers or linters may be integrated as part of the code generation procedure to provide checks along the way and avoid producing code that is visibly incorrect or dangerous" (Frontiers - [A systematic literature review on the impact of AI models on the security of code generation](https://www.frontiersin.org/journals/big-data/articles/10.3389/fdata.2024.1386720/full)) -[48] "These 30 tests generated a total of 2.23 million packages in response to our prompts, of which 440,445 (19.7%) were determined to be hallucinations, including 205,474 unique non-existent packages (i.e. packages that do not exist in PyPI or npm repositories and were distinct entries in the hallucination count, irrespective of their multiple occurrences) (Spracklen - [We Have a Package for You! A Comprehensive Analysis of Package Hallucinations by Code Generating LLMs](https://arxiv.org/abs/2406.10279)) +[josephspracklen2024e] "These 30 tests generated a total of 2.23 million packages in response to our prompts, of which 440,445 (19.7%) were determined to be hallucinations, including 205,474 unique non-existent packages (i.e. packages that do not exist in PyPI or npm repositories and were distinct entries in the hallucination count, irrespective of their multiple occurrences) (Joseph Spracklen, Raveen Wijewickrama, A H M Nazmus Sakib, Anindya Maiti, Bimal Viswanath, Murtuza Jadliwala - [We Have a Package for You! A Comprehensive Analysis of Package Hallucinations by Code Generating LLMs](https://arxiv.org/abs/2406.10279)) -[49] "A new class of supply chain attacks named 'slopsquatting' has emerged from the increased use of generative AI tools for coding and the model's tendency to "hallucinate" non-existent package names. The term slopsquatting was coined by security researcher [Seth Larson](https://mastodon.social/@andrewnez/114302875075999244) as a spin on typosquatting, an attack method that tricks developers into installing malicious packages by using names that closely resemble popular libraries. Unlike typosquatting, slopsquatting doesn't rely on misspellings. Instead, threat actors could create malicious packages on indexes like PyPI and npm named after ones commonly made up by AI models in coding examples" (Toulas - [AI-hallucinated code dependencies become new supply chain risk](https://www.bleepingcomputer.com/news/security/ai-hallucinated-code-dependencies-become-new-supply-chain-risk/) +[billtoulas2025a] "A new class of supply chain attacks named 'slopsquatting' has emerged from the increased use of generative AI tools for coding and the model's tendency to "hallucinate" non-existent package names. The term slopsquatting was coined by security researcher [Seth Larson](https://mastodon.social/@andrewnez/114302875075999244) as a spin on typosquatting, an attack method that tricks developers into installing malicious packages by using names that closely resemble popular libraries. Unlike typosquatting, slopsquatting doesn't rely on misspellings. Instead, threat actors could create malicious packages on indexes like PyPI and npm named after ones commonly made up by AI models in coding examples" (Bill Toulas - [AI-hallucinated code dependencies become new supply chain risk](https://www.bleepingcomputer.com/news/security/ai-hallucinated-code-dependencies-become-new-supply-chain-risk/) -[50] "3 of the 4 models ... proved to be highly adept in detecting their own hallucinations with detection accuracy above 75%. Table 2 displays the recall and precision values for this test, with similarly strong performance across the 3 proficient models. This phenomenon implies that each model’s specific error patterns are detectable by the same mechanisms that generate them, suggesting an inherent self-regulatory capability. The indication that these models have an implicit understanding of their own generative patterns that could be leveraged for self-improvement is an important finding for developing mitigation strategies." (Spracklen - [We Have a Package for You! A Comprehensive Analysis of Package Hallucinations by Code Generating LLMs](https://arxiv.org/abs/2406.10279)) +[josephspracklen2024f] "3 of the 4 models ... proved to be highly adept in detecting their own hallucinations with detection accuracy above 75%. Table 2 displays the recall and precision values for this test, with similarly strong performance across the 3 proficient models. This phenomenon implies that each model’s specific error patterns are detectable by the same mechanisms that generate them, suggesting an inherent self-regulatory capability. The indication that these models have an implicit understanding of their own generative patterns that could be leveraged for self-improvement is an important finding for developing mitigation strategies." (Spracklen - [We Have a Package for You! A Comprehensive Analysis of Package Hallucinations by Code Generating LLMs](https://arxiv.org/abs/2406.10279)) -[51] "). Across all the examined LLMs, the persona/memetic proxy approach has led to the highest average number of security weaknesses among all the evaluated prompting techniques excluding the baseline prompt that does not include any security specifications." (Tony - [Prompting Techniques for Secure Code Generation: A Systematic Investigation](https://arxiv.org/abs/2407.07064v2)) +[catherinetony2024] "). Across all the examined LLMs, the persona/memetic proxy approach has led to the highest average number of security weaknesses among all the evaluated prompting techniques excluding the baseline prompt that does not include any security specifications." (Catherine Tony, Nicolás E. Díaz Ferreyra, Markus Mutas, Salem Dhiff, Riccardo Scandariato - [Prompting Techniques for Secure Code Generation: A Systematic Investigation](https://arxiv.org/abs/2407.07064v2)) diff --git a/docs/index.md b/docs/index.md index 9c778163..8b4b0760 100644 --- a/docs/index.md +++ b/docs/index.md @@ -15,7 +15,6 @@ This is a list of materials (documents, services, and so on) released by the * [Simplifying Software Component Updates](https://best.openssf.org/Simplifying-Software-Component-Updates) * [The Memory Safety Continuum](https://memorysafety.openssf.org/memory-safety-continuum) * [Cyber Resilience Act (CRA) Brief Guide for Open Source Software (OSS) Developers](https://best.openssf.org/CRA-Brief-Guide-for-OSS-Developers) -* [Security Focused Guide for AI Code Assistant Instructions](https://best.openssf.org/Security-Focused-Guide-for-AI-Code-Assistant-Instructions) Note: You can also see the larger list of [Guides released by the OpenSSF](https://openssf.org/resources/guides/). @@ -39,6 +38,7 @@ guidance about memory safety through our memory safety SIG, and so on. Examples of ongoing work include: * [Secure Coding One Stop Shop for Python](Secure-Coding-Guide-for-Python/readme.md) +* [Security Focused Guide for AI Code Assistant Instructions](https://best.openssf.org/Security-Focused-Guide-for-AI-Code-Assistant-Instructions) We typically use the [Simplest Possible Process (SPP)](https://best.openssf.org/spp/Simplest-Possible-Process) to publish our results on the web. From 79cafbb4da3fd28daaeac175f5075cf09157db72 Mon Sep 17 00:00:00 2001 From: balteravishay Date: Fri, 1 Aug 2025 14:39:25 +0100 Subject: [PATCH 14/31] lint Signed-off-by: balteravishay --- ...ecurity-Focused-Guide-for-AI-Code-Assistant-Instructions.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/docs/Security-Focused-Guide-for-AI-Code-Assistant-Instructions.md b/docs/Security-Focused-Guide-for-AI-Code-Assistant-Instructions.md index 37af8838..f311416f 100644 --- a/docs/Security-Focused-Guide-for-AI-Code-Assistant-Instructions.md +++ b/docs/Security-Focused-Guide-for-AI-Code-Assistant-Instructions.md @@ -30,7 +30,8 @@ One of the first sections in your instructions should reinforce general secure c * **Call out and review stubbed code:** If the AI generates code that is stubbed or incomplete, instruct it to flag these areas for review. For example: *"If you generate placeholder code (e.g., `TODO` comments), ensure it is marked for security review before deployment".* This will help ensure that any incomplete code does not inadvertently introduce vulnerabilities. [[fossa2023]](#fossa2023) [[anssibsi2024d]](#anssibsi2024d) * **Data Protection:** When generating code, always prioritize data minimization and avoid storing or processing confidential or otherwise sensitive information (like personal data - PII) unless absolutely necessary. For that case, suggest strong encryption at rest and in transit, and recommend techniques like anonymization. For example: *"Generate a function that securely handles user input for a registration form, asking for necessary fields to avoid logging sensitive information of PII. Ensure that no sensitive or PII is stored in plaintext"*. [[swaroopdora2025f]](#swaroopdora2025f) -Note that we are *not* currently recommending in the general case that the AI be told to respond from a particular viewpoint (e.g., a role or persona) or character aka "persona pattern/memetic proxy". An example of this approach would be the instruction "Act as a software security expert. Provide outputs that a security expert would give." One set of experiments found that this approach performs poorly, producing the worst number of security weaknesses compared to other approaches. [[catherinetony2024]](#catherinetony2024) However, we encourage continued experimentation, and may change our recommendations based on future information. +Note that we are *not* currently recommending in the general case that the AI be told to respond from a particular viewpoint (e.g., a role or persona) or character aka "persona pattern/memetic proxy". An example of this approach would be the instruction "Act as a software security expert. Provide outputs that a security expert would give". One set of experiments found that this approach performs poorly, producing the worst number of security weaknesses compared to other approaches. [[catherinetony2024]](#catherinetony2024) However, we encourage continued experimentation, and may change our recommendations based on future information. + --- ## **Addressing Software Supply Chain Security** From e8ae76646add2911cdebb3d422f3df23883cbcee Mon Sep 17 00:00:00 2001 From: Avishay Balter Date: Fri, 1 Aug 2025 14:44:16 +0100 Subject: [PATCH 15/31] Update docs/Security-Focused-Guide-for-AI-Code-Assistant-Instructions.md Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> Signed-off-by: Avishay Balter --- ...ecurity-Focused-Guide-for-AI-Code-Assistant-Instructions.md | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/docs/Security-Focused-Guide-for-AI-Code-Assistant-Instructions.md b/docs/Security-Focused-Guide-for-AI-Code-Assistant-Instructions.md index f311416f..05f34bfc 100644 --- a/docs/Security-Focused-Guide-for-AI-Code-Assistant-Instructions.md +++ b/docs/Security-Focused-Guide-for-AI-Code-Assistant-Instructions.md @@ -97,8 +97,7 @@ To strengthen your AI assistant's guidance, you should point it toward establish [haoyan2025a] "These reductions suggest that newer models can effectively leverage security hints to improve code safety" (Hao Yan, Swapneel Suhas Vaidya, Xiaokuan Zhang, Ziyu Yao - [Guiding AI to Fix Its Own Flaws: An Empirical Study on LLM-Driven Secure Code Generation](https://arxiv.org/abs/2506.23034)]) -[swaroopdora2025b] "Injection attacks remain one of the most critical vulnerabilities in web applications ([OWASP Top 10](https://owasp.org/Top10/A03_2021-Injection/))... LLM- -generated code must handle user input securely to prevent exploitation"(Swaroop Dora, Deven Lunkad, Naziya Aslam, S. Venkatesan, Sandeep Kumar Shukla - [The Hidden Risks of LLM-Generated Web Application Code](https://arxiv.org/abs/2504.20612)) +[swaroopdora2025b] "Injection attacks remain one of the most critical vulnerabilities in web applications ([OWASP Top 10](https://owasp.org/Top10/A03_2021-Injection/))... LLM-generated code must handle user input securely to prevent exploitation"(Swaroop Dora, Deven Lunkad, Naziya Aslam, S. Venkatesan, Sandeep Kumar Shukla - [The Hidden Risks of LLM-Generated Web Application Code](https://arxiv.org/abs/2504.20612)) [wiz2025b]"... systematic review shows that code injection (CWE-94) and OS command injection (CWE-78) are common in AI-generated code." (Wiz Research - [Rules Files for Safer Vibe Coding](https://www.wiz.io/blog/safer-vibe-coding-rules-files)) From bc068bcffb2a4d0f5c0add287cf5eb3dd389802b Mon Sep 17 00:00:00 2001 From: Avishay Balter Date: Fri, 1 Aug 2025 14:45:16 +0100 Subject: [PATCH 16/31] Update docs/Security-Focused-Guide-for-AI-Code-Assistant-Instructions.md Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> Signed-off-by: Avishay Balter --- ...Security-Focused-Guide-for-AI-Code-Assistant-Instructions.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/Security-Focused-Guide-for-AI-Code-Assistant-Instructions.md b/docs/Security-Focused-Guide-for-AI-Code-Assistant-Instructions.md index 05f34bfc..d6722cae 100644 --- a/docs/Security-Focused-Guide-for-AI-Code-Assistant-Instructions.md +++ b/docs/Security-Focused-Guide-for-AI-Code-Assistant-Instructions.md @@ -1,6 +1,6 @@ # Security-Focused Guide for AI Code Assistant Instructions -AI code assistants can significantly speed up development, but they need guidance to produce **secure** and robust code. This guide explains how to create custom instructions (e.g. [GitHub Copilot instructions file](https://docs.github.com/en/copilot/how-tos/custom-instructions/adding-repository-custom-instructions-for-github-copilot), [Cline instructions file](https://docs.cline.bot/enterprise-solutions/custom-instructions), [Cursor rules](https://docs.cursor.com/context/rules), [Claude markdown](https://docs.anthropic.com/en/docs/claude-code/common-workflows#create-an-effective-claude-md-file), etc.) that ensure the AI assistant accounts for application code security, supply chain safety, and platform or language-specific considerations and to embed a "security conscience" into the tool. In practice, this means fewer vulnerabilities making it into your codebase. Remember that these instructions should be kept concise, specific, and actionable – the goal is to influence the AI's behaviour without overwhelming it. [[wiz2025a]](#wiz2025a) +AI code assistants can significantly speed up development. However, they need guidance to produce **secure** and robust code. This guide explains how to create custom instructions (e.g. [GitHub Copilot instructions file](https://docs.github.com/en/copilot/how-tos/custom-instructions/adding-repository-custom-instructions-for-github-copilot), [Cline instructions file](https://docs.cline.bot/enterprise-solutions/custom-instructions), [Cursor rules](https://docs.cursor.com/context/rules), [Claude markdown](https://docs.anthropic.com/en/docs/claude-code/common-workflows#create-an-effective-claude-md-file), etc.). These instructions ensure the AI assistant accounts for application code security, supply chain safety, and platform or language-specific considerations. They also help embed a "security conscience" into the tool. In practice, this means fewer vulnerabilities making it into your codebase. Remember that these instructions should be kept concise, specific, and actionable. The goal is to influence the AI's behaviour without overwhelming it. [[wiz2025a]](#wiz2025a) These recommendations are based on expert opinion and various recommendations in the literature. We encourage experimentation and feedback to improve these recommendations. We, as an industry, are together learning how to best use these tools. --- From a005149d8e4dcb74ae180bdff5851363f1f84f99 Mon Sep 17 00:00:00 2001 From: Avishay Balter Date: Fri, 1 Aug 2025 14:49:06 +0100 Subject: [PATCH 17/31] Update Security-Focused-Guide-for-AI-Code-Assistant-Instructions.md Signed-off-by: Avishay Balter --- ...Security-Focused-Guide-for-AI-Code-Assistant-Instructions.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/Security-Focused-Guide-for-AI-Code-Assistant-Instructions.md b/docs/Security-Focused-Guide-for-AI-Code-Assistant-Instructions.md index d6722cae..e1a4567f 100644 --- a/docs/Security-Focused-Guide-for-AI-Code-Assistant-Instructions.md +++ b/docs/Security-Focused-Guide-for-AI-Code-Assistant-Instructions.md @@ -30,7 +30,7 @@ One of the first sections in your instructions should reinforce general secure c * **Call out and review stubbed code:** If the AI generates code that is stubbed or incomplete, instruct it to flag these areas for review. For example: *"If you generate placeholder code (e.g., `TODO` comments), ensure it is marked for security review before deployment".* This will help ensure that any incomplete code does not inadvertently introduce vulnerabilities. [[fossa2023]](#fossa2023) [[anssibsi2024d]](#anssibsi2024d) * **Data Protection:** When generating code, always prioritize data minimization and avoid storing or processing confidential or otherwise sensitive information (like personal data - PII) unless absolutely necessary. For that case, suggest strong encryption at rest and in transit, and recommend techniques like anonymization. For example: *"Generate a function that securely handles user input for a registration form, asking for necessary fields to avoid logging sensitive information of PII. Ensure that no sensitive or PII is stored in plaintext"*. [[swaroopdora2025f]](#swaroopdora2025f) -Note that we are *not* currently recommending in the general case that the AI be told to respond from a particular viewpoint (e.g., a role or persona) or character aka "persona pattern/memetic proxy". An example of this approach would be the instruction "Act as a software security expert. Provide outputs that a security expert would give". One set of experiments found that this approach performs poorly, producing the worst number of security weaknesses compared to other approaches. [[catherinetony2024]](#catherinetony2024) However, we encourage continued experimentation, and may change our recommendations based on future information. +Note that we are *not* currently recommending in the general case that the AI be told to respond from a particular viewpoint (e.g., a role or persona) or character a.k.a. "persona pattern/memetic proxy". An example of this approach would be the instruction "Act as a software security expert. Provide outputs that a security expert would give". One set of experiments found that this approach performs poorly, producing the worst number of security weaknesses compared to other approaches. [[catherinetony2024]](#catherinetony2024) However, we encourage continued experimentation, and may change our recommendations based on future information. --- From e34182b6137cdec1feceed10e8423fbe1ebfa165 Mon Sep 17 00:00:00 2001 From: Avishay Balter Date: Mon, 4 Aug 2025 21:14:31 +0100 Subject: [PATCH 18/31] Update docs/Security-Focused-Guide-for-AI-Code-Assistant-Instructions.md Co-authored-by: David A. Wheeler Signed-off-by: Avishay Balter --- ...Security-Focused-Guide-for-AI-Code-Assistant-Instructions.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/docs/Security-Focused-Guide-for-AI-Code-Assistant-Instructions.md b/docs/Security-Focused-Guide-for-AI-Code-Assistant-Instructions.md index e1a4567f..77e97d82 100644 --- a/docs/Security-Focused-Guide-for-AI-Code-Assistant-Instructions.md +++ b/docs/Security-Focused-Guide-for-AI-Code-Assistant-Instructions.md @@ -1,5 +1,7 @@ # Security-Focused Guide for AI Code Assistant Instructions +by the OpenSSF Best Practices Working Group, 2025-08-01 + AI code assistants can significantly speed up development. However, they need guidance to produce **secure** and robust code. This guide explains how to create custom instructions (e.g. [GitHub Copilot instructions file](https://docs.github.com/en/copilot/how-tos/custom-instructions/adding-repository-custom-instructions-for-github-copilot), [Cline instructions file](https://docs.cline.bot/enterprise-solutions/custom-instructions), [Cursor rules](https://docs.cursor.com/context/rules), [Claude markdown](https://docs.anthropic.com/en/docs/claude-code/common-workflows#create-an-effective-claude-md-file), etc.). These instructions ensure the AI assistant accounts for application code security, supply chain safety, and platform or language-specific considerations. They also help embed a "security conscience" into the tool. In practice, this means fewer vulnerabilities making it into your codebase. Remember that these instructions should be kept concise, specific, and actionable. The goal is to influence the AI's behaviour without overwhelming it. [[wiz2025a]](#wiz2025a) These recommendations are based on expert opinion and various recommendations in the literature. We encourage experimentation and feedback to improve these recommendations. We, as an industry, are together learning how to best use these tools. From 87ed472d1491fbc293b54df6c83df8feeb7549a0 Mon Sep 17 00:00:00 2001 From: Avishay Balter Date: Mon, 4 Aug 2025 21:18:04 +0100 Subject: [PATCH 19/31] Update docs/Security-Focused-Guide-for-AI-Code-Assistant-Instructions.md Co-authored-by: David A. Wheeler Signed-off-by: Avishay Balter --- .../Security-Focused-Guide-for-AI-Code-Assistant-Instructions.md | 1 + 1 file changed, 1 insertion(+) diff --git a/docs/Security-Focused-Guide-for-AI-Code-Assistant-Instructions.md b/docs/Security-Focused-Guide-for-AI-Code-Assistant-Instructions.md index 77e97d82..e945d181 100644 --- a/docs/Security-Focused-Guide-for-AI-Code-Assistant-Instructions.md +++ b/docs/Security-Focused-Guide-for-AI-Code-Assistant-Instructions.md @@ -15,6 +15,7 @@ Short on time? Here's what really matters: * **Apply Engineering Best Practices Always:** AI-generated code isn't a shortcut around engineering processes such as code reviews, testing, static analysis, documentation, and version control discipline. [[markvero2025a]](#markvero2025a) * **Be Security-Conscious:** Assume AI-written code can have bugs or vulnerabilities, because it often does. AI coding assistants can introduce security issues like using outdated cryptography or outdated dependencies, ignoring error handling, or leaking secrets. Check for any secrets or sensitive data in the suggested code. Make sure dependency suggestions are safe and not pulling in known vulnerable packages. [[shihchiehdai2025a]](#shihchiehdai2025a), [[anssibsi2024b]](#anssibsi2024b) * **Guide the AI:** AI is a powerful assistant, but it works best with your guidance. Write clear prompts that specify security requirements. Don't hesitate to modify or reject AI outputs. Direct your AI tool to build its own instructions file based on this guide. [[swaroopdora2025a]](#swaroopdora2025a) [[haoyan2025a]](#haoyan2025a) +* **Ask the AI to review and improve its own work**. Once you have some AI-written code, where possible, ask it to review and improve its own work (repeating these steps as necessary). This technique is sometimes called Recursive Criticism and Improvement (RCI) and can be remarkably effective. For instance, "Review your previous answer and find problems with your answer" followed by "Based on the problems you found, improve your answer" for one or more iterations. [[catherinetony2024a]](#catherinetony2024a) By keeping these points in mind, you can harness AI code assistants effectively without sacrificing quality or security. From 94520ca86251ffa3a3e2526228a7ac732136b5d9 Mon Sep 17 00:00:00 2001 From: Avishay Balter Date: Mon, 4 Aug 2025 21:29:34 +0100 Subject: [PATCH 20/31] Update docs/Security-Focused-Guide-for-AI-Code-Assistant-Instructions.md Co-authored-by: David A. Wheeler Signed-off-by: Avishay Balter --- ...uide-for-AI-Code-Assistant-Instructions.md | 65 +++++++++++++++++++ 1 file changed, 65 insertions(+) diff --git a/docs/Security-Focused-Guide-for-AI-Code-Assistant-Instructions.md b/docs/Security-Focused-Guide-for-AI-Code-Assistant-Instructions.md index e945d181..91fc39a1 100644 --- a/docs/Security-Focused-Guide-for-AI-Code-Assistant-Instructions.md +++ b/docs/Security-Focused-Guide-for-AI-Code-Assistant-Instructions.md @@ -18,6 +18,71 @@ Short on time? Here's what really matters: * **Ask the AI to review and improve its own work**. Once you have some AI-written code, where possible, ask it to review and improve its own work (repeating these steps as necessary). This technique is sometimes called Recursive Criticism and Improvement (RCI) and can be remarkably effective. For instance, "Review your previous answer and find problems with your answer" followed by "Based on the problems you found, improve your answer" for one or more iterations. [[catherinetony2024a]](#catherinetony2024a) By keeping these points in mind, you can harness AI code assistants effectively without sacrificing quality or security. +## TL;DR Sample Instructions + +Here are sample instructions that you can copy and paste +(details are below): + + +> Provide secure code. +User inputs should be checked for expected format and length. +Always validate function arguments and use parameterized queries for database access. +Escape special characters in user-generated content before rendering it in HTML. +When generating output contexts such as HTML or SQL, use safe frameworks or encoding functions to avoid vulnerabilities. +Never include API keys, passwords, or secrets in code output, and use environment variables or secure vault references instead. +Use secure authentication flows and enforce role-based access checks where appropriate. +When generating code, handle errors gracefully and log them, but do not expose internal details or secrets in error messages +Use logging frameworks that can be configured for security. +Prefer safe defaults in configurations – for example, use HTTPS by default, require strong encryption algorithms, and disable insecure protocols or options. +Follow least privilege in any configuration or code. +When applicable, generate unit tests for security-critical functions (including negative tests to ensure the code fails safely). +If you generate placeholder code (e.g., `TODO` comments), ensure it is marked for security review before deployment. +Avoid logging sensitive information of PII. Ensure that no sensitive or PII is stored in plaintext. +Use popular, community-trusted libraries for common tasks (and avoid adding obscure dependencies if a standard library or well-known package can do the same job). Do not add dependencies that may be malicious or hallucinated. +Always use the official package manager for the given language (npm, pip, Maven, etc.) to install libraries, rather than copying code snippets. +Specify version ranges or exact versions. +When suggesting dependency versions, prefer the latest stable release and mention updating dependencies regularly to patch vulnerabilities. +Generate a Software Bill of Materials (SBOM) by using tools that support standard formats like SPDX or CycloneDX. +Where applicable, use in-toto attestations or similar frameworks to create verifiable records of your build and deployment processes. +Prefer high-level libraries for cryptography rather than rolling your own. + +> When adding important external resources (scripts, containers, etc.), include steps to verify integrity (like checksum verification or signature validation) if applicable. +When writing file or OS-level operations, use safe functions and check for errors (e.g., use secure file modes, avoid temp files without proper randomness, etc.). If running as a service, drop privileges when possible. +Always include appropriate security headers (Content Security Policy, X-Frame-Options, etc.) in web responses, and use frameworks' built-in protections for cookies and sessions. +When generating code for cloud services (AWS/Azure/GCP), follow the provider's security guidelines (e.g., use parameterized queries for cloud databases, encrypt data at rest and in transit, handle keys via cloud KMS). +When using containers, +use minimal base images and avoid running containers with the root user. Use official images from trusted sources, and pin image versions using immutable digests (e.g., SHA256 hashes) instead of mutable tags like latest. +When working with container images, verify both the integrity and authenticity of images using container signing tools like cosign or notation. Include steps to verify signatures from trusted publishers and implement admission controllers in Kubernetes to enforce signature verification policies. +When generating HTML/JS, do not include direct links to untrusted third-party hosts for critical libraries; use our locally hosted or CDN with integrity checks. +For mobile and desktop apps, +do not suggest storing sensitive data in plaintext on the device; use the platform's secure storage APIs. +When generating github actions or CI/CD pipelines, ensure secrets are stored securely (e.g., using GitHub Secrets or environment variables) and not hard-coded in the workflow files. +Include steps to run security scans (SAST/DAST) and dependency checks in the CI/CD pipeline to catch vulnerabilities early. +When generating infrastructure-as-code (IaC) scripts, ensure they follow security best practices (e.g., restrict access to resources, use secure storage for secrets, and validate inputs) and use the latest versions of devops dependencies such as GitHub actions and lock the version to specific SHA. +In C or C++ code, always use bounds-checked functions (e.g., `strncpy` over `strcpy`), avoid dangerous functions like `gets`, and include buffer size constants to prevent overflow. Enable compiler defenses (stack canaries, fortify source, DEP/NX) in any build configurations you suggest. +In Rust code, avoid using `unsafe` blocks unless absolutely necessary and document any `unsafe` usage with justification. +In any memory-safe language, prefer using safe library functions and types; don't circumvent their safety without cause. +In go code, use the data race detector when building the application. +For Python, do not use `exec`/`eval` on user input and prefer safe APIs (e.g., use the `subprocess` module with `shell=False` to avoid shell injection). +For Python, follow PEP 8 and use type hints, as this can catch misuse early. +For JavaScript/TypeScript, when generating Node.js code, use prepared statements for database queries (just like any other language) and encode any data that goes into HTML to prevent XSS. +For Java, when suggesting web code (e.g., using Spring), ensure to use built-in security annotations and avoid old, vulnerable libraries (e.g., use `BCryptPasswordEncoder` rather than writing a custom password hash). +For C\#, Use .NET's cryptography and identity libraries instead of custom solutions. + +> Never suggest turning off security features like XML entity security or type checking during deserialization. +Code suggestions should adhere to OWASP Top 10 principles (e.g., avoid injection, enforce access control) and follow the OWASP ASVS requirements where applicable. +Our project follows SAFECode's secure development practices – the AI should prioritize those (e.g., proper validation, authentication, cryptography usage per SAFECode guidance). +When generating code, consider compliance requirements (e.g., HIPAA privacy rules for medical data, PCI-DSS for credit card info) – do not output code that logs or transmits sensitive data in insecure ways. +Include comments or TODOs in code suggesting security reviews for complex logic, and note if any third-party component might need a future update or audit. +When writing or reviewing code, run or simulate the use of tools like CodeQL, Bandit, Semgrep, or OWASP Dependency-Check. Identify any flagged vulnerabilities or outdated dependencies and revise the code accordingly. Repeat this process until the code passes all simulated scans. + +Follow this with: + +> Review your previous answer and find problems with your answer. + +Follow this with: + +> Based on the problems you found, improve your answer. --- From 357899b4a4265ddbdb4931377914ae183e00a2d2 Mon Sep 17 00:00:00 2001 From: balteravishay Date: Mon, 4 Aug 2025 21:31:37 +0100 Subject: [PATCH 21/31] first fix Signed-off-by: balteravishay --- ...rity-Focused-Guide-for-AI-Code-Assistant-Instructions.md | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/docs/Security-Focused-Guide-for-AI-Code-Assistant-Instructions.md b/docs/Security-Focused-Guide-for-AI-Code-Assistant-Instructions.md index e945d181..4850e722 100644 --- a/docs/Security-Focused-Guide-for-AI-Code-Assistant-Instructions.md +++ b/docs/Security-Focused-Guide-for-AI-Code-Assistant-Instructions.md @@ -15,7 +15,7 @@ Short on time? Here's what really matters: * **Apply Engineering Best Practices Always:** AI-generated code isn't a shortcut around engineering processes such as code reviews, testing, static analysis, documentation, and version control discipline. [[markvero2025a]](#markvero2025a) * **Be Security-Conscious:** Assume AI-written code can have bugs or vulnerabilities, because it often does. AI coding assistants can introduce security issues like using outdated cryptography or outdated dependencies, ignoring error handling, or leaking secrets. Check for any secrets or sensitive data in the suggested code. Make sure dependency suggestions are safe and not pulling in known vulnerable packages. [[shihchiehdai2025a]](#shihchiehdai2025a), [[anssibsi2024b]](#anssibsi2024b) * **Guide the AI:** AI is a powerful assistant, but it works best with your guidance. Write clear prompts that specify security requirements. Don't hesitate to modify or reject AI outputs. Direct your AI tool to build its own instructions file based on this guide. [[swaroopdora2025a]](#swaroopdora2025a) [[haoyan2025a]](#haoyan2025a) -* **Ask the AI to review and improve its own work**. Once you have some AI-written code, where possible, ask it to review and improve its own work (repeating these steps as necessary). This technique is sometimes called Recursive Criticism and Improvement (RCI) and can be remarkably effective. For instance, "Review your previous answer and find problems with your answer" followed by "Based on the problems you found, improve your answer" for one or more iterations. [[catherinetony2024a]](#catherinetony2024a) +* **Ask the AI to review and improve its own work**. Once you have some AI-written code, where possible, ask it to review and improve its own work (repeating these steps as necessary). This technique is sometimes called Recursive Criticism and Improvement (RCI) and can be remarkably effective. For instance, "Review your previous answer and find problems with your answer" followed by "Based on the problems you found, improve your answer" for one or more iterations. Encourage the use of tools such as linters, SAST, dependency checkers, etc. through the improvement cycles. [[catherinetony2024a]](#catherinetony2024a) By keeping these points in mind, you can harness AI code assistants effectively without sacrificing quality or security. @@ -30,7 +30,9 @@ One of the first sections in your instructions should reinforce general secure c * **Error Handling & Logging:** Guide the AI to implement errors securely by catching exceptions and failures without revealing sensitive info (stack traces, server paths, etc.) to the end-user. In your instructions, you might include: *"When generating code, handle errors gracefully and log them, but do not expose internal details or secrets in error messages".* This ensures the assistant's suggestions include secure error-handling patterns (like generic user-facing messages and detailed logs only on the server side). Additionally, instruct the AI to use logging frameworks that can be configured for security (e.g. avoiding logging of personal data or secrets). [[swaroopdora2025d]](#swaroopdora2025d) * **Secure Defaults & Configurations:** Include guidance such as: *"Prefer safe defaults in configurations – for example, use HTTPS by default, require strong encryption algorithms, and disable insecure protocols or options".* By specifying this, the AI will be more likely to generate code that opts-in to security features. Always instruct the AI to follow the principle of least privilege (e.g. minimal file system permissions, least-privileged user accounts for services, etc.) in any configuration or code it proposes. [[wiz2025c]](#wiz2025c) [[swaroopdora2025e]](#swaroopdora2025e) * **Testing for Security:** Encourage the AI to produce or suggest tests for critical code paths including negative tests that verify that what shouldn't happen, doesn't happen. In your instructions, add: *"When applicable, generate unit tests for security-critical functions (including negative tests to ensure the code fails safely)"*. [[anssibsi2024c]](#anssibsi2024c) [[markvero2025b]](#markvero2025b) -* **Call out and review stubbed code:** If the AI generates code that is stubbed or incomplete, instruct it to flag these areas for review. For example: *"If you generate placeholder code (e.g., `TODO` comments), ensure it is marked for security review before deployment".* This will help ensure that any incomplete code does not inadvertently introduce vulnerabilities. [[fossa2023]](#fossa2023) [[anssibsi2024d]](#anssibsi2024d) +Note that we are *not* currently recommending in the general case that the AI be told to respond from a particular viewpoint (e.g., a role or persona) or character a.k.a. "persona pattern/memetic proxy". An example of this approach would be the instruction "Act as a software security expert. Provide outputs that a security expert would give". One set of experiments found that this approach performs poorly, producing the worst number of security weaknesses compared to other approaches. [[catherinetony2024b]](#catherinetony2024b) However, we encourage continued experimentation, and may change our recommendations based on future information. + +[^catherinetony2024]: https://whatever.website.local * **Data Protection:** When generating code, always prioritize data minimization and avoid storing or processing confidential or otherwise sensitive information (like personal data - PII) unless absolutely necessary. For that case, suggest strong encryption at rest and in transit, and recommend techniques like anonymization. For example: *"Generate a function that securely handles user input for a registration form, asking for necessary fields to avoid logging sensitive information of PII. Ensure that no sensitive or PII is stored in plaintext"*. [[swaroopdora2025f]](#swaroopdora2025f) Note that we are *not* currently recommending in the general case that the AI be told to respond from a particular viewpoint (e.g., a role or persona) or character a.k.a. "persona pattern/memetic proxy". An example of this approach would be the instruction "Act as a software security expert. Provide outputs that a security expert would give". One set of experiments found that this approach performs poorly, producing the worst number of security weaknesses compared to other approaches. [[catherinetony2024]](#catherinetony2024) However, we encourage continued experimentation, and may change our recommendations based on future information. From 6d46f26062519a56f6ed99b3e50bdcb96fe0524a Mon Sep 17 00:00:00 2001 From: balteravishay Date: Mon, 4 Aug 2025 21:58:02 +0100 Subject: [PATCH 22/31] more pr fixes Signed-off-by: balteravishay --- ...uide-for-AI-Code-Assistant-Instructions.md | 21 ++++++++++++------- 1 file changed, 13 insertions(+), 8 deletions(-) diff --git a/docs/Security-Focused-Guide-for-AI-Code-Assistant-Instructions.md b/docs/Security-Focused-Guide-for-AI-Code-Assistant-Instructions.md index d465d64e..3385f69f 100644 --- a/docs/Security-Focused-Guide-for-AI-Code-Assistant-Instructions.md +++ b/docs/Security-Focused-Guide-for-AI-Code-Assistant-Instructions.md @@ -18,7 +18,8 @@ Short on time? Here's what really matters: * **Ask the AI to review and improve its own work**. Once you have some AI-written code, where possible, ask it to review and improve its own work (repeating these steps as necessary). This technique is sometimes called Recursive Criticism and Improvement (RCI) and can be remarkably effective. For instance, "Review your previous answer and find problems with your answer" followed by "Based on the problems you found, improve your answer" for one or more iterations. Encourage the use of tools such as linters, SAST, dependency checkers, etc. through the improvement cycles. [[catherinetony2024a]](#catherinetony2024a) By keeping these points in mind, you can harness AI code assistants effectively without sacrificing quality or security. -## TL;DR Sample Instructions + +### TL;DR Sample Instructions Here are sample instructions that you can copy and paste (details are below): @@ -29,7 +30,7 @@ User inputs should be checked for expected format and length. Always validate function arguments and use parameterized queries for database access. Escape special characters in user-generated content before rendering it in HTML. When generating output contexts such as HTML or SQL, use safe frameworks or encoding functions to avoid vulnerabilities. -Never include API keys, passwords, or secrets in code output, and use environment variables or secure vault references instead. +Note that we are *not* currently recommending in the general case that the AI be told to respond from a particular viewpoint (e.g., a role or persona) or character a.k.a. "persona pattern/memetic proxy". An example of this approach would be the instruction "Act as a software security expert. Provide outputs that a security expert would give". One set of experiments found that this approach performs poorly, producing the worst number of security weaknesses compared to other approaches. [[catherinetony2024b]](#catherinetony2024b) However, we encourage continued experimentation, and may change our recommendations based on future information. Use secure authentication flows and enforce role-based access checks where appropriate. When generating code, handle errors gracefully and log them, but do not expose internal details or secrets in error messages Use logging frameworks that can be configured for security. @@ -95,12 +96,9 @@ One of the first sections in your instructions should reinforce general secure c * **Error Handling & Logging:** Guide the AI to implement errors securely by catching exceptions and failures without revealing sensitive info (stack traces, server paths, etc.) to the end-user. In your instructions, you might include: *"When generating code, handle errors gracefully and log them, but do not expose internal details or secrets in error messages".* This ensures the assistant's suggestions include secure error-handling patterns (like generic user-facing messages and detailed logs only on the server side). Additionally, instruct the AI to use logging frameworks that can be configured for security (e.g. avoiding logging of personal data or secrets). [[swaroopdora2025d]](#swaroopdora2025d) * **Secure Defaults & Configurations:** Include guidance such as: *"Prefer safe defaults in configurations – for example, use HTTPS by default, require strong encryption algorithms, and disable insecure protocols or options".* By specifying this, the AI will be more likely to generate code that opts-in to security features. Always instruct the AI to follow the principle of least privilege (e.g. minimal file system permissions, least-privileged user accounts for services, etc.) in any configuration or code it proposes. [[wiz2025c]](#wiz2025c) [[swaroopdora2025e]](#swaroopdora2025e) * **Testing for Security:** Encourage the AI to produce or suggest tests for critical code paths including negative tests that verify that what shouldn't happen, doesn't happen. In your instructions, add: *"When applicable, generate unit tests for security-critical functions (including negative tests to ensure the code fails safely)"*. [[anssibsi2024c]](#anssibsi2024c) [[markvero2025b]](#markvero2025b) -Note that we are *not* currently recommending in the general case that the AI be told to respond from a particular viewpoint (e.g., a role or persona) or character a.k.a. "persona pattern/memetic proxy". An example of this approach would be the instruction "Act as a software security expert. Provide outputs that a security expert would give". One set of experiments found that this approach performs poorly, producing the worst number of security weaknesses compared to other approaches. [[catherinetony2024b]](#catherinetony2024b) However, we encourage continued experimentation, and may change our recommendations based on future information. - -[^catherinetony2024]: https://whatever.website.local * **Data Protection:** When generating code, always prioritize data minimization and avoid storing or processing confidential or otherwise sensitive information (like personal data - PII) unless absolutely necessary. For that case, suggest strong encryption at rest and in transit, and recommend techniques like anonymization. For example: *"Generate a function that securely handles user input for a registration form, asking for necessary fields to avoid logging sensitive information of PII. Ensure that no sensitive or PII is stored in plaintext"*. [[swaroopdora2025f]](#swaroopdora2025f) -Note that we are *not* currently recommending in the general case that the AI be told to respond from a particular viewpoint (e.g., a role or persona) or character a.k.a. "persona pattern/memetic proxy". An example of this approach would be the instruction "Act as a software security expert. Provide outputs that a security expert would give". One set of experiments found that this approach performs poorly, producing the worst number of security weaknesses compared to other approaches. [[catherinetony2024]](#catherinetony2024) However, we encourage continued experimentation, and may change our recommendations based on future information. +Note that we are *not* currently recommending in the general case that the AI be told to respond from a particular viewpoint (e.g., a role or persona) or character a.k.a. "persona pattern/memetic proxy". An example of this approach would be the instruction "Act as a software security expert. Provide outputs that a security expert would give". One set of experiments found that this approach performs poorly, producing the worst number of security weaknesses compared to other approaches. [[catherinetony2024b]](#catherinetony2024b) However, we encourage continued experimentation, and may change our recommendations based on future information. --- @@ -108,7 +106,7 @@ Note that we are *not* currently recommending in the general case that the AI be Modern software heavily relies on third-party libraries and dependencies. It's crucial that your AI assistant's instructions cover supply chain security, ensuring that suggested dependencies and build processes are secure: -* **Safe Dependency Selection:** Instruct the AI to prefer well-vetted, reputable libraries when suggesting code that pulls in external packages. This is especially important to counter hallucinated package names; one study found 19.7% proposed packages did not exist [[josephspracklen2024e]](#josephspracklen2024e). These hallucinations enable "slopsquatting" attacks, where attackers create malicious packages with names commonly hallucinated by AI models [[billtoulas2025a]](#billtoulas2025a). While it's important to independently check new dependencies, AI tools can often self-identify these names if asked to do so [[josephspracklen2024f]](#josephspracklen2024f). For example: *"Use popular, community-trusted libraries for common tasks (and avoid adding obscure dependencies if a standard library or well-known package can do the same job). Avoid adding dependencies that may be malicious or hallucinated.".* Emphasize evaluating packages before use – as a developer would manually. [[josephspracklen2024a]](#josephspracklen2024a) +* **Safe Dependency Selection:** Instruct the AI to prefer well-vetted, reputable libraries when suggesting code that pulls in external packages. This is especially important to counter hallucinated package names; one study found 19.7% proposed packages did not exist [[josephspracklen2024e]](#josephspracklen2024e). These hallucinations enable "slopsquatting" attacks, where attackers create malicious packages with names commonly hallucinated by AI models [[billtoulas2025a]](#billtoulas2025a). While it's important to independently check new dependencies, AI tools can often self-identify these names if asked to do so [[josephspracklen2024f]](#josephspracklen2024f). For example: *"Use popular, community-trusted libraries for common tasks (and avoid adding obscure dependencies if a standard library or well-known package can do the same job). Do not add dependencies that may be malicious or hallucinated.".* Emphasize evaluating packages before use – as a developer would manually. [[josephspracklen2024a]](#josephspracklen2024a) * **Use Package Managers & Lock Versions:** Your instructions should tell the AI to use proper package management. For instance: *"Always use the official package manager for the given language (npm, pip, Maven, etc.) to install libraries, rather than copying code snippets".* Also, instruct it to specify version ranges or exact versions that are known to be secure. By doing so, the AI will generate code that, for example, uses a `requirements.txt` or `package.json` entry, which aids in maintaining supply chain integrity. [[openssf2023a]](#openssf2023a) * **Stay Updated & Monitor Vulnerabilities:** Include guidance for keeping dependencies up-to-date. For example: *"When suggesting dependency versions, prefer the latest stable release and mention updating dependencies regularly to patch vulnerabilities"*. [[arifulhaque2025a]](#arifulhaque2025a) [[josephspracklen2024b]](#josephspracklen2024b) * **Generate Software Bill of Materials (SBOM):** Instruct the AI to create and maintaining SBOMs for better visibility into your software supply chain. For example: *"Generate a Software Bill of Materials (SBOM) by using tools that support standard formats like SPDX or CycloneDX".* You can also mention provenance tracking: *"Where applicable, use in-toto attestations or similar frameworks to create verifiable records of your build and deployment processes".* This ensures comprehensive tracking of what goes into your software and provides the foundation for ongoing vulnerability monitoring and incident response. [[anssibsi2024e]](#anssibsi2024e) @@ -258,4 +256,11 @@ encryption can expose passwords, personal information, and financial data... If [josephspracklen2024f] "3 of the 4 models ... proved to be highly adept in detecting their own hallucinations with detection accuracy above 75%. Table 2 displays the recall and precision values for this test, with similarly strong performance across the 3 proficient models. This phenomenon implies that each model’s specific error patterns are detectable by the same mechanisms that generate them, suggesting an inherent self-regulatory capability. The indication that these models have an implicit understanding of their own generative patterns that could be leveraged for self-improvement is an important finding for developing mitigation strategies." (Spracklen - [We Have a Package for You! A Comprehensive Analysis of Package Hallucinations by Code Generating LLMs](https://arxiv.org/abs/2406.10279)) -[catherinetony2024] "). Across all the examined LLMs, the persona/memetic proxy approach has led to the highest average number of security weaknesses among all the evaluated prompting techniques excluding the baseline prompt that does not include any security specifications." (Catherine Tony, Nicolás E. Díaz Ferreyra, Markus Mutas, Salem Dhiff, Riccardo Scandariato - [Prompting Techniques for Secure Code Generation: A Systematic Investigation](https://arxiv.org/abs/2407.07064v2)) +[catherinetony2024a] "). "When using LLMs or LLM-powered tools like ChatGPT or Copilot... (1) Using RCI is preferable over the other techniques studied in this work, as RCI can largely improve the security of the generated code (up to an order of magnitude w.r.t weakness density) even when applied with just 2 iterations. This technique has stayed valuable over several versions of the LLM models, and, hence, there is an expectation that it will stay valid in the future as well. ... (4) In cases where multi-step techniques like RCI are not feasible, using simple zero-shot prompting with templates similar to comprehensive prompts, that specify well-established secure coding standards, can provide comparable results in relation to more complex techniques." (Catherine Tony, Nicolás E. Díaz Ferreyra, Markus Mutas, Salem Dhiff, Riccardo Scandariato - [Prompting Techniques for Secure Code Generation: A Systematic Investigation](https://arxiv.org/abs/2407.07064v2)) + +[catherinetony2024b] "). "Across all the examined LLMs, the persona/memetic proxy approach has led to the highest average number of security weaknesses among all the evaluated prompting techniques excluding the baseline prompt that does not include any security specifications." (Catherine Tony, Nicolás E. Díaz Ferreyra, Markus Mutas, Salem Dhiff, Riccardo Scandariato - [Prompting Techniques for Secure Code Generation: A Systematic Investigation](https://arxiv.org/abs/2407.07064v2)) + +## Credit + +This work's development was led by Avishay Balter. +Contributors include Avishay Balter, David A. Wheeler, Mihai Maruseac, Nell Shamrell-Harrington, Jason Goodsell, Roman Zhukov and others. \ No newline at end of file From d2f3a06324dae10f4513ff721b3ad1b17b25f429 Mon Sep 17 00:00:00 2001 From: balteravishay Date: Mon, 4 Aug 2025 22:03:15 +0100 Subject: [PATCH 23/31] lint Signed-off-by: balteravishay --- ...ity-Focused-Guide-for-AI-Code-Assistant-Instructions.md | 7 +++---- 1 file changed, 3 insertions(+), 4 deletions(-) diff --git a/docs/Security-Focused-Guide-for-AI-Code-Assistant-Instructions.md b/docs/Security-Focused-Guide-for-AI-Code-Assistant-Instructions.md index 3385f69f..2aad3345 100644 --- a/docs/Security-Focused-Guide-for-AI-Code-Assistant-Instructions.md +++ b/docs/Security-Focused-Guide-for-AI-Code-Assistant-Instructions.md @@ -24,7 +24,6 @@ By keeping these points in mind, you can harness AI code assistants effectively Here are sample instructions that you can copy and paste (details are below): - > Provide secure code. User inputs should be checked for expected format and length. Always validate function arguments and use parameterized queries for database access. @@ -46,7 +45,7 @@ When suggesting dependency versions, prefer the latest stable release and mentio Generate a Software Bill of Materials (SBOM) by using tools that support standard formats like SPDX or CycloneDX. Where applicable, use in-toto attestations or similar frameworks to create verifiable records of your build and deployment processes. Prefer high-level libraries for cryptography rather than rolling your own. - +--- > When adding important external resources (scripts, containers, etc.), include steps to verify integrity (like checksum verification or signature validation) if applicable. When writing file or OS-level operations, use safe functions and check for errors (e.g., use secure file modes, avoid temp files without proper randomness, etc.). If running as a service, drop privileges when possible. Always include appropriate security headers (Content Security Policy, X-Frame-Options, etc.) in web responses, and use frameworks' built-in protections for cookies and sessions. @@ -69,7 +68,7 @@ For Python, follow PEP 8 and use type hints, as this can catch misuse early. For JavaScript/TypeScript, when generating Node.js code, use prepared statements for database queries (just like any other language) and encode any data that goes into HTML to prevent XSS. For Java, when suggesting web code (e.g., using Spring), ensure to use built-in security annotations and avoid old, vulnerable libraries (e.g., use `BCryptPasswordEncoder` rather than writing a custom password hash). For C\#, Use .NET's cryptography and identity libraries instead of custom solutions. - +--- > Never suggest turning off security features like XML entity security or type checking during deserialization. Code suggestions should adhere to OWASP Top 10 principles (e.g., avoid injection, enforce access control) and follow the OWASP ASVS requirements where applicable. Our project follows SAFECode's secure development practices – the AI should prioritize those (e.g., proper validation, authentication, cryptography usage per SAFECode guidance). @@ -263,4 +262,4 @@ encryption can expose passwords, personal information, and financial data... If ## Credit This work's development was led by Avishay Balter. -Contributors include Avishay Balter, David A. Wheeler, Mihai Maruseac, Nell Shamrell-Harrington, Jason Goodsell, Roman Zhukov and others. \ No newline at end of file +Contributors include Avishay Balter, David A. Wheeler, Mihai Maruseac, Nell Shamrell-Harrington, Jason Goodsell, Roman Zhukov and others. From 79ad7fc170af7967d9c20273aa4468ea70f1d346 Mon Sep 17 00:00:00 2001 From: Avishay Balter Date: Mon, 4 Aug 2025 22:09:33 +0100 Subject: [PATCH 24/31] Update docs/Security-Focused-Guide-for-AI-Code-Assistant-Instructions.md Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> Signed-off-by: Avishay Balter --- .../Security-Focused-Guide-for-AI-Code-Assistant-Instructions.md | 1 - 1 file changed, 1 deletion(-) diff --git a/docs/Security-Focused-Guide-for-AI-Code-Assistant-Instructions.md b/docs/Security-Focused-Guide-for-AI-Code-Assistant-Instructions.md index 2aad3345..afe563d2 100644 --- a/docs/Security-Focused-Guide-for-AI-Code-Assistant-Instructions.md +++ b/docs/Security-Focused-Guide-for-AI-Code-Assistant-Instructions.md @@ -29,7 +29,6 @@ User inputs should be checked for expected format and length. Always validate function arguments and use parameterized queries for database access. Escape special characters in user-generated content before rendering it in HTML. When generating output contexts such as HTML or SQL, use safe frameworks or encoding functions to avoid vulnerabilities. -Note that we are *not* currently recommending in the general case that the AI be told to respond from a particular viewpoint (e.g., a role or persona) or character a.k.a. "persona pattern/memetic proxy". An example of this approach would be the instruction "Act as a software security expert. Provide outputs that a security expert would give". One set of experiments found that this approach performs poorly, producing the worst number of security weaknesses compared to other approaches. [[catherinetony2024b]](#catherinetony2024b) However, we encourage continued experimentation, and may change our recommendations based on future information. Use secure authentication flows and enforce role-based access checks where appropriate. When generating code, handle errors gracefully and log them, but do not expose internal details or secrets in error messages Use logging frameworks that can be configured for security. From 5380ba377a3c265900a5e3bb47200fb185adcba6 Mon Sep 17 00:00:00 2001 From: balteravishay Date: Mon, 4 Aug 2025 22:22:05 +0100 Subject: [PATCH 25/31] copilot change requests Signed-off-by: balteravishay --- ...cused-Guide-for-AI-Code-Assistant-Instructions.md | 12 +++++------- 1 file changed, 5 insertions(+), 7 deletions(-) diff --git a/docs/Security-Focused-Guide-for-AI-Code-Assistant-Instructions.md b/docs/Security-Focused-Guide-for-AI-Code-Assistant-Instructions.md index afe563d2..63141ec3 100644 --- a/docs/Security-Focused-Guide-for-AI-Code-Assistant-Instructions.md +++ b/docs/Security-Focused-Guide-for-AI-Code-Assistant-Instructions.md @@ -94,7 +94,7 @@ One of the first sections in your instructions should reinforce general secure c * **Error Handling & Logging:** Guide the AI to implement errors securely by catching exceptions and failures without revealing sensitive info (stack traces, server paths, etc.) to the end-user. In your instructions, you might include: *"When generating code, handle errors gracefully and log them, but do not expose internal details or secrets in error messages".* This ensures the assistant's suggestions include secure error-handling patterns (like generic user-facing messages and detailed logs only on the server side). Additionally, instruct the AI to use logging frameworks that can be configured for security (e.g. avoiding logging of personal data or secrets). [[swaroopdora2025d]](#swaroopdora2025d) * **Secure Defaults & Configurations:** Include guidance such as: *"Prefer safe defaults in configurations – for example, use HTTPS by default, require strong encryption algorithms, and disable insecure protocols or options".* By specifying this, the AI will be more likely to generate code that opts-in to security features. Always instruct the AI to follow the principle of least privilege (e.g. minimal file system permissions, least-privileged user accounts for services, etc.) in any configuration or code it proposes. [[wiz2025c]](#wiz2025c) [[swaroopdora2025e]](#swaroopdora2025e) * **Testing for Security:** Encourage the AI to produce or suggest tests for critical code paths including negative tests that verify that what shouldn't happen, doesn't happen. In your instructions, add: *"When applicable, generate unit tests for security-critical functions (including negative tests to ensure the code fails safely)"*. [[anssibsi2024c]](#anssibsi2024c) [[markvero2025b]](#markvero2025b) -* **Data Protection:** When generating code, always prioritize data minimization and avoid storing or processing confidential or otherwise sensitive information (like personal data - PII) unless absolutely necessary. For that case, suggest strong encryption at rest and in transit, and recommend techniques like anonymization. For example: *"Generate a function that securely handles user input for a registration form, asking for necessary fields to avoid logging sensitive information of PII. Ensure that no sensitive or PII is stored in plaintext"*. [[swaroopdora2025f]](#swaroopdora2025f) +* **Data Protection:** When generating code, always prioritize data minimization and avoid storing or processing confidential or otherwise sensitive information (like personal data - PII) unless absolutely necessary. For that case, suggest strong encryption at rest and in transit, and recommend techniques like anonymization. For example: *"Generate a function that securely handles user input for a registration form, asking for necessary fields to avoid logging sensitive PII. Ensure that no sensitive or PII is stored in plaintext"*. [[swaroopdora2025f]](#swaroopdora2025f) Note that we are *not* currently recommending in the general case that the AI be told to respond from a particular viewpoint (e.g., a role or persona) or character a.k.a. "persona pattern/memetic proxy". An example of this approach would be the instruction "Act as a software security expert. Provide outputs that a security expert would give". One set of experiments found that this approach performs poorly, producing the worst number of security weaknesses compared to other approaches. [[catherinetony2024b]](#catherinetony2024b) However, we encourage continued experimentation, and may change our recommendations based on future information. @@ -165,13 +165,11 @@ To strengthen your AI assistant's guidance, you should point it toward establish [swaroopdora2025b] "Injection attacks remain one of the most critical vulnerabilities in web applications ([OWASP Top 10](https://owasp.org/Top10/A03_2021-Injection/))... LLM-generated code must handle user input securely to prevent exploitation"(Swaroop Dora, Deven Lunkad, Naziya Aslam, S. Venkatesan, Sandeep Kumar Shukla - [The Hidden Risks of LLM-Generated Web Application Code](https://arxiv.org/abs/2504.20612)) -[wiz2025b]"... systematic review shows that code injection (CWE-94) and OS command injection (CWE-78) are common in AI-generated code." (Wiz Research - [Rules Files for Safer Vibe Coding](https://www.wiz.io/blog/safer-vibe-coding-rules-files)) +[wiz2025b] "... systematic review shows that code injection (CWE-94) and OS command injection (CWE-78) are common in AI-generated code." (Wiz Research - [Rules Files for Safer Vibe Coding](https://www.wiz.io/blog/safer-vibe-coding-rules-files)) -[haoyan2025b]"All models frequently generate vulnerable code across diverse vulnerability types... All models tend to generate vulnerabilities for both common (top-25) and less common types" (Hao Yan, Swapneel Suhas Vaidya, Xiaokuan Zhang, Ziyu Yao - [Guiding AI to Fix Its Own Flaws: An Empirical Study on LLM-Driven Secure Code Generation](https://arxiv.org/abs/2506.23034)) +[haoyan2025b] "All models frequently generate vulnerable code across diverse vulnerability types... All models tend to generate vulnerabilities for both common (top-25) and less common types" (Hao Yan, Swapneel Suhas Vaidya, Xiaokuan Zhang, Ziyu Yao - [Guiding AI to Fix Its Own Flaws: An Empirical Study on LLM-Driven Secure Code Generation](https://arxiv.org/abs/2506.23034)) -[hammondpearce2021a]"Overall, Copilot performed poorly in this CWE (CWE-522: Insufficiently Protected Credentials)... it frequently tried to generate code which was based on the insecure 'MD5' hashing algorithm" (Hammond Pearce, Baleegh Ahmad, Benjamin Tan, Brendan Dolan-Gavitt, Ramesh Karri - [Asleep at the Keyboard? Assessing the -Security of GitHub Copilot's Code Contributions -](https://arxiv.org/abs/2108.09293)) +[hammondpearce2021a] "Overall, Copilot performed poorly in this CWE (CWE-522: Insufficiently Protected Credentials)... it frequently tried to generate code which was based on the insecure 'MD5' hashing algorithm" (Hammond Pearce, Baleegh Ahmad, Benjamin Tan, Brendan Dolan-Gavitt, Ramesh Karri - [Asleep at the Keyboard? Assessing the Security of GitHub Copilot's Code Contributions](https://arxiv.org/abs/2108.09293)) [neilperry2022a] "Participants with access to an AI assistant were ... significantly more likely to write an insecure solution ... also less likely to authenticate the final returned value" (Neil Perry, Megha Srivastava, Deepak Kumar, Dan Boneh - [Do Users Write More Insecure Code with AI Assistants?](https://arxiv.org/abs/2211.03622)) @@ -250,7 +248,7 @@ encryption can expose passwords, personal information, and financial data... If [josephspracklen2024e] "These 30 tests generated a total of 2.23 million packages in response to our prompts, of which 440,445 (19.7%) were determined to be hallucinations, including 205,474 unique non-existent packages (i.e. packages that do not exist in PyPI or npm repositories and were distinct entries in the hallucination count, irrespective of their multiple occurrences) (Joseph Spracklen, Raveen Wijewickrama, A H M Nazmus Sakib, Anindya Maiti, Bimal Viswanath, Murtuza Jadliwala - [We Have a Package for You! A Comprehensive Analysis of Package Hallucinations by Code Generating LLMs](https://arxiv.org/abs/2406.10279)) -[billtoulas2025a] "A new class of supply chain attacks named 'slopsquatting' has emerged from the increased use of generative AI tools for coding and the model's tendency to "hallucinate" non-existent package names. The term slopsquatting was coined by security researcher [Seth Larson](https://mastodon.social/@andrewnez/114302875075999244) as a spin on typosquatting, an attack method that tricks developers into installing malicious packages by using names that closely resemble popular libraries. Unlike typosquatting, slopsquatting doesn't rely on misspellings. Instead, threat actors could create malicious packages on indexes like PyPI and npm named after ones commonly made up by AI models in coding examples" (Bill Toulas - [AI-hallucinated code dependencies become new supply chain risk](https://www.bleepingcomputer.com/news/security/ai-hallucinated-code-dependencies-become-new-supply-chain-risk/) +[billtoulas2025a] "A new class of supply chain attacks named 'slopsquatting' has emerged from the increased use of generative AI tools for coding and the model's tendency to "hallucinate" non-existent package names. The term slopsquatting was coined by security researcher [Seth Larson](https://mastodon.social/@andrewnez/114302875075999244) as a spin on typosquatting, an attack method that tricks developers into installing malicious packages by using names that closely resemble popular libraries. Unlike typosquatting, slopsquatting doesn't rely on misspellings. Instead, threat actors could create malicious packages on indexes like PyPI and npm named after ones commonly made up by AI models in coding examples" (Bill Toulas - [AI-hallucinated code dependencies become new supply chain risk](https://www.bleepingcomputer.com/news/security/ai-hallucinated-code-dependencies-become-new-supply-chain-risk/)) [josephspracklen2024f] "3 of the 4 models ... proved to be highly adept in detecting their own hallucinations with detection accuracy above 75%. Table 2 displays the recall and precision values for this test, with similarly strong performance across the 3 proficient models. This phenomenon implies that each model’s specific error patterns are detectable by the same mechanisms that generate them, suggesting an inherent self-regulatory capability. The indication that these models have an implicit understanding of their own generative patterns that could be leveraged for self-improvement is an important finding for developing mitigation strategies." (Spracklen - [We Have a Package for You! A Comprehensive Analysis of Package Hallucinations by Code Generating LLMs](https://arxiv.org/abs/2406.10279)) From 3594eff5a0aac71b0ac8b256be64db0c2d494a15 Mon Sep 17 00:00:00 2001 From: balteravishay Date: Mon, 4 Aug 2025 22:26:31 +0100 Subject: [PATCH 26/31] c# Signed-off-by: balteravishay --- ...rity-Focused-Guide-for-AI-Code-Assistant-Instructions.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/docs/Security-Focused-Guide-for-AI-Code-Assistant-Instructions.md b/docs/Security-Focused-Guide-for-AI-Code-Assistant-Instructions.md index 63141ec3..92945d69 100644 --- a/docs/Security-Focused-Guide-for-AI-Code-Assistant-Instructions.md +++ b/docs/Security-Focused-Guide-for-AI-Code-Assistant-Instructions.md @@ -66,7 +66,7 @@ For Python, do not use `exec`/`eval` on user input and prefer safe APIs (e.g., u For Python, follow PEP 8 and use type hints, as this can catch misuse early. For JavaScript/TypeScript, when generating Node.js code, use prepared statements for database queries (just like any other language) and encode any data that goes into HTML to prevent XSS. For Java, when suggesting web code (e.g., using Spring), ensure to use built-in security annotations and avoid old, vulnerable libraries (e.g., use `BCryptPasswordEncoder` rather than writing a custom password hash). -For C\#, Use .NET's cryptography and identity libraries instead of custom solutions. +For C#, Use .NET's cryptography and identity libraries instead of custom solutions. --- > Never suggest turning off security features like XML entity security or type checking during deserialization. Code suggestions should adhere to OWASP Top 10 principles (e.g., avoid injection, enforce access control) and follow the OWASP ASVS requirements where applicable. @@ -129,9 +129,9 @@ Every software runs somewhere – be it a web browser, a cloud platform, an IoT It's valuable to dedicate individual paragraphs to language-specific security considerations that are relevant to your project. The following section serves as examples, or special cases the AI should handle to highlight unique vulnerabilities or features of that language and how the AI should approach them: * **C/C++ (Memory-Unsafe Languages):** For languages without automatic memory safety, instruct the AI to be extra cautious with memory management. *"In C or C++ code, always use bounds-checked functions (e.g., `strncpy` over `strcpy`), avoid dangerous functions like `gets`, and include buffer size constants to prevent overflow. Enable compiler defenses (stack canaries, fortify source, DEP/NX) in any build configurations you suggest"*. By giving such instructions, the assistant might prefer safer standard library calls or even suggest modern C++ classes (`std::vector` instead of raw arrays) to reduce manual memory handling. It will also acknowledge when an operation is risky, possibly inserting comments like "// ensure no buffer overflow". [[connordilgren2025a]](#connordilgren2025a) [[neilperry2022c]](#neilperry2022c) -* **Rust, Go, and Memory-Safe Languages:** If the project involves memory-safe languages (Rust, Go, Java, C\#, etc.), you can note that the AI should leverage their safety features. *"In Rust code, avoid using `unsafe` blocks unless absolutely necessary and document any `unsafe` usage with justification".* Memory-safe-by-default languages enforce a lot at compile time, but you should still have the AI follow best practices of those ecosystems. For example, instruct: *"In any memory-safe language, prefer using safe library functions and types; don't circumvent their safety without cause".* If a language offers any tools to verify memory access, direct the AI assistant to use them while building or testing your code. For Example "*In go code, use the data race detector when building the application*". [[openssf2023c]](#openssf2023c) +* **Rust, Go, and Memory-Safe Languages:** If the project involves memory-safe languages (Rust, Go, Java, C#, etc.), you can note that the AI should leverage their safety features. *"In Rust code, avoid using `unsafe` blocks unless absolutely necessary and document any `unsafe` usage with justification".* Memory-safe-by-default languages enforce a lot at compile time, but you should still have the AI follow best practices of those ecosystems. For example, instruct: *"In any memory-safe language, prefer using safe library functions and types; don't circumvent their safety without cause".* If a language offers any tools to verify memory access, direct the AI assistant to use them while building or testing your code. For Example "*In go code, use the data race detector when building the application*". [[openssf2023c]](#openssf2023c) * **Python and Dynamic Languages:** Python, JavaScript, and other high-level languages manage memory for you, but come with their own security pitfalls. In your instructions, emphasize things like avoiding exec/eval with untrusted input in Python and being careful with command execution. *"For Python, do not use `exec`/`eval` on user input and prefer safe APIs (e.g., use the `subprocess` module with `shell=False` to avoid shell injection)".* Additionally, mention type checking or the use of linters: *"Follow PEP 8 and use type hints, as this can catch misuse early".* For JavaScript/TypeScript, you might add: *"When generating Node.js code, use prepared statements for database queries (just like any other language) and encode any data that goes into HTML to prevent XSS".* These instructions incorporate known best practices (like those from OWASP cheat sheets) directly into the AI's behavior. [[haoyan2025c]](#haoyan2025c) -* **Java/C\# and Enterprise Languages:** In languages often used for large applications, you might focus on frameworks and configurations. *"For Java, when suggesting web code (e.g., using Spring), ensure to use built-in security annotations and avoid old, vulnerable libraries (e.g., use `BCryptPasswordEncoder` rather than writing a custom password hash)".* For C\#, similarly: *"Use .NET's cryptography and identity libraries instead of custom solutions".* Also instruct about managing object deserialization (both Java and C\# have had vulnerabilities in this area): *"Never suggest turning off security features like XML entity security or type checking during deserialization".* These language-specific notes guide the AI to incorporate the well-known secure patterns of each ecosystem. [[neilperry2022d]](#neilperry2022d) +* **Java/C# and Enterprise Languages:** In languages often used for large applications, you might focus on frameworks and configurations. *"For Java, when suggesting web code (e.g., using Spring), ensure to use built-in security annotations and avoid old, vulnerable libraries (e.g., use `BCryptPasswordEncoder` rather than writing a custom password hash)".* For C#, similarly: *"Use .NET's cryptography and identity libraries instead of custom solutions".* Also instruct about managing object deserialization (both Java and C# have had vulnerabilities in this area): *"Never suggest turning off security features like XML entity security or type checking during deserialization".* These language-specific notes guide the AI to incorporate the well-known secure patterns of each ecosystem. [[neilperry2022d]](#neilperry2022d) --- From 8c7c6f7067c6b315d2264487bae11689f3ff608e Mon Sep 17 00:00:00 2001 From: balteravishay Date: Mon, 4 Aug 2025 22:27:49 +0100 Subject: [PATCH 27/31] line break Signed-off-by: balteravishay --- .../Security-Focused-Guide-for-AI-Code-Assistant-Instructions.md | 1 + 1 file changed, 1 insertion(+) diff --git a/docs/Security-Focused-Guide-for-AI-Code-Assistant-Instructions.md b/docs/Security-Focused-Guide-for-AI-Code-Assistant-Instructions.md index 92945d69..1e678dce 100644 --- a/docs/Security-Focused-Guide-for-AI-Code-Assistant-Instructions.md +++ b/docs/Security-Focused-Guide-for-AI-Code-Assistant-Instructions.md @@ -259,4 +259,5 @@ encryption can expose passwords, personal information, and financial data... If ## Credit This work's development was led by Avishay Balter. + Contributors include Avishay Balter, David A. Wheeler, Mihai Maruseac, Nell Shamrell-Harrington, Jason Goodsell, Roman Zhukov and others. From 5c148ccd5fcfff2c68f8752405e6f4207e882591 Mon Sep 17 00:00:00 2001 From: balteravishay Date: Tue, 5 Aug 2025 08:28:01 +0100 Subject: [PATCH 28/31] final small edits Signed-off-by: balteravishay --- ...urity-Focused-Guide-for-AI-Code-Assistant-Instructions.md | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/docs/Security-Focused-Guide-for-AI-Code-Assistant-Instructions.md b/docs/Security-Focused-Guide-for-AI-Code-Assistant-Instructions.md index 1e678dce..7f4ea333 100644 --- a/docs/Security-Focused-Guide-for-AI-Code-Assistant-Instructions.md +++ b/docs/Security-Focused-Guide-for-AI-Code-Assistant-Instructions.md @@ -1,8 +1,9 @@ # Security-Focused Guide for AI Code Assistant Instructions -by the OpenSSF Best Practices Working Group, 2025-08-01 +by the OpenSSF Best Practices and the AI/ML Working Groups, 2025-08-01 AI code assistants can significantly speed up development. However, they need guidance to produce **secure** and robust code. This guide explains how to create custom instructions (e.g. [GitHub Copilot instructions file](https://docs.github.com/en/copilot/how-tos/custom-instructions/adding-repository-custom-instructions-for-github-copilot), [Cline instructions file](https://docs.cline.bot/enterprise-solutions/custom-instructions), [Cursor rules](https://docs.cursor.com/context/rules), [Claude markdown](https://docs.anthropic.com/en/docs/claude-code/common-workflows#create-an-effective-claude-md-file), etc.). These instructions ensure the AI assistant accounts for application code security, supply chain safety, and platform or language-specific considerations. They also help embed a "security conscience" into the tool. In practice, this means fewer vulnerabilities making it into your codebase. Remember that these instructions should be kept concise, specific, and actionable. The goal is to influence the AI's behaviour without overwhelming it. [[wiz2025a]](#wiz2025a) + These recommendations are based on expert opinion and various recommendations in the literature. We encourage experimentation and feedback to improve these recommendations. We, as an industry, are together learning how to best use these tools. --- @@ -107,7 +108,7 @@ Modern software heavily relies on third-party libraries and dependencies. It's c * **Safe Dependency Selection:** Instruct the AI to prefer well-vetted, reputable libraries when suggesting code that pulls in external packages. This is especially important to counter hallucinated package names; one study found 19.7% proposed packages did not exist [[josephspracklen2024e]](#josephspracklen2024e). These hallucinations enable "slopsquatting" attacks, where attackers create malicious packages with names commonly hallucinated by AI models [[billtoulas2025a]](#billtoulas2025a). While it's important to independently check new dependencies, AI tools can often self-identify these names if asked to do so [[josephspracklen2024f]](#josephspracklen2024f). For example: *"Use popular, community-trusted libraries for common tasks (and avoid adding obscure dependencies if a standard library or well-known package can do the same job). Do not add dependencies that may be malicious or hallucinated.".* Emphasize evaluating packages before use – as a developer would manually. [[josephspracklen2024a]](#josephspracklen2024a) * **Use Package Managers & Lock Versions:** Your instructions should tell the AI to use proper package management. For instance: *"Always use the official package manager for the given language (npm, pip, Maven, etc.) to install libraries, rather than copying code snippets".* Also, instruct it to specify version ranges or exact versions that are known to be secure. By doing so, the AI will generate code that, for example, uses a `requirements.txt` or `package.json` entry, which aids in maintaining supply chain integrity. [[openssf2023a]](#openssf2023a) * **Stay Updated & Monitor Vulnerabilities:** Include guidance for keeping dependencies up-to-date. For example: *"When suggesting dependency versions, prefer the latest stable release and mention updating dependencies regularly to patch vulnerabilities"*. [[arifulhaque2025a]](#arifulhaque2025a) [[josephspracklen2024b]](#josephspracklen2024b) -* **Generate Software Bill of Materials (SBOM):** Instruct the AI to create and maintaining SBOMs for better visibility into your software supply chain. For example: *"Generate a Software Bill of Materials (SBOM) by using tools that support standard formats like SPDX or CycloneDX".* You can also mention provenance tracking: *"Where applicable, use in-toto attestations or similar frameworks to create verifiable records of your build and deployment processes".* This ensures comprehensive tracking of what goes into your software and provides the foundation for ongoing vulnerability monitoring and incident response. [[anssibsi2024e]](#anssibsi2024e) +* **Generate Software Bill of Materials (SBOM):** Instruct the AI to create and maintaining SBOMs for better visibility into your software supply chain. For example: *"Generate an SBOM by using tools that support standard formats like SPDX or CycloneDX".* You can also mention provenance tracking: *"Where applicable, use an attestations framework such as in-toto or similar to create verifiable records of your build and deployment processes".* This ensures comprehensive tracking of what goes into your software and provides the foundation for ongoing vulnerability monitoring and incident response. [[anssibsi2024e]](#anssibsi2024e) * **Integrity Verification:** To further secure the supply chain, you can instruct the assistant to show how to verify what it uses. For instance: *"When adding important external resources (scripts, containers, etc.), include steps to verify integrity (like checksum verification or signature validation) if applicable".* [[josephspracklen2024c]](#josephspracklen2024c) --- From 57f182d15abde56c13b42a54155d88f001697156 Mon Sep 17 00:00:00 2001 From: balteravishay Date: Tue, 5 Aug 2025 08:46:19 +0100 Subject: [PATCH 29/31] ununsed citations Signed-off-by: balteravishay --- ...urity-Focused-Guide-for-AI-Code-Assistant-Instructions.md | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-) diff --git a/docs/Security-Focused-Guide-for-AI-Code-Assistant-Instructions.md b/docs/Security-Focused-Guide-for-AI-Code-Assistant-Instructions.md index 7f4ea333..cd07f649 100644 --- a/docs/Security-Focused-Guide-for-AI-Code-Assistant-Instructions.md +++ b/docs/Security-Focused-Guide-for-AI-Code-Assistant-Instructions.md @@ -143,7 +143,7 @@ To strengthen your AI assistant's guidance, you should point it toward establish * **Use Industry Standards as Keywords:** Mention standards like **OWASP Top 10**, **OWASP ASVS**, **CWE/SANS Top 25**, or language-specific guidelines (Java's SEI CERT secure coding standards, for instance). For example: *"Code suggestions should adhere to OWASP Top 10 principles (e.g., avoid injection, enforce access control) and follow the OWASP ASVS requirements where applicable"*. [[arifulhaque2025c]](#arifulhaque2025c) * **Reference SAFECode and Other Guides:** SAFECode (Software Assurance Forum for Excellence in Code) publishes **Fundamental Practices for Secure Software Development**, and including a note about it can be beneficial. *"Our project follows SAFECode's secure development practices – the AI should prioritize those (e.g., proper validation, authentication, cryptography usage per SAFECode guidance)"*. This ties the assistant's suggestions to a comprehensive set of secure development principles. Similarly, you can reference the **Secure Software Development Lifecycle (SSDLC)** or standards like **ISO 27034** if your organization uses them, to align AI outputs with those processes. [[neilperry2022e]](#neilperry2022e) * **Compliance and Regulatory Security:** If you operate in an industry with specific regulations (like healthcare or finance), instruct the AI with those in mind. For instance: *"When generating code, consider compliance requirements (e.g., HIPAA privacy rules for medical data, PCI-DSS for credit card info) – do not output code that logs or transmits sensitive data in insecure ways"*. This ensures the AI's suggestions are compliant with external standards, for example, it might suggest encryption for personal data fields by default. [[arifulhaque2025d]](#arifulhaque2025d) -* **Continuous Improvement Hooks:** Encourage practices that integrate with ongoing security processes. For example: *"Include comments or TODOs in code suggesting security reviews for complex logic, and note if any third-party component might need a future update or audit".* This way, the AI not only writes code but also flags areas for human scrutiny or future action. If the AI outputs a piece of cryptography code, for instance, it might add a comment "// TODO: security review for cryptographic implementation" because you instructed it that critical code should not go un-reviewed. [[anssibsi2024h]](#anssibsi2024h) +* **Continuous Improvement Hooks:** Encourage practices that integrate with ongoing security processes. For example: *"Include comments or TODOs in code suggesting security reviews for complex logic, and note if any third-party component might need a future update or audit"*. This way, the AI not only writes code but also flags areas for human scrutiny or future action. If the AI outputs a piece of cryptography code, for instance, it might add a comment "// TODO: security review for cryptographic implementation" because you instructed it that critical code should not go un-reviewed. [[anssibsi2024h]](#anssibsi2024h) [[fossa2023]](#fossa2023)[[anssibsi2024d]](#anssibsi2024d) * **Integrate and Act on Automated Security Tooling:** Instruct the AI to proactively run automated security tools (e.g., SAST, DAST, SCA) during code generation and refinement. Example: "When writing or reviewing code, run or simulate the use of tools like CodeQL, Bandit, Semgrep, or OWASP Dependency-Check. Identify any flagged vulnerabilities or outdated dependencies and revise the code accordingly. Repeat this process until the code passes all simulated scans". This positions the AI not just as a code generator but as a security-aware assistant capable of iterating based on tool feedback. [[anssibsi2024i]](#anssibsi2024i) [[frontiers2024a]](#frontiers2024a) --- @@ -192,8 +192,7 @@ mechanisms, session management..." (Swaroop Dora, Deven Lunkad, Naziya Aslam, S. [anssibsi2024d] "It might be beneficial to flag AI generated code blocks and to document the used AI tools. This might help security testing and it can also be a useful information for third-party auditors, e.g. in the context of a security certification process" (ANSSI, BSI - [AI Coding Assistants](https://www.bsi.bund.de/SharedDocs/Downloads/EN/BSI/KI/ANSSI_BSI_AI_Coding_Assistants.pdf?__blob=publicationFile&v=7)) -[swaroopdora2025f] "Encryption safeguards sensitive data at rest and in transit. Weak encryption methods or lack of -encryption can expose passwords, personal information, and financial data... If passwords are stored in plain text or hashed without salting, attackers with database access can easily decrypt credentials, leading to mass account breaches." (Swaroop Dora, Deven Lunkad, Naziya Aslam, S. Venkatesan, Sandeep Kumar Shukla - [The Hidden Risks of LLM-Generated Web Application Code](https://arxiv.org/abs/2504.20612)) +[swaroopdora2025f] "Encryption safeguards sensitive data at rest and in transit. Weak encryption methods or lack of encryption can expose passwords, personal information, and financial data... If passwords are stored in plain text or hashed without salting, attackers with database access can easily decrypt credentials, leading to mass account breaches." (Swaroop Dora, Deven Lunkad, Naziya Aslam, S. Venkatesan, Sandeep Kumar Shukla - [The Hidden Risks of LLM-Generated Web Application Code](https://arxiv.org/abs/2504.20612)) [josephspracklen2024a] "Package hallucination occurs when an LLM generates code that recommends or contains a reference to a package that does not actually exist. An adversary can exploit package hallucinations... by publishing a package to an open-source repository with the same name as the hallucinated or fictitious package and containing some malicious code/functionality" (Joseph Spracklen, Raveen Wijewickrama, A H M Nazmus Sakib, Anindya Maiti, Bimal Viswanath, Murtuza Jadliwala - [We Have a Package for You! A Comprehensive Analysis of Package Hallucinations by Code Generating LLMs](https://arxiv.org/abs/2406.10279)) From eaccd8e382ea39372103e30f3e34ceefd00420d0 Mon Sep 17 00:00:00 2001 From: balteravishay Date: Tue, 5 Aug 2025 08:49:55 +0100 Subject: [PATCH 30/31] lint Signed-off-by: balteravishay --- ...Security-Focused-Guide-for-AI-Code-Assistant-Instructions.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/Security-Focused-Guide-for-AI-Code-Assistant-Instructions.md b/docs/Security-Focused-Guide-for-AI-Code-Assistant-Instructions.md index cd07f649..78fa3527 100644 --- a/docs/Security-Focused-Guide-for-AI-Code-Assistant-Instructions.md +++ b/docs/Security-Focused-Guide-for-AI-Code-Assistant-Instructions.md @@ -143,7 +143,7 @@ To strengthen your AI assistant's guidance, you should point it toward establish * **Use Industry Standards as Keywords:** Mention standards like **OWASP Top 10**, **OWASP ASVS**, **CWE/SANS Top 25**, or language-specific guidelines (Java's SEI CERT secure coding standards, for instance). For example: *"Code suggestions should adhere to OWASP Top 10 principles (e.g., avoid injection, enforce access control) and follow the OWASP ASVS requirements where applicable"*. [[arifulhaque2025c]](#arifulhaque2025c) * **Reference SAFECode and Other Guides:** SAFECode (Software Assurance Forum for Excellence in Code) publishes **Fundamental Practices for Secure Software Development**, and including a note about it can be beneficial. *"Our project follows SAFECode's secure development practices – the AI should prioritize those (e.g., proper validation, authentication, cryptography usage per SAFECode guidance)"*. This ties the assistant's suggestions to a comprehensive set of secure development principles. Similarly, you can reference the **Secure Software Development Lifecycle (SSDLC)** or standards like **ISO 27034** if your organization uses them, to align AI outputs with those processes. [[neilperry2022e]](#neilperry2022e) * **Compliance and Regulatory Security:** If you operate in an industry with specific regulations (like healthcare or finance), instruct the AI with those in mind. For instance: *"When generating code, consider compliance requirements (e.g., HIPAA privacy rules for medical data, PCI-DSS for credit card info) – do not output code that logs or transmits sensitive data in insecure ways"*. This ensures the AI's suggestions are compliant with external standards, for example, it might suggest encryption for personal data fields by default. [[arifulhaque2025d]](#arifulhaque2025d) -* **Continuous Improvement Hooks:** Encourage practices that integrate with ongoing security processes. For example: *"Include comments or TODOs in code suggesting security reviews for complex logic, and note if any third-party component might need a future update or audit"*. This way, the AI not only writes code but also flags areas for human scrutiny or future action. If the AI outputs a piece of cryptography code, for instance, it might add a comment "// TODO: security review for cryptographic implementation" because you instructed it that critical code should not go un-reviewed. [[anssibsi2024h]](#anssibsi2024h) [[fossa2023]](#fossa2023)[[anssibsi2024d]](#anssibsi2024d) +* **Continuous Improvement Hooks:** Encourage practices that integrate with ongoing security processes. For example: *"Include comments or TODOs in code suggesting security reviews for complex logic, and note if any third-party component might need a future update or audit"*. This way, the AI not only writes code but also flags areas for human scrutiny or future action. If the AI outputs a piece of cryptography code, for instance, it might add a comment "// TODO: security review for cryptographic implementation" because you instructed it that critical code should not go un-reviewed. [[anssibsi2024h]](#anssibsi2024h) [[fossa2023]](#fossa2023)[[anssibsi2024d]](#anssibsi2024d) * **Integrate and Act on Automated Security Tooling:** Instruct the AI to proactively run automated security tools (e.g., SAST, DAST, SCA) during code generation and refinement. Example: "When writing or reviewing code, run or simulate the use of tools like CodeQL, Bandit, Semgrep, or OWASP Dependency-Check. Identify any flagged vulnerabilities or outdated dependencies and revise the code accordingly. Repeat this process until the code passes all simulated scans". This positions the AI not just as a code generator but as a security-aware assistant capable of iterating based on tool feedback. [[anssibsi2024i]](#anssibsi2024i) [[frontiers2024a]](#frontiers2024a) --- From 6fd7832cd81c4fe383a0c1bb199fe3bc3617d4e0 Mon Sep 17 00:00:00 2001 From: balteravishay Date: Tue, 5 Aug 2025 08:52:51 +0100 Subject: [PATCH 31/31] space Signed-off-by: balteravishay --- ...Security-Focused-Guide-for-AI-Code-Assistant-Instructions.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/Security-Focused-Guide-for-AI-Code-Assistant-Instructions.md b/docs/Security-Focused-Guide-for-AI-Code-Assistant-Instructions.md index 78fa3527..aca02c62 100644 --- a/docs/Security-Focused-Guide-for-AI-Code-Assistant-Instructions.md +++ b/docs/Security-Focused-Guide-for-AI-Code-Assistant-Instructions.md @@ -143,7 +143,7 @@ To strengthen your AI assistant's guidance, you should point it toward establish * **Use Industry Standards as Keywords:** Mention standards like **OWASP Top 10**, **OWASP ASVS**, **CWE/SANS Top 25**, or language-specific guidelines (Java's SEI CERT secure coding standards, for instance). For example: *"Code suggestions should adhere to OWASP Top 10 principles (e.g., avoid injection, enforce access control) and follow the OWASP ASVS requirements where applicable"*. [[arifulhaque2025c]](#arifulhaque2025c) * **Reference SAFECode and Other Guides:** SAFECode (Software Assurance Forum for Excellence in Code) publishes **Fundamental Practices for Secure Software Development**, and including a note about it can be beneficial. *"Our project follows SAFECode's secure development practices – the AI should prioritize those (e.g., proper validation, authentication, cryptography usage per SAFECode guidance)"*. This ties the assistant's suggestions to a comprehensive set of secure development principles. Similarly, you can reference the **Secure Software Development Lifecycle (SSDLC)** or standards like **ISO 27034** if your organization uses them, to align AI outputs with those processes. [[neilperry2022e]](#neilperry2022e) * **Compliance and Regulatory Security:** If you operate in an industry with specific regulations (like healthcare or finance), instruct the AI with those in mind. For instance: *"When generating code, consider compliance requirements (e.g., HIPAA privacy rules for medical data, PCI-DSS for credit card info) – do not output code that logs or transmits sensitive data in insecure ways"*. This ensures the AI's suggestions are compliant with external standards, for example, it might suggest encryption for personal data fields by default. [[arifulhaque2025d]](#arifulhaque2025d) -* **Continuous Improvement Hooks:** Encourage practices that integrate with ongoing security processes. For example: *"Include comments or TODOs in code suggesting security reviews for complex logic, and note if any third-party component might need a future update or audit"*. This way, the AI not only writes code but also flags areas for human scrutiny or future action. If the AI outputs a piece of cryptography code, for instance, it might add a comment "// TODO: security review for cryptographic implementation" because you instructed it that critical code should not go un-reviewed. [[anssibsi2024h]](#anssibsi2024h) [[fossa2023]](#fossa2023)[[anssibsi2024d]](#anssibsi2024d) +* **Continuous Improvement Hooks:** Encourage practices that integrate with ongoing security processes. For example: *"Include comments or TODOs in code suggesting security reviews for complex logic, and note if any third-party component might need a future update or audit"*. This way, the AI not only writes code but also flags areas for human scrutiny or future action. If the AI outputs a piece of cryptography code, for instance, it might add a comment "// TODO: security review for cryptographic implementation" because you instructed it that critical code should not go un-reviewed. [[anssibsi2024h]](#anssibsi2024h) [[fossa2023]](#fossa2023) [[anssibsi2024d]](#anssibsi2024d) * **Integrate and Act on Automated Security Tooling:** Instruct the AI to proactively run automated security tools (e.g., SAST, DAST, SCA) during code generation and refinement. Example: "When writing or reviewing code, run or simulate the use of tools like CodeQL, Bandit, Semgrep, or OWASP Dependency-Check. Identify any flagged vulnerabilities or outdated dependencies and revise the code accordingly. Repeat this process until the code passes all simulated scans". This positions the AI not just as a code generator but as a security-aware assistant capable of iterating based on tool feedback. [[anssibsi2024i]](#anssibsi2024i) [[frontiers2024a]](#frontiers2024a) ---