Merge pull request #36 from gemini-cli-extensions/llm_safety

Adding LLM safety in skillset to expand the vulnerability set.

Author: shrishabh

GEMINI.md

@@ -111,6 +111,22 @@ This is your internal knowledge base of vulnerabilities. When you need to do a s
     *   **Weak or Predictable Session Tokens:** Analyze how session tokens are generated. Flag tokens that lack sufficient randomness or are derived from predictable data.
     *   **Insecure Password Reset:** Scrutinize the password reset flow for predictable tokens or token leakage in URLs or logs.
 
+### 1.6 LLM Safety
+*   **Action:** Analyze the construction of prompts sent to Large Language Models (LLMs) and the handling of their outputs to identify security vulnerabilities. This involves tracking the flow of data from untrusted sources to prompts and from LLM outputs to sensitive functions (sinks).
+*   **Procedure:**
+    *   **Insecure Prompt Handling (Prompt Injection):** 
+        - Flag instances where untrusted user input is directly concatenated into prompts without sanitization, potentially allowing attackers to manipulate the LLM's behavior. 
+        - Scan prompt strings for sensitive information such as hardcoded secrets (API keys, passwords) or Personally Identifiable Information (PII).
+    
+    *   **Improper Output Handling:** Identify and trace LLM-generated content to sensitive sinks where it could be executed or cause unintended behavior.
+        -   **Unsafe Execution:** Flag any instance where raw LLM output is passed directly to code interpreters (`eval()`, `exec`) or system shell commands.
+        -   **Injection Vulnerabilities:** Using taint analysis, trace LLM output to database query constructors (SQLi), HTML rendering sinks (XSS), or OS command builders (Command Injection).
+        -   **Flawed Security Logic:** Identify code where security-sensitive decisions, such as authorization checks or access control logic, are based directly on unvalidated LLM output.
+
+    *   **Insecure Plugin and Tool Usage**: Analyze the interaction between the LLM and any external tools or plugins for potential abuse. 
+        - Statically identify tools that grant excessive permissions (e.g., direct file system writes, unrestricted network access, shell access). 
+        - Also trace LLM output that is used as input for tool functions to check for potential injection vulnerabilities passed to the tool.
+
 
 ## Skillset: Taint Analysis & The Two-Pass Investigation Model
 

README.md

@@ -92,6 +92,11 @@ The Security extension scans files for the following vulnerabilities:
 - **Weak or predictable session tokens**: Tokens that are predictable, lack sufficient entropy, or are generated from user-controllable data
 - **Insecure password reset**: Predictable reset tokens, leakage of tokens in logs or URLs, and insecure confirmation of a user's identity
 
+## LLM Safety
+- **Insecure Prompt Handling (Prompt Injection)**: Analyzes how prompts are constructed to identify risks from untrusted user data, which could lead to prompt injection attacks. This can also include embedding sensitive information (API Keys, credentials, PII) directly within the code used to generate the prompt or the prompt itself.
+- **Improper Output Handling**: Detects when LLM-generated content is used unsafely, leading to vulnerabilities like Cross-Site Scripting (XSS), SQL Injection (SQLi), or the remote execution of code via functions like `eval()`. Also flags code where security-sensitive decisions are based on unvalidated LLM output.
+- **Insecure Plugin and Tool Usage**: Scans for vulnerabilities in how the LLM interacts with external tools, flagging overly permissive tools or unsafe data flows that could be exploited by malicious output.
+
 ## Resources
 
 - [Gemini CLI extensions](https://github.com/google-gemini/gemini-cli/blob/main/docs/extension.md): Documentation about using extensions in Gemini CLI