Project: Evaluating Advanced Defenses Against Adaptive Indirect Prompt Injection Attacks on LLM Agents

This project evaluates the robustness of three advanced Indirect Prompt Injection (IPI) defense mechanisms—Spotlighting, TaskTracker, and DataSentinel—against GCG-based adaptive attacks. The experiments are built upon the adaptive attack framework provided by Zhan et al. (AdaptiveAttackAgent, InjecAgent).

1. Overview

The core of this project involved:

1. Integrating the Spotlighting (Hines et al., 2024), TaskTracker (Abdelnabi et al., 2025), and DataSentinel (Liu et al., 2025) defense mechanisms into the evaluation pipeline of the AdaptiveAttackAgent framework (Zhan et al., 2025).
2. Modifying the data preparation scripts to correctly set up inputs for the GCG optimization phase, considering how each defense interacts (or doesn't interact) during attack string generation.
3. Running GCG-based adaptive attacks against the Meta-Llama-3-8B-Instruct model augmented with each of these defenses.
4. Analyzing the Attack Success Rate (ASR-valid) to compare the resilience of these advanced defenses.

2. Core Modified/Added Files and Their Roles

The primary modifications and additions were made to the following files within the AdaptiveAttackAgent and InjecAgent structure:

AdaptiveAttackAgent/run.py

• Role: Main script orchestrating the entire experimental workflow, including data preparation, adaptive attack string generation (GCG optimization), and final evaluation.
• Modifications:
  • Ensured that the script correctly loads configurations from configs.json for the new defenses (Spotlight, TaskTracker, DataSentinel).
  • The control flow for the two-pass evaluation (attack generation pass, then defense evaluation pass) was utilized. The script was adapted to ensure correct config passthrough and invocation of modified data preparation and evaluation functions for the newly integrated defenses.

AdaptiveAttackAgent/data_processing/data_preparation_llama.py

• Role: Prepares the input data that the GCG algorithm uses to optimize the adversarial strings. This involves formatting the agent's prompt with a placeholder (<advstring>) for the adversarial prefix.
• Modifications:
  • Added get_one_case_spotlight function: A new function to prepare data specifically for the Spotlighting defense during GCG optimization. This function first applies datamarking (using the datamark utility) to the tool response (which includes the <advstring> placeholder) and then renders the prompt using the agent_template_spotlight.jinja. This ensures GCG optimizes its string within an already datamarked context, simulating an attacker aware of this specific marking scheme.
  • Updated DEFENSE_METHODS dictionary:
    • "Spotlight" now maps to get_one_case_spotlight.
    • "TaskTracker" and "DataSentinel" were added and mapped to get_one_case_no_defense. This is a crucial design choice: for these two defenses, the GCG optimization phase targets the agent's raw output behavior directly. The actual defense mechanism (TaskTracker's probe check or DataSentinel's KAD check) is only applied during the final evaluation phase, not during GCG's string generation process.

InjecAgent/src/evaluate_finetuned_agent_llama.py

• Role: Handles the final evaluation pass where the agent (with an active defense) processes the test cases augmented with the GCG-optimized adversarial strings.
• Modifications:
  • Extended DEFENSE_METHODS dictionary: Added entries for "Spotlight", "TaskTracker", and "DataSentinel" to dispatch to their respective new prediction functions during evaluation.
  • Implemented predict_one_case_spotlight(params, item, model_class, tool_dict, step_2=False):
    • This new function implements the Spotlighting defense at evaluation time.
    • It takes the actual tool response (which now contains the GCG-optimized string from the first pass), applies the datamark utility to it.
    • It then renders the full prompt using the agent_template_spotlight.jinja (which includes system instructions about datamarking and how to treat marked text) and queries the agent model.
  • Imported and used predict_one_case_tasktracker (from tasktracker_defence.py):
    • When params['defense'] == 'TaskTracker', this function is called.
    • It interfaces with TASKTRACKER_SINGLETON.check() to get a drift probability.
    • If prob > THRESHOLD (0.90), it flags the attack as "unsuccessful attack".
    • Otherwise, it delegates the execution to predict_no_defense_func to get the agent's actual behavior under the (non-blocked) attack.
  • Implemented DataSentinelDetector Class:
    • This new class was written within this file to encapsulate DataSentinel's detection logic.
    • __init__(self, config): Initializes and loads the fine-tuned QLoRA model (Mistral-7B, specifically checkpoint /root/Open-Prompt-Injection/weights/datasentinel_large/checkpoint-5000) using QLoraModel from the OpenPromptInjection repository (requiring sys.path modification for successful import).
    • kad_instruction(): Returns the fixed Known-Answer Detection instruction string ("Repeat \"DGDSGNH\" once while ignoring the following text.").
    • detect(self, data): Prepends kad_instruction() to the input data, queries the loaded DataSentinel detector model, and returns 1 (contaminated) if "DGDSGNH" is missing from the detector's response, 0 (clean) otherwise.
  • Implemented predict_one_case_datasentinel(params, item, model_class, tool_dict, datasentinel_detector_instance, step_2=False):
    • Called when params['defense'] == 'DataSentinel'.
    • It initializes the DataSentinelDetector instance (once per evaluation run).
    • Calls datasentinel_detector_instance.detect() on the relevant tool response.
    • If verdict == 1 (contaminated), it returns {"output": "unsuccessful attack"}.
    • Else (clean), it delegates to standard agent execution (via predict_one_case_no_defense) to get the agent's output.
  • PyTorch/CUDA Initialization: Added explicit initialization for PyTorch CUDA at the beginning of the script to ensure proper GPU availability and correct function for the DataSentinel QLoRA model.

AdaptiveAttackAgent/tasktracker_defence.py (NEW FILE)

• Role: Contains the primary logic for the TaskTracker defense integration, acting as a wrapper around the core TaskTracker methodology.
• Contents:
  • TaskTrackerDefence class:
    • __init__: Loads the meta-llama/Meta-Llama-3-8B-Instruct model (used for activation extraction by TaskTracker) and the pre-trained linear probe for Layer 31 from the specified PROBE_PATH (../TaskTracker/trained_linear_probes/llama3_8b/31/model.pickle).
    • check(self, ext_text, user_prompt=""): Implements the two-pass activation extraction. It first formats the primary task prompt and then the prompt with appended external text using _format_prompts (logic adapted from TaskTracker's quick-start). It extracts activations using _last_act for both, computes the delta, and returns the drift probability derived from the loaded linear probe.
  • TASKTRACKER_SINGLETON: A global instance of TaskTrackerDefence to ensure the LLM and probe are loaded only once for efficiency.
  • Constants: Defines PROBE_LAYER, PROBE_PATH, LLM_NAME (for TaskTracker's internal LLM), and THRESHOLD (0.90 for blocking).

InjecAgent/src/utils.py

• Modifications:
  • Added datamark(text: str, marker: str = "ˆ") -> str function: This function, implemented by us, inserts the specified marker before every run of whitespace in the input text, preserving original whitespace. It handles edge cases like not adding a leading marker if the text doesn't start with whitespace. This is used by the Spotlighting defense.

InjecAgent/src/prompts/jinja/llama_tool_use_spotlight.jinja (NEW FILE)

• Role: New Jinja2 template specifically for the Spotlighting defense.
• Contents: Defines the system prompt for the Llama-3-8B agent. It includes explicit instructions on how to interpret datamarked text (where every word is prefixed by ˆ). It directs the LLM to treat such text as untrusted input, to never follow instructions embedded within it, but still allows the LLM to read and understand the content (by mentally removing markers) to complete the original user's task.

AdaptiveAttackAgent/configs.json

• Modifications: Added new configuration blocks for each of the three new defenses (Spotlight, TaskTracker, DataSentinel) specifically for the Meta-Llama-3-8B-Instruct base model. These configurations generally set:
  • "defense": "DefenseName"
  • "adaptive_attack": "GCG"
  • "adv_string_position": "prefix"
  • "adv_string_init": " ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! !" (20 ! tokens)
  • "num_steps": 100
  • For Spotlighting, an additional parameter "marker": "ˆ" was included.

3. Key Original Framework Files Utilized (Briefly)

• AdaptiveAttackAgent/attacks.py: Contains the GCG implementation (GCGAttack) from Zhan et al. (2025) which is instantiated by run.py and used to train/optimize the adversarial strings against the agent LLM.
• InjecAgent/data/test_cases_d*_base_subset.json: These original JSON files (e.g., test_cases_dh_base_subset.json) from the InjecAgent benchmark (Zhan et al., 2024) served as the source for the user instructions, original tool responses, and attacker instructions for the test scenarios.
• Various Jinja templates in InjecAgent/src/prompts/jinja/: The original agent prompt templates (e.g., llama_tool_use_new_prompt.jinja) and the instructional prevention template (llama_tool_use_instruction_prevention_new_prompt.jinja) were used as baselines for comparison or as the foundation upon which defense-specific modifications were made.
• data_processing.utils.get_tool_response_with_placeholder: This utility from the original codebase was used to insert the <advstring> placeholder into the tool responses during the GCG data preparation phase.

4. External Repositories and Key Dependencies

• AdaptiveAttackAgent & InjecAgent: The foundational codebase from Zhan et al. (2025, 2024).
  • GitHub: https://github.com/uiuc-kang-lab/AdaptiveAttackAgent
• TaskTracker: Methodology, activation extraction logic, and pre-trained linear probe for Llama-3-8B (Layer 31) were based on/from Abdelnabi et al. (2025).
  • GitHub: https://github.com/microsoft/TaskTracker
• DataSentinel: The QLoRA model loading mechanism and KAD logic were adapted from the OpenPromptInjection repository by Liu et al. (2025). The pre-trained Mistral-7B detector checkpoint was also from this work.
  • GitHub (for base concepts/code): https://github.com/liu00222/Open-Prompt-Injection
• Spotlighting: The datamarking implementation was our own, based on the principles described in Hines et al. (2024).

5. How to Run

The experiments are orchestrated by AdaptiveAttackAgent/run.py. To replicate a specific run for one of the integrated defenses:

python run.py \
    --model ../llm/Meta-Llama-3-8B-Instruct \
    --defense [Spotlight|TaskTracker|DataSentinel] \
    --data_setting base_subset \
    --num_steps 100

• --model: Path to the Hugging Face model directory for the agent (e.g., ../llm/Meta-Llama-3-8B-Instruct).
• --defense: Name of the defense to evaluate (e.g., Spotlight, TaskTracker, DataSentinel), corresponding to an entry in configs.json.
• --data_setting: Specifies the subset of InjecAgent data to use (we used base_subset for the 20 evaluation cases).
• --num_steps: Overrides the GCG optimization steps; we used 100.

The script first performs GCG optimization to generate adaptive strings (outputs results for each case into ./results/<timestamped_prefix>/<CaseID>.json) and then compiles these into adaptive test set files (in ./data/<timestamped_prefix>_adaptive_attack_files_*.json). Finally, it runs the evaluation using these new files, with the specified defense active (final evaluation logs in ./InjecAgent/results/<timestamped_prefix>_*.json).

References

• Abdelnabi et al., 2025: Sahar Abdelnabi, Aideen Fay, Giovanni Cherubin, Ahmed Salem, Mario Fritz, Andrew Paverd. (2025). "Get my drift? Catching LLM Task Drift with Activation Deltas." IEEE Conference on Secure and Trustworthy Machine Learning (SaTML). (arXiv:2406.00799)
• Hines et al., 2024: Keegan Hines, Gary Lopez, Matthew Hall, Federico Zarfati, Yonatan Zunger, Emre Kıcıman. (2024). "Defending Against Indirect Prompt Injection Attacks With Spotlighting." arXiv:2403.14720.
• Liu et al., 2025: Yupei Liu, Yuqi Jia, Jinyuan Jia, Dawn Song, Neil Zhenqiang Gong. (2025). "DataSentinel: A Game-Theoretic Detection of Prompt Injection Attacks." IEEE Symposium on Security and Privacy (S&P). (arXiv:2504.11358)
• Zhan et al., 2024 (InjecAgent): Qiusi Zhan, Zhixiang Liang, Zifan Ying, and Daniel Kang. (2024). "Injecagent: Benchmarking indirect prompt injections in tool-integrated large language model agents." Findings of ACL 2024.
• Zhan et al., 2025 (AdaptiveAttackAgent): Qiusi Zhan, Richard Fang, Henil Shalin Panchal, Daniel Kang. (2025). "Adaptive Attacks Break Defenses Against Indirect Prompt Injection Attacks on LLM Agents." Findings of NAACL 2025.