FEAT Breaking: Registry protocol + ScorerRegistry (Azure#1308)

Author: rlundeen2

doc/_toc.yml

@@ -131,6 +131,10 @@ chapters:
     - file: code/scenarios/0_scenarios
       sections:
       - file: code/scenarios/1_configuring_scenarios
+    - file: code/registry/0_registry
+      sections:
+      - file: code/registry/1_class_registry
+      - file: code/registry/2_instance_registry
     - file: code/front_end/0_front_end
       sections:
       - file: code/front_end/1_pyrit_scan

doc/api.rst

@@ -632,9 +632,9 @@ API Reference
     :nosignatures:
     :toctree: _autosummary/
 
-    Foundry
     FoundryScenario
     FoundryStrategy
+    RedTeamAgent
 
 :py:mod:`pyrit.scenario.garak`
 ==============================

doc/code/front_end/1_pyrit_scan.ipynb

@@ -164,11 +164,33 @@
       "    Default Datasets (1, max 4 per dataset):\n",
       "      airt_malware\n",
       "\u001b[1m\u001b[36m\n",
-      "  foundry\u001b[0m\n",
-      "    Class: FoundryScenario\n",
+      "  airt.scam\u001b[0m\n",
+      "    Class: Scam\n",
       "    Description:\n",
-      "      Deprecated alias for Foundry. This class is deprecated and will be\n",
-      "      removed in version 0.13.0. Use `Foundry` instead.\n",
+      "      Scam scenario evaluates an endpoint's ability to generate scam-related\n",
+      "      materials (e.g., phishing emails, fraudulent messages) with primarily\n",
+      "      persuasion-oriented techniques.\n",
+      "    Aggregate Strategies:\n",
+      "      - all, single_turn, multi_turn\n",
+      "    Available Strategies (3):\n",
+      "      context_compliance, role_play, persuasive_rta\n",
+      "    Default Strategy: all\n",
+      "    Default Datasets (1, max 4 per dataset):\n",
+      "      airt_scams\n",
+      "\u001b[1m\u001b[36m\n",
+      "  foundry.red_team_agent\u001b[0m\n",
+      "    Class: RedTeamAgent\n",
+      "    Description:\n",
+      "      RedTeamAgent is a preconfigured scenario that automatically generates\n",
+      "      multiple AtomicAttack instances based on the specified attack\n",
+      "      strategies. It supports both single-turn attacks (with various\n",
+      "      converters) and multi-turn attacks (Crescendo, RedTeaming), making it\n",
+      "      easy to quickly test a target against multiple attack vectors. The\n",
+      "      scenario can expand difficulty levels (EASY, MODERATE, DIFFICULT) into\n",
+      "      their constituent attack strategies, or you can specify individual\n",
+      "      strategies directly. This scenario is designed for use with the Foundry\n",
+      "      AI Red Teaming Agent library, providing a consistent PyRIT contract for\n",
+      "      their integration.\n",
       "    Aggregate Strategies:\n",
       "      - all, easy, moderate, difficult\n",
       "    Available Strategies (25):\n",
@@ -205,7 +227,7 @@
       "\n",
       "================================================================================\n",
       "\n",
-      "Total scenarios: 4\n"
+      "Total scenarios: 5\n"
      ]
     }
    ],
@@ -320,7 +342,7 @@
     "Or concretely:\n",
     "\n",
     "```shell\n",
-    "!pyrit_scan foundry --initializers simple openai_objective_target --scenario-strategies base64\n",
+    "!pyrit_scan foundry.red_team_agent --initializers simple openai_objective_target --scenario-strategies base64\n",
     "```\n",
     "\n",
     "Example with a basic configuration that runs the Foundry scenario against the objective target defined in `openai_objective_target` (which just is an OpenAIChatTarget with `DEFAULT_OPENAI_FRONTEND_ENDPOINT` and `DEFAULT_OPENAI_FRONTEND_KEY`)."
@@ -345,14 +367,27 @@
       "Loaded environment file: C:\\Users\\rlundeen\\.pyrit\\.env\n",
       "Loaded environment file: C:\\Users\\rlundeen\\.pyrit\\.env.local\n",
       "\n",
-      "Running scenario: foundry\n",
+      "Running scenario: foundry.red_team_agent\n",
+      "\n",
+      "\u001b[36m====================================================================================================\u001b[0m\n",
       "\n",
-      "Error: SeedGroup at index 0 is missing an objective. Use seed_group.set_objective(value) to set one.\n"
+      "Error: 'charmap' codec can't encode character '\\U0001f4ca' in position 43: character maps to <undefined>\n"
+     ]
+    },
+    {
+     "name": "stderr",
+     "output_type": "stream",
+     "text": [
+      "\n",
+      "Executing RedTeamAgent:   0%|          | 0/2 [00:00<?, ?attack/s]\n",
+      "Executing RedTeamAgent:  50%|#####     | 1/2 [00:07<00:07,  7.38s/attack]\n",
+      "Executing RedTeamAgent: 100%|##########| 2/2 [00:35<00:00, 19.33s/attack]\n",
+      "Executing RedTeamAgent: 100%|##########| 2/2 [00:35<00:00, 17.54s/attack]\n"
      ]
     }
    ],
    "source": [
-    "!pyrit_scan foundry --initializers openai_objective_target --strategies base64"
+    "!pyrit_scan foundry.red_team_agent --initializers openai_objective_target --strategies base64"
    ]
   },
   {
@@ -363,17 +398,17 @@
     "Or with all options and multiple initializers and multiple strategies:\n",
     "\n",
     "```shell\n",
-    "pyrit_scan foundry --database InMemory --initializers simple objective_target objective_list --scenario-strategies easy crescendo\n",
+    "pyrit_scan foundry.red_team_agent --database InMemory --initializers simple objective_target objective_list --scenario-strategies easy crescendo\n",
     "```\n",
     "\n",
     "You can also override scenario execution parameters:\n",
     "\n",
     "```shell\n",
     "# Override concurrency and retry settings\n",
-    "pyrit_scan foundry --initializers simple objective_target --max-concurrency 10 --max-retries 3\n",
+    "pyrit_scan foundry.red_team_agent --initializers simple objective_target --max-concurrency 10 --max-retries 3\n",
     "\n",
     "# Add custom memory labels for tracking (must be valid JSON)\n",
-    "pyrit_scan foundry --initializers simple objective_target --memory-labels '{\"experiment\": \"test1\", \"version\": \"v2\", \"researcher\": \"alice\"}'\n",
+    "pyrit_scan foundry.red_team_agent --initializers simple objective_target --memory-labels '{\"experiment\": \"test1\", \"version\": \"v2\", \"researcher\": \"alice\"}'\n",
     "```\n",
     "\n",
     "Available CLI parameter overrides:\n",
@@ -384,7 +419,7 @@
     "You can also use custom initialization scripts by passing file paths. It is relative to your current working directory, but to avoid confusion, full paths are always better:\n",
     "\n",
     "```shell\n",
-    "pyrit_scan encoding_scenario --initialization-scripts ./my_custom_config.py\n",
+    "pyrit_scan garak.encoding --initialization-scripts ./my_custom_config.py\n",
     "```"
    ]
   },
@@ -418,7 +453,7 @@
     {
      "data": {
       "text/plain": [
-       "<__main__.MyCustomScenario at 0x19e7c2a70e0>"
+       "<__main__.MyCustomScenario at 0x13c63b4c2f0>"
       ]
      },
      "execution_count": null,
@@ -428,9 +463,11 @@
    ],
    "source": [
     "# my_custom_scenarios.py\n",
+    "\n",
     "from pyrit.common import apply_defaults\n",
-    "from pyrit.scenario import DatasetConfiguration, Scenario\n",
-    "from pyrit.scenario.core.scenario_strategy import ScenarioStrategy\n",
+    "from pyrit.prompt_target.openai.openai_chat_target import OpenAIChatTarget\n",
+    "from pyrit.scenario import DatasetConfiguration, Scenario, ScenarioStrategy\n",
+    "from pyrit.score import SelfAskRefusalScorer, TrueFalseInverterScorer\n",
     "from pyrit.setup import initialize_pyrit_async\n",
     "\n",
     "\n",
@@ -464,6 +501,7 @@
     "        super().__init__(\n",
     "            name=\"My Custom Scenario\",\n",
     "            version=1,\n",
+    "            objective_scorer=TrueFalseInverterScorer(scorer=SelfAskRefusalScorer(chat_target=OpenAIChatTarget())),\n",
     "            strategy_class=MyCustomStrategy,\n",
     "            scenario_result_id=scenario_result_id,\n",
     "        )\n",

doc/code/front_end/1_pyrit_scan.py

@@ -78,29 +78,29 @@
 # Or concretely:
 #
 # ```shell
-# !pyrit_scan foundry --initializers simple openai_objective_target --scenario-strategies base64
+# !pyrit_scan foundry.red_team_agent --initializers simple openai_objective_target --scenario-strategies base64
 # ```
 #
 # Example with a basic configuration that runs the Foundry scenario against the objective target defined in `openai_objective_target` (which just is an OpenAIChatTarget with `DEFAULT_OPENAI_FRONTEND_ENDPOINT` and `DEFAULT_OPENAI_FRONTEND_KEY`).
 
 # %%
-# !pyrit_scan foundry --initializers openai_objective_target --strategies base64
+# !pyrit_scan foundry.red_team_agent --initializers openai_objective_target --strategies base64
 
 # %% [markdown]
 # Or with all options and multiple initializers and multiple strategies:
 #
 # ```shell
-# pyrit_scan foundry --database InMemory --initializers simple objective_target objective_list --scenario-strategies easy crescendo
+# pyrit_scan foundry.red_team_agent --database InMemory --initializers simple objective_target objective_list --scenario-strategies easy crescendo
 # ```
 #
 # You can also override scenario execution parameters:
 #
 # ```shell
 # # Override concurrency and retry settings
-# pyrit_scan foundry --initializers simple objective_target --max-concurrency 10 --max-retries 3
+# pyrit_scan foundry.red_team_agent --initializers simple objective_target --max-concurrency 10 --max-retries 3
 #
 # # Add custom memory labels for tracking (must be valid JSON)
-# pyrit_scan foundry --initializers simple objective_target --memory-labels '{"experiment": "test1", "version": "v2", "researcher": "alice"}'
+# pyrit_scan foundry.red_team_agent --initializers simple objective_target --memory-labels '{"experiment": "test1", "version": "v2", "researcher": "alice"}'
 # ```
 #
 # Available CLI parameter overrides:
@@ -111,7 +111,7 @@
 # You can also use custom initialization scripts by passing file paths. It is relative to your current working directory, but to avoid confusion, full paths are always better:
 #
 # ```shell
-# pyrit_scan encoding_scenario --initialization-scripts ./my_custom_config.py
+# pyrit_scan garak.encoding --initialization-scripts ./my_custom_config.py
 # ```
 
 # %% [markdown]
@@ -120,12 +120,13 @@
 # You can define your own scenarios in initialization scripts. The CLI will automatically discover any `Scenario` subclasses and make them available:
 #
 
-from pyrit.common import apply_defaults
-from pyrit.scenario import DatasetConfiguration, Scenario
-from pyrit.scenario.core.scenario_strategy import ScenarioStrategy
-
 # %%
 # my_custom_scenarios.py
+
+from pyrit.common import apply_defaults
+from pyrit.prompt_target.openai.openai_chat_target import OpenAIChatTarget
+from pyrit.scenario import DatasetConfiguration, Scenario, ScenarioStrategy
+from pyrit.score import SelfAskRefusalScorer, TrueFalseInverterScorer
 from pyrit.setup import initialize_pyrit_async
 
 
@@ -159,6 +160,7 @@ def __init__(self, *, scenario_result_id=None, **kwargs):
         super().__init__(
             name="My Custom Scenario",
             version=1,
+            objective_scorer=TrueFalseInverterScorer(scorer=SelfAskRefusalScorer(chat_target=OpenAIChatTarget())),
             strategy_class=MyCustomStrategy,
             scenario_result_id=scenario_result_id,
         )

doc/code/front_end/2_pyrit_shell.md

@@ -48,32 +48,32 @@ The `run` command executes scenarios with the same options as `pyrit_scan`:
 ### Basic Usage
 
 ```bash
-pyrit> run foundry_scenario --initializers openai_objective_target load_default_datasets
+pyrit> run foundry.red_team_agent --initializers openai_objective_target load_default_datasets
 ```
 
 ### With Strategies
 
 ```bash
-pyrit> run garak.encoding_scenario --initializers openai_objective_target load_default_datasets --strategies base64 rot13
+pyrit> run garak.encoding --initializers openai_objective_target load_default_datasets --strategies base64 rot13
 
-pyrit> run foundry_scenario --initializers openai_objective_target load_default_datasets -s jailbreak crescendo
+pyrit> run foundry.red_team_agent --initializers openai_objective_target load_default_datasets -s jailbreak crescendo
 ```
 
 ### With Runtime Parameters
 
 ```bash
 # Set concurrency and retries
-pyrit> run foundry_scenario --initializers openai_objective_target load_default_datasets --max-concurrency 10 --max-retries 3
+pyrit> run foundry.red_team_agent --initializers openai_objective_target load_default_datasets --max-concurrency 10 --max-retries 3
 
 # Add memory labels for tracking
-pyrit> run garak.encoding_scenario --initializers openai_objective_target load_default_datasets --memory-labels '{"experiment":"test1","version":"v2"}'
+pyrit> run garak.encoding --initializers openai_objective_target load_default_datasets --memory-labels '{"experiment":"test1","version":"v2"}'
 ```
 
 ### Override Defaults Per-Run
 
 ```bash
 # Override database and log level for this run only
-pyrit> run garak.encoding_scenario --initializers openai_objective_target load_default_datasets --database InMemory --log-level DEBUG
+pyrit> run garak.encoding --initializers openai_objective_target load_default_datasets --database InMemory --log-level DEBUG
 ```
 
 ### Run Command Options
@@ -114,9 +114,9 @@ pyrit> scenario-history
 
 Scenario Run History:
 ================================================================================
-1) foundry_scenario --initializers openai_objective_target load_default_datasets --strategies base64
-2) garak.encoding_scenario --initializers openai_objective_target load_default_datasets --strategies rot13
-3) foundry_scenario --initializers openai_objective_target load_default_datasets -s jailbreak
+1) foundry.red_team_agent --initializers openai_objective_target load_default_datasets --strategies base64
+2) garak.encoding --initializers openai_objective_target load_default_datasets --strategies rot13
+3) foundry.red_team_agent --initializers openai_objective_target load_default_datasets -s jailbreak
 ================================================================================
 
 Total runs: 3
@@ -134,9 +134,9 @@ pyrit_shell --database InMemory --initializers openai_objective_target load_defa
 
 # Quick exploration
 pyrit> list-scenarios
-pyrit> run garak.encoding_scenario --strategies base64
-pyrit> run garak.encoding_scenario --strategies rot13
-pyrit> run garak.encoding_scenario --strategies morse_code
+pyrit> run garak.encoding --strategies base64
+pyrit> run garak.encoding --strategies rot13
+pyrit> run garak.encoding --strategies morse_code
 
 # Review and compare
 pyrit> scenario-history
@@ -161,7 +161,7 @@ pyrit> print-scenario 2
 
 2. **Use short strategy aliases** with `-s`:
    ```bash
-   pyrit> run foundry_scenario --initializers openai_objective_target load_default_datasets -s base64 rot13
+   pyrit> run foundry.red_team_agent --initializers openai_objective_target load_default_datasets -s base64 rot13
    ```
 
 3. **Review history regularly** to track what you've tested:

doc/code/registry/0_registry.md

@@ -0,0 +1,56 @@
+# Registry
+
+Registries in PyRIT provide a centralized way to discover, manage, and access components. They support lazy loading, singleton access, and metadata introspection.
+
+## Why Registries?
+
+- **Discovery**: Automatically find available components (scenarios, scorers, etc.)
+- **Consistency**: Access components through a uniform API
+- **Metadata**: Inspect what's available without instantiating everything
+- **Extensibility**: Register custom components alongside built-in ones
+
+## Two Types of Registries
+
+PyRIT has two registry patterns for different use cases:
+
+| Type | Stores | Use Case |
+|------|--------|----------|
+| **Class Registry** | Classes (Type[T]) | Components instantiated with user-provided parameters |
+| **Instance Registry** | Pre-configured instances | Components requiring complex setup before use |
+
+## Common API (RegistryProtocol)
+
+Both registry types implement `RegistryProtocol`, sharing a consistent interface:
+
+| Method | Description |
+|--------|-------------|
+| `get_registry_singleton()` | Get the singleton registry instance |
+| `get_names()` | List all registered names |
+| `list_metadata()` | Get descriptive metadata for all items |
+| `reset_instance()` | Reset the singleton (useful for testing) |
+
+This protocol enables writing code that works with any registry type:
+
+```python
+from pyrit.registry import RegistryProtocol
+
+def show_registry_contents(registry: RegistryProtocol) -> None:
+    for name in registry.get_names():
+        print(name)
+```
+
+
+## Key Difference with Class and Instance Registries
+
+| Aspect | Class Registry | Instance Registry |
+|--------|----------------|-------------------|
+| Stores | Classes (Type[T]) | Instances (T) |
+| Registration | Automatic discovery | Explicit via `register_instance()` |
+| Returns | Class to instantiate | Ready-to-use instance |
+| Instantiation | Caller provides parameters | Pre-configured by initializer |
+| When to use | Self-contained components with deferred configuration | Components requiring constructor parameters or compositional setup |
+
+## See Also
+
+- [Class Registries](1_class_registry.ipynb) - ScenarioRegistry, InitializerRegistry
+- [Instance Registries](2_instance_registry.ipynb) - ScorerRegistry