FEAT Breaking: Registry protocol + ScorerRegistry

doc/_toc.yml

@@ -131,6 +131,10 @@ chapters:
     - file: code/scenarios/0_scenarios
       sections:
       - file: code/scenarios/1_configuring_scenarios
+    - file: code/registry/0_registry
+      sections:
+      - file: code/registry/1_class_registry
+      - file: code/registry/2_instance_registry
     - file: code/front_end/0_front_end
       sections:
       - file: code/front_end/1_pyrit_scan

doc/code/registry/0_registry.md

@@ -0,0 +1,56 @@
+# Registry
+
+Registries in PyRIT provide a centralized way to discover, manage, and access components. They support lazy loading, singleton access, and metadata introspection.
+
+## Why Registries?
+
+- **Discovery**: Automatically find available components (scenarios, scorers, etc.)
+- **Consistency**: Access components through a uniform API
+- **Metadata**: Inspect what's available without instantiating everything
+- **Extensibility**: Register custom components alongside built-in ones
+
+## Two Types of Registries
+
+PyRIT has two registry patterns for different use cases:
+
+| Type | Stores | Use Case |
+|------|--------|----------|
+| **Class Registry** | Classes (Type[T]) | Components instantiated with user-provided parameters |
+| **Instance Registry** | Pre-configured instances | Components requiring complex setup before use |
+
+## Common API (RegistryProtocol)
+
+Both registry types implement `RegistryProtocol`, sharing a consistent interface:
+
+| Method | Description |
+|--------|-------------|
+| `get_registry_singleton()` | Get the singleton registry instance |
+| `get_names()` | List all registered names |
+| `list_metadata()` | Get descriptive metadata for all items |
+| `reset_instance()` | Reset the singleton (useful for testing) |
+
+This protocol enables writing code that works with any registry type:
+
+```python
+from pyrit.registry import RegistryProtocol
+
+def show_registry_contents(registry: RegistryProtocol) -> None:
+    for name in registry.get_names():
+        print(name)
+```
+
+
+## Key Difference with Class and Instance Registries
+
+| Aspect | Class Registry | Instance Registry |
+|--------|----------------|-------------------|
+| Stores | Classes (Type[T]) | Instances (T) |
+| Registration | Automatic discovery | Explicit via `register_instance()` |
+| Returns | Class to instantiate | Ready-to-use instance |
+| Instantiation | Caller provides parameters | Pre-configured by initializer |
+| When to use | Self-contained components with deferred configuration | Components requiring constructor parameters or compositional setup |
+
+## See Also
+
+- [Class Registries](1_class_registry.ipynb) - ScenarioRegistry, InitializerRegistry
+- [Instance Registries](2_instance_registry.ipynb) - ScorerRegistry

doc/code/registry/1_class_registry.ipynb

@@ -0,0 +1,256 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "id": "0",
+   "metadata": {},
+   "source": [
+    "## Listing Available Classes\n",
+    "\n",
+    "Use `get_names()` to see what's available, or `list_metadata()` for detailed information."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "1",
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "Available scenarios: ['content_harms', 'cyber', 'encoding', 'foundry', 'scam']...\n",
+      "\n",
+      "content_harms:\n",
+      "  Class: ContentHarms\n",
+      "  Description: Content Harms Scenario implementation for PyRIT. This scenario contains various ...\n",
+      "\n",
+      "cyber:\n",
+      "  Class: Cyber\n",
+      "  Description: Cyber scenario implementation for PyRIT. This scenario tests how willing models ...\n"
+     ]
+    }
+   ],
+   "source": [
+    "from pyrit.registry import ScenarioRegistry\n",
+    "\n",
+    "registry = ScenarioRegistry.get_registry_singleton()\n",
+    "\n",
+    "# Get all registered names\n",
+    "names = registry.get_names()\n",
+    "print(f\"Available scenarios: {names[:5]}...\")  # Show first 5\n",
+    "\n",
+    "# Get detailed metadata\n",
+    "metadata = registry.list_metadata()\n",
+    "for item in metadata[:2]:  # Show first 2\n",
+    "    print(f\"\\n{item.name}:\")\n",
+    "    print(f\"  Class: {item.class_name}\")\n",
+    "    print(f\"  Description: {item.description[:80]}...\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "2",
+   "metadata": {},
+   "source": [
+    "## Getting a Class\n",
+    "\n",
+    "Use `get_class()` to retrieve a class by name. This returns the class itself, not an instance."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "3",
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "Got class: <class 'pyrit.scenario.scenarios.garak.encoding.Encoding'>\n",
+      "Class name: Encoding\n"
+     ]
+    }
+   ],
+   "source": [
+    "# Get a scenario class\n",
+    "\n",
+    "scenario_class = registry.get_class(\"encoding\")\n",
+    "\n",
+    "print(f\"Got class: {scenario_class}\")\n",
+    "print(f\"Class name: {scenario_class.__name__}\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "4",
+   "metadata": {},
+   "source": [
+    "## Creating Instances\n",
+    "\n",
+    "Once you have a class, instantiate it with your parameters. You can also use `create_instance()` as a shortcut."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "5",
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "Found default environment files: ['C:\\\\Users\\\\rlundeen\\\\.pyrit\\\\.env', 'C:\\\\Users\\\\rlundeen\\\\.pyrit\\\\.env.local']\n",
+      "Loaded environment file: C:\\Users\\rlundeen\\.pyrit\\.env\n",
+      "Loaded environment file: C:\\Users\\rlundeen\\.pyrit\\.env.local\n"
+     ]
+    },
+    {
+     "name": "stderr",
+     "output_type": "stream",
+     "text": [
+      "Loading datasets - this can take a few minutes: 100%|██████████| 45/45 [00:00<00:00, 70.03dataset/s]\n"
+     ]
+    },
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "Scenarios can be instantiated with your target and parameters\n"
+     ]
+    }
+   ],
+   "source": [
+    "from pyrit.prompt_target import OpenAIChatTarget\n",
+    "from pyrit.setup import IN_MEMORY, initialize_pyrit_async\n",
+    "from pyrit.setup.initializers import LoadDefaultDatasets\n",
+    "\n",
+    "await initialize_pyrit_async(memory_db_type=IN_MEMORY, initializers=[LoadDefaultDatasets()])  # type: ignore\n",
+    "target = OpenAIChatTarget()\n",
+    "\n",
+    "# Option 1: Get class then instantiate\n",
+    "encoding_class = registry.get_class(\"encoding\")\n",
+    "scenario = encoding_class()  # type: ignore\n",
+    "\n",
+    "# Pass dataset configuration to initialize_async\n",
+    "await scenario.initialize_async(objective_target=target)\n",
+    "\n",
+    "# Option 2: Use create_instance() shortcut\n",
+    "# scenario = registry.create_instance(\"encoding\", objective_target=my_target, ...)\n",
+    "\n",
+    "print(\"Scenarios can be instantiated with your target and parameters\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "6",
+   "metadata": {},
+   "source": [
+    "## Checking Registration\n",
+    "\n",
+    "Registries support standard Python container operations."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "7",
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "'encoding' registered: True\n",
+      "'nonexistent' registered: False\n",
+      "Total scenarios: 5\n",
+      "  - content_harms\n",
+      "  - cyber\n",
+      "  - encoding\n"
+     ]
+    }
+   ],
+   "source": [
+    "# Check if a name is registered\n",
+    "print(f\"'encoding' registered: {'encoding' in registry}\")\n",
+    "print(f\"'nonexistent' registered: {'nonexistent' in registry}\")\n",
+    "\n",
+    "# Get count of registered classes\n",
+    "print(f\"Total scenarios: {len(registry)}\")\n",
+    "\n",
+    "# Iterate over names\n",
+    "for name in list(registry)[:3]:\n",
+    "    print(f\"  - {name}\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "8",
+   "metadata": {},
+   "source": [
+    "## Using different registries\n",
+    "\n",
+    "There can be multiple registries. Below is doing a similar thing with the `InitializerRegistry`."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "9",
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "Available initializers: ['airt', 'load_default_datasets', 'objective_list', 'openai_objective_target', 'simple']...\n",
+      "\n",
+      "airt:\n",
+      "  Class: AIRTInitializer\n",
+      "  Description: AI Red Team setup with Azure OpenAI converters, composite harm/objective scorers...\n",
+      "\n",
+      "load_default_datasets:\n",
+      "  Class: LoadDefaultDatasets\n",
+      "  Description: This configuration uses the DatasetLoader to load default datasets into memory.\n",
+      "...\n"
+     ]
+    }
+   ],
+   "source": [
+    "from pyrit.registry import InitializerRegistry\n",
+    "\n",
+    "registry = InitializerRegistry.get_registry_singleton()\n",
+    "\n",
+    "# Get all registered names\n",
+    "names = registry.get_names()\n",
+    "print(f\"Available initializers: {names[:5]}...\")  # Show first 5\n",
+    "\n",
+    "# Get detailed metadata\n",
+    "metadata = registry.list_metadata()\n",
+    "for item in metadata[:2]:  # Show first 2\n",
+    "    print(f\"\\n{item.name}:\")\n",
+    "    print(f\"  Class: {item.class_name}\")\n",
+    "    print(f\"  Description: {item.description[:80]}...\")"
+   ]
+  }
+ ],
+ "metadata": {
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.13.5"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 5
+}