feat: bulk operations, connection pooling, and migration CLI enhancements

.github/workflows/build.yml

@@ -7,6 +7,22 @@ on:
     branches: [main]
 
 jobs:
+  # Dependency review for PRs - checks for vulnerabilities in dependency changes
+  dependency-review:
+    runs-on: ubuntu-latest
+    if: github.event_name == 'pull_request'
+
+    steps:
+      - name: Checkout
+        uses: actions/checkout@v6
+
+      - name: Dependency Review
+        uses: actions/dependency-review-action@v4
+        with:
+          fail-on-severity: high
+          # Allow existing vulnerabilities, only fail on new ones
+          deny-licenses: ''
+
   build:
     runs-on: windows-latest
 
@@ -23,6 +39,21 @@ jobs:
     - name: Restore dependencies
       run: dotnet restore
 
+    - name: Check for vulnerable packages
+      shell: pwsh
+      run: |
+        $output = dotnet list package --vulnerable --include-transitive 2>&1
+        $output | Write-Host
+
+        if ($output -match "has the following vulnerable packages") {
+          Write-Host ""
+          Write-Host "::warning::Vulnerable packages detected - review security advisories above"
+          # Note: Not failing the build to avoid blocking on transitive dependencies
+          # that require upstream fixes. Dependency-review-action will catch new vulns.
+        } else {
+          Write-Host "No known vulnerabilities found in packages"
+        }
+
     - name: Build
       run: dotnet build --configuration Release --no-restore
 

.github/workflows/codeql.yml

@@ -0,0 +1,75 @@
+# =============================================================================
+# Security: CodeQL Analysis
+# =============================================================================
+# Performs semantic code analysis to find security vulnerabilities and
+# code quality issues in C# code.
+#
+# Triggers:
+#   - Push to main branch
+#   - Pull requests to main
+#   - Weekly schedule (Monday 9 AM UTC)
+#
+# For more info: https://docs.github.com/en/code-security/code-scanning
+# =============================================================================
+
+name: 'Security: CodeQL'
+
+on:
+  push:
+    branches:
+      - main
+    paths:
+      - 'src/**/*.cs'
+      - '**/*.csproj'
+      - '*.sln'
+
+  pull_request:
+    branches:
+      - main
+    paths:
+      - 'src/**/*.cs'
+      - '**/*.csproj'
+      - '*.sln'
+
+  schedule:
+    # Weekly scan on Monday at 9 AM UTC
+    - cron: '0 9 * * 1'
+
+  workflow_dispatch:
+
+jobs:
+  analyze:
+    name: Analyze C#
+    runs-on: ubuntu-latest
+
+    permissions:
+      security-events: write
+      packages: read
+      actions: read
+      contents: read
+
+    steps:
+      - name: Checkout repository
+        uses: actions/checkout@v6
+
+      - name: Initialize CodeQL
+        uses: github/codeql-action/init@v4
+        with:
+          languages: csharp
+          # Use security-and-quality for comprehensive analysis
+          queries: security-and-quality
+
+      - name: Setup .NET SDK
+        uses: actions/setup-dotnet@v5
+        with:
+          dotnet-version: '8.x'
+
+      - name: Build solution
+        run: |
+          dotnet restore
+          dotnet build --no-restore --configuration Release
+
+      - name: Perform CodeQL Analysis
+        uses: github/codeql-action/analyze@v4
+        with:
+          category: "/language:csharp"

.github/workflows/publish-nuget.yml

@@ -9,10 +9,10 @@ jobs:
     runs-on: windows-latest
 
     steps:
-    - uses: actions/checkout@v4
+    - uses: actions/checkout@v6
 
     - name: Setup .NET
-      uses: actions/setup-dotnet@v4
+      uses: actions/setup-dotnet@v5
       with:
         dotnet-version: |
           8.0.x

CHANGELOG.md

@@ -32,10 +32,15 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
   - Connection selection strategies: RoundRobin, LeastConnections, ThrottleAware
   - Throttle tracking with automatic routing away from throttled connections
   - Bulk operation wrappers: CreateMultiple, UpdateMultiple, UpsertMultiple, DeleteMultiple
+  - `IProgress<ProgressSnapshot>` support for real-time progress reporting during bulk operations
   - DI integration via `AddDataverseConnectionPool()` extension method
   - Affinity cookie disabled by default for improved throughput
   - Targets: `net8.0`, `net10.0`
 
+### Documentation
+
+- Added UpsertMultiple pitfalls section to `BULK_OPERATIONS_PATTERNS.md` - documents the duplicate key error when setting alternate key columns in both `KeyAttributes` and `Attributes`
+
 ### Changed
 
 - Updated publish workflow to support multiple packages and extract version from git tag

CLAUDE.md

@@ -1,6 +1,6 @@
 # CLAUDE.md - ppds-sdk
 
-**NuGet packages for Power Platform plugin development.**
+**NuGet packages for Power Platform development: plugin attributes, Dataverse connectivity, and migration tooling.**
 
 **Part of the PPDS Ecosystem** - See `C:\VS\ppds\CLAUDE.md` for cross-project context.
 
@@ -15,6 +15,10 @@
 | Skip XML documentation on public APIs | Consumers need IntelliSense documentation |
 | Multi-target without testing all frameworks | Dataverse has specific .NET requirements |
 | Commit with failing tests | All tests must pass before merge |
+| Create new ServiceClient per request | 42,000x slower than Clone/pool pattern; wastes ~446ms per instance |
+| Guess parallelism values | Use `RecommendedDegreesOfParallelism` from server; guessing degrades performance |
+| Enable affinity cookie for bulk operations | Routes all requests to single backend node; 10x throughput loss |
+| Store pooled clients in fields | Causes connection leaks; get per operation, dispose immediately |
 
 ---
 
@@ -28,6 +32,10 @@
 | Run `dotnet test` before PR | Ensures no regressions |
 | Update `CHANGELOG.md` with changes | Release notes for consumers |
 | Follow SemVer versioning | Clear compatibility expectations |
+| Use connection pool for multi-request scenarios | Reuses connections, applies performance settings automatically |
+| Dispose pooled clients with `await using` | Returns connections to pool; prevents leaks |
+| Use bulk APIs (`CreateMultiple`, `UpdateMultiple`, `UpsertMultiple`) | 5x faster than `ExecuteMultiple` (~10M vs ~2M records/hour) |
+| Reference Microsoft Learn docs in ADRs | Authoritative source for Dataverse best practices |
 
 ---
 
@@ -47,13 +55,25 @@
 ```
 ppds-sdk/
 ├── src/
-│   └── PPDS.Plugins/
-│       ├── Attributes/          # PluginStepAttribute, PluginImageAttribute
-│       ├── Enums/               # PluginStage, PluginMode, PluginImageType
-│       ├── PPDS.Plugins.csproj
-│       └── PPDS.Plugins.snk     # Strong name key (DO NOT regenerate)
+│   ├── PPDS.Plugins/
+│   │   ├── Attributes/          # PluginStepAttribute, PluginImageAttribute
+│   │   ├── Enums/               # PluginStage, PluginMode, PluginImageType
+│   │   ├── PPDS.Plugins.csproj
+│   │   └── PPDS.Plugins.snk     # Strong name key (DO NOT regenerate)
+│   ├── PPDS.Dataverse/
+│   │   ├── BulkOperations/      # CreateMultiple, UpdateMultiple, UpsertMultiple
+│   │   ├── Client/              # DataverseClient, IDataverseClient
+│   │   ├── Pooling/             # Connection pool, strategies
+│   │   ├── Resilience/          # Throttle tracking, retry logic
+│   │   └── PPDS.Dataverse.csproj
+│   ├── PPDS.Migration/          # Migration engine library
+│   └── PPDS.Migration.Cli/      # CLI tool (ppds-migrate)
 ├── tests/
-│   └── PPDS.Plugins.Tests/
+│   ├── PPDS.Plugins.Tests/
+│   └── PPDS.Dataverse.Tests/
+├── docs/
+│   ├── adr/                     # Architecture Decision Records
+│   └── architecture/            # Pattern documentation
 ├── .github/workflows/
 │   ├── build.yml               # CI build
 │   ├── test.yml                # CI tests
@@ -200,11 +220,64 @@ namespace PPDS.Plugins.Enums;        // Enums
 |------|---------|
 | `PPDS.Plugins.csproj` | Project config, version, NuGet metadata |
 | `PPDS.Plugins.snk` | Strong name key (DO NOT regenerate) |
+| `PPDS.Dataverse.csproj` | Dataverse client library |
 | `CHANGELOG.md` | Release notes |
 | `.editorconfig` | Code style settings |
 
 ---
 
+## ⚡ Dataverse Performance (PPDS.Dataverse)
+
+### Microsoft's Required Settings for Maximum Throughput
+
+The connection pool automatically applies these settings. If bypassing the pool, you MUST apply them manually:
+
+```csharp
+ThreadPool.SetMinThreads(100, 100);           // Default is 4
+ServicePointManager.DefaultConnectionLimit = 65000;  // Default is 2
+ServicePointManager.Expect100Continue = false;
+ServicePointManager.UseNagleAlgorithm = false;
+```
+
+### Service Protection Limits (Per User, Per 5-Minute Window)
+
+| Limit | Value |
+|-------|-------|
+| Requests | 6,000 |
+| Execution time | 20 minutes |
+| Concurrent requests | 52 (check `x-ms-dop-hint` header) |
+
+### Throughput Benchmarks (Microsoft Reference)
+
+| Approach | Throughput |
+|----------|------------|
+| Single requests | ~50K records/hour |
+| ExecuteMultiple | ~2M records/hour |
+| CreateMultiple/UpdateMultiple | ~10M records/hour |
+| Elastic tables | ~120M writes/hour |
+
+### Key Documentation
+
+- [Optimize performance for bulk operations](https://learn.microsoft.com/en-us/power-apps/developer/data-platform/optimize-performance-create-update)
+- [Send parallel requests](https://learn.microsoft.com/en-us/power-apps/developer/data-platform/send-parallel-requests)
+- [Service protection API limits](https://learn.microsoft.com/en-us/power-apps/developer/data-platform/api-limits)
+- [Use bulk operation messages](https://learn.microsoft.com/en-us/power-apps/developer/data-platform/bulk-operations)
+
+### Throttle Recovery (Known Limitation)
+
+The pool handles service protection errors transparently (waits and retries). However, it currently resumes at **full parallelism** after recovery, which can cause re-throttling with extended `Retry-After` durations.
+
+**Microsoft recommends** gradual ramp-up after throttle recovery. This is planned for a future enhancement (see ADR-0004).
+
+**Workaround**: Use lower `MaxParallelBatches` to reduce throttle frequency:
+
+```csharp
+var options = new BulkOperationOptions { MaxParallelBatches = 10 };
+await executor.UpsertMultipleAsync(entities, options);
+```
+
+---
+
 ## 🧪 Testing Requirements
 
 - **Target 80% code coverage**