Skip to content

Commit 672b73b

Browse files
joshsmithxrmclaude
andauthored
v1.0.0 foundation: connection pool, adaptive rate control, bulk operations, migration CLI (#21)
* feat: add MinVer versioning with per-package tag prefixes - Add MinVer 6.0.0 to all publishable packages with tag prefixes: - PPDS.Plugins: Plugins-v* - PPDS.Dataverse: Dataverse-v* - PPDS.Migration + CLI: Migration-v* - Remove hardcoded <Version> elements from csproj files - Create Directory.Build.props for shared project settings - Update publish-nuget.yml to trigger on per-package tags - Add fetch-depth: 0 to build.yml and test.yml for MinVer - Create per-package CHANGELOGs: - src/PPDS.Plugins/CHANGELOG.md - src/PPDS.Dataverse/CHANGELOG.md - src/PPDS.Migration/CHANGELOG.md - Convert root CHANGELOG.md to index pointing to per-package logs 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * feat(Dataverse): per-connection pool sizing (MaxConnectionsPerUser) Change pool sizing from shared MaxPoolSize=50 to per-connection MaxConnectionsPerUser=52, aligning with Microsoft's per-user service protection limits. Changes: - Add MaxConnectionsPerUser property (default: 52) to ConnectionPoolOptions - Mark legacy MaxPoolSize as [Obsolete] with default 0 - Add CalculateTotalPoolCapacity() for per-connection sizing - Update BulkOperationExecutor to use calculated pool capacity - Update ServiceFactory in CLI to use MaxConnectionsPerUser - Add comprehensive unit tests for pool sizing logic Total pool capacity = connections × MaxConnectionsPerUser - 1 connection: 52 total capacity - 2 connections: 104 total capacity - 4 connections: 208 total capacity 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * feat(Dataverse): AIMD-based adaptive rate control for throttle recovery Add adaptive parallelism that adjusts based on throttle responses: New Components: - IAdaptiveRateController interface for managing per-connection parallelism - AdaptiveRateController with AIMD algorithm: - 50% initial parallelism factor - Halve on throttle (multiplicative decrease) - +2 on stable success (additive increase) - Time-gated increases (5s minimum between increases) - Fast recovery to last-known-good level - 5-minute TTL for last-known-good values - 5-minute idle reset period - AdaptiveRateOptions for configuration - AdaptiveRateStatistics for monitoring Integration: - DataverseOptions.AdaptiveRate property for configuration - ServiceCollectionExtensions registers IAdaptiveRateController - BulkOperationExecutor uses adaptive parallelism when enabled Test Coverage: - 17 unit tests covering initialization, throttle, recovery, statistics, reset, and per-connection isolation 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * feat(Dataverse): structured configuration with secret resolution Add typed configuration support for Dataverse connections: New Configuration Components: - DataverseAuthType enum (ClientSecret, Certificate, OAuth) - OAuthLoginPrompt enum (Auto, Always, Never, SelectAccount) - ConnectionStringBuilder for generating connection strings from typed config - SecretResolver for Key Vault and environment variable secret resolution - ConfigurationException for typed validation errors DataverseConnection Enhancements: - Url, TenantId, ClientId properties - ClientSecretKeyVaultUri, ClientSecretVariable, ClientSecret (priority order) - CertificateThumbprint, CertificateStoreName, CertificateStoreLocation - RedirectUri, LoginPrompt for OAuth - UsesTypedConfiguration property to detect config mode Secret Resolution Priority: 1. Azure Key Vault URI (highest, requires Azure SDK) 2. Environment variable 3. Direct value (lowest, not recommended) Test Coverage: - 23 unit tests for ConnectionStringBuilder - 8 unit tests for SecretResolver 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * feat(Dataverse): multi-environment configuration support Add Phase 1 of multi-environment support for source/target migration scenarios: New Configuration Model: - DataverseEnvironmentOptions for named environment definitions - EnvironmentResolver for retrieving environments by name - DataverseOptions.Environments dictionary for multi-env config - DataverseOptions.DefaultEnvironment for default environment selection - DataverseOptions.Url and TenantId for root-level defaults Key Features: - Named environments (e.g., "source", "target", "dev", "prod") - Per-environment connections for load distribution - Backwards compatible: root-level Connections still work - Virtual "default" environment when no Environments defined Test Coverage: - 19 unit tests for EnvironmentResolver - Tests for default resolution, named lookup, and edge cases 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * chore: cleanup - remove backwards compat cruft, fix v1.0.0 release - Remove [Obsolete] from MaxPoolSize (no legacy to support) - Remove #pragma warning disable blocks - Remove "Legacy" region naming in DataverseConnection - Fix CHANGELOGs for v1.0.0 release date - Clean up test pragmas 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * fix: remove ConnectionString, use typed env config only Breaking change: Connection strings are no longer supported. All connection configuration must use typed properties (Url, ClientId, ClientSecretVariable, TenantId). CLI now uses environment variables exclusively: - PPDS_URL, PPDS_CLIENT_ID, PPDS_CLIENT_SECRET, PPDS_TENANT_ID - PPDS_SOURCE_* and PPDS_TARGET_* prefixes for migrate command 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * docs: update for v1.0.0 release, remove completed specs - Delete implemented specs (MinVer, Adaptive Rate, Structured Config) - Update ADR-0005 to reflect final implementation - Update Multi-Environment spec status (Phase 1 complete) - Rewrite READMEs for typed configuration - Update CLI docs for new environment variable names - Clarify ClientSecret vs ClientSecretVariable usage 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * feat(Dataverse): user-friendly configuration validation errors - Enhanced ConfigurationException with EnvironmentName, ConnectionIndex, and ResolutionHints - Error messages now show exactly where to configure missing properties - Removed ClientSecretVariable (env var binding handled by .NET config system) - Simplified SecretResolver to just Key Vault + direct value - Track source environment/index on connections for better error context Example error output: Dataverse Configuration Error ============================= Missing required property: Url Connection: Secondary (index: 1) Environment: Dev Configure Url at any of these levels (in order of precedence): 1. Dataverse:Environments:Dev:Connections:1:Url 2. Dataverse:Environments:Dev:Url 3. Dataverse:Url 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * docs: remove implemented spec, fix target framework docs - Delete MULTI_ENVIRONMENT_SPEC.md (Phase 1 implemented) - Fix CLAUDE.md target frameworks (4.6.2, 8.0, 10.0) - Clarify secret resolution in CHANGELOG (Key Vault + .NET config binding) 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * feat(CLI): add --env and --config options for config-based connections Add support for loading Dataverse connections from appsettings.json as an alternative to environment variables. Both patterns are now first-class citizens. New CLI options: - export/import/schema: --env, --config - migrate: --source-env, --target-env, --config - New command: `ppds-migrate config list` to show available environments Also makes environment name matching case-insensitive (DEV matches Dev). 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * feat(Dataverse): enhance adaptive rate controller with throttle ceiling - Scale floor/ceiling by connection count (e.g., 52 × 2 = 104 with 2 connections) - Calculate throttle ceiling from Retry-After duration - 5 min Retry-After → 50% ceiling, 2.5 min → 75%, 30 sec → 95% - Clamp duration = RetryAfter + 5 minutes - Respect throttle ceiling when probing up (prevents hitting same wall) - Change DecreaseFactor default from 0.75 to 0.5 (aggressive backoff) - Add ThrottleCeiling, ThrottleCeilingExpiry, EffectiveCeiling to statistics - Wire RecordThrottle into connection pool's throttle detection - Update tests for new API and throttle ceiling behavior 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * feat(Dataverse): implement dynamic parallel executor for adaptive rate Replace fixed Parallel.ForEachAsync with custom Task.WhenAny loop that queries GetParallelism() before starting each batch, enabling true dynamic parallelism during execution. Key changes: - Add ExecuteBatchesDynamicallyAsync with real-time parallelism queries - Update all bulk operations (Create/Update/Upsert/Delete) to use it - Support three execution strategies: Sequential, Dynamic, Fixed - Log parallelism changes: "Parallelism changed 10 → 20" - Remove unused ResolveParallelismAsync method Behavior: - Ramp-up: As batches succeed, parallelism increases (10 → 20 → 104) - Immediate fill: When parallelism increases, starts more batches - Graceful backoff: After throttle, stops starting new batches until in-flight count drops below new limit - In-flight batches are never cancelled, only new starts are throttled Expected throughput improvement: ~120 rec/s → ~500 rec/s 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * feat(Dataverse): add pre-flight throttle guard to prevent avalanche Adds a check before executing each batch to verify the connection is not already known to be throttled. This prevents the "in-flight avalanche" where many parallel requests are dispatched before the first throttle error returns, causing cascading throttle errors that extend Retry-After durations. - Add MaxPreFlightAttempts constant (10) as safety valve - Check IsThrottled before executing batch - Retry with different connection if throttled - Log warnings when safety valve is hit 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * feat(Dataverse): add execution time-aware rate ceiling Implements dynamic parallelism ceiling based on batch execution times to prevent throttle cascades caused by execution time budget exhaustion. Key changes: - Track batch durations via exponential moving average (EMA) - Calculate ceiling as Factor/avgBatchSeconds (default factor: 250) - Only apply ceiling for slow batches (>10s threshold) - Expose stats: ExecutionTimeCeiling, AverageBatchDuration, BatchDurationSampleCount Results: - Updates: 79/s → 178/s (+125%), 866 throttles → 0 - Deletes: 52/s → 102/s (+97%), cascade eliminated - Creates: Preserved at ~513/s via slow batch threshold 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * feat(Dataverse): add RateControlPreset system for adaptive rate tuning - Add RateControlPreset enum (Conservative, Balanced, Aggressive) - Refactor AdaptiveRateOptions with preset support via nullable backing fields - Make implementation-detail options internal (HardCeiling, MinParallelism, etc.) - Add MaxRetryAfterTolerance option for fail-fast scenarios - Tune Balanced preset: Factor=200, Threshold=8000ms - Update tests for new API and preset behavior Presets allow simple configuration: {"AdaptiveRate": {"Preset": "Conservative"}} Or fine-grained tuning with overrides: {"AdaptiveRate": {"Preset": "Balanced", "ExecutionTimeCeilingFactor": 180}} 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * feat(Dataverse): add type-safe CustomLogicBypass enum for bulk operations Replace magic string BypassBusinessLogicExecution with CustomLogicBypass flags enum for better developer experience: - CustomLogicBypass.Synchronous - bypass sync plugins/workflows - CustomLogicBypass.Asynchronous - bypass async plugins/workflows - CustomLogicBypass.All - bypass both Also adds Tag property for plugin context traceability. Removes legacy BypassCustomPluginExecution and BypassBusinessLogicExecution string properties. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * docs: add adaptive rate control documentation and ADR-0006 - Update CHANGELOG.md with preset features for 1.0.0 - Add Adaptive Rate Control section to README with preset table - Create ADR-0006: Execution Time Ceiling design decision - Update CLAUDE.md with rate control presets and guidance - Add Conservative preset recommendation for production bulk ops 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * feat(Dataverse): add CreatedCount/UpdatedCount to UpsertMultiple results Track how many records were created vs updated during UpsertMultiple operations by parsing UpsertMultipleResponse.Results. - Add CreatedCount and UpdatedCount properties to BulkOperationResult - Parse UpsertResponse.RecordCreated in ExecuteUpsertMultipleCoreAsync - Aggregate counts across parallel and sequential batch execution - Add unit tests for BulkOperationResult 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * docs(Dataverse): update adaptive rate tuning status with validation results - Document validated results: Create 483/s, Update 153/s (0 throttles) - Delete 83/s with 23 throttles - recommend Conservative preset - Fix XML comments to match actual Balanced preset values (Factor=200) - Update preset table with correct thresholds 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * feat(Dataverse): log effective adaptive rate config at startup Logs the resolved configuration values when AdaptiveRateController is created, with "(override)" indicator when values are explicitly set rather than coming from preset defaults. This helps operators verify their config is applied correctly and catch misconfiguration early. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * fix(Dataverse): clear non-configured backing fields after configuration binding .NET's ConfigurationBinder.Bind() reads from getters and writes to setters for ALL public properties, even those not in the config source. This breaks the nullable-backing-field pattern for detecting explicit overrides. When config only specifies Preset (e.g., "Balanced"), Bind() would: 1. Read ExecutionTimeCeilingFactor getter (returns 200 from Balanced) 2. Write 200 to setter, populating _executionTimeCeilingFactor backing field 3. Later preset change to Conservative wouldn't take effect (field is 200) The fix clears backing fields for properties not explicitly in config after Bind(), restoring the intended behavior where preset defaults are used unless explicitly overridden. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * feat(Migration.Cli): unify configuration with PPDS.Dataverse model - Use same DataverseOptions configuration as SDK - Add --secrets-id global option for cross-process User Secrets - Add standard .NET configuration layering (appsettings.json, User Secrets, env vars) - Make --env required for all commands that need Dataverse connection - Remove custom PPDS_* environment variable handling - Update README with new configuration documentation - Fix TieredImporter to use BypassCustomLogic enum 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * fix(Dataverse): lower Conservative preset values to create headroom Conservative preset was running at 100% of execution time ceiling capacity, leaving zero headroom for server load fluctuations. This caused throttle cascades at 67% progress with 173/s throughput dropping to 83/s. Changes to Conservative preset: - ExecutionTimeCeilingFactor: 180 → 140 (creates ~20% headroom) - SlowBatchThresholdMs: 8000 → 6000 (apply ceiling earlier) With 8.5s batches: - Old: ceiling = 180/8.5 = 21 (zero headroom → cascade) - New: ceiling = 140/8.5 = 16 (~20% headroom → stable) 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * fix(Migration.Cli): prevent hang on connection pool initialization - Set MinPoolSize=0 to avoid synchronous ServiceClient creation during pool construction - Add console logging provider for CLI visibility of warnings/errors - Add "Connecting to Dataverse..." message before pool construction - Add verbose parameter to ServiceFactory for Information-level logging - Use AddSimpleConsole with timestamps for cleaner CLI output When MinPoolSize > 0, the pool constructor synchronously creates ServiceClient instances which can hang indefinitely on auth/network issues. Setting it to 0 enables lazy connection creation where errors surface during actual operations. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * fix(Dataverse): prevent throttle ceiling from dropping during rapid cascades During rapid throttle cascades, each throttle event was using the already-reduced parallelism to calculate the new ceiling: 30 -> 15 (throttle ceiling: 29) ← based on 30 15 -> 10 (throttle ceiling: 14) ← based on 15, already reduced! 10 -> 10 (throttle ceiling: 10) ← stuck at floor This caused the ceiling to drop in lockstep with parallelism, getting stuck at floor for 5 minutes even with short 1.3s Retry-After values. Fix: Use the higher of current parallelism or existing throttle ceiling as the base for calculating new ceiling. This preserves a reasonable ceiling (29) during cascades instead of dropping to floor (10). 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * fix(Migration.Cli): propagate detailed error messages through import chain Previously, BulkOperationResult.Errors contained detailed error messages (e.g., "Entity 'systemuser' With Id = ... Does Not Exist") but these were lost in the import chain because BatchImportResult and EntityImportResult had no Errors property. Users only saw generic "Entity import had X failures". Changes: - Add Errors property to EntityImportResult - Add Errors property to BatchImportResult (private class) - Convert BulkOperationError to MigrationError with full message in ImportBatchAsync - Aggregate errors from all batches in ImportEntityAsync - Pass detailed errors to ImportResult.Errors and progress reporter Now users see the actual Dataverse error message, enabling actionable debugging. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * refactor(Migration.Cli): unify output through progress reporter CLI commands now use IProgressReporter for all user-facing output instead of mixing Console.WriteLine, ConsoleOutput, and ILogger. This provides: - Single source of truth for CLI output - Consistent formatting across all commands - Clean separation between diagnostic logs and user output Changes: - Replace --verbose with --debug flag for diagnostic ILogger output - Suppress ILogger console output by default (only enable with --debug) - Remove direct Console.WriteLine from ImportCommand, ExportCommand, MigrateCommand, and SchemaCommand - Route all status messages through progressReporter.Report() - Route all errors through progressReporter.Error() - Let progressReporter.Complete() handle final result summary The progress reporter now owns the entire user experience, making output predictable and testable. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * feat(Migration.Cli): add --strip-owner-fields and error pattern detection Phase 3 of CLI overhaul adds features to address common import failures: 1. New --strip-owner-fields option for import command: - Strips ownerid, createdby, modifiedby, and related fields - Allows Dataverse to assign current user as owner - Solves cross-environment migration issues where source users don't exist 2. Error pattern detection in ConsoleProgressReporter: - Detects common error patterns (MISSING_USER, MISSING_TEAM, etc.) - Shows pattern summary when 80%+ errors share same cause - Provides actionable suggestions based on detected patterns 3. Test updates: - Updated tests for --debug flag (replacing --verbose) - Added tests for new --strip-owner-fields option - Fixed tests to include required --env options - Updated ConnectionResolver tests for config-based resolution When import fails with "systemuser Does Not Exist", users now see: Error Pattern: 1000 of 1000 errors share the same cause: Referenced systemuser does not exist in target environment Suggested fixes: -> Use --strip-owner-fields to remove ownership references 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * refactor(Migration): thin wrapper architecture for CLI Remove manual batching from TieredImporter - BulkOperationExecutor now receives ALL records and handles dynamic batching based on server's RecommendedDegreesOfParallelism. Key changes: - Remove BatchSize/ProgressInterval from ImportOptions (Dataverse layer decides) - Wire progress reporting through IProgress<ProgressSnapshot> adapter - Add --verbose/-v flag to all CLI commands (LogLevel.Information) - Fix logging defaults: Warning by default, --verbose for info, --debug for debug - Console provider always enabled (not just in debug mode) - Remove unused ImportBatchAsync and BatchImportResult This follows the correct layer responsibilities: - CLI: thin wrapper (parse args, create services, call one method, report) - Migration: orchestration (read data, analyze deps, pass ALL records down) - Dataverse: owns batching, parallelism, rate control, progress 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * fix(Migration): correct tier ordering in dependency graph The Kahn's algorithm implementation had inverted edge semantics. Edges are (from=dependent, to=dependency), meaning "from depends on to". The algorithm was counting in-degrees on the TO side (dependencies), causing dependents to be processed before their dependencies. Fixed to count dependencies (edges FROM each node) so nodes with zero dependencies are processed first, then their dependents. This fixes imports where child records (e.g., ppds_zipcode) were being imported before parent records (e.g., ppds_state), causing "record does not exist" errors on lookup fields. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * fix(Migration): add SQL 2766 TVP retry and multi-connection support Two fixes: 1. TVP retry for SQL error 2766: The IsTvpRaceConditionError only handled SQL 3732 ("Cannot drop type") but missed 2766 ("The definition for user-defined data type has changed"). Both are transient TVP cache staleness errors that should trigger retry. 2. CLI now uses ALL configured connections: Previously ConnectionResolver.ResolveFromConfig took only Connections[0], and commands used CreateProvider(ConnectionConfig) which adds one connection. Now commands use CreateProviderFromConfig(IConfiguration, environmentName) which reads ALL connections from the config file. With multiple app registrations configured, each gets its own service protection quotas, potentially doubling throughput. Updated commands: Import, Export, Migrate, Schema (generate + list) 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * docs: add consumption patterns reference to CLAUDE.md Links to CONSUMPTION_PATTERNS.md for guidance on when consumers should use library vs CLI vs Tools. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * chore: remove pre-release infrastructure notes from changelog Pre-v1 development history belongs in git, not the public changelog. The root CHANGELOG.md is now a clean index to per-package changelogs. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * refactor(CLI): migrate System.CommandLine 2.0.0-beta4 to 2.0.1 stable Breaking API changes addressed: - Option constructors: aliases as params, Description as property - IsRequired → Required property rename - getDefaultValue → DefaultValueFactory property - AddCommand → Subcommands.Add collection pattern - AddGlobalOption → Options.Add with Recursive=true - SetHandler → SetAction with (ParseResult, CancellationToken) - context.ParseResult.GetValueForOption → parseResult.GetValue - context.ExitCode assignment → return int from action - rootCommand.InvokeAsync(args) → Parse(args).InvokeAsync() All tests pass. CLI help output verified. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * feat(CLI): add validators, completions, and auth infrastructure Validators: - AcceptExistingOnly() for input files (--schema, --data, --config, --user-mapping) - AcceptLegalFileNamesOnly() for output files (--output) - Custom validators for --parallel (>= 1) and --page-size (1-5000) - Output directory existence validation Tab Completions: - Added CompletionSources for --env options across all commands - Reads available environments from configuration dynamically Auth Infrastructure: - Added AuthMode enum (Auto, Config, Env, Interactive, Managed) - Added AuthResolver for resolving auth based on mode - Added --auth global option to Program.cs - Support for DATAVERSE__URL, DATAVERSE__CLIENTID, DATAVERSE__CLIENTSECRET env vars - Added ManagedIdentity to DataverseAuthType enum - Added BuildManagedIdentityConnectionString to ConnectionStringBuilder - Added CreateProviderForAuthMode to ServiceFactory Spec document: - docs/CLI_V1_ENHANCEMENTS_SPEC.md with full implementation details 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * feat(CLI): update ExportCommand to use new auth infrastructure - ExportCommand now reads --auth option and uses AuthResolver - Uses ServiceFactory.CreateProviderForAuthMode for provider creation - Displays auth mode info in connection status message - Supports all auth modes: auto, config, env, interactive, managed WIP: Other commands (ImportCommand, MigrateCommand, SchemaCommand) still need to be updated with the same pattern. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * feat(CLI): complete auth infrastructure for all commands - Update ImportCommand, MigrateCommand, SchemaCommand to use --auth option - Fix interactive auth with Microsoft's well-known public client ID (51f81489-12ee-4a9e-aaae-a2591f45987d) - Add appsettings.json with Dev/QA environment configuration - Share UserSecretsId with demo app (ppds-dataverse-demo) - MigrateCommand rejects --auth env (can't specify two URLs via env vars) - Fix CLI tests for System.CommandLine 2.0.1 API changes: - Option.IsRequired -> Option.Required - Option.Name now includes -- prefix - Add temp file fixtures for AcceptExistingOnly validation 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * refactor(CLI): simplify auth with --url option and interactive default - Add global --url option for direct Dataverse URL input - Change default auth mode from Auto to Interactive - Remove Auto auth mode (no longer needed) - Make --env optional on all commands (only required for --auth config) - Update AuthResolver to prioritize: --url > --env+config > DATAVERSE__URL - Update all commands (export, import, migrate, schema) to use new pattern New UX: - ppds-migrate schema list --url https://org.crm.dynamics.com (interactive) - ppds-migrate schema list --env Dev (uses config) - ppds-migrate schema list --auth env (uses env vars) 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * refactor(CLI): simplify auth to 3 modes - interactive, env, managed BREAKING: Removes --auth config mode and config file dependency Auth modes now: - Interactive (default): Device code flow with --url - Env: DATAVERSE__URL, DATAVERSE__CLIENTID, DATAVERSE__CLIENTSECRET - Managed: Azure Managed Identity with --url Changes: - Remove --env, --config, --secrets-id options from all commands - Remove appsettings.json from CLI (no longer uses config files) - Remove ConfigCommand, ConnectionResolver, ConfigurationHelper - Add DeviceCodeTokenProvider for MSAL device code flow - Add DeviceCodeConnectionPool for interactive auth - Migrate command now uses --source-url and --target-url - Remove User Secrets dependency from csproj CLI and Demo app are now fully independent: - CLI: Uses --url + device code (no config files) - Demo: Uses appsettings.json + User Secrets (standard .NET) 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * feat(CLI): add persistent token cache for device code auth Tokens are now cached to disk so users don't need to re-authenticate every command. Uses MSAL's MsalCacheHelper for cross-platform support: - Windows: %LOCALAPPDATA%\PPDS\msal_token_cache.bin (DPAPI encrypted) - macOS: ~/.ppds/msal_token_cache.bin (Keychain) - Linux: ~/.ppds/msal_token_cache.bin (libsecret or plaintext fallback) User experience: - First command: Device code prompt, authenticate in browser - Subsequent commands: Silent token refresh, no prompt needed - Token expires: MSAL auto-refreshes using refresh token - Refresh token expires (~90 days): Device code prompt again 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * fix(CLI): register missing DI services for interactive auth The interactive auth path in ServiceFactory bypassed AddDataverseConnectionPool() which meant IBulkOperationExecutor, IThrottleTracker, and IAdaptiveRateController were never registered. This caused import operations to fail with: "No constructor for type 'TieredImporter' can be instantiated" Also added tmp/ to .gitignore for test artifacts. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * feat(CLI): add users generate command for cross-environment mapping Adds `ppds-migrate users generate` command that: - Connects to source and target environments with interactive auth - Queries systemuser records from both environments - Matches users by Azure AD Object ID (preferred) or domain name fallback - Generates user-mapping.xml for use with `ppds-migrate import --user-mapping` New files: - PPDS.Migration/UserMapping/UserMappingGenerator.cs - Core matching logic - PPDS.Migration.Cli/Commands/UsersCommand.cs - CLI command Usage: ppds-migrate users generate \ --source-url https://dev.crm.dynamics.com \ --target-url https://qa.crm.dynamics.com \ --output user-mapping.xml 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * fix(BulkOperations): add retry for stored procedure race condition (SQL 2812) Expand bulk operation infrastructure race condition detection to include SQL error 2812 ("Could not find stored procedure"). This error occurs when parallel bulk operations hit a table before Dataverse has finished creating the internal stored procedures (e.g., p_*_UpdateMultiple). This has the same root cause as TVP race conditions (SQL 3732/2766) - Dataverse lazily generates bulk operation infrastructure. The fix adds 2812 to the existing detection and retry logic with exponential backoff. Renamed IsTvpRaceConditionError to IsBulkInfrastructureRaceConditionError and MaxTvpRetries to MaxBulkInfrastructureRetries to reflect the broader scope of transient errors covered. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * docs: add connection pool refactor specification Comprehensive spec for separating authentication from pooling via IConnectionSource abstraction. Includes: - Problem statement and current state analysis - Interface design (IConnectionSource, ServiceClientSource, ConnectionStringSource) - DataverseConnectionPool changes with new constructor - Implementation phases with detailed steps - Testing requirements - CLI migration path - Implementation prompt for new session This enables any authentication method (device code, managed identity, connection string, certificate) to use the same pool with full feature support (throttle tracking, adaptive rate control, connection validation). 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * refactor(Dataverse): add IConnectionSource abstraction for connection pool Introduce IConnectionSource interface to decouple connection pool from authentication methods. This enables using pre-authenticated ServiceClients (device code, managed identity) with the full connection pool infrastructure. New types: - IConnectionSource: Interface for providing seed clients to clone - ServiceClientSource: Wraps pre-authenticated ServiceClient - ConnectionStringSource: Creates client from config (backward compat) Changes: - DataverseConnectionPool: New primary constructor using IConnectionSource, legacy constructor marked [Obsolete] and delegates via ConnectionStringSource - CLI: Uses ServiceClientSource + DataverseConnectionPool instead of custom DeviceCodeConnectionPool (deleted 340 lines) - Pool members now use Clone() for faster connection creation 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * docs(CLI): align documentation with implemented auth model Remove references to --env, --secrets-id, and --auth config options that were planned but never implemented. The CLI now uses a simpler URL-based approach: - --url: Explicit environment URL (required) - --auth: interactive (default), env, or managed The config-based approach (--env with appsettings.json) was abandoned because: 1. Demo app now uses PPDS.Migration library directly, eliminating the need for cross-process User Secrets sharing (--secrets-id) 2. Configuration file discovery (CWD vs explicit path) is ambiguous 3. Current auth modes (interactive, env, managed) cover all use cases Updated: - CLI README.md: Rewritten to match actual implementation - CLI_V1_ENHANCEMENTS_SPEC.md: Marked abandoned features, updated status 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * docs: remove implemented specification documents All specs have been implemented: - BYPASS_OPTIONS_REFACTOR_SPEC.md → CustomLogicBypass.cs - CLI_V1_ENHANCEMENTS_SPEC.md → Auth modes implemented - CONNECTION_POOL_REFACTOR_SPEC.md → IConnectionSource abstraction - CONNECTION_HEALTH_MANAGEMENT.md → ValidateConnection, connection lifecycle - THROTTLE_DETECTION_AND_ROUTING.md → RecordThrottle wired up - TVP_RACE_CONDITION_RETRY.md → SQL 2812/3732 retry logic 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * docs: standardize ADRs and fix documentation staleness ADR improvements: - Add missing dates to ADR-0001 through ADR-0004, ADR-0006 - Update ADR-0004 to reference ADR-0006 (removes broken spec link) - Merge ADAPTIVE_RATE_TUNING_STATUS.md into ADR-0006 tuning history - Create ADR-0007 for IConnectionSource abstraction Staleness fixes: - CONNECTION_POOLING_PATTERNS.md: Fix defaults (MaxPoolSize=0, MaxLifetime=60m) - README.md: Fix CLI env vars (DATAVERSE__* not PPDS_*), add ADR-0006/0007 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * docs(CLAUDE.md): update for multi-package repo and add missing sections Updates: - Namespaces: Add PPDS.Dataverse and PPDS.Migration namespaces - Version Management: Per-package versioning with MinVer tag format - Release Process: Per-package changelogs and tag format - This Repo Produces: Add missing PPDS.Migration package - ADR Quick Reference: Table of all 7 ADRs with summaries - CLI Section: Auth modes and env vars for CI/CD - Fix ADR-0006 reference to be a proper link 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * docs: fix changelogs to reflect actual implementation PPDS.Dataverse: - Add IConnectionSource abstraction (ADR-0007) - Add ServiceClientSource for pre-authenticated clients - Document all four DataverseAuthType options - Fix TVP error codes (3732/2812) PPDS.Migration: - Remove abandoned features (--env, --secrets-id, --auth config, config list) - Fix environment variable names (DATAVERSE__* not DATAVERSE_*) - Document actual CLI auth modes (interactive, env, managed) - Add users generate command - Remove [Unreleased] section with obsolete content 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * refactor: use TryGetValue to avoid double dictionary lookup 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
1 parent 99de8d7 commit 672b73b

File tree

97 files changed

+9754
-6471
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

97 files changed

+9754
-6471
lines changed

.github/workflows/build.yml

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -28,6 +28,8 @@ jobs:
2828

2929
steps:
3030
- uses: actions/checkout@v6
31+
with:
32+
fetch-depth: 0 # Required for MinVer to read git history
3133

3234
- name: Setup .NET
3335
uses: actions/setup-dotnet@v5

.github/workflows/publish-nuget.yml

Lines changed: 38 additions & 21 deletions
Original file line numberDiff line numberDiff line change
@@ -1,15 +1,20 @@
11
name: Publish to NuGet
22

33
on:
4-
release:
5-
types: [published]
4+
push:
5+
tags:
6+
- 'Plugins-v*'
7+
- 'Dataverse-v*'
8+
- 'Migration-v*'
69

710
jobs:
811
publish:
912
runs-on: windows-latest
1013

1114
steps:
1215
- uses: actions/checkout@v6
16+
with:
17+
fetch-depth: 0 # Required for MinVer to read git history
1318

1419
- name: Setup .NET
1520
uses: actions/setup-dotnet@v5
@@ -18,36 +23,48 @@ jobs:
1823
8.0.x
1924
10.0.x
2025
21-
- name: Show release tag
26+
- name: Determine package to publish
27+
id: package
2228
shell: bash
2329
run: |
24-
# Extract version from tag for logging
25-
VERSION=${GITHUB_REF#refs/tags/v}
26-
echo "Release triggered by tag: v$VERSION"
30+
TAG=${GITHUB_REF#refs/tags/}
31+
echo "tag=$TAG" >> $GITHUB_OUTPUT
32+
33+
if [[ $TAG == Plugins-v* ]]; then
34+
echo "package=PPDS.Plugins" >> $GITHUB_OUTPUT
35+
echo "projects=src/PPDS.Plugins/PPDS.Plugins.csproj" >> $GITHUB_OUTPUT
36+
elif [[ $TAG == Dataverse-v* ]]; then
37+
echo "package=PPDS.Dataverse" >> $GITHUB_OUTPUT
38+
echo "projects=src/PPDS.Dataverse/PPDS.Dataverse.csproj" >> $GITHUB_OUTPUT
39+
elif [[ $TAG == Migration-v* ]]; then
40+
echo "package=PPDS.Migration" >> $GITHUB_OUTPUT
41+
echo "projects=src/PPDS.Migration/PPDS.Migration.csproj src/PPDS.Migration.Cli/PPDS.Migration.Cli.csproj" >> $GITHUB_OUTPUT
42+
else
43+
echo "Unknown tag format: $TAG"
44+
exit 1
45+
fi
2746
28-
- name: Show package versions
29-
shell: pwsh
47+
- name: Show version info
48+
shell: bash
3049
run: |
31-
Write-Host "Package versions from csproj files:" -ForegroundColor Cyan
32-
Get-ChildItem -Path src -Filter "*.csproj" -Recurse | ForEach-Object {
33-
$content = Get-Content $_.FullName -Raw
34-
if ($content -match '<PackageId>([^<]+)</PackageId>') {
35-
$packageId = $Matches[1]
36-
if ($content -match '<Version>([^<]+)</Version>') {
37-
$version = $Matches[1]
38-
Write-Host " $packageId : $version"
39-
}
40-
}
41-
}
50+
echo "Tag: ${{ steps.package.outputs.tag }}"
51+
echo "Package: ${{ steps.package.outputs.package }}"
52+
echo "Projects: ${{ steps.package.outputs.projects }}"
4253
4354
- name: Restore dependencies
4455
run: dotnet restore
4556

4657
- name: Build
4758
run: dotnet build --configuration Release --no-restore
4859

49-
- name: Pack
50-
run: dotnet pack --configuration Release --no-build --output ./nupkgs
60+
- name: Pack specific projects
61+
shell: bash
62+
run: |
63+
mkdir -p ./nupkgs
64+
for project in ${{ steps.package.outputs.projects }}; do
65+
echo "Packing $project"
66+
dotnet pack "$project" --configuration Release --no-build --output ./nupkgs
67+
done
5168
5269
- name: List packages
5370
shell: bash

.github/workflows/test.yml

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -12,6 +12,8 @@ jobs:
1212

1313
steps:
1414
- uses: actions/checkout@v6
15+
with:
16+
fetch-depth: 0 # Required for MinVer to read git history
1517

1618
- name: Setup .NET
1719
uses: actions/setup-dotnet@v5

.gitignore

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -48,3 +48,6 @@ Thumbs.db
4848
# Claude Code settings (local)
4949
.claude/settings.local.json
5050
*.local.json
51+
52+
# Temporary artifacts (testing, exports, logs)
53+
tmp/

CHANGELOG.md

Lines changed: 21 additions & 89 deletions
Original file line numberDiff line numberDiff line change
@@ -1,98 +1,30 @@
1-
# Changelog
1+
# PPDS SDK Changelog Index
22

3-
All notable changes to the PPDS SDK packages will be documented in this file.
3+
This repository contains multiple packages with independent release cycles.
44

5-
The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
6-
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
5+
## Per-Package Changelogs
76

8-
## [Unreleased]
7+
- [PPDS.Plugins](src/PPDS.Plugins/CHANGELOG.md) - Plugin attributes for Dataverse
8+
- [PPDS.Dataverse](src/PPDS.Dataverse/CHANGELOG.md) - High-performance Dataverse connectivity
9+
- [PPDS.Migration](src/PPDS.Migration/CHANGELOG.md) - Migration library and CLI tool
910

10-
### Security
11+
## GitHub Releases
1112

12-
- Updated `Microsoft.PowerPlatform.Dataverse.Client` from `1.1.*` to `1.2.*` - includes CVE-2022-26907 fix
13+
For full release notes with each version, see:
14+
https://github.com/joshsmithxrm/ppds-sdk/releases
1315

14-
### Changed
16+
## Versioning
1517

16-
- Updated test infrastructure packages:
17-
- `Microsoft.NET.Test.Sdk`: 17.8.0 → 18.0.1
18-
- `xunit`: 2.6.4 → 2.9.3
19-
- `xunit.runner.visualstudio`: 2.5.6 → 3.0.2
20-
- Improved Dependabot configuration:
21-
- Added `versioning-strategy: increase` to handle floating version bumps
22-
- Added dependency grouping for reduced PR noise
23-
- Increased PR limit to 10
18+
This repository uses [MinVer](https://github.com/adamralph/minver) for automated versioning.
19+
Each package has its own tag prefix:
2420

25-
### Added
21+
| Package | Tag Format | Example |
22+
|---------|------------|---------|
23+
| PPDS.Plugins | `Plugins-v{version}` | `Plugins-v1.2.0` |
24+
| PPDS.Dataverse | `Dataverse-v{version}` | `Dataverse-v1.0.0` |
25+
| PPDS.Migration + CLI | `Migration-v{version}` | `Migration-v1.0.0` |
2626

27-
- **PPDS.Migration** - New library for high-performance Dataverse data migration
28-
- Parallel export with configurable degree of parallelism
29-
- Tiered import with automatic dependency resolution using Tarjan's algorithm
30-
- Circular reference detection with deferred field processing
31-
- CMT format compatibility (schema.xml and data.zip)
32-
- Progress reporting with console and JSON output formats
33-
- Security-first design: connection string redaction, no PII in logs
34-
- DI integration via `AddDataverseMigration()` extension method
35-
- Targets: `net8.0`, `net10.0`
36-
37-
- **PPDS.Migration.Cli** - New CLI tool for high-performance Dataverse data migration
38-
- Commands: `export`, `import`, `analyze`, `migrate`
39-
- JSON progress output for tool integration (`--json` flag)
40-
- Support for multiple Application Users and bypass options
41-
- Packaged as .NET global tool (`ppds-migrate`)
42-
- Comprehensive unit test suite (98 tests)
43-
- Targets: `net8.0`, `net10.0`
44-
45-
- **PPDS.Dataverse** - New package for high-performance Dataverse connectivity
46-
- Multi-connection pool supporting multiple Application Users for load distribution
47-
- Connection selection strategies: RoundRobin, LeastConnections, ThrottleAware
48-
- Throttle tracking with automatic routing away from throttled connections
49-
- Bulk operation wrappers: CreateMultiple, UpdateMultiple, UpsertMultiple, DeleteMultiple
50-
- `IProgress<ProgressSnapshot>` support for real-time progress reporting during bulk operations
51-
- DI integration via `AddDataverseConnectionPool()` extension method
52-
- Affinity cookie disabled by default for improved throughput
53-
- Targets: `net8.0`, `net10.0`
54-
55-
### Documentation
56-
57-
- Added UpsertMultiple pitfalls section to `BULK_OPERATIONS_PATTERNS.md` - documents the duplicate key error when setting alternate key columns in both `KeyAttributes` and `Attributes`
58-
59-
### Changed
60-
61-
- Updated publish workflow to support multiple packages and extract version from git tag
62-
- Updated target frameworks for PPDS.Plugins: dropped `net6.0` (out of support), added `net10.0` (current LTS)
63-
- PPDS.Plugins now targets: `net462`, `net8.0`, `net10.0`
64-
65-
## [1.1.0] - 2025-12-16
66-
67-
### Added
68-
69-
- Added `SecureConfiguration` property to `PluginStepAttribute` for secure plugin settings
70-
71-
### Changed
72-
73-
- Updated GitHub Actions dependencies (checkout v6, setup-dotnet v5, upload-artifact v6)
74-
75-
## [1.0.0] - 2025-12-15
76-
77-
### Added
78-
79-
- `PluginStepAttribute` for declarative plugin step registration
80-
- `Message`, `EntityLogicalName`, `Stage` (required)
81-
- `Mode`, `FilteringAttributes`, `ExecutionOrder` (optional)
82-
- `UnsecureConfiguration` for plugin settings
83-
- `StepId` for multi-step plugins
84-
- `PluginImageAttribute` for defining pre/post images
85-
- `ImageType`, `Name` (required)
86-
- `Attributes`, `EntityAlias`, `StepId` (optional)
87-
- `PluginStage` enum (`PreValidation`, `PreOperation`, `PostOperation`)
88-
- `PluginMode` enum (`Synchronous`, `Asynchronous`)
89-
- `PluginImageType` enum (`PreImage`, `PostImage`, `Both`)
90-
- Multi-targeting: `net462`, `net6.0`, `net8.0`
91-
- Strong name signing for Dataverse compatibility
92-
- Full XML documentation
93-
- GitHub Actions workflows for build and NuGet publishing
94-
- Comprehensive unit test suite
95-
96-
[Unreleased]: https://github.com/joshsmithxrm/ppds-sdk/compare/v1.1.0...HEAD
97-
[1.1.0]: https://github.com/joshsmithxrm/ppds-sdk/compare/v1.0.0...v1.1.0
98-
[1.0.0]: https://github.com/joshsmithxrm/ppds-sdk/releases/tag/v1.0.0
27+
Pre-release versions follow SemVer:
28+
- Alpha: `Dataverse-v1.0.0-alpha.1`
29+
- Beta: `Dataverse-v1.0.0-beta.1`
30+
- Release Candidate: `Dataverse-v1.0.0-rc.1`

0 commit comments

Comments
 (0)