Commit 672b73b
v1.0.0 foundation: connection pool, adaptive rate control, bulk operations, migration CLI (#21)
* feat: add MinVer versioning with per-package tag prefixes
- Add MinVer 6.0.0 to all publishable packages with tag prefixes:
- PPDS.Plugins: Plugins-v*
- PPDS.Dataverse: Dataverse-v*
- PPDS.Migration + CLI: Migration-v*
- Remove hardcoded <Version> elements from csproj files
- Create Directory.Build.props for shared project settings
- Update publish-nuget.yml to trigger on per-package tags
- Add fetch-depth: 0 to build.yml and test.yml for MinVer
- Create per-package CHANGELOGs:
- src/PPDS.Plugins/CHANGELOG.md
- src/PPDS.Dataverse/CHANGELOG.md
- src/PPDS.Migration/CHANGELOG.md
- Convert root CHANGELOG.md to index pointing to per-package logs
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* feat(Dataverse): per-connection pool sizing (MaxConnectionsPerUser)
Change pool sizing from shared MaxPoolSize=50 to per-connection
MaxConnectionsPerUser=52, aligning with Microsoft's per-user service
protection limits.
Changes:
- Add MaxConnectionsPerUser property (default: 52) to ConnectionPoolOptions
- Mark legacy MaxPoolSize as [Obsolete] with default 0
- Add CalculateTotalPoolCapacity() for per-connection sizing
- Update BulkOperationExecutor to use calculated pool capacity
- Update ServiceFactory in CLI to use MaxConnectionsPerUser
- Add comprehensive unit tests for pool sizing logic
Total pool capacity = connections × MaxConnectionsPerUser
- 1 connection: 52 total capacity
- 2 connections: 104 total capacity
- 4 connections: 208 total capacity
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* feat(Dataverse): AIMD-based adaptive rate control for throttle recovery
Add adaptive parallelism that adjusts based on throttle responses:
New Components:
- IAdaptiveRateController interface for managing per-connection parallelism
- AdaptiveRateController with AIMD algorithm:
- 50% initial parallelism factor
- Halve on throttle (multiplicative decrease)
- +2 on stable success (additive increase)
- Time-gated increases (5s minimum between increases)
- Fast recovery to last-known-good level
- 5-minute TTL for last-known-good values
- 5-minute idle reset period
- AdaptiveRateOptions for configuration
- AdaptiveRateStatistics for monitoring
Integration:
- DataverseOptions.AdaptiveRate property for configuration
- ServiceCollectionExtensions registers IAdaptiveRateController
- BulkOperationExecutor uses adaptive parallelism when enabled
Test Coverage:
- 17 unit tests covering initialization, throttle, recovery,
statistics, reset, and per-connection isolation
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* feat(Dataverse): structured configuration with secret resolution
Add typed configuration support for Dataverse connections:
New Configuration Components:
- DataverseAuthType enum (ClientSecret, Certificate, OAuth)
- OAuthLoginPrompt enum (Auto, Always, Never, SelectAccount)
- ConnectionStringBuilder for generating connection strings from typed config
- SecretResolver for Key Vault and environment variable secret resolution
- ConfigurationException for typed validation errors
DataverseConnection Enhancements:
- Url, TenantId, ClientId properties
- ClientSecretKeyVaultUri, ClientSecretVariable, ClientSecret (priority order)
- CertificateThumbprint, CertificateStoreName, CertificateStoreLocation
- RedirectUri, LoginPrompt for OAuth
- UsesTypedConfiguration property to detect config mode
Secret Resolution Priority:
1. Azure Key Vault URI (highest, requires Azure SDK)
2. Environment variable
3. Direct value (lowest, not recommended)
Test Coverage:
- 23 unit tests for ConnectionStringBuilder
- 8 unit tests for SecretResolver
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* feat(Dataverse): multi-environment configuration support
Add Phase 1 of multi-environment support for source/target migration scenarios:
New Configuration Model:
- DataverseEnvironmentOptions for named environment definitions
- EnvironmentResolver for retrieving environments by name
- DataverseOptions.Environments dictionary for multi-env config
- DataverseOptions.DefaultEnvironment for default environment selection
- DataverseOptions.Url and TenantId for root-level defaults
Key Features:
- Named environments (e.g., "source", "target", "dev", "prod")
- Per-environment connections for load distribution
- Backwards compatible: root-level Connections still work
- Virtual "default" environment when no Environments defined
Test Coverage:
- 19 unit tests for EnvironmentResolver
- Tests for default resolution, named lookup, and edge cases
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* chore: cleanup - remove backwards compat cruft, fix v1.0.0 release
- Remove [Obsolete] from MaxPoolSize (no legacy to support)
- Remove #pragma warning disable blocks
- Remove "Legacy" region naming in DataverseConnection
- Fix CHANGELOGs for v1.0.0 release date
- Clean up test pragmas
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* fix: remove ConnectionString, use typed env config only
Breaking change: Connection strings are no longer supported.
All connection configuration must use typed properties
(Url, ClientId, ClientSecretVariable, TenantId).
CLI now uses environment variables exclusively:
- PPDS_URL, PPDS_CLIENT_ID, PPDS_CLIENT_SECRET, PPDS_TENANT_ID
- PPDS_SOURCE_* and PPDS_TARGET_* prefixes for migrate command
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* docs: update for v1.0.0 release, remove completed specs
- Delete implemented specs (MinVer, Adaptive Rate, Structured Config)
- Update ADR-0005 to reflect final implementation
- Update Multi-Environment spec status (Phase 1 complete)
- Rewrite READMEs for typed configuration
- Update CLI docs for new environment variable names
- Clarify ClientSecret vs ClientSecretVariable usage
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* feat(Dataverse): user-friendly configuration validation errors
- Enhanced ConfigurationException with EnvironmentName, ConnectionIndex, and ResolutionHints
- Error messages now show exactly where to configure missing properties
- Removed ClientSecretVariable (env var binding handled by .NET config system)
- Simplified SecretResolver to just Key Vault + direct value
- Track source environment/index on connections for better error context
Example error output:
Dataverse Configuration Error
=============================
Missing required property: Url
Connection: Secondary (index: 1)
Environment: Dev
Configure Url at any of these levels (in order of precedence):
1. Dataverse:Environments:Dev:Connections:1:Url
2. Dataverse:Environments:Dev:Url
3. Dataverse:Url
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* docs: remove implemented spec, fix target framework docs
- Delete MULTI_ENVIRONMENT_SPEC.md (Phase 1 implemented)
- Fix CLAUDE.md target frameworks (4.6.2, 8.0, 10.0)
- Clarify secret resolution in CHANGELOG (Key Vault + .NET config binding)
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* feat(CLI): add --env and --config options for config-based connections
Add support for loading Dataverse connections from appsettings.json
as an alternative to environment variables. Both patterns are now
first-class citizens.
New CLI options:
- export/import/schema: --env, --config
- migrate: --source-env, --target-env, --config
- New command: `ppds-migrate config list` to show available environments
Also makes environment name matching case-insensitive (DEV matches Dev).
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* feat(Dataverse): enhance adaptive rate controller with throttle ceiling
- Scale floor/ceiling by connection count (e.g., 52 × 2 = 104 with 2 connections)
- Calculate throttle ceiling from Retry-After duration
- 5 min Retry-After → 50% ceiling, 2.5 min → 75%, 30 sec → 95%
- Clamp duration = RetryAfter + 5 minutes
- Respect throttle ceiling when probing up (prevents hitting same wall)
- Change DecreaseFactor default from 0.75 to 0.5 (aggressive backoff)
- Add ThrottleCeiling, ThrottleCeilingExpiry, EffectiveCeiling to statistics
- Wire RecordThrottle into connection pool's throttle detection
- Update tests for new API and throttle ceiling behavior
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* feat(Dataverse): implement dynamic parallel executor for adaptive rate
Replace fixed Parallel.ForEachAsync with custom Task.WhenAny loop that
queries GetParallelism() before starting each batch, enabling true
dynamic parallelism during execution.
Key changes:
- Add ExecuteBatchesDynamicallyAsync with real-time parallelism queries
- Update all bulk operations (Create/Update/Upsert/Delete) to use it
- Support three execution strategies: Sequential, Dynamic, Fixed
- Log parallelism changes: "Parallelism changed 10 → 20"
- Remove unused ResolveParallelismAsync method
Behavior:
- Ramp-up: As batches succeed, parallelism increases (10 → 20 → 104)
- Immediate fill: When parallelism increases, starts more batches
- Graceful backoff: After throttle, stops starting new batches until
in-flight count drops below new limit
- In-flight batches are never cancelled, only new starts are throttled
Expected throughput improvement: ~120 rec/s → ~500 rec/s
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* feat(Dataverse): add pre-flight throttle guard to prevent avalanche
Adds a check before executing each batch to verify the connection is not
already known to be throttled. This prevents the "in-flight avalanche"
where many parallel requests are dispatched before the first throttle
error returns, causing cascading throttle errors that extend Retry-After
durations.
- Add MaxPreFlightAttempts constant (10) as safety valve
- Check IsThrottled before executing batch
- Retry with different connection if throttled
- Log warnings when safety valve is hit
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* feat(Dataverse): add execution time-aware rate ceiling
Implements dynamic parallelism ceiling based on batch execution times
to prevent throttle cascades caused by execution time budget exhaustion.
Key changes:
- Track batch durations via exponential moving average (EMA)
- Calculate ceiling as Factor/avgBatchSeconds (default factor: 250)
- Only apply ceiling for slow batches (>10s threshold)
- Expose stats: ExecutionTimeCeiling, AverageBatchDuration, BatchDurationSampleCount
Results:
- Updates: 79/s → 178/s (+125%), 866 throttles → 0
- Deletes: 52/s → 102/s (+97%), cascade eliminated
- Creates: Preserved at ~513/s via slow batch threshold
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* feat(Dataverse): add RateControlPreset system for adaptive rate tuning
- Add RateControlPreset enum (Conservative, Balanced, Aggressive)
- Refactor AdaptiveRateOptions with preset support via nullable backing fields
- Make implementation-detail options internal (HardCeiling, MinParallelism, etc.)
- Add MaxRetryAfterTolerance option for fail-fast scenarios
- Tune Balanced preset: Factor=200, Threshold=8000ms
- Update tests for new API and preset behavior
Presets allow simple configuration:
{"AdaptiveRate": {"Preset": "Conservative"}}
Or fine-grained tuning with overrides:
{"AdaptiveRate": {"Preset": "Balanced", "ExecutionTimeCeilingFactor": 180}}
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* feat(Dataverse): add type-safe CustomLogicBypass enum for bulk operations
Replace magic string BypassBusinessLogicExecution with CustomLogicBypass
flags enum for better developer experience:
- CustomLogicBypass.Synchronous - bypass sync plugins/workflows
- CustomLogicBypass.Asynchronous - bypass async plugins/workflows
- CustomLogicBypass.All - bypass both
Also adds Tag property for plugin context traceability.
Removes legacy BypassCustomPluginExecution and BypassBusinessLogicExecution
string properties.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* docs: add adaptive rate control documentation and ADR-0006
- Update CHANGELOG.md with preset features for 1.0.0
- Add Adaptive Rate Control section to README with preset table
- Create ADR-0006: Execution Time Ceiling design decision
- Update CLAUDE.md with rate control presets and guidance
- Add Conservative preset recommendation for production bulk ops
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* feat(Dataverse): add CreatedCount/UpdatedCount to UpsertMultiple results
Track how many records were created vs updated during UpsertMultiple
operations by parsing UpsertMultipleResponse.Results.
- Add CreatedCount and UpdatedCount properties to BulkOperationResult
- Parse UpsertResponse.RecordCreated in ExecuteUpsertMultipleCoreAsync
- Aggregate counts across parallel and sequential batch execution
- Add unit tests for BulkOperationResult
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* docs(Dataverse): update adaptive rate tuning status with validation results
- Document validated results: Create 483/s, Update 153/s (0 throttles)
- Delete 83/s with 23 throttles - recommend Conservative preset
- Fix XML comments to match actual Balanced preset values (Factor=200)
- Update preset table with correct thresholds
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* feat(Dataverse): log effective adaptive rate config at startup
Logs the resolved configuration values when AdaptiveRateController is
created, with "(override)" indicator when values are explicitly set
rather than coming from preset defaults. This helps operators verify
their config is applied correctly and catch misconfiguration early.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* fix(Dataverse): clear non-configured backing fields after configuration binding
.NET's ConfigurationBinder.Bind() reads from getters and writes to setters
for ALL public properties, even those not in the config source. This breaks
the nullable-backing-field pattern for detecting explicit overrides.
When config only specifies Preset (e.g., "Balanced"), Bind() would:
1. Read ExecutionTimeCeilingFactor getter (returns 200 from Balanced)
2. Write 200 to setter, populating _executionTimeCeilingFactor backing field
3. Later preset change to Conservative wouldn't take effect (field is 200)
The fix clears backing fields for properties not explicitly in config after
Bind(), restoring the intended behavior where preset defaults are used unless
explicitly overridden.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* feat(Migration.Cli): unify configuration with PPDS.Dataverse model
- Use same DataverseOptions configuration as SDK
- Add --secrets-id global option for cross-process User Secrets
- Add standard .NET configuration layering (appsettings.json, User Secrets, env vars)
- Make --env required for all commands that need Dataverse connection
- Remove custom PPDS_* environment variable handling
- Update README with new configuration documentation
- Fix TieredImporter to use BypassCustomLogic enum
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* fix(Dataverse): lower Conservative preset values to create headroom
Conservative preset was running at 100% of execution time ceiling capacity,
leaving zero headroom for server load fluctuations. This caused throttle
cascades at 67% progress with 173/s throughput dropping to 83/s.
Changes to Conservative preset:
- ExecutionTimeCeilingFactor: 180 → 140 (creates ~20% headroom)
- SlowBatchThresholdMs: 8000 → 6000 (apply ceiling earlier)
With 8.5s batches:
- Old: ceiling = 180/8.5 = 21 (zero headroom → cascade)
- New: ceiling = 140/8.5 = 16 (~20% headroom → stable)
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* fix(Migration.Cli): prevent hang on connection pool initialization
- Set MinPoolSize=0 to avoid synchronous ServiceClient creation during pool construction
- Add console logging provider for CLI visibility of warnings/errors
- Add "Connecting to Dataverse..." message before pool construction
- Add verbose parameter to ServiceFactory for Information-level logging
- Use AddSimpleConsole with timestamps for cleaner CLI output
When MinPoolSize > 0, the pool constructor synchronously creates ServiceClient
instances which can hang indefinitely on auth/network issues. Setting it to 0
enables lazy connection creation where errors surface during actual operations.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* fix(Dataverse): prevent throttle ceiling from dropping during rapid cascades
During rapid throttle cascades, each throttle event was using the
already-reduced parallelism to calculate the new ceiling:
30 -> 15 (throttle ceiling: 29) ← based on 30
15 -> 10 (throttle ceiling: 14) ← based on 15, already reduced!
10 -> 10 (throttle ceiling: 10) ← stuck at floor
This caused the ceiling to drop in lockstep with parallelism, getting
stuck at floor for 5 minutes even with short 1.3s Retry-After values.
Fix: Use the higher of current parallelism or existing throttle ceiling
as the base for calculating new ceiling. This preserves a reasonable
ceiling (29) during cascades instead of dropping to floor (10).
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* fix(Migration.Cli): propagate detailed error messages through import chain
Previously, BulkOperationResult.Errors contained detailed error messages
(e.g., "Entity 'systemuser' With Id = ... Does Not Exist") but these were
lost in the import chain because BatchImportResult and EntityImportResult
had no Errors property. Users only saw generic "Entity import had X failures".
Changes:
- Add Errors property to EntityImportResult
- Add Errors property to BatchImportResult (private class)
- Convert BulkOperationError to MigrationError with full message in ImportBatchAsync
- Aggregate errors from all batches in ImportEntityAsync
- Pass detailed errors to ImportResult.Errors and progress reporter
Now users see the actual Dataverse error message, enabling actionable debugging.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* refactor(Migration.Cli): unify output through progress reporter
CLI commands now use IProgressReporter for all user-facing output instead
of mixing Console.WriteLine, ConsoleOutput, and ILogger. This provides:
- Single source of truth for CLI output
- Consistent formatting across all commands
- Clean separation between diagnostic logs and user output
Changes:
- Replace --verbose with --debug flag for diagnostic ILogger output
- Suppress ILogger console output by default (only enable with --debug)
- Remove direct Console.WriteLine from ImportCommand, ExportCommand,
MigrateCommand, and SchemaCommand
- Route all status messages through progressReporter.Report()
- Route all errors through progressReporter.Error()
- Let progressReporter.Complete() handle final result summary
The progress reporter now owns the entire user experience, making output
predictable and testable.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* feat(Migration.Cli): add --strip-owner-fields and error pattern detection
Phase 3 of CLI overhaul adds features to address common import failures:
1. New --strip-owner-fields option for import command:
- Strips ownerid, createdby, modifiedby, and related fields
- Allows Dataverse to assign current user as owner
- Solves cross-environment migration issues where source users don't exist
2. Error pattern detection in ConsoleProgressReporter:
- Detects common error patterns (MISSING_USER, MISSING_TEAM, etc.)
- Shows pattern summary when 80%+ errors share same cause
- Provides actionable suggestions based on detected patterns
3. Test updates:
- Updated tests for --debug flag (replacing --verbose)
- Added tests for new --strip-owner-fields option
- Fixed tests to include required --env options
- Updated ConnectionResolver tests for config-based resolution
When import fails with "systemuser Does Not Exist", users now see:
Error Pattern: 1000 of 1000 errors share the same cause:
Referenced systemuser does not exist in target environment
Suggested fixes:
-> Use --strip-owner-fields to remove ownership references
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* refactor(Migration): thin wrapper architecture for CLI
Remove manual batching from TieredImporter - BulkOperationExecutor now
receives ALL records and handles dynamic batching based on server's
RecommendedDegreesOfParallelism.
Key changes:
- Remove BatchSize/ProgressInterval from ImportOptions (Dataverse layer decides)
- Wire progress reporting through IProgress<ProgressSnapshot> adapter
- Add --verbose/-v flag to all CLI commands (LogLevel.Information)
- Fix logging defaults: Warning by default, --verbose for info, --debug for debug
- Console provider always enabled (not just in debug mode)
- Remove unused ImportBatchAsync and BatchImportResult
This follows the correct layer responsibilities:
- CLI: thin wrapper (parse args, create services, call one method, report)
- Migration: orchestration (read data, analyze deps, pass ALL records down)
- Dataverse: owns batching, parallelism, rate control, progress
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* fix(Migration): correct tier ordering in dependency graph
The Kahn's algorithm implementation had inverted edge semantics.
Edges are (from=dependent, to=dependency), meaning "from depends on to".
The algorithm was counting in-degrees on the TO side (dependencies),
causing dependents to be processed before their dependencies.
Fixed to count dependencies (edges FROM each node) so nodes with
zero dependencies are processed first, then their dependents.
This fixes imports where child records (e.g., ppds_zipcode) were
being imported before parent records (e.g., ppds_state), causing
"record does not exist" errors on lookup fields.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* fix(Migration): add SQL 2766 TVP retry and multi-connection support
Two fixes:
1. TVP retry for SQL error 2766:
The IsTvpRaceConditionError only handled SQL 3732 ("Cannot drop type")
but missed 2766 ("The definition for user-defined data type has changed").
Both are transient TVP cache staleness errors that should trigger retry.
2. CLI now uses ALL configured connections:
Previously ConnectionResolver.ResolveFromConfig took only Connections[0],
and commands used CreateProvider(ConnectionConfig) which adds one connection.
Now commands use CreateProviderFromConfig(IConfiguration, environmentName)
which reads ALL connections from the config file. With multiple app
registrations configured, each gets its own service protection quotas,
potentially doubling throughput.
Updated commands: Import, Export, Migrate, Schema (generate + list)
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* docs: add consumption patterns reference to CLAUDE.md
Links to CONSUMPTION_PATTERNS.md for guidance on when consumers
should use library vs CLI vs Tools.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* chore: remove pre-release infrastructure notes from changelog
Pre-v1 development history belongs in git, not the public changelog.
The root CHANGELOG.md is now a clean index to per-package changelogs.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* refactor(CLI): migrate System.CommandLine 2.0.0-beta4 to 2.0.1 stable
Breaking API changes addressed:
- Option constructors: aliases as params, Description as property
- IsRequired → Required property rename
- getDefaultValue → DefaultValueFactory property
- AddCommand → Subcommands.Add collection pattern
- AddGlobalOption → Options.Add with Recursive=true
- SetHandler → SetAction with (ParseResult, CancellationToken)
- context.ParseResult.GetValueForOption → parseResult.GetValue
- context.ExitCode assignment → return int from action
- rootCommand.InvokeAsync(args) → Parse(args).InvokeAsync()
All tests pass. CLI help output verified.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* feat(CLI): add validators, completions, and auth infrastructure
Validators:
- AcceptExistingOnly() for input files (--schema, --data, --config, --user-mapping)
- AcceptLegalFileNamesOnly() for output files (--output)
- Custom validators for --parallel (>= 1) and --page-size (1-5000)
- Output directory existence validation
Tab Completions:
- Added CompletionSources for --env options across all commands
- Reads available environments from configuration dynamically
Auth Infrastructure:
- Added AuthMode enum (Auto, Config, Env, Interactive, Managed)
- Added AuthResolver for resolving auth based on mode
- Added --auth global option to Program.cs
- Support for DATAVERSE__URL, DATAVERSE__CLIENTID, DATAVERSE__CLIENTSECRET env vars
- Added ManagedIdentity to DataverseAuthType enum
- Added BuildManagedIdentityConnectionString to ConnectionStringBuilder
- Added CreateProviderForAuthMode to ServiceFactory
Spec document:
- docs/CLI_V1_ENHANCEMENTS_SPEC.md with full implementation details
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* feat(CLI): update ExportCommand to use new auth infrastructure
- ExportCommand now reads --auth option and uses AuthResolver
- Uses ServiceFactory.CreateProviderForAuthMode for provider creation
- Displays auth mode info in connection status message
- Supports all auth modes: auto, config, env, interactive, managed
WIP: Other commands (ImportCommand, MigrateCommand, SchemaCommand) still
need to be updated with the same pattern.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* feat(CLI): complete auth infrastructure for all commands
- Update ImportCommand, MigrateCommand, SchemaCommand to use --auth option
- Fix interactive auth with Microsoft's well-known public client ID
(51f81489-12ee-4a9e-aaae-a2591f45987d)
- Add appsettings.json with Dev/QA environment configuration
- Share UserSecretsId with demo app (ppds-dataverse-demo)
- MigrateCommand rejects --auth env (can't specify two URLs via env vars)
- Fix CLI tests for System.CommandLine 2.0.1 API changes:
- Option.IsRequired -> Option.Required
- Option.Name now includes -- prefix
- Add temp file fixtures for AcceptExistingOnly validation
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* refactor(CLI): simplify auth with --url option and interactive default
- Add global --url option for direct Dataverse URL input
- Change default auth mode from Auto to Interactive
- Remove Auto auth mode (no longer needed)
- Make --env optional on all commands (only required for --auth config)
- Update AuthResolver to prioritize: --url > --env+config > DATAVERSE__URL
- Update all commands (export, import, migrate, schema) to use new pattern
New UX:
- ppds-migrate schema list --url https://org.crm.dynamics.com (interactive)
- ppds-migrate schema list --env Dev (uses config)
- ppds-migrate schema list --auth env (uses env vars)
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* refactor(CLI): simplify auth to 3 modes - interactive, env, managed
BREAKING: Removes --auth config mode and config file dependency
Auth modes now:
- Interactive (default): Device code flow with --url
- Env: DATAVERSE__URL, DATAVERSE__CLIENTID, DATAVERSE__CLIENTSECRET
- Managed: Azure Managed Identity with --url
Changes:
- Remove --env, --config, --secrets-id options from all commands
- Remove appsettings.json from CLI (no longer uses config files)
- Remove ConfigCommand, ConnectionResolver, ConfigurationHelper
- Add DeviceCodeTokenProvider for MSAL device code flow
- Add DeviceCodeConnectionPool for interactive auth
- Migrate command now uses --source-url and --target-url
- Remove User Secrets dependency from csproj
CLI and Demo app are now fully independent:
- CLI: Uses --url + device code (no config files)
- Demo: Uses appsettings.json + User Secrets (standard .NET)
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* feat(CLI): add persistent token cache for device code auth
Tokens are now cached to disk so users don't need to re-authenticate
every command. Uses MSAL's MsalCacheHelper for cross-platform support:
- Windows: %LOCALAPPDATA%\PPDS\msal_token_cache.bin (DPAPI encrypted)
- macOS: ~/.ppds/msal_token_cache.bin (Keychain)
- Linux: ~/.ppds/msal_token_cache.bin (libsecret or plaintext fallback)
User experience:
- First command: Device code prompt, authenticate in browser
- Subsequent commands: Silent token refresh, no prompt needed
- Token expires: MSAL auto-refreshes using refresh token
- Refresh token expires (~90 days): Device code prompt again
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* fix(CLI): register missing DI services for interactive auth
The interactive auth path in ServiceFactory bypassed AddDataverseConnectionPool()
which meant IBulkOperationExecutor, IThrottleTracker, and IAdaptiveRateController
were never registered. This caused import operations to fail with:
"No constructor for type 'TieredImporter' can be instantiated"
Also added tmp/ to .gitignore for test artifacts.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* feat(CLI): add users generate command for cross-environment mapping
Adds `ppds-migrate users generate` command that:
- Connects to source and target environments with interactive auth
- Queries systemuser records from both environments
- Matches users by Azure AD Object ID (preferred) or domain name fallback
- Generates user-mapping.xml for use with `ppds-migrate import --user-mapping`
New files:
- PPDS.Migration/UserMapping/UserMappingGenerator.cs - Core matching logic
- PPDS.Migration.Cli/Commands/UsersCommand.cs - CLI command
Usage:
ppds-migrate users generate \
--source-url https://dev.crm.dynamics.com \
--target-url https://qa.crm.dynamics.com \
--output user-mapping.xml
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* fix(BulkOperations): add retry for stored procedure race condition (SQL 2812)
Expand bulk operation infrastructure race condition detection to include
SQL error 2812 ("Could not find stored procedure"). This error occurs when
parallel bulk operations hit a table before Dataverse has finished creating
the internal stored procedures (e.g., p_*_UpdateMultiple).
This has the same root cause as TVP race conditions (SQL 3732/2766) -
Dataverse lazily generates bulk operation infrastructure. The fix adds
2812 to the existing detection and retry logic with exponential backoff.
Renamed IsTvpRaceConditionError to IsBulkInfrastructureRaceConditionError
and MaxTvpRetries to MaxBulkInfrastructureRetries to reflect the broader
scope of transient errors covered.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* docs: add connection pool refactor specification
Comprehensive spec for separating authentication from pooling via
IConnectionSource abstraction. Includes:
- Problem statement and current state analysis
- Interface design (IConnectionSource, ServiceClientSource, ConnectionStringSource)
- DataverseConnectionPool changes with new constructor
- Implementation phases with detailed steps
- Testing requirements
- CLI migration path
- Implementation prompt for new session
This enables any authentication method (device code, managed identity,
connection string, certificate) to use the same pool with full feature
support (throttle tracking, adaptive rate control, connection validation).
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* refactor(Dataverse): add IConnectionSource abstraction for connection pool
Introduce IConnectionSource interface to decouple connection pool from
authentication methods. This enables using pre-authenticated ServiceClients
(device code, managed identity) with the full connection pool infrastructure.
New types:
- IConnectionSource: Interface for providing seed clients to clone
- ServiceClientSource: Wraps pre-authenticated ServiceClient
- ConnectionStringSource: Creates client from config (backward compat)
Changes:
- DataverseConnectionPool: New primary constructor using IConnectionSource,
legacy constructor marked [Obsolete] and delegates via ConnectionStringSource
- CLI: Uses ServiceClientSource + DataverseConnectionPool instead of
custom DeviceCodeConnectionPool (deleted 340 lines)
- Pool members now use Clone() for faster connection creation
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* docs(CLI): align documentation with implemented auth model
Remove references to --env, --secrets-id, and --auth config options that
were planned but never implemented. The CLI now uses a simpler URL-based
approach:
- --url: Explicit environment URL (required)
- --auth: interactive (default), env, or managed
The config-based approach (--env with appsettings.json) was abandoned
because:
1. Demo app now uses PPDS.Migration library directly, eliminating the
need for cross-process User Secrets sharing (--secrets-id)
2. Configuration file discovery (CWD vs explicit path) is ambiguous
3. Current auth modes (interactive, env, managed) cover all use cases
Updated:
- CLI README.md: Rewritten to match actual implementation
- CLI_V1_ENHANCEMENTS_SPEC.md: Marked abandoned features, updated status
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* docs: remove implemented specification documents
All specs have been implemented:
- BYPASS_OPTIONS_REFACTOR_SPEC.md → CustomLogicBypass.cs
- CLI_V1_ENHANCEMENTS_SPEC.md → Auth modes implemented
- CONNECTION_POOL_REFACTOR_SPEC.md → IConnectionSource abstraction
- CONNECTION_HEALTH_MANAGEMENT.md → ValidateConnection, connection lifecycle
- THROTTLE_DETECTION_AND_ROUTING.md → RecordThrottle wired up
- TVP_RACE_CONDITION_RETRY.md → SQL 2812/3732 retry logic
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* docs: standardize ADRs and fix documentation staleness
ADR improvements:
- Add missing dates to ADR-0001 through ADR-0004, ADR-0006
- Update ADR-0004 to reference ADR-0006 (removes broken spec link)
- Merge ADAPTIVE_RATE_TUNING_STATUS.md into ADR-0006 tuning history
- Create ADR-0007 for IConnectionSource abstraction
Staleness fixes:
- CONNECTION_POOLING_PATTERNS.md: Fix defaults (MaxPoolSize=0, MaxLifetime=60m)
- README.md: Fix CLI env vars (DATAVERSE__* not PPDS_*), add ADR-0006/0007
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* docs(CLAUDE.md): update for multi-package repo and add missing sections
Updates:
- Namespaces: Add PPDS.Dataverse and PPDS.Migration namespaces
- Version Management: Per-package versioning with MinVer tag format
- Release Process: Per-package changelogs and tag format
- This Repo Produces: Add missing PPDS.Migration package
- ADR Quick Reference: Table of all 7 ADRs with summaries
- CLI Section: Auth modes and env vars for CI/CD
- Fix ADR-0006 reference to be a proper link
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* docs: fix changelogs to reflect actual implementation
PPDS.Dataverse:
- Add IConnectionSource abstraction (ADR-0007)
- Add ServiceClientSource for pre-authenticated clients
- Document all four DataverseAuthType options
- Fix TVP error codes (3732/2812)
PPDS.Migration:
- Remove abandoned features (--env, --secrets-id, --auth config, config list)
- Fix environment variable names (DATAVERSE__* not DATAVERSE_*)
- Document actual CLI auth modes (interactive, env, managed)
- Add users generate command
- Remove [Unreleased] section with obsolete content
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* refactor: use TryGetValue to avoid double dictionary lookup
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>1 parent 99de8d7 commit 672b73b
File tree
97 files changed
+9754
-6471
lines changed- .github/workflows
- docs
- adr
- architecture
- specs
- src
- PPDS.Dataverse
- BulkOperations
- Configuration
- DependencyInjection
- Pooling
- Resilience
- PPDS.Migration.Cli
- Commands
- Infrastructure
- PPDS.Migration
- Analysis
- Formats
- Import
- Progress
- UserMapping
- PPDS.Plugins
- tests
- PPDS.Dataverse.Tests
- BulkOperations
- Configuration
- DependencyInjection
- Pooling
- Strategies
- Resilience
- Security
- PPDS.Migration.Cli.Tests/Commands
Some content is hidden
Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.
97 files changed
+9754
-6471
lines changed| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
28 | 28 | | |
29 | 29 | | |
30 | 30 | | |
| 31 | + | |
| 32 | + | |
31 | 33 | | |
32 | 34 | | |
33 | 35 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
1 | 1 | | |
2 | 2 | | |
3 | 3 | | |
4 | | - | |
5 | | - | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
6 | 9 | | |
7 | 10 | | |
8 | 11 | | |
9 | 12 | | |
10 | 13 | | |
11 | 14 | | |
12 | 15 | | |
| 16 | + | |
| 17 | + | |
13 | 18 | | |
14 | 19 | | |
15 | 20 | | |
| |||
18 | 23 | | |
19 | 24 | | |
20 | 25 | | |
21 | | - | |
| 26 | + | |
| 27 | + | |
22 | 28 | | |
23 | 29 | | |
24 | | - | |
25 | | - | |
26 | | - | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
| 33 | + | |
| 34 | + | |
| 35 | + | |
| 36 | + | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
| 42 | + | |
| 43 | + | |
| 44 | + | |
| 45 | + | |
27 | 46 | | |
28 | | - | |
29 | | - | |
| 47 | + | |
| 48 | + | |
30 | 49 | | |
31 | | - | |
32 | | - | |
33 | | - | |
34 | | - | |
35 | | - | |
36 | | - | |
37 | | - | |
38 | | - | |
39 | | - | |
40 | | - | |
41 | | - | |
| 50 | + | |
| 51 | + | |
| 52 | + | |
42 | 53 | | |
43 | 54 | | |
44 | 55 | | |
45 | 56 | | |
46 | 57 | | |
47 | 58 | | |
48 | 59 | | |
49 | | - | |
50 | | - | |
| 60 | + | |
| 61 | + | |
| 62 | + | |
| 63 | + | |
| 64 | + | |
| 65 | + | |
| 66 | + | |
| 67 | + | |
51 | 68 | | |
52 | 69 | | |
53 | 70 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
12 | 12 | | |
13 | 13 | | |
14 | 14 | | |
| 15 | + | |
| 16 | + | |
15 | 17 | | |
16 | 18 | | |
17 | 19 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
48 | 48 | | |
49 | 49 | | |
50 | 50 | | |
| 51 | + | |
| 52 | + | |
| 53 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
1 | | - | |
| 1 | + | |
2 | 2 | | |
3 | | - | |
| 3 | + | |
4 | 4 | | |
5 | | - | |
6 | | - | |
| 5 | + | |
7 | 6 | | |
8 | | - | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
9 | 10 | | |
10 | | - | |
| 11 | + | |
11 | 12 | | |
12 | | - | |
| 13 | + | |
| 14 | + | |
13 | 15 | | |
14 | | - | |
| 16 | + | |
15 | 17 | | |
16 | | - | |
17 | | - | |
18 | | - | |
19 | | - | |
20 | | - | |
21 | | - | |
22 | | - | |
23 | | - | |
| 18 | + | |
| 19 | + | |
24 | 20 | | |
25 | | - | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
26 | 26 | | |
27 | | - | |
28 | | - | |
29 | | - | |
30 | | - | |
31 | | - | |
32 | | - | |
33 | | - | |
34 | | - | |
35 | | - | |
36 | | - | |
37 | | - | |
38 | | - | |
39 | | - | |
40 | | - | |
41 | | - | |
42 | | - | |
43 | | - | |
44 | | - | |
45 | | - | |
46 | | - | |
47 | | - | |
48 | | - | |
49 | | - | |
50 | | - | |
51 | | - | |
52 | | - | |
53 | | - | |
54 | | - | |
55 | | - | |
56 | | - | |
57 | | - | |
58 | | - | |
59 | | - | |
60 | | - | |
61 | | - | |
62 | | - | |
63 | | - | |
64 | | - | |
65 | | - | |
66 | | - | |
67 | | - | |
68 | | - | |
69 | | - | |
70 | | - | |
71 | | - | |
72 | | - | |
73 | | - | |
74 | | - | |
75 | | - | |
76 | | - | |
77 | | - | |
78 | | - | |
79 | | - | |
80 | | - | |
81 | | - | |
82 | | - | |
83 | | - | |
84 | | - | |
85 | | - | |
86 | | - | |
87 | | - | |
88 | | - | |
89 | | - | |
90 | | - | |
91 | | - | |
92 | | - | |
93 | | - | |
94 | | - | |
95 | | - | |
96 | | - | |
97 | | - | |
98 | | - | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
0 commit comments