Lightweight scale controller package for DTS backend#3387
Open
Lightweight scale controller package for DTS backend#3387
Conversation
Contributor
There was a problem hiding this comment.
Pull request overview
This PR adds DTS (Durable Task Scheduler) backend support to the lightweight scale controller package. It introduces a new AzureManaged scalability provider alongside the existing AzureStorage provider, enabling target-based autoscaling for the Azure Managed backend.
Changes:
- Added
AzureManagedScalabilityProvider,AzureManagedScalabilityProviderFactory, andAzureManagedTargetScalerclasses implementing scale controller integration for the DTS backend. - Registered the new
AzureManagedScalabilityProviderFactoryin DI and added theMicrosoft.DurableTask.AzureManagedBackendpackage dependency. - Added a new CI workflow job (
dts) that runs Azure Managed scale tests using a DTS emulator Docker container, plus new unit and E2E tests.
Reviewed changes
Copilot reviewed 10 out of 10 changed files in this pull request and generated 4 comments.
Show a summary per file
| File | Description |
|---|---|
src/.../AzureManaged/AzureManagedScalabilityProvider.cs |
New scalability provider for AzureManaged backend with target scaler and dummy scale monitor |
src/.../AzureManaged/AzureManagedScalabilityProviderFactory.cs |
Factory that creates/caches AzureManaged providers with connection string resolution and token credential support |
src/.../AzureManaged/AzureManagedTargetScaler.cs |
Target scaler computing worker count from task hub metrics |
src/.../DurableTaskScaleConfigurationExtensions.cs |
Registers AzureManagedScalabilityProviderFactory in DI |
src/.../Microsoft.Azure.WebJobs.Extensions.DurableTask.FunctionsScale.csproj |
Adds AzureManagedBackend package reference |
test/ScaleTests/DurableTaskScaleConfigurationExtensionsTests.cs |
Asserts AzureManaged factory is registered |
test/ScaleTests/AzureManaged/AzureManagedScalabilityProviderFactoryTests.cs |
Unit tests for factory creation, caching, and selection |
test/ScaleTests/AzureManaged/AzureManagedTargetScalerTests.cs |
E2E test for target-based scaling with pending orchestrations |
test/ScaleTests/TestHelpers.cs |
Removed unused import and redundant comments |
.github/workflows/validate-build-scale.yml |
Added DTS CI job with emulator and renamed existing job |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
...Extensions.DurableTask.FunctionsScale/AzureManaged/AzureManagedScalabilityProviderFactory.cs
Outdated
Show resolved
Hide resolved
![github-code-quality[bot]](https://avatars.githubusercontent.com/u/9919?s=60&v=4){kind=small}
...ebJobs.Extensions.DurableTask.FunctionsScale/AzureManaged/AzureManagedScalabilityProvider.cs
Fixed
Show fixed
Hide fixed
...ebJobs.Extensions.DurableTask.FunctionsScale/AzureManaged/AzureManagedScalabilityProvider.cs
Fixed
Show fixed
Hide fixed
...Extensions.DurableTask.FunctionsScale/AzureManaged/AzureManagedScalabilityProviderFactory.cs
Fixed
Show fixed
Hide fixed
Contributor
There was a problem hiding this comment.
Pull request overview
Adds Azure Managed (DTS) backend support to the Microsoft.Azure.WebJobs.Extensions.DurableTask.FunctionsScale “lightweight scale controller” package, including DI registration, implementation types, tests, and CI workflow coverage.
Changes:
- Register and ship an Azure Managed scalability provider + target scaler implementation.
- Add Azure Managed-focused unit/integration tests and split CI scale-test workflow into Azure Storage vs DTS emulator runs.
- Add
Microsoft.DurableTask.AzureManagedBackenddependency to the FunctionsScale project.
Reviewed changes
Copilot reviewed 10 out of 10 changed files in this pull request and generated 5 comments.
Show a summary per file
| File | Description |
|---|---|
| test/ScaleTests/TestHelpers.cs | Removes unused import and trims helper comments around connection-string retrieval. |
| test/ScaleTests/DurableTaskScaleConfigurationExtensionsTests.cs | Extends DI registration test to assert Azure Managed factory is registered. |
| test/ScaleTests/AzureManaged/AzureManagedTargetScalerTests.cs | Adds DTS-emulator-backed test validating target-based scaling behavior. |
| test/ScaleTests/AzureManaged/AzureManagedScalabilityProviderFactoryTests.cs | Adds unit tests for Azure Managed factory behavior, caching, and factory selection logic. |
| src/Microsoft.Azure.WebJobs.Extensions.DurableTask.FunctionsScale/Microsoft.Azure.WebJobs.Extensions.DurableTask.FunctionsScale.csproj | Adds Azure Managed backend package reference. |
| src/Microsoft.Azure.WebJobs.Extensions.DurableTask.FunctionsScale/DurableTaskScaleConfigurationExtensions.cs | Registers AzureManagedScalabilityProviderFactory in AddDurableTask(). |
| src/.../AzureManaged/AzureManagedTargetScaler.cs | Implements Azure Managed target scaler based on backend task hub metrics. |
| src/.../AzureManaged/AzureManagedScalabilityProviderFactory.cs | Implements Azure Managed scalability provider factory with connection resolution + caching. |
| src/.../AzureManaged/AzureManagedScalabilityProvider.cs | Implements Azure Managed scalability provider wiring scale monitor/target scaler. |
| .github/workflows/validate-build-scale.yml | Splits scale tests into separate Azure Storage and DTS emulator jobs with filters. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
+28
to
+34
| public async Task<TargetScalerResult> GetScaleResultAsync(TargetScalerContext context) | ||
| { | ||
| TaskHubMetrics metrics = await this.service.GetTaskHubMetricsAsync(default); | ||
| if (metrics is null) | ||
| { | ||
| this.logger?.LogWarning("Task hub metrics returned null from Azure Managed backend. This may indicate the DTS emulator is being used which may not support metrics. Returning 0 worker count."); | ||
| return new TargetScalerResult { TargetWorkerCount = 0 }; |
Collaborator
Author
There was a problem hiding this comment.
Do we need this? seems pretty nit
Collaborator
There was a problem hiding this comment.
This is how it was, so we are not regressing anything, right? I recommend ignoring this one.
...Azure.WebJobs.Extensions.DurableTask.FunctionsScale/AzureManaged/AzureManagedTargetScaler.cs
Show resolved
Hide resolved
...Extensions.DurableTask.FunctionsScale/AzureManaged/AzureManagedScalabilityProviderFactory.cs
Outdated
Show resolved
Hide resolved
...Extensions.DurableTask.FunctionsScale/AzureManaged/AzureManagedScalabilityProviderFactory.cs
Outdated
Show resolved
Hide resolved
...ebJobs.Extensions.DurableTask.FunctionsScale/AzureManaged/AzureManagedScalabilityProvider.cs
Show resolved
Hide resolved
AnatoliB
reviewed
Mar 20, 2026
...Extensions.DurableTask.FunctionsScale/AzureManaged/AzureManagedScalabilityProviderFactory.cs
Outdated
Show resolved
Hide resolved
AnatoliB
reviewed
Mar 21, 2026
...Extensions.DurableTask.FunctionsScale/AzureManaged/AzureManagedScalabilityProviderFactory.cs
Outdated
Show resolved
Hide resolved
...Extensions.DurableTask.FunctionsScale/AzureManaged/AzureManagedScalabilityProviderFactory.cs
Fixed
Show fixed
Hide fixed
Contributor
There was a problem hiding this comment.
Pull request overview
Adds Azure Managed (DTS) backend support to the FunctionsScale “lightweight scale controller” package by introducing an AzureManaged scalability provider + factory + target scaler, and wiring this into DI/tests/CI so Scale Controller can select and exercise the DTS path.
Changes:
- Register
AzureManagedScalabilityProviderFactoryinAddDurableTask()and add theMicrosoft.DurableTask.AzureManagedBackenddependency. - Add Azure Managed scalability implementation (
AzureManagedScalabilityProviderFactory,AzureManagedScalabilityProvider,AzureManagedTargetScaler). - Add Azure Managed scale tests and update the scale-test GitHub workflow to run Azure Storage tests and DTS tests separately.
Reviewed changes
Copilot reviewed 10 out of 10 changed files in this pull request and generated 5 comments.
Show a summary per file
| File | Description |
|---|---|
| test/ScaleTests/TestHelpers.cs | Simplifies connection-string helpers used by scale tests. |
| test/ScaleTests/DurableTaskScaleConfigurationExtensionsTests.cs | Verifies AzureManaged factory is registered in DI. |
| test/ScaleTests/AzureManaged/AzureManagedTargetScalerTests.cs | Adds end-to-end-ish target-scaler validation against DTS emulator. |
| test/ScaleTests/AzureManaged/AzureManagedScalabilityProviderFactoryTests.cs | Adds unit coverage for AzureManaged factory behavior (defaults, caching, selection). |
| src/.../Microsoft.Azure.WebJobs.Extensions.DurableTask.FunctionsScale.csproj | Adds Azure Managed backend package reference. |
| src/.../DurableTaskScaleConfigurationExtensions.cs | Registers AzureManaged scalability provider factory in DI. |
| src/.../AzureManaged/AzureManagedTargetScaler.cs | Implements target-based scaling using AzureManaged backend metrics. |
| src/.../AzureManaged/AzureManagedScalabilityProviderFactory.cs | Creates/caches AzureManaged scalability providers and resolves credentials/options. |
| src/.../AzureManaged/AzureManagedScalabilityProvider.cs | Exposes AzureManaged target-scaler integration via ScalabilityProvider. |
| .github/workflows/validate-build-scale.yml | Splits CI into Azure Storage scale tests vs DTS scale tests (emulator). |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
...Extensions.DurableTask.FunctionsScale/AzureManaged/AzureManagedScalabilityProviderFactory.cs
Outdated
Show resolved
Hide resolved
Comment on lines
+112
to
+121
| // If a provider has already been created for this connection name, task hub, and client ID, return it. | ||
| if (this.cachedProviders.TryGetValue(cacheKey, out AzureManagedScalabilityProvider? cachedProvider)) | ||
| { | ||
| this.logger.LogDebug( | ||
| "Returning cached durability provider for connection '{Connection}', task hub '{TaskHub}', and client ID '{ClientId}'", | ||
| cacheKey.Item1, | ||
| cacheKey.Item2, | ||
| cacheKey.Item3 ?? "null"); | ||
| return cachedProvider; | ||
| } |
There was a problem hiding this comment.
The cache key doesn’t include max-concurrency settings, and on a cache hit the provider/service aren’t updated with the current metadata values. If Scale Controller sends different concurrency limits over time (or across triggers) for the same connection+hub, the factory will keep using stale limits. Consider including concurrency limits in the cache key, or updating the cached provider/service settings before returning it.
...Extensions.DurableTask.FunctionsScale/AzureManaged/AzureManagedScalabilityProviderFactory.cs
Outdated
Show resolved
Hide resolved
...Azure.WebJobs.Extensions.DurableTask.FunctionsScale/AzureManaged/AzureManagedTargetScaler.cs
Show resolved
Hide resolved
|
|
||
| - name: Start Azurite | ||
| run: azurite --silent --blobPort 10000 --queuePort 10001 --tablePort 10002 & | ||
|
|
There was a problem hiding this comment.
The dts job starts Azurite but doesn’t wait for it to accept connections before running tests. This can make the DTS scale tests flaky on slower runners. Add the same readiness loop used in the azure-storage job (e.g., nc -z on port 10000 with retries) before invoking dotnet test.
Suggested change
| - name: Wait for Azurite to be ready | |
| run: | | |
| echo "Waiting for Azurite to be ready on port 10000..." | |
| for i in {1..30}; do | |
| if nc -z localhost 10000; then | |
| echo "Azurite is ready." | |
| break | |
| fi | |
| echo "Azurite not ready yet. Retrying in 1 second..." | |
| sleep 1 | |
| done | |
| if ! nc -z localhost 10000; then | |
| echo "::error::Azurite did not become ready in time." | |
| exit 1 | |
| fi |
Comment on lines
+173
to
+180
| catch (Exception ex) | ||
| { | ||
| // Unexpected exception types. Fall back to use connection string. | ||
| this.logger.LogWarning( | ||
| ex, | ||
| "Unexpected error while getting token credential from trigger metadata for connection '{Connection}'", | ||
| connectionName); | ||
| } |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
As titled. This PR adds DTS backend support for the new lightweight scale controller package.