Implementing a resolver that resolves worker configurations from specified probing paths

[surgupta-msft]

Issue describing the changes in this PR

resolves #10944

Validation

E2E scenarios have been validated as outlined in the doc. We can run another round of E2E validations if there are significant changes in the code. Otherwise, these are not required for general changes and tests in the PR are sufficient.

• Update 9/22 - Ran another round of E2E validations
• Update 9/8 - Ran another round of E2E validations till commit id - c8982d0

Gists

A lot of code has been moved from RPCWorkerConfigFactory to WorkerConfigurationHelper so it can be shared by 2 resolvers. Adding below gists help with the review. Most of it is just refactoring and no logic changes.

• WorkerConfigurationHelper.cs - https://gist.github.com/surgupta-msft/b1bfc518f46114a1913fc2327a52ccc4/revisions
• WorkerConfigurationHelper.GetWorkerDescription() - https://gist.github.com/surgupta-msft/5b67056f47222ff67d527c3e743ef978/revisions

Pull request checklist

IMPORTANT: Currently, changes must be backported to the in-proc branch to be included in Core Tools and non-Flex deployments.

• Backporting to the in-proc branch is not required
  • Otherwise: Link to backporting PR
• My changes do not require documentation changes
  • Otherwise: Documentation issue linked to PR
• My changes should not be added to the release notes for the next release
  • Otherwise: I've added my notes to release_notes.md
• My changes do not need to be backported to a previous version
  • Otherwise: Backport tracked by issue/PR #issue_or_pr
• My changes do not require diagnostic events changes
  • Otherwise: I have added/updated all related diagnostic events and their documentation (Documentation issue linked to PR)
• I have added all required tests (Unit tests, E2E tests)

Additional information

This pull request introduces changes to decouple language workers from the Host release and enable dynamic worker resolution.

Backlog issues - Link
Design doc - Link

Flows covered in this PR -

1. Resolving Worker Configurations -
   • Introduced DynamicWorkerConfigurationResolver, which dynamically resolves worker configurations based on -
     • Environment settings and probing paths
     • Host-Worker compatibility check
     • Leveraging Release channel concept to point to a previous version of the worker
     • Fallback to the old flow if worker is not found at the probing path level
     • Ignoring versions of a worker specified via Hosting config
2. RPCWorkerConfigFactory -
   • Moved some methods to WorkerConfigurationHelper.cs to enable reusing of workers profile evaluation logic between 2 resolvers.
3. Environment Updates -
   • Added a new feature flag FeatureFlagDisableWorkerProbingPaths to disable dynamic resolution feature by users.
   • Added 2 Hosting config settings
     • To get list of workers available for resolution.
     • To get list of workers versions to be ignored.
4. WorkerConfigurationResolverOptionsSetup -
   • Configures WorkerConfigurationResolverOptions by getting required values from IConfiguration and IEnvironment such as worker probing paths, release channel, workers available for resolution via hosting config.

Note: The decoupling workers flow is disabled by default in this PR. The flow can be enabled by setting the Hosting config which will be done after completing other relevant backlog items.

[jviau]

Is IsDynamicWorkerResolutionEnabled expected to change value in the lifetime of a host process? You are using IOptionsMonitor and .CurrentValue, which makes it look like you expect it to change. But you are registering a singleton here. This factory will only ever be run once. So, once a dynamic or default config resolver is selected it will never change.

[jviau]

This is immutable, readonly, and has no values. It will never contain a value. What is the purpose of it?

[surgupta-msft]

Today, we don't have any capability that requires negotiation. But if in future, we introduce a capability then we can add in this structure?

For example, there is a new feature "feature1" and during worker probing we want to make sure that host supports "feature1" then we can add as follows in that Host version.

HostCapabilities = ImmutableHashSet.Create<string>("feature1",  "feature2");

[jviau]

Consider returning IReadOnlyDictionary, unless you expect callers to mutate it.

Preferring IReadOnlyDictionary opens up the door for implementations to cache their result and return it in future calls, as they can assert no caller has mutated it underneath them

[surgupta-msft]

Caller can modify it - Link1, Link2

[jviau]

I only ever use immutable collections for a very specific case: when you expect callers to take your value AND "mutate" their own copy of it, but you don't want the original instance to be affected.

If these are expected to be purely read only (you don't expect callers to make adjustments), then use a readonly interface.

IReadOnlySet<T>, IReadOnlyDictionary<TKey, TValue>

To give an example of a good immutable collection use case is Roslyn analyzers. Roslyn will provide a bunch of collections to all the different analyzers that are expected to make changes to them and return those changes. But these analyzers are all ran in parallel, so regular collections wouldn't be thread safe! So Roslyn uses immutable collects to send the initial values to all the analyzers, then evaluates any changes the analyzers made afterwards and merges them together. (this is my very basic understanding of Roslyn, it may not be 100% accurate)

[soninaren]

nit: ArgumentNullException.ThrowIfNull(metricsLogger)

similar recommendation for other arguments as well.