refactor: provide structure to client configuration (#778)

* Proposes a new convention doc for client configuration that describes how new configurations should be added and in general how they are used by the SDK. This isn't a normal design doc but a convention/guidelines doc.
* Define new builder interfaces for specific domain configurations (e.g. `HttpClientConfig.Builder`, `TracingClientConfig.Builder`, etc). Previously these configuration interfaces were inherited on the generated `Config` type but not on the configuration builder. This is pre-requisite for runtime plugins to work because without it every `Config.Builder` generated shares nothing with any other client. This prevents any kind of common code (e.g. like a plugin) to configure a service client without knowing the concrete type of client.
* Rename `ServiceGenerator` to `ServiceClientGenerator`
* Refactor `ClientConfigGenerator` into an abstract base class that makes less assumptions about the type of configuration being generated. 
* Generate a builder for service clients
* Introduce new runtime types that generated service client `companion` objects inherit from. This moves the DSL `operator invoke` function into the runtime.
* Introduce an abstract base class for service client builders. This will allow us to inject code into all clients instantiated (e.g. load plugins using SPI is one use case).
* Moved a few things around (e.g. everything in core `config` package moved to the `client` subpackage).
* Refactors `RuntimeTypes` to reduce boilerplate
* Refactor our section generator abstraction to be type safe when getting section keys

Author: aajtodd

.changes/cbe1ef3e-6a6b-467d-b15b-a50eb17c4dbb.json

@@ -0,0 +1,5 @@
+{
+    "id": "cbe1ef3e-6a6b-467d-b15b-a50eb17c4dbb",
+    "type": "misc",
+    "description": "Refactor the way service client configuration is generated"
+}

docs/design/README.md

@@ -10,13 +10,12 @@ Start here for an overview:
 
 ## Detailed sub-designs
 
+* [Client Configuration](client-configuration.md)
 * [Document type](document-type.md)
 * [Domain class types in Kotlin SDK](domain-class-types-in-kotlin-sdk.md)
 * [Binary request/response streams](binary-streaming.md)
 * [Event streams](event-streams.md)
-* [Marshaling/serde](marshalling-serde.md) (coming soon)
 * [Modeled errors](modeled-errors.md)
-* [Nullable properties in SDK domain types](nullable-properties-in-sdk-domain-types.md) (coming soon)
 * [Pagination](paginators.md)
 * [Retries](retries.md)
 * [Tracing](tracing.md)

docs/design/client-configuration.md

@@ -0,0 +1,329 @@
+# Client Configuration
+
+* **Type**: Convention
+* **Author(s)**: Aaron Todd
+
+# Abstract
+
+This document describes the structure and conventions used for service client configuration. 
+
+# Conventions
+
+This section describes general conventions and guidelines to be used for service client configuration. See the section
+below for a concrete example of these conventions in practice.
+
+* All generated service client config builders implement `SdkClientConfig.Builder`
+* Domain specific configuration (e.g. HTTP, tracing, etc) should be used as a "mixin" rather than form a hierarchy
+    * e.g. `HttpClientConfig` should not inherit from `SdkClientConfig`
+* Configuration interfaces should live in a subpackage `config` of whatever root package they belong to.
+* Configuration interfaces should have a nested `Builder` interface (e.g. `FooConfig.Builder`)
+
+## Example Domain Specific Configuration
+
+This section walks through what a new configuration interface and builder should look like in the runtime.
+
+```kotlin
+
+/**
+ * Description (1)
+ */
+public interface FooConfig {  // 2
+    /**
+     * Property use description // 3
+     */
+    public val fooConfigProp: String  // 4
+
+    public interface Builder {  // 5
+
+        /**
+         * Property configuration description // 6
+         */
+        public var fooConfigProp: String?  // 7
+    }
+}
+```
+
+1. A detailed description of what the configuration is used for and what might be found in it.
+2. Configuration interface should be `public` and generally be `Xyz` with the suffix `Config` (e.g. `XyzConfig`).
+3. A detailed description of how the property is used, which _may_ differ from the description of how to configure it.
+    * This may delegate to the builder property docs. e.g. `Controls the how the foo does blah. See [Builder.fooConfigProp] for more details.`.
+4. Configuration fields must be read-only (`val`) and should be immutable. Their type may differ from the builder type, e.g. `List` vs `MutableList` or be non-null where the configuration field is nullable.
+    * These differences are handled in codegen by setting default values or mapping the builder type as required.
+5. The configuration builder interface should be public and nested inside the configuration interface it is meant to build.
+6. A detailed description of how the property is configured and how it is used. 
+7. Builder fields should generally be `var` and/or mutable. Their type may be different from the immutable configuration property.
+
+## Example Service Client
+
+This section walks through an example service client configuration class, describing each of its components at a high level.
+
+```kotlin
+public interface BazClient : SdkClient { // 1
+
+    override val serviceName: String
+        get() = "baz"
+
+    public override val config: Config // 2
+
+    public companion object : SdkClientFactory<Config, Config.Builder, BazClient, Builder>() { // 3
+        @JvmStatic
+        override fun builder(): Builder = Builder()
+    }
+
+    public class Builder internal constructor() : AbstractSdkClientBuilder<Config, Config.Builder, BazClient>() { // 4
+        override val config: Config.Builder = Config.Builder()
+        override fun newClient(config: Config): BazClient = DefaultBazClient(config)  // 5
+    }
+
+    public class Config private constructor(builder: Builder) : SdkClientConfig, HttpClientConfig {  // 6
+        override val sdkLogMode: SdkLogMode = builder.sdkLogMode  // 7
+        override val httpClientEngine: HttpClientEngine? = builder.httpClientEngine
+        val bazSpecificConfig: String? = builder.bazSpecificConfig
+
+        public companion object { // 8
+            public inline operator fun invoke(block: Builder.() -> kotlin.Unit): Config = Builder().apply(block).build()
+        }
+
+        public class Builder : SdkClientConfig.Builder<Config>, HttpClientConfig.Builder { // 9
+            /**
+             * Description
+             */
+            override var httpClientEngine: HttpClientEngine? = null  // 10
+
+            /**
+             * Description
+             */
+            override var sdkLogMode: SdkLogMode = SdkLogMode.Default
+
+            /**
+             * Description
+             */
+            var bazSpecificConfig: String? = null
+
+            override fun build(): Config = Config(this)
+        }
+    }
+}
+```
+
+NOTE: The example client interface above has intentionally left off KDoc comments for many of the fields for brevity.
+NOTE: See the Appendix for definitions of some of these inherited interfaces (e.g. `SdkClientFactory`).
+
+1. All generated service clients inherit from `SdkClient`
+2. The `config` property (from `SdkClient`) may be overridden with a more specific configuration than what the inherited interface specifies (`SdkClientConfig`)
+3. A companion object is generated that inherits default behavior from the runtime. See [client creation patterns](#client-creation-patterns).
+4. A builder for creating a service client is generated. It inherits behavior from `AbstractSdkClientBuilder`
+5. The internal concrete (generated) implementation of `BazClient` is provided. 
+6. The immutable service client configuration container type. 
+    * This will inherit from `SdkClientConfig` and `N` "mixin" configurations. Mixins are pulled in based on the model/customizations.
+7. Configuration properties are set from the builder, possibly providing defaults when not set or mapping to different types.
+8. The configuration companion object provides a convenience DSL like experience for instantiation, e.g. `val config = BazClient.Config { ... }`
+9. The configuration builder inherits from `SdkClientConfig.Builder` and `N` "mixin" configuration builders. 
+10. Default values are set for the configuration builder properties
+
+## Client Creation Patterns
+
+This section describes in more detail how service clients are created and configured using the `BazClient` described
+in the [example service client](#example-service-client) section. These behaviors are a combination of runtime types and
+codegen.
+
+```kotlin
+// explicit using DSL syntax inherited from companion `SdkClientFactory`
+val c1 = BazClient { // this: BazClient.Config.Builder
+    sdkLogMode = SdkLogMode.LogRequest
+}
+
+// use of a builder explicitly, this could be passed around for example
+val c2 = BazClient.builder().apply { // this: BazClient.Builder
+    config.sdkLogMode = SdkLogMode.LogRequest
+}.build()
+
+// "vended" using common code
+val c3 = ClientVendingMachine.getClient(BazClient)
+
+
+private object ClientVendingMachine {
+    fun <
+        TConfig: SdkClientConfig,
+        TConfigBuilder: SdkClientConfig.Builder<TConfig>,
+        TClient: SdkClient,
+        TClientBuilder: SdkClient.Builder<TConfig, TConfigBuilder, TClient>
+    > getClient(factory: SdkClientFactory<TConfig, TConfigBuilder, TClient, TClientBuilder>): TClient {
+        val builder = factory.builder()
+        // setting various configuration
+        when(val config = builder.config) {
+            is BazClient.Config.Builder -> {
+                // access to configuration that may be specific to BazClient
+                config.bazSpecificConfig = "specific"
+            }
+            is HttpClientConfig.Builder -> {
+                // access to http related configuration
+                config.httpClientEngine
+            }
+            else -> {
+                // always available from generic bounds:
+                config.sdkLogMode
+            }
+        }
+        return builder.build()
+    }
+}
+```
+
+1. `c1` is created using DSL like syntax inherited from [SdkClientFactory](#sdkclientfactory)
+2. `c2` is created by instantiating a service client builder explicitly using `BazClient.builder()`. 
+This comes from the companion object and is a static method (defined in [SdkClientFactory](#sdkclientfactory))
+3. `c3` is an example of what's possible. It is created using common code that inspects the builder type for specific
+configuration (builder) mixins. This type of pattern could be used to centralize configuring a service client without
+knowing the concrete type.
+
+# Appendices
+
+## Appendix: FAQ
+
+**Why builders**?
+
+Builders are useful for dealing with evolution as well as the ability to insert additional logic into the creation of 
+some entity at the time it is built. The API of most builders should be "DSL" like in nature (properties over functions).
+When possible the SDK will offer a DSL like experience for creating instances of types.
+
+**Why no hierarchy of config builder classes to inherit from**?
+
+The config interface (e.g. `FooConfig`) is an immutable collection of properties to be
+consumed by something else (usually a service client). The configuration builder interface (`FooConfig.Builder`)
+describes how to configure the properties. The builder interfaces (e.g. `FooConfig.Builder`) generally have no 
+concrete implementation because the SDK will codegen a specific implementation by describing the configuration 
+properties (including the default values that should be used when not given). 
+
+## Appendix: Runtime Types
+
+This section provides concrete examples of runtime types referenced in this document. These types may differ or evolve
+from when this document was written without loss of understanding of conventions and guidelines described herein. 
+
+### SdkClient
+
+```kotlin
+/**
+ * Common interface all generated service clients implement
+ */
+public interface SdkClient : Closeable {
+    /**
+     * The name of the service client
+     */
+    public val serviceName: String
+
+    /**
+     * The client's configuration
+     */
+    public val config: SdkClientConfig
+
+    public interface Builder<
+            TConfig : SdkClientConfig,
+            TConfigBuilder : SdkClientConfig.Builder<TConfig>,
+            out TClient : SdkClient,
+            > : Buildable<TClient> {
+
+        /**
+         * The configuration builder for this client
+         */
+        public val config: TConfigBuilder
+    }
+}
+```
+
+### SdkClientConfig
+
+```kotlin
+/**
+ * Common configuration options for any generated SDK client
+ */
+public interface SdkClientConfig {
+    /**
+     * Controls the events that will be logged by the SDK, see [Builder.sdkLogMode].
+     */
+    public val sdkLogMode: SdkLogMode
+        get() = SdkLogMode.Default
+
+    public interface Builder<TConfig : SdkClientConfig> : Buildable<TConfig> {
+        /**
+         * Configure events that will be logged. By default, clients will not output
+         * raw requests or responses. Use this setting to opt in to additional debug logging.
+         *
+         * This can be used to configure logging of requests, responses, retries, etc of SDK clients.
+         *
+         * **NOTE**: Logging of raw requests or responses may leak sensitive information! It may also have
+         * performance considerations when dumping the request/response body. This is primarily a tool for
+         * debug purposes.
+         */
+        public var sdkLogMode: SdkLogMode
+    }
+}
+```
+
+### SdkClientFactory
+
+```kotlin
+/**
+ * Interface all generated [SdkClient] companion objects inherit from.
+ */
+public interface SdkClientFactory<
+        TConfig : SdkClientConfig,
+        TConfigBuilder : SdkClientConfig.Builder<TConfig>,
+        TClient : SdkClient,
+        out TClientBuilder : SdkClient.Builder<TConfig, TConfigBuilder, TClient>,
+        > {
+    /**
+     * Return a [TClientBuilder] that can create a new [TClient] instance
+     */
+    public fun builder(): TClientBuilder
+
+    /**
+     * Configure a new [TClient] with [block].
+     *
+     * Example
+     * ```
+     * val client = FooClient { ... }
+     * ```
+     */
+    public operator fun invoke(block: TConfigBuilder.() -> Unit): TClient = builder().apply {
+        config.apply(block)
+    }.build()
+}
+```
+
+
+### AbstractSdkClientBuilder
+
+Abstract base class that all service client builder implementations inherit from. This allows the runtime to add
+additional logic over time before or after a client is instantiated (e.g. this could be used to load and apply plugins
+using SPI on the JVM).
+
+```kotlin
+/**
+ * Abstract base class all [SdkClient] builders should inherit from
+ */
+@InternalApi
+public abstract class AbstractSdkClientBuilder<
+        TConfig : SdkClientConfig,
+        TConfigBuilder : SdkClientConfig.Builder<TConfig>,
+        out TClient : SdkClient,
+        > : SdkClient.Builder<TConfig, TConfigBuilder, TClient> {
+
+    final override fun build(): TClient {
+        return newClient(config.build())
+    }
+
+    /**
+     * Return a new [TClient] instance with the given [config]
+     */
+    protected abstract fun newClient(config: TConfig): TClient
+}
+```
+
+## Appendix: Additional References
+
+* [Kotlin Smithy SDK](kotlin-smithy-sdk.md)
+
+# Revision history
+
+* 01/12/2023 - Created

runtime/protocol/http/common/src/aws/smithy/kotlin/runtime/http/config/HttpClientConfig.kt

@@ -6,13 +6,46 @@
 package aws.smithy.kotlin.runtime.http.config
 
 import aws.smithy.kotlin.runtime.http.engine.HttpClientEngine
+import aws.smithy.kotlin.runtime.http.interceptors.HttpInterceptor
 
 /**
  * The user-accessible configuration properties for the SDKs internal HTTP client facility.
  */
 public interface HttpClientConfig {
     /**
-     * Allows for overriding the default HTTP client engine.
+     * Explicit HTTP engine to use when making SDK requests, when not set a default engine will be created and managed
+     * on behalf of the caller.
+     *
+     * **NOTE**: The caller is responsible for managing the lifetime of the engine when set. The SDK
+     * client will not close it when the client is closed.
      */
     public val httpClientEngine: HttpClientEngine?
+
+    /**
+     * Interceptors that will be executed for each SDK operation.
+     * An [aws.smithy.kotlin.runtime.client.Interceptor] has access to read and modify
+     * the request and response objects as they are processed by the SDK.
+     * Interceptors are executed in the order they are configured and are always later than any added automatically by
+     * the SDK.
+     */
+    public val interceptors: List<HttpInterceptor>
+
+    public interface Builder {
+        /**
+         * Override the default HTTP client engine used to make SDK requests (e.g. configure proxy behavior, timeouts,
+         * concurrency, etc).
+         *
+         * **NOTE**: The caller is responsible for managing the lifetime of the engine when set. The SDK
+         * client will not close it when the client is closed.
+         */
+        public var httpClientEngine: HttpClientEngine?
+
+        /**
+         * Add an [aws.smithy.kotlin.runtime.client.Interceptor] that will have access to read and modify
+         * the request and response objects as they are processed by the SDK.
+         * Interceptors added using this method are executed in the order they are configured and are always
+         * later than any added automatically by the SDK.
+         */
+        public var interceptors: MutableList<HttpInterceptor>
+    }
 }