Fixing 8.19 asciidoc content

README.md

@@ -17,20 +17,21 @@ The .NET client for Elasticsearch provides strongly typed requests and responses
 
 ## Compatibility
 
-Language clients are forward compatible; meaning that the clients support
-communicating with greater or equal minor versions of Elasticsearch without
-breaking. It does not mean that the clients automatically support new features
-of newer Elasticsearch versions; it is only possible after a release of a new
-client version. For example, a 8.12 client version won't automatically support
-the new features of the 8.13 version of Elasticsearch, the 8.13 client version
-is required for that. Elasticsearch language clients are only backwards
-compatible with default distributions and without guarantees made.
-
-| Elasticsearch Version | Elasticsearch-NET Branch  | Supported |
-| --------------------- | ------------------------- | --------- |
-| main                  | main                      |           |
-| 8.x                   | 8.x                       | 8.x       |
-| 7.x                   | 7.x                       | 7.17      |
+Language clients are **forward compatible**: 
+
+Given a constant major version of the client, each related minor version is compatible with its equivalent- and all later Elasticsearch minor versions of the **same or next higher** major version.
+
+For example:
+
+| Client Version | Compatible with Elasticsearch `7.x` | Compatible with Elasticsearch `8.x` | Compatible with Elasticsearch `9.x` |
+| ---: | :-- | :-- | :-- |
+| 8.x | ❌ no | ✅ yes | ✅ yes |
+| 7.x | ✅ yes | ✅ yes | ❌ no |
+
+Language clients are also **backward compatible** across minor versions within the **same** major version (without strong guarantees), but **never** backward compatible with earlier Elasticsearch major versions.
+
+> [!NOTE]
+> Compatibility does not imply feature parity. For example, an `8.12` client is compatible with `8.13`, but does not support any of the new features introduced in Elasticsearch `8.13`.
 
 ## Installation
 

docs/client-concepts/client-concepts.asciidoc

@@ -0,0 +1,26 @@
+[[client-concepts]]
+= Client concepts
+
+The .NET client for {es} maps closely to the original {es} API. All 
+requests and responses are exposed through types, making it ideal for getting up and running quickly.
+
+[[serialization]]
+== Serialization
+
+By default, the .NET client for {es} uses the Microsoft System.Text.Json library for serialization. The client understands how to serialize and 
+deserialize the request and response types correctly. It also handles (de)serialization of user POCO types representing documents read or written to {es}.
+
+The client has two distinct serialization responsibilities - serialization of the types owned by the `Elastic.Clients.Elasticsearch` library and serialization of source documents, modeled in application code. The first responsibility is entirely internal; the second is configurable.
+
+[[source-serialization]]
+=== Source serialization
+
+Source serialization refers to the process of (de)serializing POCO types in consumer applications as source documents indexed and retrieved from {es}. A source serializer implementation handles serialization, with the default implementation using the `System.Text.Json` library. As a result, you may use `System.Text.Json` attributes and converters to control the serialization behavior. 
+
+* <<modeling-documents-with-types,Modelling documents with types>>
+
+* <<customizing-source-serialization,Customizing source serialization>>
+
+include::serialization/modeling-documents-with-types.asciidoc[]
+
+include::serialization/custom-serialization.asciidoc[]

docs/client-concepts/serialization/custom-serialization.asciidoc

@@ -0,0 +1,232 @@
+[[customizing-source-serialization]]
+==== Customizing source serialization
+
+The built-in source serializer handles most POCO document models correctly. Sometimes, you may need further control over how your types are serialized.
+
+NOTE: The built-in source serializer uses the https://learn.microsoft.com/en-us/dotnet/standard/serialization/system-text-json/overview[Microsoft `System.Text.Json` library] internally. You can apply `System.Text.Json` attributes and converters to control the serialization of your document types.
+
+[discrete]
+[[system-text-json-attributes]]
+===== Using `System.Text.Json` attributes
+
+`System.Text.Json` includes attributes that can be applied to types and properties to control their serialization. These can be applied to your POCO document types to perform actions such as controlling the name of a property or ignoring a property entirely. Visit the https://learn.microsoft.com/en-us/dotnet/standard/serialization/system-text-json/overview[Microsoft documentation for further examples].
+
+We can model a document to represent data about a person using a regular class (POCO), applying `System.Text.Json` attributes as necessary.
+
+[source,csharp]
+----
+include::{doc-tests-src}/ClientConcepts/Serialization/CustomSerializationTests.cs[tag=usings-serialization]
+include::{doc-tests-src}/ClientConcepts/Serialization/CustomSerializationTests.cs[tag=person-class-with-attributes]
+----
+<1> The `JsonPropertyName` attribute ensures the `FirstName` property uses the JSON name `forename` when serialized.
+<2> The `JsonIgnore` attribute prevents the `Age` property from appearing in the serialized JSON.
+
+We can then index an instance of the document into {es}.
+
+[source,csharp]
+----
+include::{doc-tests-src}/ClientConcepts/Serialization/CustomSerializationTests.cs[tag=usings]
+include::{doc-tests-src}/ClientConcepts/Serialization/CustomSerializationTests.cs[tag=index-person-with-attributes]
+----
+
+The index request is serialized, with the source serializer handling the `Person` type, serializing the POCO property named `FirstName` to the JSON object member named `forename`. The `Age` property is ignored and does not appear in the JSON.
+
+[source,javascript]
+----
+{
+  "forename": "Steve"
+}
+----
+
+[discrete]
+[[configuring-custom-jsonserializeroptions]]
+===== Configuring custom `JsonSerializerOptions`
+
+The default source serializer applies a set of standard `JsonSerializerOptions` when serializing source document types. In some circumstances, you may need to override some of our defaults. This is achievable by creating an instance of `DefaultSourceSerializer` and passing an `Action<JsonSerializerOptions>`, which is applied after our defaults have been set. This mechanism allows you to apply additional settings or change the value of our defaults.
+
+The `DefaultSourceSerializer` includes a constructor that accepts the current `IElasticsearchClientSettings` and a `configureOptions` `Action`.
+
+[source,csharp]
+----
+public DefaultSourceSerializer(IElasticsearchClientSettings settings, Action<JsonSerializerOptions> configureOptions);
+----
+
+Our application defines the following `Person` class, which models a document we will index to {es}.
+
+[source,csharp]
+----
+include::{doc-tests-src}/ClientConcepts/Serialization/CustomSerializationTests.cs[tag=person-class]
+----
+
+We want to serialize our source document using Pascal Casing for the JSON properties. Since the options applied in the `DefaultSouceSerializer` set the `PropertyNamingPolicy` to `JsonNamingPolicy.CamelCase`, we must override this setting. After configuring the `ElasticsearchClientSettings`, we index our document to {es}.
+
+[source,csharp]
+----
+include::{doc-tests-src}/ClientConcepts/Serialization/CustomSerializationTests.cs[tag=usings]
+include::{doc-tests-src}/ClientConcepts/Serialization/CustomSerializationTests.cs[tag=custom-options-local-function]
+include::{doc-tests-src}/ClientConcepts/Serialization/CustomSerializationTests.cs[tag=create-client]
+include::{doc-tests-src}/ClientConcepts/Serialization/CustomSerializationTests.cs[tag=index-person]
+----
+<1> A local function can be defined, accepting a `JsonSerializerOptions` parameter. Here, we set `PropertyNamingPolicy` to `null`. This returns to the default behavior for `System.Text.Json`, which uses Pascal Case.
+<2> When creating the `ElasticsearchClientSettings`, we supply a `SourceSerializerFactory` using a lambda. The factory function creates a new instance of `DefaultSourceSerializer`, passing in the `settings` and our `ConfigureOptions` local function. We have now configured the settings with a custom instance of the source serializer.
+
+The `Person` instance is serialized, with the source serializer serializing the POCO property named `FirstName` using Pascal Case.
+
+[source,javascript]
+----
+{
+  "FirstName": "Steve"
+}
+----
+
+As an alternative to using a local function, we could store an `Action<JsonSerializerOptions>` into a variable instead, which can be passed to the `DefaultSouceSerializer` constructor.
+
+[source,csharp]
+----
+include::{doc-tests-src}/ClientConcepts/Serialization/CustomSerializationTests.cs[tag=custom-options-action]
+----
+
+[discrete]
+[[registering-custom-converters]]
+===== Registering custom `System.Text.Json` converters
+
+In certain more advanced situations, you may have types which require further customization during serialization than is possible using `System.Text.Json` property attributes. In these cases, the recommendation from Microsoft is to leverage a custom `JsonConverter`. Source document types serialized using the  `DefaultSourceSerializer` can leverage the power of custom converters.
+
+For this example, our application has a document class that should use a legacy JSON structure to continue operating with existing indexed documents. Several options are available, but we'll apply a custom converter in this case.
+
+Our class is defined, and the `JsonConverter` attribute is applied to the class type, specifying the type of a custom converter.
+
+[source,csharp]
+----
+include::{doc-tests-src}/ClientConcepts/Serialization/CustomSerializationTests.cs[tag=usings-serialization]
+include::{doc-tests-src}/ClientConcepts/Serialization/CustomSerializationTests.cs[tag=customer-with-jsonconverter-attribute]
+----
+<1> The `JsonConverter` attribute signals to `System.Text.Json` that it should use a converter of type `CustomerConverter` when serializing instances of this class.
+
+When serializing this class, rather than include a string value representing the value of the `CustomerType` property, we must send a boolean property named `isStandard`. This requirement can be achieved with a custom JsonConverter implementation.
+
+[source,csharp]
+----
+include::{doc-tests-src}/ClientConcepts/Serialization/CustomSerializationTests.cs[tag=converter-usings]
+include::{doc-tests-src}/ClientConcepts/Serialization/CustomSerializationTests.cs[tag=customer-converter]
+----
+<1> When reading, this converter reads the `isStandard` boolean and translate this to the correct `CustomerType` enum value.
+<2> When writing, this converter translates the `CustomerType` enum value to an `isStandard` boolean property.
+
+We can then index a customer document into {es}.
+
+[source,csharp]
+----
+include::{doc-tests-src}/ClientConcepts/Serialization/CustomSerializationTests.cs[tag=usings]
+include::{doc-tests-src}/ClientConcepts/Serialization/CustomSerializationTests.cs[tag=index-customer-with-converter]
+----
+
+The `Customer` instance is serialized using the custom converter, creating the following JSON document.
+
+[source,javascript]
+----
+{
+  "customerName": "Customer Ltd",
+  "isStandard": false
+}
+----
+
+[discrete]
+[[creating-custom-system-text-json-serializer]]
+===== Creating a custom `SystemTextJsonSerializer`
+
+The built-in `DefaultSourceSerializer` includes the registration of `JsonConverter` instances which apply during source serialization. In most cases, these provide the proper behavior for serializing source documents, including those which use `Elastic.Clients.Elasticsearch` types on their properties.
+
+An example of a situation where you may require more control over the converter registration order is for serializing `enum` types. The `DefaultSourceSerializer` registers the `System.Text.Json.Serialization.JsonStringEnumConverter`, so enum values are serialized using their string representation. Generally, this is the preferred option for types used to index documents to {es}.
+
+In some scenarios, you may need to control the string value sent for an enumeration value. That is not directly supported in `System.Text.Json` but can be achieved by creating a custom `JsonConverter` for the `enum` type you wish to customize. In this situation, it is not sufficient to use the `JsonConverterAttribute` on the `enum` type to register the converter. `System.Text.Json` will prefer the converters added to the `Converters` collection on the `JsonSerializerOptions` over an attribute applied to an `enum` type. It is, therefore, necessary to either remove the `JsonStringEnumConverter` from the `Converters` collection or register a specialized converter for your `enum` type before the `JsonStringEnumConverter`.
+
+The latter is possible via several techniques. When using the {es} .NET library, we can achieve this by deriving from the abstract `SystemTextJsonSerializer` class.
+
+Here we have a POCO which uses the `CustomerType` enum as the type for a property.
+
+[source,csharp]
+----
+include::{doc-tests-src}/ClientConcepts/Serialization/CustomSerializationTests.cs[tag=usings-serialization]
+include::{doc-tests-src}/ClientConcepts/Serialization/CustomSerializationTests.cs[tag=customer-without-jsonconverter-attribute]
+----
+
+To customize the strings used during serialization of the `CustomerType`, we define a custom `JsonConverter` specific to our `enum` type.
+
+[source,csharp]
+----
+include::{doc-tests-src}/ClientConcepts/Serialization/CustomSerializationTests.cs[tag=usings-serialization]
+include::{doc-tests-src}/ClientConcepts/Serialization/CustomSerializationTests.cs[tag=customer-type-converter]
+----
+<1> When reading, this converter translates the string used in the JSON to the matching enum value.
+<2> When writing, this converter translates the `CustomerType` enum value to a custom string value written to the JSON.
+
+We create a serializer derived from `SystemTextJsonSerializer` to give us complete control of converter registration order.
+
+[source,csharp]
+----
+include::{doc-tests-src}/ClientConcepts/Serialization/CustomSerializationTests.cs[tag=derived-converter-usings]
+include::{doc-tests-src}/ClientConcepts/Serialization/CustomSerializationTests.cs[tag=my-custom-serializer]
+----
+<1> Inherit from `SystemTextJsonSerializer`.
+<2> In the constructor, use the factory method `DefaultSourceSerializer.CreateDefaultJsonSerializerOptions` to create default options for serialization. No default converters are registered at this stage because we pass `false` as an argument.
+<3> Register our `CustomerTypeConverter` as the first converter.
+<4> To apply any default converters, call the `DefaultSourceSerializer.AddDefaultConverters` helper method, passing the options to modify.
+<5> Implement the `CreateJsonSerializerOptions` method returning the stored `JsonSerializerOptions`.
+
+Because we have registered our `CustomerTypeConverter` before the default converters (which include the `JsonStringEnumConverter`), our converter takes precedence when serializing `CustomerType` instances on source documents.
+
+The base `SystemTextJsonSerializer` class handles the implementation details of binding, which is required to ensure that the built-in converters can access the `IElasticsearchClientSettings` where needed.
+
+We can then index a customer document into {es}.
+
+[source,csharp]
+----
+include::{doc-tests-src}/ClientConcepts/Serialization/CustomSerializationTests.cs[tag=usings]
+include::{doc-tests-src}/ClientConcepts/Serialization/CustomSerializationTests.cs[tag=index-customer-without-jsonconverter-attribute]
+----
+
+The `Customer` instance is serialized using the custom `enum` converter, creating the following JSON document.
+
+[source,javascript]
+----
+{
+  "customerName": "Customer Ltd",
+  "customerType": "premium" // <1>
+}
+----
+<1> The string value applied during serialization is provided by our custom converter.
+
+[discrete]
+[[creating-custom-serializers]]
+===== Creating a custom `Serializer`
+
+Suppose you prefer using an alternative JSON serialization library for your source types. In that case, you can inject an isolated serializer only to be called for the serialization of `_source`, `_fields`, or wherever a user-provided value is expected to be written and returned.
+
+Implementing `Elastic.Transport.Serializer` is technically enough to create a custom source serializer.
+
+[source,csharp]
+----
+include::{doc-tests-src}/ClientConcepts/Serialization/CustomSerializationTests.cs[tag=vanilla-serializer-using-directives]
+include::{doc-tests-src}/ClientConcepts/Serialization/CustomSerializationTests.cs[tag=vanilla-serializer]
+----
+
+Registering up the serializer is performed in the `ConnectionSettings` constructor.
+
+[source,csharp]
+----
+include::{doc-tests-src}/ClientConcepts/Serialization/CustomSerializationTests.cs[tag=usings]
+include::{doc-tests-src}/ClientConcepts/Serialization/CustomSerializationTests.cs[tag=register-vanilla-serializer]
+----
+<1> If implementing `Serializer` is enough, why must we provide an instance wrapped in a factory `Func`?
+
+There are various cases where you might have a POCO type that contains an `Elastic.Clients.Elasticsearch` type as one of its properties. The `SourceSerializerFactory` delegate provides access to the default built-in serializer so you can access it when necessary. For example, consider if you want to use percolation; you need to store {es} queries as part of the `_source` of your document, which means you need to have a POCO that looks like this.
+
+[source,csharp]
+----
+include::{doc-tests-src}/ClientConcepts/Serialization/CustomSerializationTests.cs[tag=querydsl-using-directives]
+include::{doc-tests-src}/ClientConcepts/Serialization/CustomSerializationTests.cs[tag=percolation-document]
+----
+
+A custom serializer would not know how to serialize `Query` or other `Elastic.Clients.Elasticsearch` types that could appear as part of
+the `_source` of a document. Therefore, your custom `Serializer` would need to store a reference to our built-in serializer and delegate serialization of Elastic types back to it.

docs/client-concepts/serialization/modeling-documents-with-types.asciidoc

@@ -0,0 +1,37 @@
+[[modeling-documents-with-types]]
+==== Modeling documents with types
+
+{es} provides search and aggregation capabilities on the documents that it is sent and indexes. These documents are sent as
+JSON objects within the request body of a HTTP request. It is natural to model documents within the {es} .NET client using
+https://en.wikipedia.org/wiki/Plain_Old_CLR_Object[POCOs (__Plain Old CLR Objects__)].
+
+This section provides an overview of how types and type hierarchies can be used to model documents.
+
+[[default-behaviour]]
+===== Default behaviour
+
+The default behaviour is to serialize type property names as camelcase JSON object members.
+
+We can model documents using a regular class (POCO).
+
+[source,csharp]
+----
+include-tagged::{doc-tests-src}/ClientConcepts/Serialization/ModellingDocumentsWithTypesTests.cs[my-document-poco]
+----
+
+We can then index the an instance of the document into {es}.
+
+[source,csharp]
+----
+include-tagged::{doc-tests-src}/ClientConcepts/Serialization/ModellingDocumentsWithTypesTests.cs[usings]
+include-tagged::{doc-tests-src}/ClientConcepts/Serialization/ModellingDocumentsWithTypesTests.cs[index-my-document]
+----
+
+The index request is serialized, with the source serializer handling the `MyDocument` type, serializing the POCO property named `StringProperty` to the JSON object member named `stringProperty`.
+
+[source,javascript]
+----
+{
+  "stringProperty": "value"
+}
+----