Merge remote-tracking branch 'origin/main' into eedugon/kustomize_troubleshooting

Author: eedugon

.backportrc.json

@@ -1,7 +1,7 @@
 {
   "repoOwner": "elastic",
   "repoName": "ingest-docs",
-  "targetBranchChoices": ["main", "8.x","8.15","8.14","8.13","8.12", "8.11", "8.10", "8.9", "8.8", "8.7", "8.6", "7.17"],
+  "targetBranchChoices": ["main", "8.x","8.16","8.15","8.14","8.13","8.12", "8.11", "8.10", "8.9", "8.8", "8.7", "8.6", "7.17"],
   "autoMerge": true,
   "autoMergeMethod": "squash",
   "branchLabelMapping": {

.mergify.yml

@@ -29,6 +29,20 @@ pull_request_rules:
         title: "[{{ destination_branch }}] {{ title }} (backport #{{ number }})"
         labels:
           - backport
+  - name: backport patches to 8.16 branch
+    conditions:
+      - merged
+      - base=main
+      - label=backport-8.16
+    actions:
+      backport:
+        assignees:
+          - "{{ author }}"
+        branches:
+          - "8.16"
+        title: "[{{ destination_branch }}] {{ title }} (backport #{{ number }})"
+        labels:
+          - backport
   - name: backport patches to 8.15 branch
     conditions:
       - merged

docs/en/ingest-management/commands.asciidoc

@@ -1106,16 +1106,19 @@ to remove files.
 
 [IMPORTANT]
 ====
-Be sure to run the `uninstall` command from the directory where {agent} is installed and not from the directory where you previously ran the `install` command.
+Be sure to run the `uninstall` command from a directory outside of where {agent} is installed.
 
---
-
-include::{ingest-docs-root}/docs/en/ingest-management/tab-widgets/uninstall-widget.asciidoc[]
+For example, on a Windows system the install location is `C:\Program Files\Elastic\Agent`. Run the uninstall command from `C:\Program Files\Elastic` or `\tmp`, or even your default home directory:
 
---
+[source,shell]
+----
+C:\"Program Files"\Elastic\Agent\elastic-agent.exe uninstall
+----
 
 ====
 
+include::{ingest-docs-root}/docs/en/ingest-management/tab-widgets/uninstall-widget.asciidoc[]
+
 [discrete]
 === Synopsis
 

docs/en/ingest-management/elastic-agent/configuration/autodiscovery/elastic-agent-kubernetes-autodiscovery.asciidoc

@@ -3,28 +3,28 @@
 
 When you run applications on containers, they become moving targets to the monitoring system. Autodiscover allows you to track them and adapt settings as changes happen. By defining configuration templates, the autodiscover subsystem can monitor services as they start running.
 
-To use autodiscovery, you will need to modify the manifest file of the {agent}. Refer to <<running-on-kubernetes-standalone>> to learn how to retrieve and configure it.
+To use autodiscover, you will need to modify the manifest file of the {agent}. Refer to <<running-on-kubernetes-standalone>> to learn how to retrieve and configure it.
 
-There are two different ways to use autodiscovery:
+There are two different ways to use autodiscover:
 
 * <<conditions-based-autodiscover>>
 
 * <<hints-annotations-autodiscovery>>
 
 
 [discrete]
-== How to configure autodiscovery
+== How to configure autodiscover
 
-`Conditions Based Autodiscovery` is more suitable for scenarios when users know the different group of containers they want to monitor in advance. It is advisable to choose conditions-based configuration when administrators can configure specific conditions that match their needs. Conditions are supported in both Managed and Standalone {agent}.
+`Conditions Based Autodiscover` is more suitable for scenarios when users know the different group of containers they want to monitor in advance. It is advisable to choose conditions-based configuration when administrators can configure specific conditions that match their needs. Conditions are supported in both Managed and Standalone {agent}.
 
-`Hints Based Autodiscovry` is suitable for more generic scenarios, especially when users don't know the exact configuration of the system to monitor and can not create in advance conditions. Additionally a big advantage of Hints Autodiscovery is the ability to offer dynamic configuration of inputs based on annotations from Pods/Containers. If dynamic configuration is needed, then Hints should be enabled. Hints are supported only in Standalone {agent} mode.
+`Hints Based Autodiscover` is suitable for more generic scenarios, especially when users don't know the exact configuration of the system to monitor and can not create in advance conditions. Additionally a big advantage of Hints Autodiscover is the ability to offer dynamic configuration of inputs based on annotations from Pods/Containers. If dynamic configuration is needed, then Hints should be enabled. Hints are supported only in Standalone {agent} mode.
 
-*Best Practises when you configure autodiscovery:*
+*Best Practises when you configure autodiscover:*
 
 - Always define alternatives and default values to your variables that are used in conditions or [hint templates](eg. See `auth.basic` set as `auth.basic.user: ${kubernetes.hints.nginx.access.username|kubernetes.hints.nginx.username|''}`` in [nginx.yml](https://github.com/elastic/elastic-agent/blob/main/deploy/kubernetes/elastic-agent-standalone/templates.d/nginx.yml#L8))
 
 IMPORTANT: When an input uses a variable substitution that is not present in the current key/value mappings being evaluated, the input is removed in the result. (See more information in <<dynamic-input-configuration>>)   
 
 - To debug configurations that include variable substitution and conditions, use the inspect command of {agent}. (See more information in <<dynamic-input-configuration>> in *Debugging* Section)
 
-- In Condition Based autodiscovery is advisable to define a generic last condition that will act as your default condition and will be validated when all others fail or don't apply. If applicable, such conditions might help to identify processing and troubleshoot possible problems.
+- In Condition Based autodiscover is advisable to define a generic last condition that will act as your default condition and will be validated when all others fail or don't apply. If applicable, such conditions might help to identify processing and troubleshoot possible problems.

docs/en/ingest-management/elastic-agent/configuration/env/container-envs.asciidoc

@@ -68,8 +68,6 @@ Settings used to prepare the {fleet} plugin in {kib}.
 |===
 | Settings | Description
 
-include::shared-env.asciidoc[tag=kibana-fleet-setup]
-
 include::shared-env.asciidoc[tag=kibana-fleet-host]
 
 include::shared-env.asciidoc[tag=kibana-fleet-username]
@@ -100,6 +98,12 @@ include::shared-env.asciidoc[tag=fleet-server-elasticsearch-host]
 
 include::shared-env.asciidoc[tag=fleet-server-elasticsearch-ca]
 
+include::shared-env.asciidoc[tag=fleet-server-es-cert]
+
+include::shared-env.asciidoc[tag=fleet-server-es-cert-key]
+
+include::shared-env.asciidoc[tag=fleet-server-insecure-http]
+
 include::shared-env.asciidoc[tag=fleet-server-service-token]
 
 include::shared-env.asciidoc[tag=fleet-server-service-token-path]
@@ -118,8 +122,14 @@ include::shared-env.asciidoc[tag=fleet-server-cert-key]
 
 include::shared-env.asciidoc[tag=fleet-server-cert-key-passphrase]
 
+include::shared-env.asciidoc[tag=fleet-server-client-auth]
+
 include::shared-env.asciidoc[tag=fleet-server-es-ca-trusted-fingerprint]
 
+include::shared-env.asciidoc[tag=fleet-daemon-timeout]
+
+include::shared-env.asciidoc[tag=fleet-server-timeout]
+
 |===
 
 [discrete]
@@ -135,10 +145,16 @@ Settings used to enroll an {agent} into a {fleet-server}.
 |===
 | Settings | Description
 
+include::shared-env.asciidoc[tag=elastic-agent-cert]
+
+include::shared-env.asciidoc[tag=elastic-agent-cert-key]
+
 include::shared-env.asciidoc[tag=elastic-agent-tag]
 
 include::shared-env.asciidoc[tag=fleet-enroll]
 
+include::shared-env.asciidoc[tag=fleet-force]
+
 include::shared-env.asciidoc[tag=fleet-url]
 
 include::shared-env.asciidoc[tag=fleet-enrollment-token]

docs/en/ingest-management/elastic-agent/configuration/env/shared-env.asciidoc

@@ -17,30 +17,37 @@ OPTIONAL INFO AND EXAMPLE
 
 // =============================================================================
 
-// tag::elastic-agent-tag[]
+// tag::elastic-agent-cert[]
 |
-[id="env-{type}-elastic-agent-tag"]
-`ELASTIC_AGENT_TAGS`
+[id="env-{type}-elastic-agent-cert"]
+`ELASTIC_AGENT_CERT`
 
-| (string) A comma-separated list of tags to apply to {fleet}-managed {agent}s.
-You can use these tags to filter the list of agents in {fleet}.
+| (string) The path to the mutual TLS client certificate that {agent} will use to connect to {fleet-server}.
 
-// end::elastic-agent-tag[]
+// end::elastic-agent-cert[]
 
 // =============================================================================
 
-// tag::kibana-fleet-setup[]
+// tag::elastic-agent-cert-key[]
 |
-[id="env-{type}-kibana-fleet-setup"]
-`KIBANA_FLEET_SETUP`
+[id="env-{type}-elastic-agent-cert-key"]
+`ELASTIC_AGENT_CERT_KEY`
 
-| (int) Set to `1` to enable {fleet} setup.
-Enabling {fleet} is required before {fleet-server} will start.
-When this action is not performed, a user must manually log in to {kib} and visit the {fleet} page to enable setup.
+| (string) The path to the mutual TLS private key that {agent} will use to connect to {fleet-server}.
 
-*Default:* none
+// end::elastic-agent-cert-key[]
+
+// =============================================================================
 
-// end::kibana-fleet-setup[]
+// tag::elastic-agent-tag[]
+|
+[id="env-{type}-elastic-agent-tag"]
+`ELASTIC_AGENT_TAGS`
+
+| (string) A comma-separated list of tags to apply to {fleet}-managed {agent}s.
+You can use these tags to filter the list of agents in {fleet}.
+
+// end::elastic-agent-tag[]
 
 // =============================================================================
 
@@ -105,6 +112,20 @@ contains your CA's certificate.
 
 // =============================================================================
 
+// tag::fleet-force[]
+|
+[id="env-{type}-fleet-force"]
+`FLEET_FORCE`
+
+| (bool) Set to `true` to force overwrite of the current {agent} configuration without prompting for confirmation.
+This flag is helpful when using automation software or scripted deployments.
+
+*Default:* `false`
+
+// end::fleet-force[]
+
+// =============================================================================
+
 // tag::fleet-server-enable[]
 |
 [id="env-{type}-fleet-server-enable"]
@@ -273,6 +294,21 @@ Overrides the port defined in the policy.
 
 // =============================================================================
 
+// tag::fleet-server-client-auth[]
+|
+[id="env-{type}-fleet-server-client-auth"]
+`FLEET_SERVER_CLIENT_AUTH`
+
+| (string) One of `none`, `optional`, or `required`.
+{fleet-server}'s client authentication option for client mTLS connections.
+If `optional` or `required` is specified, client certificates are verified using CAs.
+
+*Default:* `none`
+
+// end::fleet-server-client-auth[]
+
+// =============================================================================
+
 // tag::fleet-server-es-ca-trusted-fingerprint[]
 |
 [id="env-{type}-fleet-server-es-ca-trusted-fingerprint"]
@@ -288,6 +324,68 @@ by {agent} for communication. This flag is required when using self-signed certi
 
 // =============================================================================
 
+// tag::fleet-server-es-cert[]
+|
+[id="env-{type}-fleet-server-es-cert"]
+`FLEET_SERVER_ES_CERT`
+
+| (string) The path to the mutual TLS client certificate that {fleet-server} will use to connect to {es}.
+
+*Default:* `""`
+
+// end::fleet-server-es-cert[]
+
+// =============================================================================
+
+// tag::fleet-server-es-cert-key[]
+|
+[id="env-{type}-fleet-server-es-cert-key"]
+`FLEET_SERVER_ES_CERT_KEY`
+
+| (string) The path to the mutual TLS private key that {fleet-server} will use to connect to {es}.
+
+*Default:* `""`
+
+// end::fleet-server-es-cert-key[]
+
+// =============================================================================
+
+// tag::fleet-server-insecure-http[]
+|
+[id="env-{type}-fleet-server-insecure-http"]
+`FLEET_SERVER_INSECURE_HTTP`
+
+| (bool) When `true`, {fleet-server} is exposed over insecure or unverified HTTP.
+Setting this to `true` is not recommended.
+
+*Default:* `false`
+
+// end::fleet-server-insecure-http[]
+
+// =============================================================================
+
+// tag::fleet-daemon-timeout[]
+|
+[id="env-{type}-fleet-daemon-timeout"]
+`FLEET_DAEMON_TIMEOUT`
+
+| (duration) Set to indicate how long {fleet-server} will wait during the bootstrap process for {elastic-agent}.
+
+// end::fleet-daemon-timeout[]
+
+// =============================================================================
+
+// tag::fleet-server-timeout[]
+|
+[id="env-{type}-fleet-server-timeout"]
+`FLEET_SERVER_TIMEOUT`
+
+| (duration) Set to indicate how long {agent} will wait for {fleet-server} to check in as healthy.
+
+// end::fleet-server-timeout[]
+
+// =============================================================================
+
 // tag::fleet-enroll[]
 |
 [id="env-{type}-fleet-enroll"]

docs/en/ingest-management/elastic-agent/configuration/outputs/output-kafka.asciidoc

@@ -46,11 +46,13 @@ outputs:
 
 == Kafka output and using {ls} to index data to {es}
 
-If you are considering using {ls} to ship the data from `kafka` to {es}, please
-be aware Elastic is not currently testing this kind of setup.
+If you are considering using {ls} to ship the data from `kafka` to {es}, please be aware the 
+structure of the documents sent from {agent} to `kafka` must not be modified by {ls}.
+We suggest disabling `ecs_compatibility` on both the `kafka` input and the `json` codec in order
+to make sure the input doesn't edit the fields and their contents.
 
-The structure of the documents sent from {agent} to `kafka` must not be modified by {ls}.
-We suggest disabling `ecs_compatibility` on both the `kafka` input and the `json` codec.
+The data streams set up by the integrations expect to receive events having the same structure and
+field names as they were sent directly from an {agent}.
 
 Refer to <<logstash-output,{ls} output for {agent}>> documentation for more details.
 

docs/en/ingest-management/elastic-agent/uninstall-elastic-agent.asciidoc

@@ -7,7 +7,18 @@
 To uninstall {agent}, run the `uninstall` command from the directory where
 {agent} is running.
 
-IMPORTANT: Be sure to run the `uninstall` command from the directory where {agent} is running, as shown in the example below, and not from the directory where you previously ran the `install` command. Running the command from the wrong directory can leave the agent in an inconsistent state.
+[IMPORTANT]
+====
+Be sure to run the `uninstall` command from a directory outside of where {agent} is installed.
+
+For example, on a Windows system the install location is `C:\Program Files\Elastic\Agent`. Run the uninstall command from `C:\Program Files\Elastic` or `\tmp`, or even your default home directory:
+
+[source,shell]
+----
+C:\"Program Files"\Elastic\Agent\elastic-agent.exe uninstall
+----
+
+====
 
 --
 include::{ingest-docs-root}/docs/en/ingest-management/tab-widgets/uninstall-widget.asciidoc[]

docs/en/ingest-management/fleet/fleet-settings-output-kafka.asciidoc

@@ -8,7 +8,12 @@ Specify these settings to send data over a secure connection to Kafka. In the {f
 == Kafka output and using {ls} to index data to {es}
 
 If you are considering using {ls} to ship the data from `kafka` to {es}, please
-be aware Elastic is not currently testing this kind of setup.
+be aware the structure of the documents sent from {agent} to `kafka` must not be modified by {ls}.
+We suggest disabling `ecs_compatibility` on both the `kafka` input and the `json` codec in order
+to make sure the input doesn't edit the fields and their contents.
+
+The data streams setup by the integrations expect to receive events having the same structure and
+field names as they were sent directly from an {agent}.
 
 The structure of the documents sent from {agent} to `kafka` must not be modified by {ls}.
 We suggest disabling `ecs_compatibility` on both the `kafka` input and the `json` codec.
@@ -214,7 +219,18 @@ Use this option to set the Kafka topic for each {agent} event.
 [id="kafka-output-topics-default"]
 **Default topic**
 
-| Set a default topic to use for events sent by {agent} to the Kafka output, for example `elastic-agent`.
+| Set a default topic to use for events sent by {agent} to the Kafka output.
+
+You can set a static topic, for example `elastic-agent`, or you can choose to set a topic dynamically based on an {ecs-ref}/ecs-reference.html[Elastic Common Scheme (ECS)] field. Available fields include:
+
+* `data_stream_type`
+* `data_stream.dataset`
+* `data_stream.namespace`
+* `@timestamp`
+* `event-dataset`
+
+You can also set a custom field. This is useful if you're using the <<add_fields-processor,`add_fields` processor>> as part of your {agent} input.
+Otherwise, setting a custom field is not recommended.
 
 |===
 

docs/en/ingest-management/fleet/fleet-settings.asciidoc

@@ -132,3 +132,16 @@ in the agent policy.
 
 You can specify a proxy server to be used in {fleet-server}, {agent} outputs, or for any agent binary download sources. 
 For full details about proxy configuration refer to <<fleet-agent-proxy-support,Using a proxy server with {agent} and {fleet}>>.
+
+[discrete]
+[[delete-unenrolled-agents-setting]]
+== Delete unenrolled agents
+
+After an {agent} has been unenrolled in {fleet}, a number of documents about the agent are retained just in case the agent needs to be recovered at some point. You can choose to have all data related to an unenrolled agent deleted automatically.
+
+Note that this option can also be enabled by adding the `xpack.fleet.enableDeleteUnenrolledAgents: true` setting to the {kibana-ref}/[{kib} settings file].
+
+To enable automatic deletion of unenrolled agents:
+
+. Go to **{fleet} -> Settings**.
+. Under **Advanced Settings**, enable the **Delete unenrolled agents** option.