Deploy + manage: self-managed ES

deploy-manage/cloud-organization/billing/manage-subscription.md

@@ -41,9 +41,10 @@ To change your subscription level:
 
 ### Feature usage notifications [ec_feature_usage_notifications]
 
-:::{applies_to}
-:hosted: all
-:::
+```{applies_to}
+deployment:
+  ess: all
+```
 
 If you try to change your subscription to a lower level, but you are using features that belong either to your current level or to a higher one, you need to make some changes before you can proceed, as described in **Review required feature changes**.
 

deploy-manage/deploy/_snippets/installation-order.md

@@ -0,0 +1,10 @@
+If you're deploying the {{stack}} in a self-managed cluster, then install the {{stack}} products you want to use in the following order:
+
+* {{es}}
+* {{kib}}
+* [Logstash](logstash://reference/index.md)
+* [{{agent}}](/reference/ingestion-tools/fleet/index.md) or [Beats](asciidocalypse://docs/beats/docs/reference/index.md)
+* [APM](/solutions/observability/apps/application-performance-monitoring-apm.md)
+* [Elasticsearch Hadoop](elasticsearch-hadoop://reference/index.md)
+
+Installing in this order ensures that the components each product depends on are in place.

deploy-manage/deploy/_snippets/stack-version-compatibility.md

@@ -0,0 +1,3 @@
+When installing the {{stack}}, you must use the same version across the entire stack. For example, if you are using {{es}} {{stack-version}}, you install Beats {{stack-version}}, APM Server {{stack-version}}, Elasticsearch Hadoop {{stack-version}}, {{kib}} {{stack-version}}, and Logstash {{stack-version}}.
+
+If you’re upgrading an existing installation, see [](/deploy-manage/upgrade.md) for information about how to ensure compatibility with {{stack-version}}.

deploy-manage/deploy/cloud-enterprise/configure-host-ubuntu.md

@@ -238,7 +238,7 @@ You can specify `--log-opt max-size` and `--log-opt max-file` to define the Dock
 
 1. Update `/etc/systemd/system/docker.service.d/docker.conf`. If the file path and file do not exist, create them first.
 
-    ```sh
+    ```ini
     [Unit]
     Description=Docker Service
     After=multi-user.target

deploy-manage/deploy/cloud-on-k8s/create-custom-images.md

@@ -50,7 +50,7 @@ If your custom images follow the naming convention adopted by the official image
 
 For more information, check the following references:
 
-* [Elasticsearch documentation on Using custom Docker images](/deploy-manage/deploy/self-managed/install-elasticsearch-with-docker.md#_c_customized_image)
+* [Elasticsearch documentation on Using custom Docker images](/deploy-manage/deploy/self-managed/install-elasticsearch-docker-configure.md#_c_customized_image)
 * [Google Container Registry](https://cloud.google.com/container-registry/docs/how-to)
 * [Azure Container Registry](https://docs.microsoft.com/en-us/azure/container-registry/)
 * [Amazon Elastic Container Registry](https://docs.aws.amazon.com/AmazonECR/latest/userguide/what-is-ecr.html)

deploy-manage/deploy/deployment-comparison.md

@@ -1,5 +1,5 @@
 
-# Compare deployment options
+# Detailed deployment comparison
 
 This reference provides detailed comparisons of features and capabilities across Elastic's deployment options: self-managed deployments, Elastic Cloud Hosted, and Serverless. For a high-level overview of deployment types and guidance on choosing between them, see the [overview](../deploy.md).
 

deploy-manage/deploy/elastic-cloud/restrictions-known-problems.md

@@ -55,7 +55,7 @@ Elasticsearch APIs
 $$$ec-restrictions-apis-kibana$$$
 
 Kibana APIs
-:   There are no rate limits restricting your use of the Kibana APIs. However, Kibana features are affected by the [Kibana configuration settings](../self-managed/configure.md), not all of which are supported in {{ecloud}}. For a list of what settings are currently supported, check [Add Kibana user settings](edit-stack-settings.md). For all details about using the Kibana APIs, check the [Kibana API reference documentation](https://www.elastic.co/guide/en/kibana/current/api.html).
+:   There are no rate limits restricting your use of the Kibana APIs. However, Kibana features are affected by the [Kibana configuration settings](kibana://reference/configuration-reference.md), not all of which are supported in {{ecloud}}. For a list of what settings are currently supported, check [Add Kibana user settings](edit-stack-settings.md). For all details about using the Kibana APIs, check the [Kibana API reference documentation](https://www.elastic.co/docs/api/doc/kibana/).
 
 
 ## Transport client [ec-restrictions-transport-client]

deploy-manage/deploy/self-managed.md

@@ -1,8 +1,113 @@
 ---
 mapped_pages:
   - https://www.elastic.co/guide/en/elasticsearch/reference/current/dependencies-versions.html
+sub:
+  stack-version: "9.0.0"
+applies_to:
+  deployment:
+    self:
 ---
 
-# Self-managed cluster [dependencies-versions]
+# Self-managed cluster 
 
-See [Elastic Stack Third-party Dependencices](https://artifacts.elastic.co/reports/dependencies/dependencies-current.html) for the complete list of dependencies for {{es}}.
+If you want to install Elastic on your own premises without the assistance of an [orchestrator](/deploy-manage/deploy.md#about-orchestration), then you can deploy a self-managed cluster. If you deploy a self-managed cluster, then you have complete control and responsibility over every aspect of your Elastic deployment.
+
+To quickly set up {{es}} and {{kib}} in Docker for local development or testing, jump to [](/deploy-manage/deploy/self-managed/local-development-installation-quickstart.md).
+
+:::{admonition} Simplify the deployment process
+Self-managed clusters are useful for local development, and for exploring Elastic features. However, Elastic offer several deployment options that can simplify the process of deploying and managing multi-node deployments, especially in production. They also allow you to deploy and manage multiple deployments from a single surface.
+
+Managed by Elastic: 
+* [{{serverless-full}}](/deploy-manage/deploy/elastic-cloud/serverless.md)
+* [{{ech}}](/deploy-manage/deploy/elastic-cloud/cloud-hosted.md)
+
+Self-hosted options:
+* [{{eck}}](/deploy-manage/deploy/cloud-on-k8s.md)
+* [{{ece}}](/deploy-manage/deploy/cloud-enterprise.md)
+
+For a comparison of these deployment options, refer to [Choosing your deployment type](/deploy-manage/deploy.md#choosing-your-deployment-type) and [](/deploy-manage/deploy/deployment-comparison.md).
+:::
+
+## Section overview
+
+This section focuses on deploying {{es}} and {{kib}} without an orchestrator.
+
+Depending on your use case, you might need to deploy other components, such as APM, Fleet, or Logstash. Deploying those components is not covered in this section. [Learn more about optional components](/get-started/the-stack.md).
+
+This section covers the following tasks: 
+
+### Deploying Elasticsearch
+
+Learn how to install and configure {{es}}. {{es}} is the distributed search and analytics engine, scalable data store, and vector database at the heart of all Elastic solutions.
+
+* [](/deploy-manage/deploy/self-managed/installing-elasticsearch.md)
+  * [](/deploy-manage/deploy/self-managed/important-system-configuration.md): Prepare your environment for an {{es}} installation.
+  * [](/deploy-manage/deploy/self-managed/installing-elasticsearch.md#installation-methods): Install and run {{es}} using one of our install packages or container images.
+  * [](/deploy-manage/deploy/self-managed/local-development-installation-quickstart.md): Quickly set up {{es}} and {{kib}} in Docker for local development or testing.
+* [](/deploy-manage/deploy/self-managed/configure-elasticsearch.md): Learn how to make configuration changes to {{es}}
+  * [](/deploy-manage/deploy/self-managed/important-settings-configuration.md): Learn about key settings required for production environments.
+  * [](/deploy-manage/deploy/self-managed/plugins.md): Learn about how to extend {{es}} functionality with plugins.
+
+    :::{note}
+    For a complete list of settings that you can apply to your {{es}} cluster, refer to the [Elasticsearch configuration reference](elasticsearch://reference/elasticsearch/configuration-reference/index.md).
+    :::
+
+### Deploying Kibana
+
+After you deploy {{es}}, you can install {{kib}}. {{kib}} provides the user interface for all Elastic solutions. It’s a powerful tool for visualizing and analyzing your data, and for managing and monitoring the {{stack}}. Although {{kib}} is not required to use {{es}}, it's required for most use cases.
+
+* [](/deploy-manage/deploy/self-managed/install-kibana.md): Install {{kib}} using one of our install packages or container images, and enroll {{kib}} with your {{es}} cluster.
+* [](/deploy-manage/deploy/self-managed/configure-kibana.md): Learn how to make configuration changes to {{kib}}.
+* [](/deploy-manage/deploy/self-managed/access-kibana.md): Learn how to access {{kib}} using a web browser.
+
+### Installing in air gapped environments
+
+Some components of the {{stack}} require additional configuration and local dependencies in order to deploy in environments without internet access. 
+
+Refer to [](/deploy-manage/deploy/self-managed/air-gapped-install.md) to learn how to install {{es}}, {{kib}}, and optional components in an environment without internet access.
+
+### Tools and APIs
+
+Review a list of all of the resources that you can use to interact with your self-managed cluster, including tools, APIs, client libraries, and more.
+
+[](/deploy-manage/deploy/self-managed/tools-apis.md).
+
+## Other important sections
+
+Review these other sections for critical information about securing and managing your self-managed cluster.
+
+### Secure and control access
+
+Learn how to secure your Elastic environment to restrict access to only authorized parties, and allow communication between your environment and external parties.
+
+* [](/deploy-manage/security.md): Learn about security features that prevent bad actors from tampering with your data, and encrypt communications to, from, and within your cluster.
+* [](/deploy-manage/users-roles/cluster-or-deployment-auth.md): Set up authentication and authorization for your cluster, and learn about the underlying security technologies that {{es}} uses to authenticate and authorize requests internally and across services.
+* [](/deploy-manage/manage-spaces.md): Learn how to organize content in {{kib}}, and restrict access to this content to specific users.
+* [](/deploy-manage/api-keys.md): Authenticate and authorize programmatic access to your deployments and {{es}} resources.
+* [](/deploy-manage/manage-connectors.md): Manage connection information between Elastic and third-party systems.
+* [](/deploy-manage/remote-clusters/remote-clusters-self-managed.md): Enable communication between {{es}} clusters to support [cross-cluster replication](/deploy-manage/tools/cross-cluster-replication.md) and [cross-cluster search](/solutions/search/cross-cluster-search.md).
+
+### Administer and maintain
+
+Monitor the performance of your Elastic environment, administer your license, set up backup and resilience tools, and maintain the health of your environment.
+
+* [](/deploy-manage/tools.md): Learn about the tools available to safeguard data, ensure continuous availability, and maintain resilience in your {{es}} environment.
+* [](/deploy-manage/monitor.md): View health and performance data for Elastic components, and receive recommendations and insights.
+* [](/deploy-manage/license.md): Learn how to manage your Elastic license.
+* [](/deploy-manage/maintenance/start-stop-services.md): Learn how to isolate or deactivate parts of your Elastic environment to perform maintenance, or restart parts of Elastic.
+* [](/deploy-manage/maintenance/add-and-remove-elasticsearch-nodes.md): Learn how to add nodes to a cluster or remove them from a cluster to change the size and capacity of {{es}}.
+
+### Upgrade
+
+You can [upgrade your Elastic environment](/deploy-manage/upgrade.md) to gain access to the latest features.
+
+### Design guidance
+
+Learn how to design a production-ready Elastic environment.
+
+* [](/deploy-manage/production-guidance.md): Review tips and guidance that you can use to design a production environment that matches your workloads, policies, and deployment needs.
+* [](/deploy-manage/reference-architectures.md): Explore blueprints for deploying clusters tailored to different use cases.
+
+### Architectural information
+
+In the [](/deploy-manage/distributed-architecture.md) section, learn about the architecture of {{es}} and {{kib}}, and how Elastic stores and retrieves data and executes tasks in clusters with multiple nodes.

deploy-manage/deploy/self-managed/_snippets/auto-security-config.md

@@ -0,0 +1,12 @@
+When you start {{es}} for the first time, the following security configuration occurs automatically:
+
+* [Certificates and keys](/deploy-manage/security/security-certificates-keys.md#stack-security-certificates) for TLS are generated for the transport and HTTP layers.
+* The TLS configuration settings are written to `elasticsearch.yml`.
+* A password is generated for the `elastic` user.
+* An enrollment token is generated for {{kib}}, which is valid for 30 minutes.
+
+You can then start {{kib}} and enter the enrollment token. This token automatically applies the security settings from your {{es}} cluster, authenticates to {{es}} with the built-in `kibana` service account, and writes the security configuration to `kibana.yml`.
+
+::::{note}
+There are [some cases](/deploy-manage/security/security-certificates-keys.md#stack-skip-auto-configuration) where security can’t be configured automatically because the node startup process detects that the node is already part of a cluster, or that security is already configured or explicitly disabled.
+::::

deploy-manage/deploy/self-managed/_snippets/ca-cert.md

@@ -0,0 +1,7 @@
+If your library doesn’t support a method of validating the fingerprint, the auto-generated CA certificate is created in the following directory on each {{es}} node:
+
+```sh
+{{es-conf}}{{slash}}certs{{slash}}http_ca.crt
+```
+
+Copy the `http_ca.crt` file to your machine and configure your client to use this certificate to establish trust when it connects to {{es}}.

deploy-manage/deploy/self-managed/_snippets/ca-fingerprint.md

@@ -0,0 +1,14 @@
+Copy the fingerprint value that’s output to your terminal when {{es}} starts, and configure your client to use this fingerprint to establish trust when it connects to {{es}}.
+
+If the auto-configuration process already completed, you can still obtain the fingerprint of the security certificate by running the following command. The path is to the auto-generated CA certificate for the HTTP layer.
+
+```sh
+openssl x509 -fingerprint -sha256 -in config/certs/http_ca.crt
+```
+
+The command returns the security certificate, including the fingerprint. The `issuer` should be `{{es}} security auto-configuration HTTP CA`.
+
+```sh
+issuer= /CN={{es}} security auto-configuration HTTP CA
+SHA256 Fingerprint=<fingerprint>
+```

deploy-manage/deploy/self-managed/_snippets/check-es-running.md

@@ -0,0 +1,32 @@
+You can test that your {{es}} node is running by sending an HTTPS request to port `9200` on `localhost`:
+
+```sh
+curl --cacert {{es-conf}}{{slash}}certs{{slash}}http_ca.crt {{escape}} <1>
+-u elastic:$ELASTIC_PASSWORD https://localhost:9200 <2>
+```
+1. `--cacert`: Path to the generated `http_ca.crt` certificate for the HTTP layer.
+2. Ensure that you use `https` in your call, or the request will fail.
+
+
+
+The call returns a response like this:
+
+```js
+{
+  "name" : "Cp8oag6",
+  "cluster_name" : "elasticsearch",
+  "cluster_uuid" : "AT69_T_DTp-1qgIJlatQqA",
+  "version" : {
+    "number" : "9.0.0-SNAPSHOT",
+    "build_type" : "tar",
+    "build_hash" : "f27399d",
+    "build_flavor" : "default",
+    "build_date" : "2016-03-30T09:51:41.449Z",
+    "build_snapshot" : false,
+    "lucene_version" : "10.0.0",
+    "minimum_wire_compatibility_version" : "1.2.3",
+    "minimum_index_compatibility_version" : "1.2.3"
+  },
+  "tagline" : "You Know, for Search"
+}
+```

deploy-manage/deploy/self-managed/_snippets/cmd-line-config.md

@@ -0,0 +1,21 @@
+{{es}} loads its configuration from the following location by default:
+
+```
+{{es-conf}}{{slash}}elasticsearch.yml
+```
+
+The format of this config file is explained in [](/deploy-manage/deploy/self-managed/configure-elasticsearch.md).
+
+Any settings that can be specified in the config file can also be specified on the command line, using the `-E` syntax as follows:
+
+```sh
+.\bin\elasticsearch.bat -Ecluster.name=my_cluster -Enode.name=node_1
+```
+
+:::{note}
+Values that contain spaces must be surrounded with quotes. For instance `-Epath.logs="C:\My Logs\logs"`.
+:::
+
+:::{tip}
+Typically, any cluster-wide settings (like `cluster.name`) should be added to the `elasticsearch.yml` config file, while any node-specific settings such as `node.name` could be specified on the command line.
+::::

deploy-manage/deploy/self-managed/_snippets/connect-clients.md

@@ -0,0 +1,12 @@
+% This file is reused in each of the installation pages. Ensure that any changes
+% you make to this file are applicable across all installation environments.
+
+When you start {{es}} for the first time, TLS is configured automatically for the HTTP layer. A CA certificate is generated and stored on disk at:
+
+```sh
+{{es-conf}}{{slash}}certs{{slash}}http_ca.crt
+```
+
+The hex-encoded SHA-256 fingerprint of this certificate is also output to the terminal. Any clients that connect to {{es}}, such as the [{{es}} Clients](https://www.elastic.co/guide/en/elasticsearch/client/index.html), {{beats}}, standalone {{agent}}s, and {{ls}} must validate that they trust the certificate that {{es}} uses for HTTPS. {{fleet-server}} and {{fleet}}-managed {{agent}}s are automatically configured to trust the CA certificate. Other clients can establish trust by using either the fingerprint of the CA certificate or the CA certificate itself.
+
+If the auto-configuration process already completed, you can still obtain the fingerprint of the security certificate. You can also copy the CA certificate to your machine and configure your client to use it.

deploy-manage/deploy/self-managed/_snippets/dedicated-hosts.md

@@ -0,0 +1 @@
+In production, we recommend you run {{es}} on a dedicated host or as a primary service. Several {{es}} features, such as automatic JVM heap sizing, assume that {{es}} is the only resource-intensive application on the host or container. For example, you might run {{metricbeat}} alongside {{es}} for cluster statistics, but a resource-heavy {{ls}} deployment should be on its own host.

deploy-manage/deploy/self-managed/_snippets/enable-auto-indices.md

@@ -0,0 +1,9 @@
+Some features automatically create indices within {{es}}. By default, {{es}} is configured to allow automatic index creation, and no additional steps are required. However, if you have disabled automatic index creation in {{es}}, you must configure [`action.auto_create_index`](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-create) in `elasticsearch.yml` to allow features to create the following indices:
+
+```yaml
+action.auto_create_index: .monitoring*,.watches,.triggered_watches,.watcher-history*,.ml*
+```
+
+::::{important}
+If you are using [Logstash](https://www.elastic.co/products/logstash) or [Beats](https://www.elastic.co/products/beats) then you will most likely require additional index names in your `action.auto_create_index` setting, and the exact value will depend on your local configuration. If you are unsure of the correct value for your environment, you may consider setting the value to `*` which will allow automatic creation of all indices.
+::::