Deploy + manage: self-managed ES

deploy-manage/deploy/self-managed.md

@@ -1,8 +1,20 @@
 ---
 mapped_pages:
   - https://www.elastic.co/guide/en/elasticsearch/reference/current/dependencies-versions.html
+sub:
+  stack-version: "9.0.0"
 ---
 
 # Self-managed cluster [dependencies-versions]
 
-See [Elastic Stack Third-party Dependencices](https://artifacts.elastic.co/reports/dependencies/dependencies-current.md) for the complete list of dependencies for {{es}}.
+See [Elastic Stack Third-party Dependencices](https://artifacts.elastic.co/reports/dependencies/dependencies-current.md) for the complete list of dependencies for {{es}}.
+
+
+```sh
+{{stack-version}}
+```
+
+{{stack-version}}
+
+1. Compares the SHA of the downloaded `.tar.gz` archive and the published checksum, which should output `elasticsearch-<version>-linux-x86_64.tar.gz: OK`.
+2. This directory is known as `$ES_HOME`.

deploy-manage/deploy/self-managed/_snippets/auto-security-config.md

@@ -0,0 +1,12 @@
+When you start {{es}} for the first time, the following security configuration occurs automatically:
+
+* [Certificates and keys](../../../deploy-manage/security/security-certificates-keys.md#stack-security-certificates) for TLS are generated for the transport and HTTP layers.
+* The TLS configuration settings are written to `elasticsearch.yml`.
+* A password is generated for the `elastic` user.
+* An enrollment token is generated for {{kib}}, which is valid for 30 minutes.
+
+You can then start {{kib}} and enter the enrollment token. This token automatically applies the security settings from your {{es}} cluster, authenticates to {{es}} with the built-in `kibana` service account, and writes the security configuration to `kibana.yml`.
+
+::::{note}
+There are [some cases](../../../deploy-manage/security/security-certificates-keys.md#stack-skip-auto-configuration) where security can’t be configured automatically because the node startup process detects that the node is already part of a cluster, or that security is already configured or explicitly disabled.
+::::

deploy-manage/deploy/self-managed/_snippets/ca-cert.md

@@ -0,0 +1,7 @@
+If your library doesn’t support a method of validating the fingerprint, the auto-generated CA certificate is created in the following directory on each {{es}} node:
+
+```sh
+{{es-conf}}{{slash}}certs{{slash}}http_ca.crt
+```
+
+Copy the `http_ca.crt` file to your machine and configure your client to use this certificate to establish trust when it connects to {{es}}.

deploy-manage/deploy/self-managed/_snippets/ca-fingerprint.md

@@ -0,0 +1,14 @@
+Copy the fingerprint value that’s output to your terminal when {{es}} starts, and configure your client to use this fingerprint to establish trust when it connects to {{es}}.
+
+If the auto-configuration process already completed, you can still obtain the fingerprint of the security certificate by running the following command. The path is to the auto-generated CA certificate for the HTTP layer.
+
+```sh
+openssl x509 -fingerprint -sha256 -in config/certs/http_ca.crt
+```
+
+The command returns the security certificate, including the fingerprint. The `issuer` should be `{{es}} security auto-configuration HTTP CA`.
+
+```sh
+issuer= /CN={{es}} security auto-configuration HTTP CA
+SHA256 Fingerprint=<fingerprint>
+```

deploy-manage/deploy/self-managed/_snippets/check-es-running.md

@@ -0,0 +1,32 @@
+You can test that your {{es}} node is running by sending an HTTPS request to port `9200` on `localhost`:
+
+```sh
+curl --cacert {{es-conf}}{{slash}}certs{{slash}}http_ca.crt {{escape}} <1>
+-u elastic:$ELASTIC_PASSWORD https://localhost:9200 <2>
+```
+1. `--cacert`: Path to the generated `http_ca.crt` certificate for the HTTP layer.
+2. Ensure that you use `https` in your call, or the request will fail.
+
+
+
+The call returns a response like this:
+
+```js
+{
+  "name" : "Cp8oag6",
+  "cluster_name" : "elasticsearch",
+  "cluster_uuid" : "AT69_T_DTp-1qgIJlatQqA",
+  "version" : {
+    "number" : "9.0.0-SNAPSHOT",
+    "build_type" : "tar",
+    "build_hash" : "f27399d",
+    "build_flavor" : "default",
+    "build_date" : "2016-03-30T09:51:41.449Z",
+    "build_snapshot" : false,
+    "lucene_version" : "10.0.0",
+    "minimum_wire_compatibility_version" : "1.2.3",
+    "minimum_index_compatibility_version" : "1.2.3"
+  },
+  "tagline" : "You Know, for Search"
+}
+```

deploy-manage/deploy/self-managed/_snippets/cmd-line-config.md

@@ -0,0 +1,21 @@
+{{es}} loads its configuration from the following location by default:
+
+```
+{{es-conf}}{{slash}}elasticsearch.yml
+```
+
+The format of this config file is explained in [](/deploy-manage/deploy/self-managed/configure-elasticsearch.md).
+
+Any settings that can be specified in the config file can also be specified on the command line, using the `-E` syntax as follows:
+
+```sh
+.\bin\elasticsearch.bat -Ecluster.name=my_cluster -Enode.name=node_1
+```
+
+:::{note}
+Values that contain spaces must be surrounded with quotes. For instance `-Epath.logs="C:\My Logs\logs"`.
+:::
+
+:::{tip}
+Typically, any cluster-wide settings (like `cluster.name`) should be added to the `elasticsearch.yml` config file, while any node-specific settings such as `node.name` could be specified on the command line.
+::::

deploy-manage/deploy/self-managed/_snippets/connect-clients.md

@@ -0,0 +1,12 @@
+% This file is reused in each of the installation pages. Ensure that any changes
+% you make to this file are applicable across all installation environments.
+
+When you start {{es}} for the first time, TLS is configured automatically for the HTTP layer. A CA certificate is generated and stored on disk at:
+
+```sh
+{{es-conf}}{{slash}}certs{{slash}}http_ca.crt
+```
+
+The hex-encoded SHA-256 fingerprint of this certificate is also output to the terminal. Any clients that connect to {{es}}, such as the [{{es}} Clients](https://www.elastic.co/guide/en/elasticsearch/client/index.html), {{beats}}, standalone {{agent}}s, and {{ls}} must validate that they trust the certificate that {{es}} uses for HTTPS. {{fleet-server}} and {{fleet}}-managed {{agent}}s are automatically configured to trust the CA certificate. Other clients can establish trust by using either the fingerprint of the CA certificate or the CA certificate itself.
+
+If the auto-configuration process already completed, you can still obtain the fingerprint of the security certificate. You can also copy the CA certificate to your machine and configure your client to use it.

deploy-manage/deploy/self-managed/_snippets/enable-auto-indices.md

@@ -0,0 +1,9 @@
+Some features automatically create indices within {{es}}. By default, {{es}} is configured to allow automatic index creation, and no additional steps are required. However, if you have disabled automatic index creation in {{es}}, you must configure [`action.auto_create_index`](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-create) in `elasticsearch.yml` to allow features to create the following indices:
+
+```yaml
+action.auto_create_index: .monitoring*,.watches,.triggered_watches,.watcher-history*,.ml*
+```
+
+::::{important}
+If you are using [Logstash](https://www.elastic.co/products/logstash) or [Beats](https://www.elastic.co/products/beats) then you will most likely require additional index names in your `action.auto_create_index` setting, and the exact value will depend on your local configuration. If you are unsure of the correct value for your environment, you may consider setting the value to `*` which will allow automatic creation of all indices.
+::::

deploy-manage/deploy/self-managed/_snippets/enroll-nodes.md

@@ -0,0 +1,29 @@
+When {{es}} starts for the first time, the security auto-configuration process binds the HTTP layer to `0.0.0.0`, but only binds the transport layer to localhost. This intended behavior ensures that you can start a single-node cluster with security enabled by default without any additional configuration.
+
+Before enrolling a new node, additional actions such as binding to an address other than `localhost` or satisfying bootstrap checks are typically necessary in production clusters. During that time, an auto-generated enrollment token could expire, which is why enrollment tokens aren’t generated automatically.
+
+Additionally, only nodes on the same host can join the cluster without additional configuration. If you want nodes from another host to join your cluster, you need to set `transport.host` to a [supported value](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/configuration-reference/networking-settings.md#network-interface-values) (such as uncommenting the suggested value of `0.0.0.0`), or an IP address that’s bound to an interface where other hosts can reach it. Refer to [transport settings](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/configuration-reference/networking-settings.md#transport-settings) for more information.
+
+To enroll new nodes in your cluster, create an enrollment token with the `elasticsearch-create-enrollment-token` tool on any existing node in your cluster. You can then start a new node with the `--enrollment-token` parameter so that it joins an existing cluster.
+
+1. In a separate terminal from where {{es}} is running, navigate to the directory where you installed {{es}} and run the [`elasticsearch-create-enrollment-token`](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/command-line-tools/create-enrollment-token.md) tool to generate an enrollment token for your new nodes.
+
+    ```sh
+    bin{{slash}}elasticsearch-create-enrollment-token -s node
+    ```
+
+    Copy the enrollment token, which you’ll use to enroll new nodes with your {{es}} cluster.
+
+2. From the installation directory of your new node, start {{es}} and pass the enrollment token with the `--enrollment-token` parameter.
+
+    ```sh
+    bin{{slash}}elasticsearch --enrollment-token <enrollment-token>
+    ```
+
+    {{es}} automatically generates certificates and keys in the following directory:
+
+    ```sh
+    config{{slash}}certs
+    ```
+
+3. Repeat the previous step for any new nodes that you want to enroll.

deploy-manage/deploy/self-managed/_snippets/es-releases.md

@@ -0,0 +1 @@
+The latest stable version of {{es}} can be found on the [Download {{es}}](https://elastic.co/downloads/elasticsearch) page. Other versions can be found on the [Past Releases page](https://elastic.co/downloads/past-releases).

deploy-manage/deploy/self-managed/_snippets/etc-elasticsearch.md

@@ -0,0 +1,18 @@
+The `/etc/elasticsearch` directory contains the default runtime configuration for {{es}}. The ownership of this directory and all contained files are set to `root:elasticsearch` on package installations.
+
+The `setgid` flag applies group permissions on the `/etc/elasticsearch` directory to ensure that {{es}} can read any contained files and subdirectories. All files and subdirectories inherit the `root:elasticsearch` ownership. Running commands from this directory or any subdirectories, such as the [elasticsearch-keystore tool](/deploy-manage/security/secure-settings.md), requires `root:elasticsearch` permissions.
+
+{{es}} loads its configuration from the `/etc/elasticsearch/elasticsearch.yml` file by default. The format of this config file is explained in [](/deploy-manage/deploy/self-managed/configure-elasticsearch.md).
+
+The {{distro}} package also has a system configuration file (`/etc/sysconfig/elasticsearch`), which allows you to set the following parameters:
+
+| Parameter | Description |
+| --- | --- |
+| `ES_JAVA_HOME` | Set a custom Java path to be used. |
+| `ES_PATH_CONF` | Configuration file directory (which needs to include `elasticsearch.yml`, `jvm.options`, and `log4j2.properties` files); defaults to `/etc/elasticsearch`. |
+| `ES_JAVA_OPTS` | Any additional JVM system properties you may want to apply. |
+| `RESTART_ON_UPGRADE` | Configure restart on package upgrade, defaults to `false`. This means you will have to restart your {{es}} instance after installing a package  manually. The reason for this is to ensure, that upgrades in a cluster do not result in a continuous shard reallocation resulting in high network traffic and reducing the response times of your cluster. |
+
+::::{note}
+Distributions that use `systemd` require that system resource limits be configured via `systemd` rather than via the `/etc/sysconfig/elasticsearch` file. See [Systemd configuration](/deploy-manage/deploy/self-managed/setting-system-settings.md#systemd) for more information.
+::::

deploy-manage/deploy/self-managed/_snippets/install-next-steps.md

@@ -0,0 +1,5 @@
+You now have a test {{es}} environment set up. Before you start serious development or go into production with {{es}}, you must do some additional setup:
+
+* Learn how to [configure {{es}}](configure-elasticsearch.md).
+* Configure [important {{es}} settings](important-settings-configuration.md).
+* Configure [important system settings](important-system-configuration.md).

deploy-manage/deploy/self-managed/_snippets/java-version.md

@@ -0,0 +1,3 @@
+::::{note}
+{{es}} includes a bundled version of [OpenJDK](https://openjdk.java.net) from the JDK maintainers (GPLv2+CE). To use your own version of Java, see the [JVM version requirements](installing-elasticsearch.md#jvm-version).
+::::

deploy-manage/deploy/self-managed/_snippets/join-existing-cluster.md

@@ -0,0 +1,18 @@
+When you install {{es}}, the installation process configures a single-node cluster by default. If you want a node to join an existing cluster instead, generate an enrollment token on an existing node *before* you start the new node for the first time.
+
+1. On any node in your existing cluster, generate a node enrollment token:
+
+    ```sh
+    /usr/share/elasticsearch/bin/elasticsearch-create-enrollment-token -s node
+    ```
+
+2. Copy the enrollment token, which is output to your terminal.
+3. On your new {{es}} node, pass the enrollment token as a parameter to the `elasticsearch-reconfigure-node` tool:
+
+    ```sh
+    /usr/share/elasticsearch/bin/elasticsearch-reconfigure-node --enrollment-token <enrollment-token>
+    ```
+
+    {{es}} is now configured to join the existing cluster.
+
+4. [Start your new node using `systemd`](#running-systemd).

deploy-manage/deploy/self-managed/_snippets/pgp-key.md

@@ -0,0 +1,6 @@
+We sign all of our packages with the {{es}} signing key (PGP key [D88E42B4](https://pgp.mit.edu/pks/lookup?op=vindex&search=0xD27D666CD88E42B4), available from [https://pgp.mit.edu](https://pgp.mit.edu)) with fingerprint:
+
+```
+4609 5ACC 8548 582C 1A26 99A9 D27D 666C D88E 42B4
+```
+Download and install the public signing key:

deploy-manage/deploy/self-managed/_snippets/prereqs.md

@@ -0,0 +1,4 @@
+Before you install {{es}}, do the following: 
+
+* Review the [supported operating systems](https://www.elastic.co/support/matrix). {{es}} is tested on the listed platforms, but it is possible that it will work on other platforms too.
+* Configure your operating system using the [](/deploy-manage/deploy/self-managed/important-system-configuration.md) guidelines.

deploy-manage/deploy/self-managed/_snippets/pw-env-var.md

@@ -0,0 +1,11 @@
+The password for the `elastic` user and the enrollment token for {{kib}} are output to your terminal.
+
+We recommend storing the `elastic` password as an environment variable in your shell. For example:
+
+```sh
+{{export}}ELASTIC_PASSWORD="your_password"
+```
+
+If you have password-protected the {{es}} keystore, you will be prompted to enter the keystore’s password. See [Secure settings](../../security/secure-settings.md) for more details.
+
+To learn how to reset this password, refer to [](/deploy-manage/users-roles/cluster-or-deployment-auth/built-in-sm.md).

deploy-manage/deploy/self-managed/_snippets/security-files.md

@@ -0,0 +1,24 @@
+When you install {{es}}, the following certificates and keys are generated in the {{es}} configuration directory. These files are used to connect a {{kib}} instance to your secured {{es}} cluster and to encrypt internode communication. The files are listed here for reference.
+
+`http_ca.crt`
+:   The CA certificate that is used to sign the certificates for the HTTP layer of this {{es}} cluster.
+
+`http.p12`
+:   Keystore that contains the key and certificate for the HTTP layer for this node.
+
+`transport.p12`
+:   Keystore that contains the key and certificate for the transport layer for all the nodes in your cluster.
+
+`http.p12` and `transport.p12` are password-protected PKCS#12 keystores. {{es}} stores the passwords for these keystores as [secure settings](../../security/secure-settings.md). To retrieve the passwords so that you can inspect or change the keystore contents, use the [`bin/elasticsearch-keystore`](asciidocalypse://docs/elasticsearch/docs/reference/elasticsearch/command-line-tools/elasticsearch-keystore.md) tool.
+
+Use the following command to retrieve the password for `http.p12`:
+
+```sh
+bin/elasticsearch-keystore show xpack.security.http.ssl.keystore.secure_password
+```
+
+Use the following command to retrieve the password for `transport.p12`:
+
+```sh
+bin/elasticsearch-keystore show xpack.security.transport.ssl.keystore.secure_password
+```

deploy-manage/deploy/self-managed/_snippets/skip-set-kernel-params.md

@@ -0,0 +1,3 @@
+::::{note}
+On systemd-based distributions, the installation scripts will attempt to set kernel parameters (e.g., `vm.max_map_count`). You can skip this by masking the `systemd-sysctl.service` unit.
+::::

deploy-manage/deploy/self-managed/_snippets/start-local.md

@@ -0,0 +1,46 @@
+::::{warning}
+**DO NOT USE THESE INSTRUCTIONS FOR PRODUCTION DEPLOYMENTS**
+
+The instructions on this page are for **local development only**. Do not use this configuration for production deployments, because it is not secure. Refer to [deployment options](../../get-started/deployment-options.md) for a list of production deployment options.
+
+::::
+
+Quickly set up {{es}} and {{kib}} in Docker for local development or testing, using the [`start-local` script](https://github.com/elastic/start-local?tab=readme-ov-file#-try-elasticsearch-and-kibana-locally).
+
+This setup comes with a one-month trial license that includes all Elastic features. After the trial period, the license reverts to **Free and open - Basic**. Refer to [Elastic subscriptions](https://www.elastic.co/subscriptions) for more information.
+
+## Prerequisites [local-dev-prerequisites]
+
+* If you don’t have Docker installed, [download and install Docker Desktop](https://www.docker.com/products/docker-desktop) for your operating system.
+* If you’re using Microsoft Windows, then install [Windows Subsystem for Linux (WSL)](https://learn.microsoft.com/en-us/windows/wsl/install).
+
+## Run `start-local` script [local-dev-quick-start]
+
+To set up {{es}} and {{kib}} locally, run the `start-local` script:
+
+```sh
+curl -fsSL https://elastic.co/start-local | sh
+```
+
+This script creates an `elastic-start-local` folder containing configuration files and starts both {{es}} and {{kib}} using Docker.
+
+After running the script, you can access Elastic services at the following endpoints:
+
+* **{{es}}**: [http://localhost:9200](http://localhost:9200)
+* **{{kib}}**: [http://localhost:5601](http://localhost:5601)
+
+The script generates a random password for the `elastic` user, and an API key, stored in the `.env` file.
+
+::::{warning}
+This setup is for local testing only. HTTPS is disabled, and Basic authentication is used for {{es}}. For security, {{es}} and {{kib}} are accessible only through `localhost`.
+
+::::
+
+## Learn more [local-dev-additional-info]
+
+For more detailed information about the `start-local` setup, refer to the [README on GitHub](https://github.com/elastic/start-local). Learn about customizing the setup, logging, and more.
+
+
+## Next steps [local-dev-next-steps]
+
+Use our [quick start guides](https://www.elastic.co/guide/en/elasticsearch/reference/current/quickstart.html) to learn the basics of {{es}}.

deploy-manage/deploy/self-managed/_snippets/systemd-journal.md

@@ -0,0 +1,23 @@
+By default, the {{es}} service doesn’t log information in the `systemd` journal. To enable `journalctl` logging, the `--quiet` option must be removed from the `ExecStart` command line in the `elasticsearch.service` file.
+
+When `systemd` logging is enabled, the logging information are available using the `journalctl` commands:
+
+To tail the journal:
+
+```sh
+sudo journalctl -f
+```
+
+To list journal entries for the elasticsearch service:
+
+```sh
+sudo journalctl --unit elasticsearch
+```
+
+To list journal entries for the elasticsearch service starting from a given time:
+
+```sh
+sudo journalctl --unit elasticsearch --since  "2016-10-30 18:17:16"
+```
+
+Check `man journalctl` or [https://www.freedesktop.org/software/systemd/man/journalctl.html](https://www.freedesktop.org/software/systemd/man/journalctl.md) for more command line options.