Skip to content

Commit 474b4aa

Browse files
mashhursmergify[bot]
mashhurs
authored and
mergify[bot]
committed
Improve the key validation in secret identifier. (#17351)
* Improve the key validation in secret identifier. * Restrict adding invalid (from ConfigVariableExpander point of view) key names to the keystore through keystore CLI. Keystore now warns on invalid keys if they were already added. * Key pattern is moved to a central config expander class with its description. Co-authored-by: Rye Biesemeyer <[email protected]> (cherry picked from commit d24675a) # Conflicts: # docs/reference/keystore.md
1 parent 21593be commit 474b4aa

File tree

5 files changed

+259
-55
lines changed

5 files changed

+259
-55
lines changed

‎docs/reference/keystore.md‎

Lines changed: 159 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,159 @@
1+
---
2+
mapped_pages:
3+
- https://www.elastic.co/guide/en/logstash/current/keystore.html
4+
---
5+
6+
# Secrets keystore for secure settings [keystore]
7+
8+
When you configure Logstash, you might need to specify sensitive settings or configuration, such as passwords. Rather than relying on file system permissions to protect these values, you can use the Logstash keystore to securely store secret values for use in configuration settings.
9+
10+
After adding a key and its secret value to the keystore, you can use the key in place of the secret value when you configure sensitive settings.
11+
12+
The syntax for referencing keys is identical to the syntax for [environment variables](/reference/environment-variables.md):
13+
14+
```txt
15+
${KEY}
16+
```
17+
18+
Where KEY is the name of the key.
19+
20+
**Example**
21+
22+
Imagine that the keystore contains a key called `ES_PWD` with the value `yourelasticsearchpassword`.
23+
24+
In configuration files, use:
25+
26+
```shell
27+
output { elasticsearch {...password => "${ES_PWD}" } } }
28+
```
29+
30+
In `logstash.yml`, use:
31+
32+
```shell
33+
xpack.management.elasticsearch.password: ${ES_PWD}
34+
```
35+
36+
Notice that the Logstash keystore differs from the Elasticsearch keystore. Whereas the Elasticsearch keystore lets you store `elasticsearch.yml` values by name, the Logstash keystore lets you specify arbitrary names that you can reference in the Logstash configuration.
37+
38+
::::{note}
39+
There are some configuration fields that have no secret meaning, so not every field could leverage the secret store for variables substitution. Plugin’s `id` field is a field of this kind
40+
::::
41+
42+
43+
::::{note}
44+
Referencing keystore data from `pipelines.yml` or the command line (`-e`) is not currently supported.
45+
::::
46+
47+
48+
::::{note}
49+
Referencing keystore data from [centralized pipeline management](/reference/logstash-centralized-pipeline-management.md) requires each Logstash deployment to have a local copy of the keystore.
50+
::::
51+
52+
53+
::::{note}
54+
The {{ls}} keystore needs to be protected, but the {{ls}} user must have access to the file. While most things in {{ls}} can be protected with `chown -R root:root <foo>`, the keystore itself must be accessible from the {{ls}} user. Use `chown logstash:root <keystore> && chmod 0600 <keystore>`.
55+
::::
56+
57+
58+
When Logstash parses the settings (`logstash.yml`) or configuration (`/etc/logstash/conf.d/*.conf`), it resolves keys from the keystore before resolving environment variables.
59+
60+
61+
## Keystore password [keystore-password]
62+
63+
You can protect access to the Logstash keystore by storing a password in an environment variable called `LOGSTASH_KEYSTORE_PASS`. If you create the Logstash keystore after setting this variable, the keystore will be password protected. This means that the environment variable needs to be accessible to the running instance of Logstash. This environment variable must also be correctly set for any users who need to issue keystore commands (add, list, remove, etc.).
64+
65+
Using a keystore password is recommended, but optional. The data will be encrypted even if you do not set a password. However, it is highly recommended to configure the keystore password and grant restrictive permissions to any files that may contain the environment variable value. If you choose not to set a password, then you can skip the rest of this section.
66+
67+
For example:
68+
69+
```sh
70+
set +o history
71+
export LOGSTASH_KEYSTORE_PASS=mypassword
72+
set -o history
73+
bin/logstash-keystore create
74+
```
75+
76+
This setup requires the user running Logstash to have the environment variable `LOGSTASH_KEYSTORE_PASS=mypassword` defined. If the environment variable is not defined, Logstash cannot access the keystore.
77+
78+
When you run Logstash from an RPM or DEB package installation, the environment variables are sourced from `/etc/sysconfig/logstash`.
79+
80+
::::{note}
81+
You might need to create `/etc/sysconfig/logstash`. This file should be owned by `root` with `600` permissions. The expected format of `/etc/sysconfig/logstash` is `ENVIRONMENT_VARIABLE=VALUE`, with one entry per line.
82+
::::
83+
84+
85+
For other distributions, such as Docker or ZIP, see the documentation for your runtime environment (Windows, Docker, etc) to learn how to set the environment variable for the user that runs Logstash. Ensure that the environment variable (and thus the password) is only accessible to that user.
86+
87+
88+
## Keystore location [keystore-location]
89+
90+
The keystore must be located in the Logstash `path.settings` directory. This is the same directory that contains the `logstash.yml` file. When performing any operation against the keystore, it is recommended to set `path.settings` for the keystore command. For example, to create a keystore on a RPM/DEB installation:
91+
92+
```sh
93+
set +o history
94+
export LOGSTASH_KEYSTORE_PASS=mypassword
95+
set -o history
96+
sudo -E /usr/share/logstash/bin/logstash-keystore --path.settings /etc/logstash create
97+
```
98+
99+
See [Logstash Directory Layout](/reference/dir-layout.md) for more about the default directory locations.
100+
101+
::::{note}
102+
You will see a warning if the `path.settings` is not pointed to the same directory as the `logstash.yml`.
103+
::::
104+
105+
106+
107+
## Create or overwrite a keystore [creating-keystore]
108+
109+
The `create` command creates a new keystore or overwrites an existing keystore:
110+
111+
```sh
112+
bin/logstash-keystore create
113+
```
114+
115+
Creates the keystore in the directory defined in the `path.settings` setting.
116+
117+
::::{important}
118+
If a keystore already exists, the `create` command can overwrite it (after a Y/N prompt). Selecting `Y` clears all keys and secrets that were previously stored.
119+
::::
120+
121+
122+
::::{tip}
123+
Set a [keystore password](#keystore-password) when you create the keystore.
124+
::::
125+
126+
127+
128+
## Add keys [add-keys-to-keystore]
129+
130+
To store sensitive values, such as authentication credentials for Elasticsearch, use the `add` command:
131+
132+
```sh
133+
bin/logstash-keystore add ES_USER ES_PWD
134+
```
135+
136+
When prompted, enter a value for each key.
137+
138+
::::{note}
139+
Key values are limited to ASCII letters (`a`-`z`, `A`-`Z`), numbers (`0`-`9`), underscores (`_`), and dots (`.`); they must be at least one character long and cannot begin with a number.
140+
::::
141+
142+
143+
144+
## List keys [list-settings]
145+
146+
To list the keys defined in the keystore, use:
147+
148+
```sh
149+
bin/logstash-keystore list
150+
```
151+
152+
153+
## Remove keys [remove-settings]
154+
155+
To remove keys from the keystore, use:
156+
157+
```sh
158+
bin/logstash-keystore remove ES_USER ES_PWD
159+
```

‎logstash-core/src/main/java/org/logstash/plugins/ConfigVariableExpander.java‎

Lines changed: 8 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -35,11 +35,14 @@
3535
* */
3636
public class ConfigVariableExpander implements AutoCloseable {
3737

38-
private static String SUBSTITUTION_PLACEHOLDER_REGEX = "\\$\\{(?<name>[a-zA-Z_.][a-zA-Z0-9_.]*)(:(?<default>[^}]*))?}";
39-
40-
private Pattern substitutionPattern = Pattern.compile(SUBSTITUTION_PLACEHOLDER_REGEX);
41-
private SecretStore secretStore;
42-
private EnvironmentVariableProvider envVarProvider;
38+
public static final Pattern KEY_PATTERN = Pattern.compile("[a-zA-Z_.][a-zA-Z0-9_.]*");
39+
public static final String KEY_PATTERN_DESCRIPTION = "Key names are limited to ASCII letters (`a`-`z`, `A`-`Z`), numbers (`0`-`9`), " +
40+
"underscores (`_`), and dots (`.`); they must be at least one character long and cannot begin with a number";
41+
private static final String SUBSTITUTION_PLACEHOLDER_REGEX = "\\$\\{(?<name>" + KEY_PATTERN + ")(:(?<default>[^}]*))?}";
42+
43+
private static final Pattern substitutionPattern = Pattern.compile(SUBSTITUTION_PLACEHOLDER_REGEX);
44+
private final SecretStore secretStore;
45+
private final EnvironmentVariableProvider envVarProvider;
4346

4447
/**
4548
* Creates a ConfigVariableExpander that doesn't lookup any secreted placeholder.

‎logstash-core/src/main/java/org/logstash/secret/SecretIdentifier.java‎

Lines changed: 27 additions & 16 deletions
Original file line numberDiff line numberDiff line change
@@ -20,6 +20,11 @@
2020

2121
package org.logstash.secret;
2222

23+
import org.apache.logging.log4j.Logger;
24+
import org.apache.logging.log4j.LogManager;
25+
import org.apache.logging.log4j.util.Strings;
26+
import org.logstash.plugins.ConfigVariableExpander;
27+
2328
import java.util.Locale;
2429
import java.util.regex.Pattern;
2530

@@ -34,12 +39,14 @@ public class SecretIdentifier {
3439
private final static Pattern urnPattern = Pattern.compile("urn:logstash:secret:"+ VERSION + ":.*$");
3540
private final String key;
3641

42+
private static final Logger logger = LogManager.getLogger(SecretIdentifier.class);
43+
3744
/**
3845
* Constructor
3946
*
4047
* @param key The unique part of the identifier. This is the key to reference the secret, and the key itself should not be sensitive. For example: {@code db.pass}
4148
*/
42-
public SecretIdentifier(String key) {
49+
public SecretIdentifier(final String key) {
4350
this.key = validateWithTransform(key, "key");
4451
}
4552

@@ -49,28 +56,14 @@ public SecretIdentifier(String key) {
4956
* @param urn The {@link String} formatted identifier obtained originally from {@link SecretIdentifier#toExternalForm()}
5057
* @return The {@link SecretIdentifier} object used to identify secrets, null if not valid external form.
5158
*/
52-
public static SecretIdentifier fromExternalForm(String urn) {
59+
public static SecretIdentifier fromExternalForm(final String urn) {
5360
if (urn == null || !urnPattern.matcher(urn).matches()) {
5461
throw new IllegalArgumentException("Invalid external form " + urn);
5562
}
5663
String[] parts = colonPattern.split(urn, 5);
5764
return new SecretIdentifier(validateWithTransform(parts[4], "key"));
5865
}
5966

60-
/**
61-
* Minor validation and downcases the parts
62-
*
63-
* @param part The part of the URN to validate
64-
* @param partName The name of the part used for logging.
65-
* @return The validated and transformed part.
66-
*/
67-
private static String validateWithTransform(String part, String partName) {
68-
if (part == null || part.isEmpty()) {
69-
throw new IllegalArgumentException(String.format("%s may not be null or empty", partName));
70-
}
71-
return part.toLowerCase(Locale.US);
72-
}
73-
7467
@Override
7568
public boolean equals(Object o) {
7669
if (this == o) return true;
@@ -118,4 +111,22 @@ public String toExternalForm() {
118111
public String toString() {
119112
return toExternalForm();
120113
}
114+
115+
/**
116+
* Minor validation and downcases the parts
117+
*
118+
* @param key The key, a part of the URN to validate.
119+
* @param partName The name of the part used for logging.
120+
* @return The validated and transformed part.
121+
*/
122+
private static String validateWithTransform(final String key, final String partName) {
123+
if (key == null || key.isEmpty() || Strings.isBlank(key)) {
124+
throw new IllegalArgumentException(String.format("%s may not be null or empty", partName));
125+
}
126+
127+
if (!ConfigVariableExpander.KEY_PATTERN.matcher(key).matches()) {
128+
logger.warn(String.format("Invalid secret key name `%s` provided. %s", key, ConfigVariableExpander.KEY_PATTERN_DESCRIPTION));
129+
}
130+
return key.toLowerCase(Locale.US);
131+
}
121132
}

‎logstash-core/src/main/java/org/logstash/secret/cli/SecretStoreCli.java‎

Lines changed: 25 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -20,13 +20,23 @@
2020

2121
package org.logstash.secret.cli;
2222

23+
import org.logstash.plugins.ConfigVariableExpander;
2324
import org.logstash.secret.SecretIdentifier;
24-
import org.logstash.secret.store.*;
25+
import org.logstash.secret.store.SecretStore;
26+
import org.logstash.secret.store.SecretStoreFactory;
27+
import org.logstash.secret.store.SecretStoreUtil;
28+
import org.logstash.secret.store.SecureConfig;
2529

2630
import java.nio.CharBuffer;
2731
import java.nio.charset.CharsetEncoder;
2832
import java.nio.charset.StandardCharsets;
29-
import java.util.*;
33+
import java.util.Arrays;
34+
import java.util.Collection;
35+
import java.util.Collections;
36+
import java.util.EnumSet;
37+
import java.util.List;
38+
import java.util.Optional;
39+
import java.util.Set;
3040
import java.util.stream.Collectors;
3141

3242
import static org.logstash.secret.store.SecretStoreFactory.LOGSTASH_MARKER;
@@ -178,7 +188,7 @@ Set<CommandOptions> getValidOptions() {
178188
}
179189
}
180190

181-
public SecretStoreCli(Terminal terminal){
191+
public SecretStoreCli(Terminal terminal) {
182192
this(terminal, SecretStoreFactory.fromEnvironment());
183193
}
184194

@@ -189,9 +199,10 @@ public SecretStoreCli(Terminal terminal){
189199

190200
/**
191201
* Entry point to issue a command line command.
202+
*
192203
* @param primaryCommand The string representation of a {@link SecretStoreCli.Command}, if the String does not map to a {@link SecretStoreCli.Command}, then it will show the help menu.
193-
* @param config The configuration needed to work a secret store. May be null for help.
194-
* @param allArguments This can be either identifiers for a secret, or a sub command like --help. May be null.
204+
* @param config The configuration needed to work a secret store. May be null for help.
205+
* @param allArguments This can be either identifiers for a secret, or a sub command like --help. May be null.
195206
*/
196207
public void command(String primaryCommand, SecureConfig config, String... allArguments) {
197208
terminal.writeLine("");
@@ -212,7 +223,7 @@ public void command(String primaryCommand, SecureConfig config, String... allArg
212223
final CommandLine commandLine = commandParseResult.get();
213224
switch (commandLine.getCommand()) {
214225
case CREATE: {
215-
if (commandLine.hasOption(CommandOptions.HELP)){
226+
if (commandLine.hasOption(CommandOptions.HELP)) {
216227
terminal.writeLine("Creates a new keystore. For example: 'bin/logstash-keystore create'");
217228
return;
218229
}
@@ -227,7 +238,7 @@ public void command(String primaryCommand, SecureConfig config, String... allArg
227238
break;
228239
}
229240
case LIST: {
230-
if (commandLine.hasOption(CommandOptions.HELP)){
241+
if (commandLine.hasOption(CommandOptions.HELP)) {
231242
terminal.writeLine("List all secret identifiers from the keystore. For example: " +
232243
"`bin/logstash-keystore list`. Note - only the identifiers will be listed, not the secrets.");
233244
return;
@@ -239,7 +250,7 @@ public void command(String primaryCommand, SecureConfig config, String... allArg
239250
break;
240251
}
241252
case ADD: {
242-
if (commandLine.hasOption(CommandOptions.HELP)){
253+
if (commandLine.hasOption(CommandOptions.HELP)) {
243254
terminal.writeLine("Add secrets to the keystore. For example: " +
244255
"`bin/logstash-keystore add my-secret`, at the prompt enter your secret. You will use the identifier ${my-secret} in your Logstash configuration.");
245256
return;
@@ -251,6 +262,9 @@ public void command(String primaryCommand, SecureConfig config, String... allArg
251262
if (secretStoreFactory.exists(config.clone())) {
252263
final SecretStore secretStore = secretStoreFactory.load(config);
253264
for (String argument : commandLine.getArguments()) {
265+
if (!ConfigVariableExpander.KEY_PATTERN.matcher(argument).matches()) {
266+
throw new IllegalArgumentException(String.format("Invalid secret key name `%s` provided. %s", argument, ConfigVariableExpander.KEY_PATTERN_DESCRIPTION));
267+
}
254268
final SecretIdentifier id = new SecretIdentifier(argument);
255269
final byte[] existingValue = secretStore.retrieveSecret(id);
256270
if (existingValue != null) {
@@ -263,7 +277,7 @@ public void command(String primaryCommand, SecureConfig config, String... allArg
263277

264278
final String enterValueMessage = String.format("Enter value for %s: ", argument);
265279
char[] secret = null;
266-
while(secret == null) {
280+
while (secret == null) {
267281
terminal.write(enterValueMessage);
268282
final char[] readSecret = terminal.readSecret();
269283

@@ -288,7 +302,7 @@ public void command(String primaryCommand, SecureConfig config, String... allArg
288302
break;
289303
}
290304
case REMOVE: {
291-
if (commandLine.hasOption(CommandOptions.HELP)){
305+
if (commandLine.hasOption(CommandOptions.HELP)) {
292306
terminal.writeLine("Remove secrets from the keystore. For example: " +
293307
"`bin/logstash-keystore remove my-secret`");
294308
return;
@@ -314,7 +328,7 @@ public void command(String primaryCommand, SecureConfig config, String... allArg
314328
}
315329
}
316330

317-
private void printHelp(){
331+
private void printHelp() {
318332
terminal.writeLine("Usage:");
319333
terminal.writeLine("--------");
320334
terminal.writeLine("bin/logstash-keystore [option] command [argument]");

0 commit comments

Comments
 (0)