You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
The `expand` parameter controls whether to expand equivalent synonym rules.
100
+
Consider a synonym defined like:
101
+
102
+
`foo, bar, baz`
103
+
104
+
Using `expand: true`, the synonym rule would be expanded into:
120
105
121
-
With the above request the word `bar` gets skipped but a mapping `foo => baz` is still added. However, if the mapping
122
-
being added was `foo, baz => bar` nothing would get added to the synonym list. This is because the target word for the
123
-
mapping is itself eliminated because it was a stop word. Similarly, if the mapping was "bar, foo, baz" and `expand` was
124
-
set to `false` no mapping would get added as when `expand=false` the target mapping is the first word. However, if
125
-
`expand=true` then the mappings added would be equivalent to `foo, baz => foo, baz` i.e, all mappings other than the
126
-
stop word.
106
+
```
107
+
foo => foo
108
+
foo => bar
109
+
foo => baz
110
+
bar => foo
111
+
bar => bar
112
+
bar => baz
113
+
baz => foo
114
+
baz => bar
115
+
baz => baz
116
+
```
117
+
118
+
When `expand` is set to `false`, the synonym rule is not expanded and the first synonym is treated as the canonical representation. The synonym would be equivalent to:
119
+
120
+
```
121
+
foo => foo
122
+
bar => foo
123
+
baz => foo
124
+
```
125
+
126
+
The `expand` parameter does not affect explicit synonym rules, like `foo, bar => baz`.
@@ -160,12 +160,65 @@ Text will be processed first through filters preceding the synonym filter before
160
160
{es} will also use the token filters preceding the synonym filter in a tokenizer chain to parse the entries in a synonym file or synonym set.
161
161
In the above example, the synonyms graph token filter is placed after a stemmer. The stemmer will also be applied to the synonym entries.
162
162
163
-
The synonym rules should not contain words that are removed by a filter that appears later in the chain (like a `stop` filter).
164
-
Removing a term from a synonym rule means there will be no matching for it at query time.
165
-
166
163
Because entries in the synonym map cannot have stacked positions, some token filters may cause issues here.
167
164
Token filters that produce multiple versions of a token may choose which version of the token to emit when parsing synonyms.
168
165
For example, `asciifolding` will only produce the folded version of the token.
169
166
Others, like `multiplexer`, `word_delimiter_graph` or `ngram` will throw an error.
170
167
171
168
If you need to build analyzers that include both multi-token filters and synonym filters, consider using the <<analysis-multiplexer-tokenfilter,multiplexer>> filter, with the multi-token filters in one branch and the synonym filter in the other.
169
+
170
+
[discrete]
171
+
[[synonym-graph-tokenizer-stop-token-filter]]
172
+
===== Synonyms and `stop` token filters
173
+
174
+
Synonyms and <<analysis-stop-tokenfilter,stop token filters>> interact with each other in the following ways:
The stop filter will remove the terms from the resulting synonym expansion.
221
+
222
+
For example, a synonym rule like `foo, bar => baz` and a stop filter that removes `baz` will get no matches for `foo` or `bar`, as both would get expanded to `baz` which is removed by the stop filter.
223
+
224
+
If the stop filter removed `foo` instead, then searching for `foo` would get expanded to `baz`, which is not removed by the stop filter thus potentially providing matches for `baz`.
See <<synonym-tokenizer-expand-equivalent-synonyms,expand equivalent synonyms>>.
78
+
* `lenient` (defaults to `false`).
79
+
If `true` ignores errors while parsing the synonym configuration.
80
+
It is important to note that only those synonym rules which cannot get parsed are ignored.
81
+
See <<synonym-tokenizer-stop-token-filter,synonyms and stop token filters>> for an example of `lenient` behaviour for invalid synonym rules.
82
+
83
+
[discrete]
84
+
[[synonym-tokenizer-expand-equivalent-synonyms]]
85
+
===== `expand` equivalent synonym rules
86
+
87
+
The `expand` parameter controls whether to expand equivalent synonym rules.
88
+
Consider a synonym defined like:
89
+
90
+
`foo, bar, baz`
91
+
92
+
Using `expand: true`, the synonym rule would be expanded into:
109
93
110
-
With the above request the word `bar` gets skipped but a mapping `foo => baz` is still added. However, if the mapping
111
-
being added was `foo, baz => bar` nothing would get added to the synonym list. This is because the target word for the
112
-
mapping is itself eliminated because it was a stop word. Similarly, if the mapping was "bar, foo, baz" and `expand` was
113
-
set to `false` no mapping would get added as when `expand=false` the target mapping is the first word. However, if
114
-
`expand=true` then the mappings added would be equivalent to `foo, baz => foo, baz` i.e, all mappings other than the
115
-
stop word.
94
+
```
95
+
foo => foo
96
+
foo => bar
97
+
foo => baz
98
+
bar => foo
99
+
bar => bar
100
+
bar => baz
101
+
baz => foo
102
+
baz => bar
103
+
baz => baz
104
+
```
116
105
106
+
When `expand` is set to `false`, the synonym rule is not expanded and the first synonym is treated as the canonical representation. The synonym would be equivalent to:
107
+
108
+
```
109
+
foo => foo
110
+
bar => foo
111
+
baz => foo
112
+
```
113
+
114
+
The `expand` parameter does not affect explicit synonym rules, like `foo, bar => baz`.
117
115
118
116
[discrete]
119
117
[[synonym-tokenizer-ignore_case-deprecated]]
@@ -135,7 +133,7 @@ To apply synonyms, you will need to include a synonym token filters into an anal
135
133
"my_analyzer": {
136
134
"type": "custom",
137
135
"tokenizer": "standard",
138
-
"filter": ["stemmer", "synonym_graph"]
136
+
"filter": ["stemmer", "synonym"]
139
137
}
140
138
}
141
139
----
@@ -148,14 +146,67 @@ Order is important for your token filters.
148
146
Text will be processed first through filters preceding the synonym filter before being processed by the synonym filter.
149
147
150
148
{es} will also use the token filters preceding the synonym filter in a tokenizer chain to parse the entries in a synonym file or synonym set.
151
-
In the above example, the synonyms graph token filter is placed after a stemmer. The stemmer will also be applied to the synonym entries.
152
-
153
-
The synonym rules should not contain words that are removed by a filter that appears later in the chain (like a `stop` filter).
154
-
Removing a term from a synonym rule means there will be no matching for it at query time.
149
+
In the above example, the synonyms token filter is placed after a stemmer. The stemmer will also be applied to the synonym entries.
155
150
156
151
Because entries in the synonym map cannot have stacked positions, some token filters may cause issues here.
157
152
Token filters that produce multiple versions of a token may choose which version of the token to emit when parsing synonyms.
158
153
For example, `asciifolding` will only produce the folded version of the token.
159
154
Others, like `multiplexer`, `word_delimiter_graph` or `ngram` will throw an error.
160
155
161
156
If you need to build analyzers that include both multi-token filters and synonym filters, consider using the <<analysis-multiplexer-tokenfilter,multiplexer>> filter, with the multi-token filters in one branch and the synonym filter in the other.
157
+
158
+
[discrete]
159
+
[[synonym-tokenizer-stop-token-filter]]
160
+
===== Synonyms and `stop` token filters
161
+
162
+
Synonyms and <<analysis-stop-tokenfilter,stop token filters>> interact with each other in the following ways:
The stop filter will remove the terms from the resulting synonym expansion.
209
+
210
+
For example, a synonym rule like `foo, bar => baz` and a stop filter that removes `baz` will get no matches for `foo` or `bar`, as both would get expanded to `baz` which is removed by the stop filter.
211
+
212
+
If the stop filter removed `foo` instead, then searching for `foo` would get expanded to `baz`, which is not removed by the stop filter thus potentially providing matches for `baz`.
Copy file name to clipboardExpand all lines: docs/reference/analysis/tokenfilters/synonyms-format.asciidoc
+1-1Lines changed: 1 addition & 1 deletion
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -15,7 +15,7 @@ This format uses two different definitions:
15
15
ipod, i-pod, i pod
16
16
computer, pc, laptop
17
17
----
18
-
* Explicit mappings: Matches a group of words to other words. Words on the left hand side of the rule definition are expanded into all the possibilities described on the right hand side. Example:
18
+
* Explicit synonyms: Matches a group of words to other words. Words on the left hand side of the rule definition are expanded into all the possibilities described on the right hand side. Example:
Copy file name to clipboardExpand all lines: docs/reference/search/search-your-data/search-with-synonyms.asciidoc
+13Lines changed: 13 additions & 0 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -82,6 +82,19 @@ If an index is created referencing a nonexistent synonyms set, the index will re
82
82
The only way to recover from this scenario is to ensure the synonyms set exists then either delete and re-create the index, or close and re-open the index.
83
83
======
84
84
85
+
[WARNING]
86
+
====
87
+
Invalid synonym rules can cause errors when applying analyzer changes.
88
+
For reloadable analyzers, this prevents reloading and applying changes.
89
+
You must correct errors in the synonym rules and reload the analyzer.
90
+
91
+
An index with invalid synonym rules cannot be reopened, making it inoperable when:
92
+
93
+
* A node containing the index starts
94
+
* The index is opened from a closed state
95
+
* A node restart occurs (which reopens the node assigned shards)
96
+
====
97
+
85
98
{es} uses synonyms as part of the <<analysis-overview,analysis process>>.
86
99
You can use two types of <<analysis-tokenfilters,token filter>> to include synonyms:
Copy file name to clipboardExpand all lines: docs/reference/synonyms/apis/synonyms-apis.asciidoc
+17Lines changed: 17 additions & 0 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -24,6 +24,23 @@ NOTE: Synonyms sets are limited to a maximum of 10,000 synonym rules per set.
24
24
Synonym sets with more than 10,000 synonym rules will provide inconsistent search results.
25
25
If you need to manage more synonym rules, you can create multiple synonyms sets.
26
26
27
+
WARNING: Synonyms sets must exist before they can be added to indices.
28
+
If an index is created referencing a nonexistent synonyms set, the index will remain in a partially created and inoperable state.
29
+
The only way to recover from this scenario is to ensure the synonyms set exists then either delete and re-create the index, or close and re-open the index.
30
+
31
+
[WARNING]
32
+
====
33
+
Invalid synonym rules can cause errors when applying analyzer changes.
34
+
For reloadable analyzers, this prevents reloading and applying changes.
35
+
You must correct errors in the synonym rules and reload the analyzer.
36
+
37
+
An index with invalid synonym rules cannot be reopened, making it inoperable when:
38
+
39
+
* A node containing the index starts
40
+
* The index is opened from a closed state
41
+
* A node restart occurs (which reopens the node assigned shards)
0 commit comments