outputs: elasticsearch: aws updates (fluent#341)

* add docs around v1.5 out_es change to ES type; ce

- in v1.5, the default `type` was changed, which can cause problems. Highlighted this in the upgrade notes and added a 'FAQ' to the `out_es` page.
- since v1.5 introduces aws auth support, added more text to highlight that `AWS\_Region` is required. If not, it dies at runtime. Seems like it could be inferred, but at least the docs reflect it a little better.
- minor copyedits throughout.

Signed-off-by: ted <[email protected]>

Author: tedder

concepts/buffering.md

@@ -4,7 +4,7 @@ description: Performance and Data Safety
 
 # Buffering
 
-When [Fluent Bit](https://fluentbit.io) process data, it uses the system memory \(heap\) as a primary and temporal place to store the record logs before they get delivered, on this private memory area the records are processed.
+When [Fluent Bit](https://fluentbit.io) processes data, it uses the system memory \(heap\) as a primary and temporal place to store the record logs before they get delivered, on this private memory area the records are processed.
 
 Buffering refers to the ability to store the records somewhere, and while they are processed and delivered, still be able to store more. Buffering in memory is the fastest mechanism, but there are certain scenarios where the mechanism requires special strategies to deal with [backpressure](../administration/backpressure.md), data safety or reduce memory consumption by the service in constraint environments.
 

concepts/data-pipeline/buffer.md

@@ -11,7 +11,7 @@ The `buffer` phase already contains the data in an immutable state, meaning, no
 ![](../../.gitbook/assets/logging_pipeline_buffer%20%281%29.png)
 
 {% hint style="info" %}
-Note that buffered data is not longer a raw text, instead it's in Fluent Bit internal binary representation.
+Note that buffered data is not raw text, it's in Fluent Bit's internal binary representation.
 {% endhint %}
 
 Fluent Bit offers a buffering mechanism in the file system that acts as a _backup system_ to avoid data loss in case of system failures.

concepts/key-concepts.md

@@ -103,7 +103,7 @@ Source events can have or not have a structure. A structure defines a set of _ke
 At a low level both are just an array of bytes, but the Structured message defines _keys_ and _values_, having a structure helps to implement faster operations on data modifications.
 
 {% hint style="info" %}
-Fluent Bit **always** handle every Event message as a structured message. For performance reasons, we use a binary serialization data format called [MessagePack](https://msgpack.org/).
+Fluent Bit **always** handles every Event message as a structured message. For performance reasons, we use a binary serialization data format called [MessagePack](https://msgpack.org/).
 
 Consider [MessagePack](https://msgpack.org/) as a binary version of JSON on steroids.
 {% endhint %}

installation/upgrade-notes.md

@@ -8,15 +8,16 @@ For more details about changes on each release please refer to the [Official Rel
 
 The migration from v1.4 to v1.5 is pretty straightforward. 
 
--  If you enabled ```keepalive``` mode in your configuration, note that this configuration property has been renamed to ```net.keepalive```. Now all Network I/O is keepalive default enabled, to learn more about this and other associated configuration properties read the [Networking Administration](https://docs.fluentbit.io/manual/administration/networking#tcp-keepalive) section.
+-  If you enabled `keepalive` mode in your configuration, note that this configuration property has been renamed to `net.keepalive`. Now all Network I/O keepalive is enabled by default, to learn more about this and other associated configuration properties read the [Networking Administration](https://docs.fluentbit.io/manual/administration/networking#tcp-keepalive) section.
+- If you use the Elasticsearch output plugin, note the default value of `type` [changed from `flb_type` to `_doc`](https://github.com/fluent/fluent-bit/commit/04ed3d8104ca8a2f491453777ae6e38e5377817e#diff-c9ae115d3acaceac5efb949edbb21196). Many versions of Elasticsearch will tolerate this, but ES v5.6 through v6.1 require a type *without* a leading underscore. See the [Elasticsearch output plugin documentation FAQ entry](https://docs.fluentbit.io/manual/pipeline/outputs/elasticsearch#faq-underscore) for more.
 
 ## Fluent Bit v1.4
 
-If you are migrating from Fluent Bit v1.3, there are not breaking changes. Just new exciting features to enjoy :\)
+If you are migrating from Fluent Bit v1.3, there are no breaking changes. Just new exciting features to enjoy :\)
 
 ## Fluent Bit v1.3
 
-If you are migrating from Fluent Bit v1.2 to v1.3, there are not breaking changes. If you are upgrading from an older version please review the incremental changes below.
+If you are migrating from Fluent Bit v1.2 to v1.3, there are no breaking changes. If you are upgrading from an older version please review the incremental changes below.
 
 ## Fluent Bit v1.2
 

pipeline/outputs/elasticsearch.md

@@ -122,9 +122,26 @@ If you see an error message like below, you'll need to fix your configuration to
 
 For details, please read [the official blog post on that issue](https://www.elastic.co/guide/en/elasticsearch/reference/6.7/removal-of-types.html).
 
-### Fluent Bit + Amazon Elasticsearch Service
+### Elasticsearch rejects requests saying "Document mapping type name can't start with '\_'" <a id="faq-underscore"></a>
 
-Amazon ElasticSearch Service adds an extra security layer where HTTP requests must be signed with AWS Sigv4. Fluent Bit v1.5 introduced full support for Amazon ElasticSearch Service with IAM Authentication.
+Fluent Bit v1.5 changed the default mapping type from `flb_type` to `_doc`, which matches the recommendation from Elasticsearch from version 6.2 forwards \([see commit with rationale](https://github.com/fluent/fluent-bit/commit/04ed3d8104ca8a2f491453777ae6e38e5377817e#diff-c9ae115d3acaceac5efb949edbb21196)\). This doesn't work in Elasticsearch versions 5.6 through 6.1 \([see Elasticsearch discussion and fix](https://discuss.elastic.co/t/cant-use-doc-as-type-despite-it-being-declared-the-preferred-method/113837/9)\). Ensure you set an explicit map (such as `doc` or `flb_type`) in the configuration, as seen on the last line:
+
+```text
+[OUTPUT]
+    Name  es
+    Match *
+    Host  vpc-test-domain-ke7thhzoo7jawsrhmm6mb7ite7y.us-west-2.es.amazonaws.com
+    Port  443
+    Index my_index
+    AWS_Auth On
+    AWS_Region us-west-2
+    tls   On
+    Type  doc
+```
+
+### Fluent Bit + Amazon Elasticsearch Service <a id="#aws-es"></a>
+
+The Amazon ElasticSearch Service adds an extra security layer where HTTP requests must be signed with AWS Sigv4. Fluent Bit v1.5 introduced full support for Amazon ElasticSearch Service with IAM Authentication.
 
 Fluent Bit supports sourcing AWS credentials from any of the standard sources \(for example, an [Amazon EKS IAM Role for a Service Account](https://docs.aws.amazon.com/eks/latest/userguide/iam-roles-for-service-accounts.html)\).
 
@@ -143,5 +160,5 @@ Example configuration:
     tls     On
 ```
 
-Notice that the `Port` is set to `443`, and that `tls` is enabled.
+Notice that the `Port` is set to `443`, `tls` is enabled, and `AWS_Region` is set.