Merge pull request #453 from wrijeff/trace

Updated trace analytics documentation to match recent changes.

Author: aetter

docs/trace/data-prepper-reference.md

@@ -7,7 +7,17 @@ nav_order: 25
 
 # Data Prepper configuration reference
 
-This page lists all supported Data Prepper sources, buffers, processors, and sinks, along with their associated options. For example configuration files, see [Data Prepper](../data-prepper/).
+This page lists all supported Data Prepper sources, buffers, preppers, and sinks, along with their associated options. For example configuration files, see [Data Prepper](../data-prepper/).
+
+
+## Data Prepper server options
+Option | Required | Description
+:--- | :--- | :---
+ssl | No | Boolean, indicating whether TLS should be used for server APIs. Defaults to true.
+keyStoreFilePath | No | String, path to a .jks or .p12 keystore file. Required if ssl is true.
+keyStorePassword | No | String, password for keystore. Optional, defaults to empty string.
+privateKeyPassword | No | String, password for private key within keystore. Optional, defaults to empty string.
+serverPort | No | Integer, port number to use for server APIs. Defaults to 4900
 
 
 ## General pipeline options
@@ -72,12 +82,12 @@ buffer_size | No | Integer, default 512. The maximum number of records the buffe
 batch_size | No | Integer, default 8. The maximum number of records the buffer drains after each read.
 
 
-## Processors
+## Preppers
 
-Processors perform some action on your data: filter, transform, enrich, etc.
+Preppers perform some action on your data: filter, transform, enrich, etc.
 
 
-### otel_trace_raw_processor
+### otel_trace_raw_prepper
 
 Converts OpenTelemetry data to Elasticsearch-compatible JSON documents. No options.
 
@@ -86,10 +96,22 @@ Converts OpenTelemetry data to Elasticsearch-compatible JSON documents. No optio
 
 Uses OpenTelemetry data to create a distributed service map for visualization in Kibana. No options.
 
+### peer_forwarder
+Forwards ExportTraceServiceRequests via gRPC to other Data Prepper instances. Required for operating Data Prepper in a clustered deployment.
+
+Option | Required | Description
+:--- | :--- | :---
+time_out | No | Integer, forwarded request timeout in seconds. Defaults to 3 seconds.
+span_agg_count | No | Integer, batch size for number of spans per request. Defaults to 48.
+discovery_mode | No | String, peer discovery mode to be used. Allowable values are `static` and `dns`. Defaults to `static`.
+static_endpoints | No | List, containing string endpoints of all Data Prepper instances.
+domain_name | No | String, single domain name to query DNS against. Typically used by creating multiple DNS A Records for the same domain.
+ssl | No | Boolean, indicating whether TLS should be used. Default is true.
+sslKeyCertChainFile | No | String, path to the security certificate
 
 ### string_converter
 
-Converts strings to uppercase or lowercase. Mostly useful as an example if you want to develop your own processor.
+Converts strings to uppercase or lowercase. Mostly useful as an example if you want to develop your own prepper.
 
 Option | Required | Description
 :--- | :--- | :---
@@ -116,7 +138,7 @@ aws_region | No | String, AWS region for the cluster (e.g. `"us-east-1"`) if you
 trace_analytics_raw | No | Boolean, default false. Whether to export as trace data to the `otel-v1-apm-span-*` index pattern (alias `otel-v1-apm-span`) for use with the Trace Analytics Kibana plugin.
 trace_analytics_service_map | No | Boolean, default false. Whether to export as trace data to the `otel-v1-apm-service-map` index for use with the service map component of the Trace Analytics Kibana plugin.
 index | No | String, name of the index to export to. Only required if you don't use the `trace_analytics_raw` or `trace_analytics_service_map` presets.
-template_file | No | String, the path to a JSON [index template](https://opendistro.github.io/for-elasticsearch-docs/docs/elasticsearch/index-templates/) file (e.g. `/your/local/template-file.json` if you do not use the `trace_analytics_raw` or `trace_analytics_service_map`. See [otel-v1-apm-span-index-template.json](https://github.com/opendistro-for-elasticsearch/simple-ingest-transformation-utility-pipeline/blob/master/situp-plugins/elasticsearch/src/main/resources/otel-v1-apm-span-index-template.json) for an example.
+template_file | No | String, the path to a JSON [index template](https://opendistro.github.io/for-elasticsearch-docs/docs/elasticsearch/index-templates/) file (e.g. `/your/local/template-file.json` if you do not use the `trace_analytics_raw` or `trace_analytics_service_map`. See [otel-v1-apm-span-index-template.json](https://github.com/opendistro-for-elasticsearch/data-prepper/blob/main/data-prepper-plugins/elasticsearch/src/main/resources/otel-v1-apm-span-index-template.json) for an example.
 document_id_field | No | String, the field from the source data to use for the Elasticsearch document ID (e.g. `"my-field"`) if you don't use the `trace_analytics_raw` or `trace_analytics_service_map` presets.
 dlq_file | No | String, the path to your preferred dead letter queue file (e.g. `/your/local/dlq-file`). Data Prepper writes to this file when it fails to index a document on the Elasticsearch cluster.
 bulk_size | No | Integer (long), default 5. The maximum size (in MiB) of bulk requests to the Elasticsearch cluster. Values below 0 indicate an unlimited size. If a single document exceeds the maximum bulk request size, Data Prepper sends it individually.

docs/trace/data-prepper.md

@@ -23,7 +23,7 @@ Otherwise, [download](https://opendistro.github.io/for-elasticsearch/downloads.h
 
 ## Configure pipelines
 
-To use Data Prepper, you define pipelines in a configuration YAML file. Each pipeline is a combination of a source, a buffer, zero or more processors, and one or more sinks:
+To use Data Prepper, you define pipelines in a configuration YAML file. Each pipeline is a combination of a source, a buffer, zero or more preppers, and one or more sinks:
 
 ```yml
 sample-pipeline:
@@ -38,8 +38,8 @@ sample-pipeline:
     bounded_blocking:
       buffer_size: 1024 # max number of records the buffer accepts
       batch_size: 256 # max number of records the buffer drains after each read
-  processor:
-    - otel_trace_raw_processor:
+  prepper:
+    - otel_trace_raw_prepper:
   sink:
     - elasticsearch:
         hosts: ["https:localhost:9200"]
@@ -55,9 +55,9 @@ sample-pipeline:
 
   By default, Data Prepper uses its one and only buffer, the `bounded_blocking` buffer, so you can omit this section unless you developed a custom buffer or need to tune the buffer settings.
 
-- Processors perform some action on your data: filter, transform, enrich, etc.
+- Preppers perform some action on your data: filter, transform, enrich, etc.
 
-  You can have multiple processors, which run sequentially from top to bottom, not in parallel. The `otel_trace_raw_processor` processor converts OpenTelemetry data into Elasticsearch-compatible JSON documents.
+  You can have multiple preppers, which run sequentially from top to bottom, not in parallel. The `otel_trace_raw_prepper` prepper converts OpenTelemetry data into Elasticsearch-compatible JSON documents.
 
 - Sinks define where your data goes. In this case, the sink is an Open Distro for Elasticsearch cluster.
 
@@ -68,6 +68,9 @@ entry-pipeline:
   delay: "100"
   source:
     otel_trace_source:
+      ssl: true
+      sslKeyCertChainFile: "config/demo-data-prepper.crt"
+      sslKeyFile: "config/demo-data-prepper.key"
   sink:
     - pipeline:
         name: "raw-pipeline"
@@ -77,10 +80,10 @@ raw-pipeline:
   source:
     pipeline:
       name: "entry-pipeline"
-  processor:
+  prepper:
     - string_converter:
         upper_case: true
-    - otel_trace_raw_processor:
+    - otel_trace_raw_prepper:
   sink:
     - elasticsearch:
         hosts: ["https://localhost:9200" ]
@@ -93,7 +96,7 @@ service-map-pipeline:
   source:
     pipeline:
       name: "entry-pipeline"
-  processor:
+  prepper:
     - service_map_stateful:
   sink:
     - elasticsearch:
@@ -106,19 +109,29 @@ service-map-pipeline:
 
 To learn more, see the [Data Prepper configuration reference](../data-prepper-reference/).
 
+## Configure the Data Prepper server
+Data Prepper itself provides administrative HTTP endpoints such as `/list` to list pipelines and `/metrics/prometheus` to provide Prometheus-compatible metrics data. The port which serves these endpoints, as well as TLS configuration, is specified by a separate YAML file. Example:
+
+```yml
+ssl: true
+keyStoreFilePath: "/usr/share/data-prepper/keystore.jks"
+keyStorePassword: "password"
+privateKeyPassword: "other_password"
+serverPort: 1234
+```
 
 ## Start Data Prepper
 
 **Docker**
 
 ```bash
-docker run --name data-prepper --expose 21890 --read-only -v /full/path/to/my-data-prepper-config.yml:/usr/share/data-prepper/data-prepper.yml amazon/opendistro-for-elasticsearch-data-prepper:latest
+docker run --name data-prepper --expose 21890 -v /full/path/to/pipelines.yaml:/usr/share/data-prepper/pipelines.yaml -v /full/path/to/data-prepper-config.yaml:/usr/share/data-prepper/data-prepper-config.yaml amazon/opendistro-for-elasticsearch-data-prepper:latest
 ```
 
 **macOS and Linux**
 
 ```bash
-./data-prepper-tar-install.sh config/my-data-prepper-config.yml
+./data-prepper-tar-install.sh config/pipelines.yaml config/data-prepper-config.yaml
 ```
 
-For production workloads, you likely want to run Data Prepper on a dedicated machine, which makes connectivity a concern. Data Prepper uses port 21890 and must be able to connect to both the OpenTelemetry Collector and the Elasticsearch cluster. In the [sample applications](https://github.com/opendistro-for-elasticsearch/Data-Prepper/tree/master/examples), you can see that all components use the same Docker network and expose the appropriate ports.
+For production workloads, you likely want to run Data Prepper on a dedicated machine, which makes connectivity a concern. Data Prepper uses port 21890 and must be able to connect to both the OpenTelemetry Collector and the Elasticsearch cluster. In the [sample applications](https://github.com/opendistro-for-elasticsearch/Data-Prepper/tree/main/examples), you can see that all components use the same Docker network and expose the appropriate ports.

docs/trace/get-started.md

@@ -7,7 +7,7 @@ nav_order: 1
 
 # Get started with Trace Analytics
 
-Open Distro for Elasticsearch Trace Analytics consists of two components---Data Prepper and the Trace Analytics Kibana plugin---that fit into the OpenTelemetry and Elasticsearch ecosystems. The Data Prepper repository has several [sample applications](https://github.com/opendistro-for-elasticsearch/Data-Prepper/tree/master/examples) to help you get started.
+Open Distro for Elasticsearch Trace Analytics consists of two components---Data Prepper and the Trace Analytics Kibana plugin---that fit into the OpenTelemetry and Elasticsearch ecosystems. The Data Prepper repository has several [sample applications](https://github.com/opendistro-for-elasticsearch/data-prepper/tree/main/examples) to help you get started.
 
 
 ## Basic flow of data
@@ -29,7 +29,7 @@ Open Distro for Elasticsearch Trace Analytics consists of two components---Data
 
 One Trace Analytics sample application is the Jaeger HotROD demo, which mimics the flow of data through a distributed application.
 
-Download or clone the [Data Prepper repository](https://github.com/opendistro-for-elasticsearch/Data-Prepper/tree/master/examples). Then navigate to `examples/jaeger-hotrod/` and open `docker-compose.yml` in a text editor. This file contains a container for each element from [Basic flow of data](#basic-flow-of-data):
+Download or clone the [Data Prepper repository](https://github.com/opendistro-for-elasticsearch/data-prepper). Then navigate to `examples/jaeger-hotrod/` and open `docker-compose.yml` in a text editor. This file contains a container for each element from [Basic flow of data](#basic-flow-of-data):
 
 - A distributed application (`jaeger-hot-rod`) with the Jaeger agent (`jaeger-agent`)
 - The [OpenTelemetry Collector](https://opentelemetry.io/docs/collector/getting-started/) (`otel-collector`)