Services map support for traces and fields mapping sample

docs/Integration-API.md

@@ -244,9 +244,14 @@ the current state of the integration:
   ]
 }
 ```
+The next screen shows the Integration assets status:
+
+![Integration - assets - status](img/Integration-assets-status.png)
+
+
 The next screen shows the maintenance issues:
 
-![todo](...)
+![Maintaining - issue](img/maintaining-issue.png)
 
 Once all the issues are manually resolved by the User, the UX can continue the loading process by the next API
 `PUT _integration/store/$instance_name/activate`
@@ -272,8 +277,6 @@ The loading assets phase will use the existing bulk load api for all the existin
 
 The User can chery pick specific assets to load and use the next UX window for this purpose
 
-![todo](...)
-
 Using the next API
 `PUT _integration/store/$instance_name/load`
 ```json

docs/TableOfContent.md → docs/README.md

@@ -1,4 +1,12 @@
-## Integrations Folder
+## Integrations Documentation Table of Content
+
+The integration repository contains the list of the supported catalogs ([Observability](../schema/observability/README.md) for example)
+
+In addition, it also contains the list of [Integrations](../integrations/README.md) such as [Nginx](../integrations/nginx/info/README.md)
+
+---
+The next documents describe the user workflows, design and architecture of the project. 
+
 - [Integration introduction](Integrations.md)
 - [Simple-Schema](Simple-schema.md)
 - [Integration Structure](Integration-structure.md)

integrations/nginx/info/fluet-bit/README.md

@@ -2,7 +2,8 @@
 
 ## Fluent-bit 
 
-Fluent-bit is a lightweight and flexible log data collector and forwarder, designed to handle a large volume of log data in real-time. It is an open-source project maintained by Fluentd Inc. and has gained popularity among developers due to its simplicity and ease of use.
+Fluent-bit is a lightweight and flexible data collector and forwarder, designed to handle a large volume of log data in real-time.
+It is an open-source projectpart of the Cloud Native Computing Foundation (CNCF). and has gained popularity among developers for simplicity and ease of use.
 
 Fluent-bit is designed to be lightweight, which means that it has a small footprint and can be installed on resource-constrained environments like embedded systems or containers. It is written in C language, making it fast and efficient, and it has a low memory footprint, which allows it to consume minimal system resources.
 
@@ -41,18 +42,20 @@ For setting up a fluent-bit agent on Nginx, please follow the next instructions
     Skip_Long_Lines   On
 
 [OUTPUT]
-    Name        os
+    Name        opensearch
     Match       nginx.*
     Host        <OSS_HOST>
     Port        <OSS_PORT>
     Index       sso_nginx-access-%Y.%m.%d
 ```
 Here, we specify the input plugin as tail, set the path to the Nginx access log file, and specify a tag to identify the logs in Fluent-bit. We also set some additional parameters such as memory buffer limit and skipping long lines.
 
-For the output, we use the `os` plugin to send the logs to Opensearch. We specify the Opensearch host, port, and index name.
+For the output, we use the `opensearch` plugin to send the logs to Opensearch. We specify the Opensearch host, port, and index name.
+
+   - Modify the Opensearch host and port in the configuration file to match your Opensearch installation.
+   - Depending on the system where Fluent Bit is installed:
+     - Start the Fluent-bit service by running the following command:
 
-- Modify the Opensearch host and port in the configuration file to match your Opensearch installation.
-- Start the Fluent-bit service by running the following command:
 ```text
 sudo systemctl start fluent-bit
 ```

integrations/service-map/config.json

@@ -0,0 +1,47 @@
+{
+  "name": "data-prepper",
+  "version": {
+    "integ": "0.1.0",
+    "schema": "1.0.0",
+    "resource": "^2.6.0"
+  },
+  "description": "Data-Prepper Service Map and Tracing correlations Integration",
+  "catalog": "observability",
+  "components": [
+    "traces,traceGroups,metrics"
+  ],
+  "collection":[
+    {
+      "traces": [{
+        "info": "traces signals",
+        "input_type":"traces",
+        "dataset":"spans",
+        "labels" :["traceGroups","traces"],
+        "fields-mapping" : [
+          {"alias":"attributes.serviceName","field":"serviceName"} ,
+          {"alias":"events.@timestamp","field":"events.time"}
+        ]
+      }]
+    },
+    {
+      "services": [{
+          "info": "service map",
+          "input_type": "services",
+          "dataset": "service",
+          "labels": ["services"]
+        }]
+    },
+    {
+      "metrics": [{
+        "info": "status metrics",
+        "input_type":"metrics",
+        "dataset":"metrics",
+        "labels" :["metrics"]
+      }]
+    }
+  ],
+  "repo": {
+    "github": "https://github.com/opensearch-project/observability/tree/main/integrarions/service-map"
+  }
+}
+

integrations/service-map/info/README.md

@@ -0,0 +1,81 @@
+![data-prepper](dataPrepper.svg)
+
+# What is Data Prepper
+
+[Data Prepper](https://github.com/opensearch-project/data-prepper/blob/main/docs/overview.md) is an open source utility service. Data Prepper is a server side data collector with abilities to filter, enrich, transform, normalize and aggregate data for downstream analytics and visualization. The broader vision for Data Prepper is to enable an end-to-end data analysis life cycle from gathering raw logs to facilitating sophisticated and actionable interactive ad-hoc analyses on the data.
+
+# What is Data Prepper Integration
+
+Data Prepper integration is concerned with the following aspects
+
+ - Allow simple and automatic generation of all schematic structured
+   - traces ( including specific fields mapping to map to SS4O schema)
+   - services ( adding support for specific service mapping category)
+   - metrics (using the standard SS4O schema)
+
+ - Add Dashboard Assets for correlation between traces-services-metrics 
+
+ - Add correlation queries to investigate traces based metrics
+
+# Data - Prepper Trace Fields
+Data Prepper uses the following [Traces](https://github.com/opensearch-project/data-prepper/blob/main/docs/schemas/trace-analytics/otel-v1-apm-span-index-template.md) mapping file
+The next fields are used:
+```text
+
+- traceId - A unique identifier for a trace. All spans from the same trace share the same traceId.
+- spanId - A unique identifier for a span within a trace, assigned when the span is created.
+- traceState - Conveys information about request position in multiple distributed tracing graphs.
+- parentSpanId - The spanId of this span's parent span. If this is a root span, then this field must be empty.
+- name - A description of the span's operation.
+- kind - The type of span. See OpenTelemetry - SpanKind.
+- startTime - The start time of the span.
+- endTime - The end time of the span.
+- durationInNanos - Difference in nanoseconds between startTime and endTime.
+- serviceName - Currently derived from the opentelemetry.proto.resource.v1.Resource associated with the span, the resource from the span originates.
+- events - A list of events. See OpenTelemetry - Events.
+- links - A list of linked spans. See OpenTelemetry - Links.
+- droppedAttributesCount - The number of attributes that were discarded.
+- droppedEventsCount - The number of events that were discarded.
+- droppedLinksCount - The number of links that were dropped.
+- span.attributes.* - All span attributes are split into a list of keywords.
+- resource.attributes.* - All resource attributes are split into a list of keywords.
+- status.code - The status of the span. See OpenTelemetry - Status.
+
+```
+These fields have a high overlap with the [`sso_traces`](../../../schema/observability/traces/README.md) fields from the observability catalog  
+
+There are some additional `trace.group` related fields which are not part of the [OTEL spec](https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/trace/api.md) for traces
+```text
+- traceGroup - A derived field, the name of the trace's root span.
+- traceGroupFields.endTime - A derived field, the endTime of the trace's root span.
+- traceGroupFields.statusCode - A derived field, the status.code of the trace's root span.
+- traceGroupFields.durationInNanos - A derived field, the durationInNanos of the trace's root span.
+
+```
+
+# Data - Prepper Trace Fields Mapping to SSO Trace Schema
+To compensate for the minor differences in fields naming between the [Data-Prepper-Trace mapping](https://github.com/opensearch-project/data-prepper/blob/main/docs/schemas/trace-analytics/otel-v1-apm-span-index-template.md) and [SSO Traces mapping](../../../schema/observability/traces/traces.mapping)
+We will define the next mapping alias that will be part of the [config.json](../config.json) for this integration
+
+```json5
+    ...
+     "fields-mapping" : [
+       {"alias":"attributes.serviceName","field":"serviceName"} ,
+       {"alias":"events.@timestamp","field":"events.time"}
+     ]
+    ...
+```
+
+
+To address this difference, the `trace` signal can be augmented with additional trace-component - in this case it will be the [traceGroup](../../../schema/observability/traces/traceGroup).
+
+## Service dashboard
+Service will have a dashboard comprised of the following visualizations:
+ - Latency per service with traces / metrics links
+ - Latency per trace-group with traces / metrics links
+ - Error rate per service with traces / metrics links
+ - Error rate per trace-group with traces / metrics links
+ - Throughput rate per service with traces / metrics links
+ - Throughput rate per trace-group with traces / metrics links
+
+ - Services view including error rate, latency, throughput, connected-services, traces, metrics

integrations/service-map/samples/README.md

@@ -0,0 +1,7 @@
+# Samples
+The sample folder contains any type of sampled data that explains and demonstrates the expected input signals.
+
+Specifically this folder contains two inner folder
+- **preloaded** containing a ready-made signals with detailed instructions on how to load them into the appropriate opensearch data-streams.
+
+Any other internal folder can be added that represents additional aspects of this integration expected ingesting content.

integrations/service-map/samples/docker-compose.yaml

@@ -0,0 +1,56 @@
+version: '3'
+
+services:
+  opensearch:
+    image: opensearchproject/opensearch:2.5.0
+    container_name: opensearch
+    environment:
+      - cluster.name=opensearch-cluster # Name the cluster
+      - node.name=opensearch # Name the node that will run in this container
+      - discovery.seed_hosts=opensearch # Nodes to look for when discovering the cluster
+      - cluster.initial_cluster_manager_nodes=opensearch # Nodes eligibile to serve as cluster manager
+      - bootstrap.memory_lock=true # Disable JVM heap memory swapping
+      - "OPENSEARCH_JAVA_OPTS=-Xms512m -Xmx512m" # Set min and max JVM heap sizes to at least 50% of system RAM
+      - "DISABLE_INSTALL_DEMO_CONFIG=true" # Prevents execution of bundled demo script which installs demo certificates and security configurations to OpenSearch
+      - "DISABLE_SECURITY_PLUGIN=true" # Disables security plugin
+    ulimits:
+      memlock:
+        soft: -1 # Set memlock to unlimited (no soft or hard limit)
+        hard: -1
+      nofile:
+        soft: 65536 # Maximum number of open files for the opensearch user - set to at least 65536
+        hard: 65536
+    volumes:
+      - opensearch:/usr/share/opensearch/data # Creates volume called opensearch-data1 and mounts it to the container
+    ports:
+      - 9200:9200
+      - 9600:9600
+    expose:
+      - "9200"
+    healthcheck:
+      test: ["CMD", "curl", "-f", "http://localhost:9200/_cluster/health?wait_for_status=yellow"]
+      interval: 5s
+      timeout: 25s
+      retries: 4
+    networks:
+      - opensearch-net # All of the containers will join the same Docker bridge network
+  opensearch-dashboards:
+    image: opensearchproject/opensearch-dashboards:latest
+    container_name: opensearch-dashboards
+    ports:
+      - 5601:5601 # Map host port 5601 to container port 5601
+    expose:
+      - "5601" # Expose port 5601 for web access to OpenSearch Dashboards
+    environment:
+      - 'OPENSEARCH_HOSTS=["http://opensearch:9200"]'
+      - "DISABLE_SECURITY_DASHBOARDS_PLUGIN=true" # disables security dashboards plugin in OpenSearch Dashboards
+    depends_on:
+      - opensearch
+    networks:
+      - opensearch-net
+
+volumes:
+  opensearch:
+
+networks:
+  opensearch-net: