diff --git a/.github/workflows/update-kube-stack-version.yml b/.github/workflows/update-kube-stack-version.yml new file mode 100644 index 0000000000..d3a02a0b06 --- /dev/null +++ b/.github/workflows/update-kube-stack-version.yml @@ -0,0 +1,65 @@ +name: Update Kube-Stack Version + +on: + schedule: + # Run every Monday at 9:00 AM UTC + - cron: '0 9 * * 1' + workflow_dispatch: # Allow manual triggering + +jobs: + update-kube-stack-version: + runs-on: ubuntu-latest + + steps: + - name: Checkout repository + uses: actions/checkout@v4 + with: + token: ${{ secrets.GITHUB_TOKEN }} + + - name: Set up Python + uses: actions/setup-python@v4 + with: + python-version: '3.9' + + - name: Run kube-stack version update script + run: | + cd scripts + python update_kube_stack_version.py + + - name: Check for changes + id: verify-changed-files + run: | + if [ -n "$(git status --porcelain)" ]; then + echo "changed=true" >> $GITHUB_OUTPUT + else + echo "changed=false" >> $GITHUB_OUTPUT + fi + + - name: Commit and push changes + if: steps.verify-changed-files.outputs.changed == 'true' + run: | + git config --local user.email "action@github.com" + git config --local user.name "GitHub Action" + git add docs/docset.yml + git commit -m "chore: update kube-stack version [skip ci]" + git push + + - name: Create Pull Request + if: steps.verify-changed-files.outputs.changed == 'true' + uses: peter-evans/create-pull-request@v5 + with: + token: ${{ secrets.GITHUB_TOKEN }} + commit-message: "chore: update kube-stack version" + title: "chore: update kube-stack version" + body: | + This PR automatically updates the kube-stack version in `docs/docset.yml` based on the latest version from the elastic-agent repository. + + **Changes:** + - Updated kube-stack version in docset.yml + + This PR was created automatically by the weekly kube-stack version update workflow. + branch: update-kube-stack-version + delete-branch: true + labels: | + automated + documentation diff --git a/deploy-manage/deploy/elastic-cloud/differences-from-other-elasticsearch-offerings.md b/deploy-manage/deploy/elastic-cloud/differences-from-other-elasticsearch-offerings.md index fdd7330742..7c115fb066 100644 --- a/deploy-manage/deploy/elastic-cloud/differences-from-other-elasticsearch-offerings.md +++ b/deploy-manage/deploy/elastic-cloud/differences-from-other-elasticsearch-offerings.md @@ -111,7 +111,7 @@ This table compares Observability capabilities between {{ech}} deployments and O | **APM integration** | ✅ | ✅ | Use **Managed Intake Service** (supports Elastic APM and OTLP protocols)
Refer to [Managed OTLP endpoint](opentelemetry://reference/motlp.md) for OTLP data ingestion | | [**APM Agent Central Configuration**](/solutions/observability/apm/apm-agent-central-configuration.md) | ✅ | ❌ | Not available in Serverless | | [**APM Tail-based sampling**](/solutions/observability/apm/transaction-sampling.md#apm-tail-based-sampling) | ✅ | ❌ | - Not available in Serverless
- Consider **OpenTelemetry** tail sampling processor as an alternative | -| [**Android agent/SDK instrumentation**](opentelemetry://reference/edot-sdks/android/index.md) | ✅ | ✅ | | +| [**Android agent/SDK instrumentation**](apm-agent-android://reference/edot-android/index.md) | ✅ | ✅ | | | [**AWS Firehose integration**](/solutions/observability/cloud/monitor-amazon-web-services-aws-with-amazon-data-firehose.md) | ✅ | ✅ | | | [**Custom roles for Kibana Spaces**](/deploy-manage/manage-spaces.md#spaces-control-user-access) | ✅ | ✅ | | | [**Data stream lifecycle**](/manage-data/lifecycle/data-stream.md) | ✅ | ✅ | Primary lifecycle management method in Serverless | @@ -121,7 +121,7 @@ This table compares Observability capabilities between {{ech}} deployments and O | **[Fleet Agent policies](/reference/fleet/agent-policy.md)** | ✅ | ✅ | | | **[Fleet server](/reference/fleet/fleet-server.md)** | - Self-hosted
- Hosted | ✅ | Fully managed by Elastic | | [**Index lifecycle management**](/manage-data/lifecycle/index-lifecycle-management.md) | ✅ | ❌ | Use [**Data stream lifecycle**](/manage-data/lifecycle/data-stream.md) instead | -| **[iOS agent/SDK instrumentation](opentelemetry://reference/edot-sdks/ios/index.md)** | ✅ | ✅ | | +| **[iOS agent/SDK instrumentation](apm-agent-ios://reference/edot-ios/index.md)** | ✅ | ✅ | | | **[Kibana Alerts](/deploy-manage/monitor/monitoring-data/configure-stack-monitoring-alerts.md)** | ✅ | ✅ | | | **[LogsDB index mode](/manage-data/data-store/data-streams/logs-data-stream.md)** | ✅ | ✅ | - Reduces storage footprint
- Enabled by default
- Cannot be disabled | | **[Logs management](/solutions/observability/logs.md)** | ✅ | ✅ | | diff --git a/docset.yml b/docset.yml index bae66b731f..bacc98d4cf 100644 --- a/docset.yml +++ b/docset.yml @@ -294,3 +294,4 @@ subs: ece-apis: https://www.elastic.co/docs/api/doc/cloud-enterprise/ intake-apis: https://www.elastic.co/docs/api/doc/observability-serverless/ models-app: "Trained Models" + kube-stack-version: 0.6.3 \ No newline at end of file diff --git a/scripts/update_kube_stack_version.py b/scripts/update_kube_stack_version.py new file mode 100755 index 0000000000..a75f3d86f9 --- /dev/null +++ b/scripts/update_kube_stack_version.py @@ -0,0 +1,161 @@ +#!/usr/bin/env python3 +""" +Kube-Stack Version Update Script + +This script automatically updates the kube-stack version in the docset.yml file +by fetching the latest version from the elastic-agent repository. + +Usage: + python update_kube_stack_version.py [--dry-run] + +Options: + --dry-run Show what would be updated without making changes +""" + +import urllib.request +import re +import sys +import argparse +from pathlib import Path + + +def fetch_url_content(url): + """Fetch content from a URL""" + try: + print(f"Attempting to fetch: {url}") + with urllib.request.urlopen(url) as response: + content = response.read().decode('utf-8') + return content + except urllib.error.URLError as e: + print(f"Failed to retrieve content: {e.reason}") + return None + + +def get_collector_version(file_path): + """Extract the collector version from docset.yml""" + try: + with open(file_path, 'r', encoding='utf-8') as file: + content = file.read() + + lines = content.splitlines() + for line in lines: + if line.strip().startswith('edot-collector-version:'): + return line.split(':', 1)[1].strip() + + # If no specific version is found, use a default version that we know works + return '9.1.2' + except FileNotFoundError: + print(f"Error: Could not find {file_path}") + return None + except Exception as e: + print(f"Error reading {file_path}: {e}") + return None + + +def get_kube_stack_version(version='main'): + """Extract KubeStackChartVersion from elastic-agent repository""" + # Try different URL formats for the k8s.go file + # First try with the version as-is (in case it already has 'v' prefix) + url = f'https://raw.githubusercontent.com/elastic/elastic-agent/{version}/testing/integration/k8s/k8s.go' + print(f"Trying k8s.go URL: {url}") + content = fetch_url_content(url) + + # If first attempt fails and version doesn't start with 'v', try with 'v' prefix + if content is None and not version.startswith('v') and version != 'main': + url = f'https://raw.githubusercontent.com/elastic/elastic-agent/v{version}/testing/integration/k8s/k8s.go' + print(f"Retrying k8s.go with URL: {url}") + content = fetch_url_content(url) + + # If that fails too, try with main branch + if content is None: + url = 'https://raw.githubusercontent.com/elastic/elastic-agent/main/testing/integration/k8s/k8s.go' + print(f"Falling back to main branch for k8s.go: {url}") + content = fetch_url_content(url) + + if content is None: + print(f"Could not fetch k8s.go from any URL") + return None + + # Look for the KubeStackChartVersion line + lines = content.splitlines() + for line in lines: + if 'KubeStackChartVersion' in line and '=' in line: + # Extract the version from the line like: KubeStackChartVersion = "0.6.3" + match = re.search(r'KubeStackChartVersion\s*=\s*"([^"]+)"', line) + if match: + return match.group(1) + + print("Could not find KubeStackChartVersion in k8s.go") + return None + + +def update_docset_kube_stack_version(version, docset_path, dry_run=False): + """Update the kube-stack-version substitution in docset.yml""" + try: + with open(docset_path, 'r', encoding='utf-8') as file: + content = file.read() + + # Replace the kube-stack-version line + pattern = r'(kube-stack-version:\s*)[0-9]+\.[0-9]+\.[0-9]+' + replacement = f'\\g<1>{version}' + new_content = re.sub(pattern, replacement, content) + + if new_content != content: + if dry_run: + print(f"[DRY RUN] Would update kube-stack-version to {version} in {docset_path}") + return True + else: + with open(docset_path, 'w', encoding='utf-8') as file: + file.write(new_content) + print(f"Updated kube-stack-version to {version} in {docset_path}") + return True + else: + print(f"kube-stack-version already up to date: {version}") + return False + + except Exception as e: + print(f"Error updating {docset_path}: {e}") + return False + + +def main(): + parser = argparse.ArgumentParser(description='Update kube-stack version in docset.yml') + parser.add_argument('--dry-run', action='store_true', + help='Show what would be updated without making changes') + args = parser.parse_args() + + # Get the script directory and construct paths relative to it + script_dir = Path(__file__).parent + docset_path = script_dir.parent / 'docset.yml' + + print(f"Using docset.yml path: {docset_path}") + + # Get the current collector version from docset.yml + col_version = get_collector_version(docset_path) + if col_version is None: + print("Error: Could not determine collector version") + sys.exit(1) + + print(f"Collector version: {col_version}") + + # Get the kube-stack version from elastic-agent repository + kube_stack_version = get_kube_stack_version(col_version) + if kube_stack_version is None: + print("Error: Could not fetch kube-stack version") + sys.exit(1) + + print(f"Found kube-stack version: {kube_stack_version}") + + # Update the docset.yml file + success = update_docset_kube_stack_version(kube_stack_version, docset_path, args.dry_run) + + if success: + print("Kube-stack version update completed successfully") + sys.exit(0) + else: + print("No update was needed or update failed") + sys.exit(1) + + +if __name__ == '__main__': + main() diff --git a/solutions/observability/apm/api-keys.md b/solutions/observability/apm/api-keys.md index ee5f7484f3..fb8b392235 100644 --- a/solutions/observability/apm/api-keys.md +++ b/solutions/observability/apm/api-keys.md @@ -151,10 +151,10 @@ To view all API keys for your project: You can now apply your newly created API keys in the configuration of each of your APM agents. See the relevant agent documentation for additional information: -* **Android**: [`apiKey`](opentelemetry://reference/edot-sdks/android/configuration.md) +* **Android**: [`apiKey`](apm-agent-android://reference/edot-android/configuration.md) * **Go agent**: [`ELASTIC_APM_API_KEY`](apm-agent-go://reference/configuration.md#config-api-key) * **.NET agent**: [`ApiKey`](apm-agent-dotnet://reference/config-reporter.md#config-api-key) -* **iOS**: [`withApiKey`](opentelemetry://reference/edot-sdks/ios/configuration.md#withapikey) +* **iOS**: [`withApiKey`](apm-agent-ios://reference/edot-ios/configuration.md#withapikey) * **Java agent**: [`api_key`](apm-agent-java://reference/config-reporter.md#config-api-key) * **Node.js agent**: [`apiKey`](apm-agent-nodejs://reference/configuration.md#api-key) * **PHP agent**: [`api_key`](apm-agent-php://reference/configuration-reference.md#config-api-key) diff --git a/solutions/observability/apm/apm-agent-central-configuration.md b/solutions/observability/apm/apm-agent-central-configuration.md index 2f36666282..4c42f8c1df 100644 --- a/solutions/observability/apm/apm-agent-central-configuration.md +++ b/solutions/observability/apm/apm-agent-central-configuration.md @@ -35,13 +35,13 @@ Each APM agent has a list of supported configurations. After selecting a Service Supported configurations are also tagged with the ![dynamic config](/solutions/images/observability-dynamic-config.svg "") badge in each APM agent’s configuration reference: Android agent -: [Configuration reference](opentelemetry://reference/edot-sdks/android/configuration.md) +: [Configuration reference](apm-agent-android://reference/edot-android/configuration.md) Go agent : [Configuration reference](apm-agent-go://reference/configuration.md) iOS agent -: [Configuration reference](opentelemetry://reference/edot-sdks/ios/configuration.md) +: [Configuration reference](apm-agent-ios://reference/edot-ios/configuration.md) Java agent : [Configuration reference](apm-agent-java://reference/configuration.md) diff --git a/solutions/observability/apm/apm-server-binary.md b/solutions/observability/apm/apm-server-binary.md index 9a1c79f24d..8eb10af7de 100644 --- a/solutions/observability/apm/apm-server-binary.md +++ b/solutions/observability/apm/apm-server-binary.md @@ -214,7 +214,7 @@ All that’s left is to compile and run your application. That’s it! **Learn more in the agent reference** -Read more in the [APM Android Agent Reference](opentelemetry://reference/edot-sdks/android/index.md). +Read more in the [APM Android Agent Reference](apm-agent-android://reference/edot-android/index.md). :::::: ::::::{tab-item} Go @@ -340,7 +340,7 @@ var config = AgentConfigBuilder() **Learn more in the agent reference** -Read more in the [APM iOS Agent Reference](opentelemetry://reference/edot-sdks/ios/index.md). +Read more in the [APM iOS Agent Reference](apm-agent-ios://reference/edot-ios/index.md). :::::: ::::::{tab-item} Java diff --git a/solutions/observability/apm/apm-server-fleet-managed.md b/solutions/observability/apm/apm-server-fleet-managed.md index 1401a437e1..f6c5a4fd69 100644 --- a/solutions/observability/apm/apm-server-fleet-managed.md +++ b/solutions/observability/apm/apm-server-fleet-managed.md @@ -261,7 +261,7 @@ All that’s left is to compile and run your application. That’s it! **Learn more in the agent reference** -Read more in the [APM Android Agent Reference](opentelemetry://reference/edot-sdks/android/index.md). +Read more in the [APM Android Agent Reference](apm-agent-android://reference/edot-android/index.md). :::::: ::::::{tab-item} Go @@ -387,7 +387,7 @@ var config = AgentConfigBuilder() **Learn more in the agent reference** -Read more in the [APM iOS Agent Reference](opentelemetry://reference/edot-sdks/ios/index.md). +Read more in the [APM iOS Agent Reference](apm-agent-ios://reference/edot-ios/index.md). :::::: ::::::{tab-item} Java diff --git a/solutions/observability/apm/edot-sdks-central-configuration.md b/solutions/observability/apm/edot-sdks-central-configuration.md index 3cf2d179b5..a9ad7b35cb 100644 --- a/solutions/observability/apm/edot-sdks-central-configuration.md +++ b/solutions/observability/apm/edot-sdks-central-configuration.md @@ -34,12 +34,12 @@ Supported configurations are also tagged with the ![dynamic config](/solutions/i | Language/Platform | EDOT SDK | Configuration Reference | | --- | --- | --- | -| Android | EDOT Android SDK | [Configuration reference](opentelemetry://reference/edot-sdks/android/configuration.md) | -| iOS | EDOT iOS SDK | [Configuration reference](opentelemetry://reference/edot-sdks/ios/configuration.md) | -| Java | EDOT Java SDK | [Configuration reference](opentelemetry://reference/edot-sdks/java/configuration.md) | -| Node.js | EDOT Node.js SDK | [Configuration reference](opentelemetry://reference/edot-sdks/nodejs/configuration.md) | -| PHP | EDOT PHP SDK | [Configuration reference](opentelemetry://reference/edot-sdks/php/configuration.md) | -| Python | EDOT Python SDK | [Configuration reference](opentelemetry://reference/edot-sdks/python/configuration.md) | +| Android | EDOT Android SDK | [Configuration reference](apm-agent-android://reference/edot-android/configuration.md) | +| iOS | EDOT iOS SDK | [Configuration reference](apm-agent-ios://reference/edot-ios/configuration.md) | +| Java | EDOT Java SDK | [Configuration reference](elastic-otel-java://reference/edot-java/configuration.md) | +| Node.js | EDOT Node.js SDK | [Configuration reference](elastic-otel-node://reference/edot-node/configuration.md) | +| PHP | EDOT PHP SDK | [Configuration reference](elastic-otel-php://reference/edot-php/configuration.md) | +| Python | EDOT Python SDK | [Configuration reference](elastic-otel-python://reference/edot-python/configuration.md) | ## EDOT configuration [_edot_configuration] diff --git a/solutions/observability/apm/secret-token.md b/solutions/observability/apm/secret-token.md index 9cbacd1226..c5928008ac 100644 --- a/solutions/observability/apm/secret-token.md +++ b/solutions/observability/apm/secret-token.md @@ -62,9 +62,9 @@ apm-server.auth.secret_token: Each Elastic {{apm-agent}} has a configuration option to set the value of the secret token: -* **Android agent**: [`secretToken`](opentelemetry://reference/edot-sdks/android/configuration.md) +* **Android agent**: [`secretToken`](apm-agent-android://reference/edot-android/configuration.md) * **Go agent**: [`ELASTIC_APM_SECRET_TOKEN`](apm-agent-go://reference/configuration.md#config-secret-token) -* **iOS agent**: [`secretToken`](opentelemetry://reference/edot-sdks/ios/configuration.md#secrettoken) +* **iOS agent**: [`secretToken`](apm-agent-ios://reference/edot-ios/configuration.md#secrettoken) * **Java agent**: [`secret_token`](apm-agent-java://reference/config-reporter.md#config-secret-token) * **.NET agent**: [`ELASTIC_APM_SECRET_TOKEN`](apm-agent-dotnet://reference/config-reporter.md#config-secret-token) * **Node.js agent**: [`Secret Token`](apm-agent-nodejs://reference/configuration.md#secret-token) diff --git a/solutions/observability/apm/use-opentelemetry-with-apm.md b/solutions/observability/apm/use-opentelemetry-with-apm.md index 3bd23d9874..90c92f0921 100644 --- a/solutions/observability/apm/use-opentelemetry-with-apm.md +++ b/solutions/observability/apm/use-opentelemetry-with-apm.md @@ -31,11 +31,11 @@ With an [Elastic Distribution of OpenTelemetry language SDK](opentelemetry://ref To set up OpenTelemetry with Elastic, refer to these guides for each language: -* [**Elastic Distribution of OpenTelemetry Java**](opentelemetry://reference/edot-sdks/java/index.md) -* [**Elastic Distribution of OpenTelemetry .NET**](opentelemetry://reference/edot-sdks/dotnet/index.md) -* [**Elastic Distribution of OpenTelemetry Node.js**](opentelemetry://reference/edot-sdks/nodejs/index.md) -* [**Elastic Distribution of OpenTelemetry Python**](opentelemetry://reference/edot-sdks/python/index.md) -* [**Elastic Distribution of OpenTelemetry PHP**](opentelemetry://reference/edot-sdks/php/index.md) +* [**Elastic Distribution of OpenTelemetry Java**](elastic-otel-java://reference/edot-java/index.md) +* [**Elastic Distribution of OpenTelemetry .NET**](elastic-otel-dotnet://reference/edot-dotnet/index.md) +* [**Elastic Distribution of OpenTelemetry Node.js**](elastic-otel-node://reference/edot-node/index.md) +* [**Elastic Distribution of OpenTelemetry Python**](elastic-otel-python://reference/edot-python/index.md) +* [**Elastic Distribution of OpenTelemetry PHP**](elastic-otel-php://reference/edot-php/index.md) ::::{important} For a complete overview of OpenTelemetry and Elastic, explore [**Elastic Distributions of OpenTelemetry**](opentelemetry://reference/index.md). diff --git a/solutions/observability/applications/llm-observability.md b/solutions/observability/applications/llm-observability.md index 4b486e2289..710380e34c 100644 --- a/solutions/observability/applications/llm-observability.md +++ b/solutions/observability/applications/llm-observability.md @@ -33,11 +33,11 @@ You can instrument the application with one of the following Elastic Distributio - [Node.js](https://github.com/elastic/elastic-otel-node) - [Java](https://github.com/elastic/elastic-otel-java) -EDOT includes many types of LLM instrumentation. This [table](opentelemetry://reference/use-cases/llms/index.md#supported-technologies) lists the supported technologies. +EDOT includes many types of LLM instrumentation. This [table](/solutions/observability/get-started/opentelemetry/use-cases/llms/index.md#supported-technologies) lists the supported technologies. ### Getting started -Check [these instructions](opentelemetry://reference/use-cases/llms/index.md) on how to setup and collect OpenTelemetry data for your LLM applications. +Check [these instructions](/solutions/observability/get-started/opentelemetry/use-cases/llms/index.md) on how to setup and collect OpenTelemetry data for your LLM applications. ## Use cases diff --git a/solutions/observability/get-started/opentelemetry/_snippets/edot-collector-auth.md b/solutions/observability/get-started/opentelemetry/_snippets/edot-collector-auth.md new file mode 100644 index 0000000000..5b1e3ffb35 --- /dev/null +++ b/solutions/observability/get-started/opentelemetry/_snippets/edot-collector-auth.md @@ -0,0 +1,22 @@ +```yaml +extensions: + bearertokenauth: + scheme: "APIKey" + token: "" + + apmconfig: + opamp: + protocols: + http: + # Default is localhost:4320 + # endpoint: "" + source: + elasticsearch: + endpoint: "" + auth: + authenticator: bearertokenauth +``` + +:::{note} +For comprehensive authentication configuration options, see [Authentication methods](elastic-agent://reference/edot-collector/config/authentication-methods.md). +::: \ No newline at end of file diff --git a/solutions/observability/get-started/opentelemetry/_snippets/guided-instructions.md b/solutions/observability/get-started/opentelemetry/_snippets/guided-instructions.md new file mode 100644 index 0000000000..877754b140 --- /dev/null +++ b/solutions/observability/get-started/opentelemetry/_snippets/guided-instructions.md @@ -0,0 +1,6 @@ +Use the **Add data** screen in Elastic Observability to generate install commands that are already configured with the values you need. + +1. Open Elastic Observability. +2. Go to **Add data**. +3. Select what you want to monitor. +4. Follow the instructions. \ No newline at end of file diff --git a/solutions/observability/get-started/opentelemetry/_snippets/process-config.md b/solutions/observability/get-started/opentelemetry/_snippets/process-config.md new file mode 100644 index 0000000000..ef316f3520 --- /dev/null +++ b/solutions/observability/get-started/opentelemetry/_snippets/process-config.md @@ -0,0 +1,17 @@ +Process metrics are turned off by default to avoid generating a large volume of timeseries data. To turn on process metrics, uncomment or add the following section inside the `hostmetrics` receiver configuration: + +```yaml + process: + mute_process_exe_error: true + mute_process_io_error: true + mute_process_user_error: true + metrics: + process.threads: + enabled: true + process.open_file_descriptors: + enabled: true + process.memory.utilization: + enabled: true + process.disk.operations: + enabled: true +``` \ No newline at end of file diff --git a/solutions/observability/get-started/opentelemetry/_snippets/retrieve-credentials.md b/solutions/observability/get-started/opentelemetry/_snippets/retrieve-credentials.md new file mode 100644 index 0000000000..043d0b0299 --- /dev/null +++ b/solutions/observability/get-started/opentelemetry/_snippets/retrieve-credentials.md @@ -0,0 +1,9 @@ +Retrieve your {{es}} URL and your API key: + +1. Retrieve the {{es}} URL for your {{ecloud}} deployment: + + 1. Go to the [{{ecloud}} console](https://cloud.elastic.co/). + 2. Next to your deployment, select **Manage**. + 3. Under **Applications** next to **{{es}}**, select **Copy endpoint**. + +2. Create an API Key following [these instructions](/deploy-manage/api-keys/elasticsearch-api-keys.md). \ No newline at end of file diff --git a/solutions/observability/get-started/opentelemetry/_snippets/serverless-endpoint-api.md b/solutions/observability/get-started/opentelemetry/_snippets/serverless-endpoint-api.md new file mode 100644 index 0000000000..b36a1151ae --- /dev/null +++ b/solutions/observability/get-started/opentelemetry/_snippets/serverless-endpoint-api.md @@ -0,0 +1,7 @@ +Follow these steps to retrieve the managed OTLP endpoint URL for your Serverless project: + +1. In {{serverless-full}}, open your Observability project. +2. Go to **Add data** → **Application** → **OpenTelemetry**. +3. Select **Managed OTLP Endpoint** in the second step. +4. Copy the OTLP endpoint configuration value. +5. Select **Create API Key** to generate an API key. \ No newline at end of file diff --git a/solutions/observability/get-started/opentelemetry/images/EDOT-K8s-architecture.png b/solutions/observability/get-started/opentelemetry/images/EDOT-K8s-architecture.png new file mode 100644 index 0000000000..19b399218b Binary files /dev/null and b/solutions/observability/get-started/opentelemetry/images/EDOT-K8s-architecture.png differ diff --git a/solutions/observability/get-started/opentelemetry/quickstart/ech/docker.md b/solutions/observability/get-started/opentelemetry/quickstart/ech/docker.md new file mode 100644 index 0000000000..90a17cf895 --- /dev/null +++ b/solutions/observability/get-started/opentelemetry/quickstart/ech/docker.md @@ -0,0 +1,129 @@ +--- +navigation_title: Docker +description: Learn how to set up the EDOT Collector and EDOT SDKs in a Docker environment with Elastic Cloud Hosted to collect host metrics, logs and application traces. +applies_to: + stack: + deployment: + ess: ga + product: + edot_collector: ga +products: + - id: cloud-hosted + - id: observability + - id: edot-collector +--- + +# Quickstart for Docker on Elastic Cloud Hosted + +Learn how to set up the EDOT Collector and EDOT SDKs in a Docker environment with {{ech}} (ECH) to collect host metrics, logs, and application traces. + +## Guided setup + +:::{include} ../../_snippets/guided-instructions.md +::: + +## Manual installation + +Follow these steps to deploy the EDOT Collector and EDOT OTel SDKs in Docker with ECH. + +:::::{stepper} + +::::{step} Create the config file + +Create the `otel-collector-config.yml` file with your EDOT Collector configuration. Refer to the [configuration reference](elastic-agent://reference/edot-collector/config/default-config-standalone.md). +:::: + +::::{step} Retrieve your settings + +:::{include} ../../_snippets/retrieve-credentials.md +::: +:::: + +::::{step}Create the .env file + +Create an `.env` file with the following content. Replace the placeholder values with your Elastic Cloud credentials: + + ```bash subs=true + HOST_FILESYSTEM=/ + DOCKER_SOCK=/var/run/docker.sock + ELASTIC_AGENT_OTEL=true + COLLECTOR_CONTRIB_IMAGE=elastic/elastic-agent:{{version.edot_collector}} + ELASTIC_API_KEY= + ELASTIC_ENDPOINT= + OTEL_COLLECTOR_CONFIG=/path/to/otel-collector-config.yml + ``` +:::: + +::::{step} Create the compose file + +Create a `compose.yml` file with the following content: + + ```yaml + services: + otel-collector: + image: ${COLLECTOR_CONTRIB_IMAGE} + container_name: otel-collector + deploy: + resources: + limits: + memory: 1.5G + restart: unless-stopped + command: ["--config", "/etc/otelcol-config.yml" ] + network_mode: host + user: 0:0 + volumes: + - ${HOST_FILESYSTEM}:/hostfs:ro + - ${DOCKER_SOCK}:/var/run/docker.sock:ro + - ${OTEL_COLLECTOR_CONFIG}:/etc/otelcol-config.yml + environment: + - HOST_FILESYSTEM + - ELASTIC_API_KEY + - ELASTIC_ENDPOINT + - ELASTIC_AGENT_OTEL + - STORAGE_DIR=/usr/share/elastic-agent + ``` +:::: + +::::{step} Start the Collector + +Start the Collector by running: + + ```bash + docker compose up -d + ``` +:::: + +::::{step} (Optional) Instrument your applications + +To collect telemetry from applications and use the EDOT Collector as a gateway, +instrument your target applications following the setup instructions: + + - [Android](apm-agent-android://reference/edot-android/index.md) + - [.NET](elastic-otel-dotnet://reference/edot-dotnet/setup/index.md) + - [iOS](apm-agent-ios://reference/edot-ios/index.md) + - [Java](elastic-otel-java://reference/edot-java/setup/index.md) + - [Node.js](elastic-otel-node://reference/edot-node/setup/index.md) + - [PHP](elastic-otel-php://reference/edot-php/setup/index.md) + - [Python](elastic-otel-python://reference/edot-python/setup/index.md) + + Configure your SDKs to send the data to the local EDOT Collector using OTLP/gRPC (`http://localhost:4317`) or OTLP/HTTP (`http://localhost:4318`). + +:::: + +::::{step} Install the content packs + +Install the **[System OpenTelemetry Assets](integration-docs://reference/system_otel.md)** integration and the **[Docker OpenTelemetry Assets](integration-docs://reference/docker_otel.md)** integration in {{kib}}. + +:::: + +::::{step} Explore your data + +Go to {{kib}} and select **Dashboards** to explore your newly collected data. + +:::: +::::: + + +## Troubleshooting + +Having issues with EDOT? Refer to the [Troubleshooting common issues with the EDOT Collector](/troubleshoot/ingest/opentelemetry/edot-collector/index.md) and [Troubleshooting the EDOT SDKs](/troubleshoot/ingest/opentelemetry/edot-sdks/index.md) for help. diff --git a/solutions/observability/get-started/opentelemetry/quickstart/ech/hosts_vms.md b/solutions/observability/get-started/opentelemetry/quickstart/ech/hosts_vms.md new file mode 100644 index 0000000000..ba49e245f1 --- /dev/null +++ b/solutions/observability/get-started/opentelemetry/quickstart/ech/hosts_vms.md @@ -0,0 +1,142 @@ +--- +navigation_title: Hosts and VMs +description: Learn how to set up the EDOT Collector and EDOT SDKs with Elastic Cloud Hosted to collect host metrics, logs and application traces. +applies_to: + stack: + deployment: + ess: ga + product: + edot_collector: ga +products: + - id: cloud-hosted + - id: observability + - id: edot-collector +--- + +# Quickstart for hosts and VMs on Elastic Cloud Hosted + +Learn how to set up the EDOT Collector and EDOT SDKs with {{ech}} (ECH) to collect host metrics, logs and application traces. + +## Guided setup + +:::{include} ../../_snippets/guided-instructions.md +::: + +## Manual installation + +Follow these steps to deploy the EDOT Collector and EDOT OTel SDKs with ECH. + +::::::{stepper} + +:::::{step} Download the EDOT Collector + +[Download the EDOT Collector](elastic-agent://reference/edot-collector/download.md) for your operating system, extract the file, and change directory to the extracted files. + +::::: + +:::::{step} Configure the EDOT Collector + +:::{include} ../../_snippets/retrieve-credentials.md +::: + +Replace `` and `` before applying the following commands. + +::::{tab-set} + +:::{tab-item} Linux +```bash +ELASTICSEARCH_ENDPOINT= && \ +ELASTIC_API_KEY= && \ +cp ./otel_samples/logs_metrics_traces.yml ./otel.yml && \ +mkdir -p ./data/otelcol && \ +sed -i "s#\${env:STORAGE_DIR}#${PWD}/data/otelcol#g" ./otel.yml && \ +sed -i "s#\${env:ELASTIC_ENDPOINT}#${ELASTICSEARCH_ENDPOINT}#g" ./otel.yml && \ +sed -i "s#\${env:ELASTIC_API_KEY}#${ELASTIC_API_KEY}#g" ./otel.yml +``` +::: + +:::{tab-item} macOS +```bash +ELASTICSEARCH_ENDPOINT= && \ +ELASTIC_API_KEY= && \ +cp ./otel_samples/logs_metrics_traces.yml ./otel.yml && \ +mkdir -p ./data/otelcol && \ +sed -i '' "s#\${env:STORAGE_DIR}#${PWD}/data/otelcol#g" ./otel.yml && \ +sed -i '' "s#\${env:ELASTIC_ENDPOINT}#${ELASTICSEARCH_ENDPOINT}#g" ./otel.yml && \ +sed -i '' "s#\${env:ELASTIC_API_KEY}#${ELASTIC_API_KEY}#g" ./otel.yml +``` +::: + +:::{tab-item} Windows +```powershell +Remove-Item -Path .\otel.yml -ErrorAction SilentlyContinue +Copy-Item .\otel_samples\logs_metrics_traces.yml .\otel.yml +New-Item -ItemType Directory -Force -Path .\data\otelcol | Out-Null + +$content = Get-Content .\otel.yml +$content = $content -replace '\${env:STORAGE_DIR}', "$PWD\data\otelcol" +$content = $content -replace '\${env:ELASTIC_ENDPOINT}', "" +$content = $content -replace '\${env:ELASTIC_API_KEY}', "" +$content | Set-Content .\otel.yml +``` +::: +:::: +::::: + +:::::{step} Run the EDOT Collector + +Run the following command to run the EDOT Collector. + +::::{tab-set} + +:::{tab-item} Linux and macOS +```bash +sudo ./otelcol --config otel.yml +``` +::: + +:::{tab-item} Windows +```powershell +.\elastic-agent.exe otel --config otel.yml +``` +::: +:::: + +:::{note} +By default, the Collector opens ports `4317` and `4318` to receive application data from locally running OTel SDKs. +::: + +::::: + +:::::{step} (Optional) Instrument your applications + +To collect telemetry from applications and use the EDOT Collector as a gateway, +instrument your target applications following the setup instructions: + +- [Android](apm-agent-android://reference/edot-android/index.md) +- [.NET](elastic-otel-dotnet://reference/edot-dotnet/setup/index.md) +- [iOS](apm-agent-ios://reference/edot-ios/index.md) +- [Java](elastic-otel-java://reference/edot-java/setup/index.md) +- [Node.js](elastic-otel-node://reference/edot-node/setup/index.md) +- [PHP](elastic-otel-php://reference/edot-php/setup/index.md) +- [Python](elastic-otel-python://reference/edot-python/setup/index.md) + +Configure your SDKs to send the data to the local EDOT Collector using OTLP/gRPC (`http://localhost:4317`) or OTLP/HTTP (`http://localhost:4318`). +::::: + +:::::{step} Install the content pack + +Install the **[System OpenTelemetry Assets](integration-docs://reference/system_otel.md)** integration in {{kib}}. + +::::: + +:::::{step} Explore your data + +Go to {{kib}} and select **Dashboards** to explore your newly collected data. + +::::: +:::::: + +## Troubleshooting + +Having issues with EDOT? Refer to the [Troubleshooting common issues with the EDOT Collector](/troubleshoot/ingest/opentelemetry/edot-collector/index.md) and [Troubleshooting the EDOT SDKs](/troubleshoot/ingest/opentelemetry/edot-sdks/index.md) for help. \ No newline at end of file diff --git a/solutions/observability/get-started/opentelemetry/quickstart/ech/index.md b/solutions/observability/get-started/opentelemetry/quickstart/ech/index.md new file mode 100644 index 0000000000..f50972781b --- /dev/null +++ b/solutions/observability/get-started/opentelemetry/quickstart/ech/index.md @@ -0,0 +1,26 @@ +--- +navigation_title: Elastic Cloud Hosted +description: Quickstart guide for setting up EDOT on Elastic Cloud Hosted. +applies_to: + stack: + serverless: + observability: + product: + edot_collector: ga +products: + - id: cloud-serverless + - id: observability + - id: edot-collector +--- + +# Quickstart on Elastic Cloud Hosted + +Select the quickstart guide for your environment from the following list: + +- [Kubernetes on hosted](k8s.md) +- [Docker on hosted](docker.md) +- [Hosts or VMs on hosted](hosts_vms.md) + +## Troubleshooting + +Having issues with EDOT? Refer to the [Troubleshooting common issues with the EDOT Collector](/troubleshoot/ingest/opentelemetry/edot-collector/index.md) and [Troubleshooting the EDOT SDKs](/troubleshoot/ingest/opentelemetry/edot-sdks/index.md) for help. \ No newline at end of file diff --git a/solutions/observability/get-started/opentelemetry/quickstart/ech/k8s.md b/solutions/observability/get-started/opentelemetry/quickstart/ech/k8s.md new file mode 100644 index 0000000000..0607a721e8 --- /dev/null +++ b/solutions/observability/get-started/opentelemetry/quickstart/ech/k8s.md @@ -0,0 +1,100 @@ +--- +navigation_title: Kubernetes +description: Learn how to set up the EDOT Collector and EDOT SDKs in a Kubernetes environment with {{ech}} to collect host metrics, logs and application traces. +applies_to: + stack: + serverless: + observability: + product: + edot_collector: ga +products: + - id: cloud-serverless + - id: observability + - id: edot-collector +--- + +# Quickstart for Kubernetes on Elastic Cloud Hosted + +Learn how to set up the EDOT Collector and EDOT SDKs in a Kubernetes environment with {{ech}} (ECH) to collect host metrics, logs and application traces. + +## Guided setup + +:::{include} ../../_snippets/guided-instructions.md +::: + +## Manual installation + +Follow these steps to deploy the EDOT Collector and EDOT OTel SDKs in Kubernetes with ECH. + +:::::{stepper} + +::::{step} Add the repository to Helm + +Run the following command to add the charts repository to Helm: + +```bash +helm repo add open-telemetry "https://open-telemetry.github.io/opentelemetry-helm-charts" --force-update +``` +:::: + +::::{step} Configure your credentials + +:::{include} ../../_snippets/retrieve-credentials.md +::: + +Replace both the `` and `` placeholders in the following command to create a namespace and a secret with your credentials. + +```bash +kubectl create namespace opentelemetry-operator-system +kubectl create secret generic elastic-secret-otel \ +--namespace opentelemetry-operator-system \ +--from-literal=elastic_endpoint='' \ +--from-literal=elastic_api_key='' +``` + +:::{note} +On Windows PowerShell, replace backslashes (`\`) with backticks (`` ` ``) for line continuation and single quotes (`'`) with double quotes (`"`). +::: +:::: + +::::{step} Install the Operator + +Install the OpenTelemetry Operator using the `kube-stack` Helm chart with the configured `values.yaml` file. + +```bash subs=true +helm install opentelemetry-kube-stack open-telemetry/opentelemetry-kube-stack \ +--namespace opentelemetry-operator-system \ +--values 'https://raw.githubusercontent.com/elastic/elastic-agent/refs/tags/v{{version.edot_collector}}/deploy/helm/edot-collector/kube-stack/values.yaml' \ +--version '{{kube-stack-version}}' +``` +:::: + +::::{step} Auto-instrument applications + +Add a language-specific annotation to your namespace by replacing `` with one of the supported values: `nodejs`, `java`, `python`, `dotnet` or `go`: + +```bash +kubectl annotate namespace YOUR_NAMESPACE instrumentation.opentelemetry.io/inject-="opentelemetry-operator-system/elastic-instrumentation" +``` + +Restart your deployment to ensure the annotations and auto-instrumentations are applied. + +For languages where auto-instrumentation is not available, you need to manually instrument your application. See the [Setup section for the corresponding SDK](opentelemetry://reference/edot-sdks/index.md). +:::: + +::::{step} Install the content packs + +Install the **[Kubernetes OpenTelemetry Assets](integration-docs://reference/kubernetes_otel.md)** and **[System OpenTelemetry Assets](integration-docs://reference/system_otel.md)** integrations in {{kib}}. + +:::: + +::::{step} Explore your data + +Go to {{kib}} and select **Dashboards** to explore your newly collected data. + +:::: +::::: + +## Troubleshooting + +Having issues with EDOT? Refer to the [Troubleshooting common issues with the EDOT Collector](/troubleshoot/ingest/opentelemetry/edot-collector/index.md) and [Troubleshooting the EDOT SDKs](/troubleshoot/ingest/opentelemetry/edot-sdks/index.md) for help. \ No newline at end of file diff --git a/solutions/observability/get-started/opentelemetry/quickstart/index.md b/solutions/observability/get-started/opentelemetry/quickstart/index.md new file mode 100644 index 0000000000..5706780ffc --- /dev/null +++ b/solutions/observability/get-started/opentelemetry/quickstart/index.md @@ -0,0 +1,43 @@ +--- +navigation_title: OpenTelemetry quickstarts +description: Learn how to set up the Elastic Distributions for OpenTelemetry (EDOT) to monitor Kubernetes, applications, and hosts. The guides cover installing the EDOT Collector, enabling auto-instrumentation, and configuring data collection for metrics, logs, and traces in Elastic Observability. +applies_to: + stack: + serverless: + observability: + product: + edot_collector: ga +products: + - id: cloud-serverless + - id: observability + - id: edot-collector +--- + +# OpenTelemetry quickstarts + +Learn how to set up the Elastic Distributions for OpenTelemetry (EDOT) to monitor Kubernetes, applications, and hosts. + +## Add data from the UI + +You can quickly add data from hosts, Kubernetes, applications, and cloud services from the Observability UI. + +1. Open Elastic Observability. +2. Go to **Add data**. +3. Select what you want to monitor. +4. Follow the instructions. + +## Manual installation guides + +The guides cover how to install the EDOT Collector, turn on auto-instrumentation, and configure data collection for metrics, logs, and traces in Elastic Observability. + +Select a guide based on the environment of your target system and your Elastic deployment model. + +| Deployment Model | Kubernetes | Docker | Hosts or VMs | +|-------------------------|-----------------------------------------|-----------------------------------------|---------------------------------------| +| Self-managed Elastic Stack | [Kubernetes on self-managed](/solutions/observability/get-started/opentelemetry/quickstart/self-managed/k8s.md) | [Docker on self-managed](/solutions/observability/get-started/opentelemetry/quickstart/self-managed/docker.md) | [Hosts or VMs on self-managed](/solutions/observability/get-started/opentelemetry/quickstart/self-managed/hosts_vms.md) | +| {{serverless-full}} | [Kubernetes on serverless](/solutions/observability/get-started/opentelemetry/quickstart/serverless/k8s.md) | [Docker on serverless](/solutions/observability/get-started/opentelemetry/quickstart/serverless/docker.md) | [Hosts or VMs on serverless](/solutions/observability/get-started/opentelemetry/quickstart/serverless/hosts_vms.md) | +| {{ech}} | [Kubernetes on hosted](/solutions/observability/get-started/opentelemetry/quickstart/ech/k8s.md) | [Docker on hosted](/solutions/observability/get-started/opentelemetry/quickstart/ech/docker.md) | [Hosts or VMs on hosted](/solutions/observability/get-started/opentelemetry/quickstart/ech/hosts_vms.md) | + +## Troubleshooting + +Having issues with the EDOT Collector? Refer to the [Troubleshooting common issues with the EDOT Collector](/troubleshoot/ingest/opentelemetry/edot-collector/index.md) guide for help. \ No newline at end of file diff --git a/solutions/observability/get-started/opentelemetry/quickstart/self-managed/docker.md b/solutions/observability/get-started/opentelemetry/quickstart/self-managed/docker.md new file mode 100644 index 0000000000..0185139693 --- /dev/null +++ b/solutions/observability/get-started/opentelemetry/quickstart/self-managed/docker.md @@ -0,0 +1,127 @@ +--- +navigation_title: Docker +description: Learn how to set up the EDOT Collector and EDOT SDKs in a Docker environment to collect host metrics, logs and application traces. +applies_to: + stack: + serverless: + observability: + product: + edot_collector: ga +products: + - id: cloud-serverless + - id: observability + - id: edot-collector +--- + +# Quickstart for Docker on self-managed deployments + +Learn how to set up the EDOT Collector and EDOT SDKs in a Docker environment to collect host metrics, logs and application traces. + +## Guided setup + +:::{include} ../../_snippets/guided-instructions.md +::: + +## Manual installation + +Follow these steps to deploy the EDOT Collector and EDOT OTel SDKs in Docker. + +:::::{stepper} + +::::{step} Create the config file + +Create the `otel-collector-config.yml` file with your EDOT Collector configuration. Refer to the [configuration reference](elastic-agent://reference/edot-collector/config/default-config-standalone.md). +:::: + +::::{step} Retrieve your settings + +Retrieve your [{{es}} endpoint](/solutions/search/search-connection-details.md) and [API key](/deploy-manage/api-keys/elasticsearch-api-keys.md). +:::: + +::::{step} Create the .env file + +Create an `.env` file with the following content. Replace the placeholder values with your Elastic Cloud credentials: + +```bash subs=true +HOST_FILESYSTEM=/ +DOCKER_SOCK=/var/run/docker.sock +ELASTIC_AGENT_OTEL=true +COLLECTOR_CONTRIB_IMAGE=elastic/elastic-agent:{{version.edot_collector}} +ELASTIC_API_KEY= +ELASTIC_ENDPOINT= +OTEL_COLLECTOR_CONFIG=/path/to/otel-collector-config.yml + ``` +:::: + +::::{step} Create the compose file + +Create a `compose.yml` file with the following content: + +```yaml +services: + otel-collector: + image: ${COLLECTOR_CONTRIB_IMAGE} + container_name: otel-collector + deploy: + resources: + limits: + memory: 1.5G + restart: unless-stopped + command: ["--config", "/etc/otelcol-config.yml" ] + network_mode: host + user: 0:0 + volumes: + - ${HOST_FILESYSTEM}:/hostfs:ro + - ${DOCKER_SOCK}:/var/run/docker.sock:ro + - ${OTEL_COLLECTOR_CONFIG}:/etc/otelcol-config.yml + environment: + - HOST_FILESYSTEM + - ELASTIC_AGENT_OTEL + - ELASTIC_API_KEY + - ELASTIC_ENDPOINT + - STORAGE_DIR=/usr/share/elastic-agent +``` +:::: + +::::{step} Start the Collector + +Start the Collector by running the following command: + +```bash +docker compose up -d +``` +:::: + +::::{step} (Optional) Instrument your applications + +To collect telemetry from applications and use the EDOT Collector as a gateway, +instrument your target applications following the setup instructions: + +- [Android](apm-agent-android://reference/edot-android/index.md) +- [.NET](elastic-otel-dotnet://reference/edot-dotnet/setup/index.md) +- [iOS](apm-agent-ios://reference/edot-ios/index.md) +- [Java](elastic-otel-java://reference/edot-java/setup/index.md) +- [Node.js](elastic-otel-node://reference/edot-node/setup/index.md) +- [PHP](elastic-otel-php://reference/edot-php/setup/index.md) +- [Python](elastic-otel-python://reference/edot-python/setup/index.md) + +Configure your SDKs to send the data to the local EDOT Collector using OTLP/gRPC (`http://localhost:4317`) or OTLP/HTTP (`http://localhost:4318`). +:::: + +::::{step} Install the content packs + +Install the **[System OpenTelemetry Assets](integration-docs://reference/system_otel.md)** integration and the **[Docker OpenTelemetry Assets](integration-docs://reference/docker_otel.md)** integration in {{kib}}. + +:::: + + +::::{step} Explore your data + +Go to {{kib}} and select **Dashboards** to explore your newly collected data. + +:::: +::::: + +## Troubleshooting + +Having issues with EDOT? Refer to the [Troubleshooting common issues with the EDOT Collector](/troubleshoot/ingest/opentelemetry/edot-collector/index.md) and [Troubleshooting the EDOT SDKs](/troubleshoot/ingest/opentelemetry/edot-sdks/index.md) for help. \ No newline at end of file diff --git a/solutions/observability/get-started/opentelemetry/quickstart/self-managed/hosts_vms.md b/solutions/observability/get-started/opentelemetry/quickstart/self-managed/hosts_vms.md new file mode 100644 index 0000000000..69f3dd3fb0 --- /dev/null +++ b/solutions/observability/get-started/opentelemetry/quickstart/self-managed/hosts_vms.md @@ -0,0 +1,140 @@ +--- +navigation_title: Hosts / VMs +description: Learn how to set up the EDOT Collector and EDOT SDKs to collect host metrics, logs and application traces. +applies_to: + stack: + serverless: + observability: + product: + edot_collector: ga +products: + - id: cloud-serverless + - id: observability + - id: edot-collector +--- + +# Quickstart for hosts / VMs on self-managed deployments + +Learn how to set up the EDOT Collector and EDOT SDKs to collect host metrics, logs and application traces. + +## Guided setup + +:::{include} ../../_snippets/guided-instructions.md +::: + +## Manual installation + +Follow these steps to deploy the EDOT Collector and EDOT OTel SDKs. + +::::::{stepper} + +:::::{step} Download the EDOT Collector + +[Download the EDOT Collector](elastic-agent://reference/edot-collector/download.md) for your operating system, extract the archive and move to the extracted directory. +::::: + +:::::{step} Configure the EDOT Collector + +Retrieve your [{{es}} endpoint](/solutions/search/search-connection-details.md) and [API key](/deploy-manage/api-keys/elasticsearch-api-keys.md) and replace `` and `` before applying the following command. + +::::{tab-set} + +:::{tab-item} Linux +```bash +ELASTICSEARCH_ENDPOINT= && \ +ELASTIC_API_KEY= && \ +cp ./otel_samples/logs_metrics_traces.yml ./otel.yml && \ +mkdir -p ./data/otelcol && \ +sed -i "s#\${env:STORAGE_DIR}#${PWD}/data/otelcol#g" ./otel.yml && \ +sed -i "s#\${env:ELASTIC_ENDPOINT}#${ELASTICSEARCH_ENDPOINT}#g" ./otel.yml && \ +sed -i "s#\${env:ELASTIC_API_KEY}#${ELASTIC_API_KEY}#g" ./otel.yml +``` +::: + +:::{tab-item} macOS +```bash +ELASTICSEARCH_ENDPOINT= && \ +ELASTIC_API_KEY= && \ +cp ./otel_samples/logs_metrics_traces.yml ./otel.yml && \ +mkdir -p ./data/otelcol && \ +sed -i '' "s#\${env:STORAGE_DIR}#${PWD}/data/otelcol#g" ./otel.yml && \ +sed -i '' "s#\${env:ELASTIC_ENDPOINT}#${ELASTICSEARCH_ENDPOINT}#g" ./otel.yml && \ +sed -i '' "s#\${env:ELASTIC_API_KEY}#${ELASTIC_API_KEY}#g" ./otel.yml +``` +::: + +:::{tab-item} Windows +```powershell +Remove-Item -Path .\otel.yml -ErrorAction SilentlyContinue +Copy-Item .\otel_samples\logs_metrics_traces.yml .\otel.yml +New-Item -ItemType Directory -Force -Path .\data\otelcol | Out-Null + +$content = Get-Content .\otel.yml +$content = $content -replace '\${env:STORAGE_DIR}', "$PWD\data\otelcol" +$content = $content -replace '\${env:ELASTIC_ENDPOINT}', "" +$content = $content -replace '\${env:ELASTIC_API_KEY}', "" +$content | Set-Content .\otel.yml +``` +::: + +:::: + +::::: + +:::::{step} Run the EDOT Collector + +Run the following command to run the EDOT Collector. + +:::{note} +The Collector will open the ports `4317` and `4318` to receive application data from locally running OTel SDKs. +::: + +::::{tab-set} + +:::{tab-item} Linux and macOS +```bash +sudo ./otelcol --config otel.yml +``` +::: + +:::{tab-item} Windows +```powershell +.\elastic-agent.exe otel --config otel.yml +``` +::: +:::: +::::: + +:::::{step} (Optional) Instrument your applications + +To collect telemetry from applications and use the EDOT Collector as a gateway, +instrument your target applications following the setup instructions: + +- [Android](apm-agent-android://reference/edot-android/index.md) +- [.NET](elastic-otel-dotnet://reference/edot-dotnet/setup/index.md) +- [iOS](apm-agent-ios://reference/edot-ios/index.md) +- [Java](elastic-otel-java://reference/edot-java/setup/index.md) +- [Node.js](elastic-otel-node://reference/edot-node/setup/index.md) +- [PHP](elastic-otel-php://reference/edot-php/setup/index.md) +- [Python](elastic-otel-python://reference/edot-python/setup/index.md) + +Configure your SDKs to send the data to the local EDOT Collector using OTLP/gRPC (`http://localhost:4317`) or OTLP/HTTP (`http://localhost:4318`). + +::::: + +:::::{step} Install the content pack + +Install the **[System OpenTelemetry Assets](integration-docs://reference/system_otel.md)** integration in {{kib}}. + +::::: + +:::::{step} Explore your data + +Go to {{kib}} and select **Dashboards** to explore your newly collected data. + +::::: +:::::: + +## Troubleshooting + +Having issues with EDOT? Refer to the [Troubleshooting common issues with the EDOT Collector](/troubleshoot/ingest/opentelemetry/edot-collector/index.md) and [Troubleshooting the EDOT SDKs](/troubleshoot/ingest/opentelemetry/edot-sdks/index.md) for help. \ No newline at end of file diff --git a/solutions/observability/get-started/opentelemetry/quickstart/self-managed/index.md b/solutions/observability/get-started/opentelemetry/quickstart/self-managed/index.md new file mode 100644 index 0000000000..3e119a7368 --- /dev/null +++ b/solutions/observability/get-started/opentelemetry/quickstart/self-managed/index.md @@ -0,0 +1,28 @@ +--- +navigation_title: Self-managed +description: Before following the quickstart guides to set up your EDOT deployment, make sure to install your self-managed Elastic Stack. +applies_to: + stack: + serverless: + observability: + product: + edot_collector: ga +products: + - id: cloud-serverless + - id: observability + - id: edot-collector +--- + +# Quickstart with a self-managed Elastic Stack + +Select the quickstart guide for your environment from the following list: + +- [Kubernetes on self-managed](k8s.md) +- [Docker on self-managed](docker.md) +- [Hosts or VMs on self-managed](hosts_vms.md) + +Before following the quickstart guides to set up your EDOT deployment, make sure to [install your self-managed Elastic Stack](/solutions/observability/apm/use-opentelemetry-with-apm.md). + +## Troubleshooting + +Having issues with EDOT? Refer to the [Troubleshooting common issues with the EDOT Collector](/troubleshoot/ingest/opentelemetry/edot-collector/index.md) and [Troubleshooting the EDOT SDKs](/troubleshoot/ingest/opentelemetry/edot-sdks/index.md) for help. \ No newline at end of file diff --git a/solutions/observability/get-started/opentelemetry/quickstart/self-managed/k8s.md b/solutions/observability/get-started/opentelemetry/quickstart/self-managed/k8s.md new file mode 100644 index 0000000000..ed605e25f4 --- /dev/null +++ b/solutions/observability/get-started/opentelemetry/quickstart/self-managed/k8s.md @@ -0,0 +1,93 @@ +--- +navigation_title: Kubernetes +description: Learn how to set up the EDOT Collector and EDOT SDKs in a Kubernetes environment to collect host metrics, logs and application traces. +applies_to: + stack: + serverless: + observability: + product: + edot_collector: ga +products: + - id: cloud-serverless + - id: observability + - id: edot-collector +--- + +# Quickstart for Kubernetes on self-managed deployments + +Learn how to set up the EDOT Collector and EDOT SDKs in a Kubernetes environment to collect host metrics, logs and application traces. + +## Guided setup + +:::{include} ../../_snippets/guided-instructions.md +::: + +## Manual installation + +Follow these steps to deploy the EDOT Collector and EDOT OTel SDKs in Docker. + +:::::{stepper} + +::::{step} Add the OpenTelemetry repository to Helm + +```bash +helm repo add open-telemetry 'https://open-telemetry.github.io/opentelemetry-helm-charts' --force-update +``` +:::: + +::::{step} Set up credentials + +Retrieve your [{{es}} endpoint](/solutions/search/search-connection-details.md) and [API key](/deploy-manage/api-keys/elasticsearch-api-keys.md) and replace both in the following command to create a namespace and a secret with your credentials. + +```bash +kubectl create namespace opentelemetry-operator-system +kubectl create secret generic elastic-secret-otel \ +--namespace opentelemetry-operator-system \ +--from-literal=elastic_endpoint='' \ +--from-literal=elastic_api_key='' +``` + +:::: + +::::{step} Install the operator + +Install the OpenTelemetry Operator using the `kube-stack` Helm chart with the pre-configured `values.yaml` file. + +```bash subs=true +helm install opentelemetry-kube-stack open-telemetry/opentelemetry-kube-stack \ +--namespace opentelemetry-operator-system \ +--values 'https://raw.githubusercontent.com/elastic/elastic-agent/refs/tags/v{{version.edot_collector}}/deploy/helm/edot-collector/kube-stack/values.yaml' \ +--version '{{kube-stack-version}}' +``` + +:::: + +::::{step} Auto-instrument applications + +Add a language-specific annotation to your namespace by replacing `` with one of the supported values (`nodejs`, `java`, `python`, `dotnet` or `go`) in the following command. + +```bash +kubectl annotate namespace YOUR_NAMESPACE instrumentation.opentelemetry.io/inject-="opentelemetry-operator-system/elastic-instrumentation" +``` + +Restart your deployment to ensure the annotations and auto-instrumentations are applied. + +For languages where auto-instrumentation is not available, you will need to manually instrument your application. See the [Setup section in the corresponding SDK](opentelemetry://reference/edot-sdks/index.md). +:::: + +::::{step} Install the content packs + +Install the **[Kubernetes OpenTelemetry Assets](integration-docs://reference/kubernetes_otel.md)** and **[System OpenTelemetry Assets](integration-docs://reference/system_otel.md)** integrations in {{kib}}. + +:::: + +::::{step} Explore your data + +Go to {{kib}} and select **Dashboards** to explore your newly collected data. + +:::: +::::: + +## Troubleshooting + +Having issues with EDOT? Refer to the [Troubleshooting common issues with the EDOT Collector](/troubleshoot/ingest/opentelemetry/edot-collector/index.md) and [Troubleshooting the EDOT SDKs](/troubleshoot/ingest/opentelemetry/edot-sdks/index.md) for help. \ No newline at end of file diff --git a/solutions/observability/get-started/opentelemetry/quickstart/serverless/docker.md b/solutions/observability/get-started/opentelemetry/quickstart/serverless/docker.md new file mode 100644 index 0000000000..1da1864f9c --- /dev/null +++ b/solutions/observability/get-started/opentelemetry/quickstart/serverless/docker.md @@ -0,0 +1,150 @@ +--- +navigation_title: Docker +description: Learn how to set up the EDOT Collector and EDOT SDKs in a Docker environment with {{serverless-full}} to collect host metrics, logs, and application traces. +applies_to: + stack: + serverless: + observability: + product: + edot_collector: ga +products: + - id: cloud-serverless + - id: observability + - id: edot-collector +--- + +# Quickstart for Docker on Elastic Cloud Serverless + +Learn how to set up the EDOT Collector and EDOT SDKs in a Docker environment with {{serverless-full}} to collect host metrics, logs, and application traces. + + +## Guided setup + +:::{include} ../../_snippets/guided-instructions.md +::: + +## Manual installation + +Follow these steps to deploy the EDOT Collector and EDOT OTel SDKs in Docker with {{serverless-full}}. + +:::::{stepper} + +::::{step} Create the config file + +Create a `otel-collector-config.yml` file with your EDOT collector configuration. For more details, refer to the [configuration reference](elastic-agent://reference/edot-collector/config/default-config-standalone.md) for {{motlp}}. + +:::: + +::::{step} Retrieve your settings + +:::{include} ../../_snippets/serverless-endpoint-api.md +::: + +:::: + +::::{step} Create the .env file + +Create a `.env` file with the following content, replacing the placeholder values with your actual Elastic Cloud credentials: + +```bash subs=true +HOST_FILESYSTEM=/ +DOCKER_SOCK=/var/run/docker.sock +ELASTIC_AGENT_OTEL=true +COLLECTOR_CONTRIB_IMAGE=elastic/elastic-agent:{{version.edot_collector}} +ELASTIC_API_KEY= +ELASTIC_OTLP_ENDPOINT= +OTEL_COLLECTOR_CONFIG=/path/to/otel-collector-config.yml +``` +:::: + +::::{step} Create the compose file + +Create a `compose.yml` file with the following content: + +```yaml +services: + otel-collector: + image: ${COLLECTOR_CONTRIB_IMAGE} + container_name: otel-collector + deploy: + resources: + limits: + memory: 1.5G + restart: unless-stopped + command: ["--config", "/etc/otelcol-config.yml" ] + network_mode: host + user: 0:0 + volumes: + - ${HOST_FILESYSTEM}:/hostfs:ro + - ${DOCKER_SOCK}:/var/run/docker.sock:ro + - ${OTEL_COLLECTOR_CONFIG}:/etc/otelcol-config.yml + environment: + - HOST_FILESYSTEM + - ELASTIC_AGENT_OTEL + - ELASTIC_API_KEY + - ELASTIC_OTLP_ENDPOINT + - STORAGE_DIR=/usr/share/elastic-agent +``` + +:::: + +::::{step} Start the Collector + +Start the collector by running: + +```bash +docker compose up -d +``` + +:::: + +::::{step} (Optional) Instrument your applications + +To collect telemetry from applications and use the EDOT Collector as a gateway, +instrument your target applications following the setup instructions: + +- [Android](apm-agent-android://reference/edot-android/index.md) +- [.NET](elastic-otel-dotnet://reference/edot-dotnet/setup/index.md) +- [iOS](apm-agent-ios://reference/edot-ios/index.md) +- [Java](elastic-otel-java://reference/edot-java/setup/index.md) +- [Node.js](elastic-otel-node://reference/edot-node/setup/index.md) +- [PHP](elastic-otel-php://reference/edot-php/setup/index.md) +- [Python](elastic-otel-python://reference/edot-python/setup/index.md) + +Configure your SDKs to send the data to the local EDOT Collector using OTLP/gRPC (`http://localhost:4317`) or OTLP/HTTP (`http://localhost:4318`). + +:::: + +::::{step} Install the content pack + +Install the **[Docker OpenTelemetry Assets](integration-docs://reference/docker_otel.md)** integration in {{kib}}. + +:::: + + +::::{step} Explore your data + +Go to {{kib}} and select **Dashboards** to explore your newly collected data. + +:::: +::::: + +## Troubleshooting + +The following issues might occur. + +### API Key prefix not found + +The following error is due to an improperly formatted API key: + +```txt +Exporting failed. Dropping data. +{"kind": "exporter", "data_type": } +"Unauthenticated desc = ApiKey prefix not found" +``` + +Format your API key as `"Authorization": "ApiKey "` or `"Authorization=ApiKey "` depending on whether you're using a Collector or SDK. + +### Error: too many requests + +The managed endpoint has per-project rate limits in place. If you reach this limit, contact our [support team](https://support.elastic.co). diff --git a/solutions/observability/get-started/opentelemetry/quickstart/serverless/hosts_vms.md b/solutions/observability/get-started/opentelemetry/quickstart/serverless/hosts_vms.md new file mode 100644 index 0000000000..48e3e89145 --- /dev/null +++ b/solutions/observability/get-started/opentelemetry/quickstart/serverless/hosts_vms.md @@ -0,0 +1,152 @@ +--- +navigation_title: Hosts and VMs +description: Learn how to set up the EDOT Collector and EDOT SDKs in a Docker environment with {{serverless-full}} to collect host metrics, logs, and application traces. Send the data through OTLP to your Elastic Serverless Project. +applies_to: + stack: + serverless: + observability: + product: + edot_collector: ga +products: + - id: cloud-serverless + - id: observability + - id: edot-collector +--- + +# Quickstart for hosts / VMs on Elastic Cloud Serverless + +Learn how to set up the EDOT Collector and EDOT SDKs in a Docker environment with {{serverless-full}} to collect host metrics, logs, and application traces. Send the data through OTLP to your Elastic Serverless Project. + +## Guided setup + +:::{include} ../../_snippets/guided-instructions.md +::: + +## Manual installation + +Follow these steps to deploy the EDOT Collector and EDOT OTel SDKs with {{serverless-full}}. + +::::::{stepper} + +:::::{step} Download the EDOT Collector + +[Download the EDOT Collector](elastic-agent://reference/edot-collector/download.md) for your operating system. + +::::: + +:::::{step} Configure the EDOT Collector + +:::{include} ../../_snippets/serverless-endpoint-api.md +::: + +Replace `` and `` before applying the following command. + +::::{tab-set} + +:::{tab-item} Linux +```bash +ELASTIC_OTLP_ENDPOINT= && \ +ELASTIC_API_KEY= && \ +cp ./otel_samples/managed_otlp/logs_metrics_traces.yml ./otel.yml && \ +mkdir -p ./data/otelcol && \ +sed -i "s#\${env:STORAGE_DIR}#${PWD}/data/otelcol#g" ./otel.yml && \ +sed -i "s#\${env:ELASTIC_OTLP_ENDPOINT}#${ELASTIC_OTLP_ENDPOINT}#g" ./otel.yml && \ +sed -i "s#\${env:ELASTIC_API_KEY}#${ELASTIC_API_KEY}#g" ./otel.yml +``` +::: + +:::{tab-item} macOS +```bash +ELASTIC_OTLP_ENDPOINT= && \ +ELASTIC_API_KEY= && \ +cp ./otel_samples/managed_otlp/logs_metrics_traces.yml ./otel.yml && \ +mkdir -p ./data/otelcol && \ +sed -i '' "s#\${env:STORAGE_DIR}#${PWD}/data/otelcol#g" ./otel.yml && \ +sed -i '' "s#\${env:ELASTIC_OTLP_ENDPOINT}#${ELASTIC_OTLP_ENDPOINT}#g" ./otel.yml && \ +sed -i '' "s#\${env:ELASTIC_API_KEY}#${ELASTIC_API_KEY}#g" ./otel.yml +``` +::: + +:::{tab-item} Windows +```powershell +Remove-Item -Path .\otel.yml -ErrorAction SilentlyContinue +Copy-Item .\otel_samples\managed_otlp\logs_metrics_traces.yml .\otel.yml +New-Item -ItemType Directory -Force -Path .\data\otelcol | Out-Null + +$content = Get-Content .\otel.yml +$content = $content -replace '\${env:STORAGE_DIR}', "$PWD\data\otelcol" +$content = $content -replace '\${env:ELASTIC_OTLP_ENDPOINT}', "" +$content = $content -replace '\${env:ELASTIC_API_KEY}', "" +$content | Set-Content .\otel.yml +``` +::: +:::: +::::: + +:::::{step} Run the EDOT Collector + +Run the following command to run the EDOT Collector. + +::::{tab-set} + +:::{tab-item} Linux and macOS + +```bash +sudo ./otelcol --config otel.yml +``` +::: + +:::{tab-item} Windows +```powershell +.\elastic-agent.exe otel --config otel.yml +``` +::: +:::: + +::::{note} +The Collector opens ports `4317` and `4318` to receive application data from locally running OTel SDKs without authentication. This allows the SDKs to send data without any further configuration needed as they use this endpoint by default. +:::: +::::: + +:::::{step} (Optional) Instrument your applications + +To collect telemetry from applications and use the EDOT Collector as a gateway, +instrument your target applications following the setup instructions: + +- [Android](apm-agent-android://reference/edot-android/index.md) +- [.NET](elastic-otel-dotnet://reference/edot-dotnet/setup/index.md) +- [iOS](apm-agent-ios://reference/edot-ios/index.md) +- [Java](elastic-otel-java://reference/edot-java/setup/index.md) +- [Node.js](elastic-otel-node://reference/edot-node/setup/index.md) +- [PHP](elastic-otel-php://reference/edot-php/setup/index.md) +- [Python](elastic-otel-python://reference/edot-python/setup/index.md) + +Configure your SDKs to send the data to the local EDOT Collector using OTLP/gRPC (`http://localhost:4317`) or OTLP/HTTP (`http://localhost:4318`). +::::: + +:::::{step} Explore your data + +Go to {{kib}} and select **Dashboards** to explore your newly collected data. + +::::: +:::::: + +## Troubleshooting + +The following issues might occur. + +### API Key prefix not found + +The following error is due to an improperly formatted API key: + +```txt +Exporting failed. Dropping data. +{"kind": "exporter", "data_type": } +"Unauthenticated desc = ApiKey prefix not found" +``` + +Format your API key as `"Authorization": "ApiKey "` or `"Authorization=ApiKey "` depending on whether you're using a Collector or SDK. + +### Error: too many requests + +The managed endpoint has per-project rate limits in place. If you reach this limit, contact our [support team](https://support.elastic.co). diff --git a/solutions/observability/get-started/opentelemetry/quickstart/serverless/index.md b/solutions/observability/get-started/opentelemetry/quickstart/serverless/index.md new file mode 100644 index 0000000000..85f00269fa --- /dev/null +++ b/solutions/observability/get-started/opentelemetry/quickstart/serverless/index.md @@ -0,0 +1,51 @@ +--- +navigation_title: Elastic Cloud Serverless +description: Quickstart guide for setting up EDOT on Elastic Cloud Serverless. +applies_to: + stack: + serverless: + observability: + product: + edot_collector: ga +products: + - id: cloud-serverless + - id: observability + - id: edot-collector +--- + +# Quickstart on Elastic Cloud Serverless + +The [{{motlp}}](opentelemetry://reference/motlp.md) simplifies OpenTelemetry data ingestion. It provides an endpoint for OpenTelemetry SDKs and Collectors to send telemetry data, with Elastic handling scaling, data processing, and storage. The endpoint is exclusively for Elastic Cloud users, initially available in {{serverless-full}} only. + +The {{motlp}} is designed for the following use cases: + +* Logs and Infrastructure Monitoring: Logs forwarded in OTLP format and host and Kubernetes metrics in OTLP format. +* APM: Application telemetry in OTLP format. + +## Prerequisites + +* An Elastic Observability Serverless project. +* An {{edot}} or any system that can forward logs, metrics, or traces in OTLP format. + +You also need to retrieve your OTLP endpoint address and an API key. + +:::{include} ../../_snippets/serverless-endpoint-api.md +::: + +## Quickstart guides + +Select the quickstart guide for your environment from the following list: + +- [Kubernetes on serverless](k8s.md) +- [Docker on serverless](docker.md) +- [Hosts or VMs on serverless](hosts_vms.md) + +## Differences with Elastic APM Endpoint + +The {{motlp}} ensures that OpenTelemetry data is stored without any schema translation, preserving both OpenTelemetry semantic conventions and resource attributes. It supports ingesting OTLP logs, metrics, and traces in a unified manner, ensuring consistent treatment across all telemetry data. + +## Provide feedback + +Help improve the {{motlp}} by sending us feedback in our [discussion forum](https://discuss.elastic.co/c/apm) or [community Slack](https://elasticstack.slack.com/signup). + +For EDOT Collector feedback, open an issue in the [elastic-agent repository](https://github.com/elastic/elastic-agent/issues). diff --git a/solutions/observability/get-started/opentelemetry/quickstart/serverless/k8s.md b/solutions/observability/get-started/opentelemetry/quickstart/serverless/k8s.md new file mode 100644 index 0000000000..d09678b6d5 --- /dev/null +++ b/solutions/observability/get-started/opentelemetry/quickstart/serverless/k8s.md @@ -0,0 +1,109 @@ +--- +navigation_title: Kubernetes +description: Learn how to set up the EDOT Collector and EDOT SDKs in a Kubernetes environment with {{serverless-full}} to collect host metrics, logs and application traces. +applies_to: + stack: + serverless: + observability: + product: + edot_collector: ga +products: + - id: cloud-serverless + - id: observability + - id: edot-collector +--- + +# Quickstart for Kubernetes on Elastic Cloud Serverless + +Learn how to set up the EDOT Collector and EDOT SDKs in a Kubernetes environment with {{serverless-full}} to collect host metrics, logs and application traces. + +## Guided setup + +:::{include} ../../_snippets/guided-instructions.md +::: + +## Manual installation + +Follow these steps to deploy the EDOT Collector and EDOT OTel SDKs in Kubernetes. + +:::::{stepper} + +::::{step} Add the OpenTelemetry repository to Helm + +Run the following command to add the charts repository to Helm: + +```bash +helm repo add open-telemetry 'https://open-telemetry.github.io/opentelemetry-helm-charts' --force-update +``` +:::: + +::::{step} Set up connection and credentials + +:::{include} ../../_snippets/serverless-endpoint-api.md +::: + +Replace `` and `` in the following command to create a namespace and a secret with your credentials. + +```bash +kubectl create namespace opentelemetry-operator-system +kubectl create secret generic elastic-secret-otel \ +--namespace opentelemetry-operator-system \ +--from-literal=elastic_otlp_endpoint='' \ +--from-literal=elastic_api_key='' +``` +:::: + +::::{step} Install the operator + +Install the OpenTelemetry Operator using the `kube-stack` Helm chart with the configured `values.yaml` file. + +```bash subs=true +helm install opentelemetry-kube-stack open-telemetry/opentelemetry-kube-stack \ +--namespace opentelemetry-operator-system \ +--values 'https://raw.githubusercontent.com/elastic/elastic-agent/refs/tags/v{{version.edot_collector}}/deploy/helm/edot-collector/kube-stack/managed_otlp/values.yaml' \ +--version '{{kube-stack-version}}' +``` + +The Operator provides a deployment of the EDOT Collector and configuration environment variables. This allows SDKs and instrumentation to send data to the EDOT Collector without further configuration. +:::: + +::::{step} Auto-instrument applications + +Add a language-specific annotation to your namespace by replacing `` with one of the supported values (`nodejs`, `java`, `python`, `dotnet` or `go`) in the following command. + +```bash +kubectl annotate namespace YOUR_NAMESPACE instrumentation.opentelemetry.io/inject-="opentelemetry-operator-system/elastic-instrumentation" +``` + +The OpenTelemetry Operator automatically provides the OTLP endpoint configuration and authentication to the SDKs through environment variables. Restart your deployment to ensure the annotations and auto-instrumentations are applied. + +For languages where auto-instrumentation is not available, manually instrument your application. See the [Setup section in the corresponding SDK](opentelemetry://reference/edot-sdks/index.md). +:::: + +::::{step} Explore your data + +Go to {{kib}} and select **Dashboards** to explore your newly collected data. + +:::: + +::::: + +## Troubleshooting + +The following issues might occur. + +### API Key prefix not found + +The following error is due to an improperly formatted API key: + +```txt +Exporting failed. Dropping data. +{"kind": "exporter", "data_type": } +"Unauthenticated desc = ApiKey prefix not found" +``` + +Format your API key as `"Authorization": "ApiKey "` or `"Authorization=ApiKey "` depending on whether you're using a Collector or SDK. + +### Error: too many requests + +The managed endpoint has per-project rate limits in place. If you reach this limit, contact our [support team](https://support.elastic.co). diff --git a/solutions/observability/get-started/opentelemetry/use-cases/index.md b/solutions/observability/get-started/opentelemetry/use-cases/index.md new file mode 100644 index 0000000000..19155763c5 --- /dev/null +++ b/solutions/observability/get-started/opentelemetry/use-cases/index.md @@ -0,0 +1,22 @@ +--- +navigation_title: OpenTelemetry use cases +description: Overview of different use cases for the {{edot}}. +applies_to: + stack: + serverless: + observability: + product: + edot_collector: ga +products: + - id: cloud-serverless + - id: observability +--- + +# OpenTelemetry use cases + +Explore specific use cases for the {{edot}}: + +- [Kubernetes observability](/solutions/observability/get-started/opentelemetry/use-cases/kubernetes/index.md) +- [LLM observability](/solutions/observability/get-started/opentelemetry/use-cases/llms/index.md) + + diff --git a/solutions/observability/get-started/opentelemetry/use-cases/kubernetes/components.md b/solutions/observability/get-started/opentelemetry/use-cases/kubernetes/components.md new file mode 100644 index 0000000000..9f9b7849c3 --- /dev/null +++ b/solutions/observability/get-started/opentelemetry/use-cases/kubernetes/components.md @@ -0,0 +1,70 @@ +--- +navigation_title: Components description +description: Description of components involved in Kubernetes observability with OpenTelemetry, including Operator, Collectors, Helm Chart, and auto-instrumentation. +applies_to: + stack: + serverless: + observability: + product: + edot_collector: ga +products: + - id: cloud-serverless + - id: observability + - id: edot-collector +--- + +# Components description + +Getting started with OpenTelemetry for Kubernetes observability requires an understanding of the following components, their functions, and interactions: + +- OpenTelemetry Operator +- Collectors +- Kube-stack Helm Chart +- Auto-instrumentation resources + +## OpenTelemetry Operator + +The OpenTelemetry Operator is a [Kubernetes Operator](https://kubernetes.io/docs/concepts/extend-kubernetes/operator/) implementation designed to manage OpenTelemetry resources in a Kubernetes environment. + +The operator defines and oversees the following Custom Resource Definitions (CRDs): + +- [OpenTelemetry Collectors](https://github.com/open-telemetry/opentelemetry-collector): Agents responsible for receiving, processing, and exporting telemetry data such as logs, metrics, and traces. +- [Instrumentation](https://opentelemetry.io/docs/kubernetes/operator/automatic): Leverages OpenTelemetry instrumentation libraries to automatically instrument workloads. + +All signals including logs, metrics, and traces are processed by the collectors and sent directly to {{es}} using the [Elasticsearch exporter](elastic-agent://reference/edot-collector/components/elasticsearchexporter.md). A collector's processor pipeline replaces the traditional APM server functionality for handling application traces. + +## Kube-stack Helm Chart + +The [kube-stack Helm chart](https://github.com/open-telemetry/opentelemetry-helm-charts/tree/main/charts/opentelemetry-kube-stack) manages the installation of the operator, including its CRDs. It also configures a suite of collectors, which instrument various Kubernetes components to add observability and monitoring. + +The chart is installed with a provided default `values.yaml` file that can be customized when needed. + +## DaemonSet collectors + +The OpenTelemetry components deployed within the DaemonSet collectors are responsible for observing specific signals from each node. To ensure complete data collection, these components must be deployed on every node in the cluster. Failing to do so results in partial and potentially incomplete data. + +The DaemonSet collectors handle the following data: + +- Host Metrics: Collects host metrics specific to each node, through the hostmetrics receiver. +- Kubernetes Metrics: Captures metrics related to the Kubernetes infrastructure on each node. +- Logs: Uses the filelog receiver to gather logs from all Pods running on the respective node. +- OTLP Traces Receiver: Opens an HTTP and a GRPC port on the node to receive OTLP trace data. + +## Deployment collector + +The OpenTelemetry components deployed within a Deployment collector focus on gathering data at the cluster level rather than at individual nodes. Unlike DaemonSet collectors, which need to be deployed on every node, a Deployment collector operates as a standalone instance. + +The Deployment collector handles the following data: + +- Kubernetes Events: Monitors and collects events occurring across the entire Kubernetes cluster. +- Cluster Metrics: Captures metrics that provide insights into the overall health and performance of the Kubernetes cluster. + +## Auto-instrumentation + +The Helm chart is configured to enable zero-code instrumentation using the [Operator's Instrumentation resource](https://github.com/open-telemetry/opentelemetry-operator/?tab=readme-ov-file#opentelemetry-auto-instrumentation-injection) for the following programming languages: + +- Go +- Java +- Node.js +- Python +- .NET diff --git a/solutions/observability/get-started/opentelemetry/use-cases/kubernetes/customization.md b/solutions/observability/get-started/opentelemetry/use-cases/kubernetes/customization.md new file mode 100644 index 0000000000..180688d529 --- /dev/null +++ b/solutions/observability/get-started/opentelemetry/use-cases/kubernetes/customization.md @@ -0,0 +1,89 @@ +--- +navigation_title: Customization +description: Guide on customizing the EDOT installation parameters for Kubernetes monitoring. +applies_to: + stack: + serverless: + observability: + product: + edot_collector: ga +products: + - id: cloud-serverless + - id: observability + - id: edot-collector +--- + +# Customize the configuration + +To customize the installation parameters, change the configuration values provided in `values.yaml` file, or override them using `--set parameter=value` during the installation. + +To update an installed release, run a `helm upgrade` with the updated `values.yaml` file. Depending on the changes, some Pods may need to be restarted for the updates to take effect. Refer to [upgrades](/solutions/observability/get-started/opentelemetry/use-cases/kubernetes/upgrade.md) for a command example. + +## Configurable parameters + +The following table lists common parameters that might be relevant for your use case: + +| `values.yaml` parameter | Description | +|----------------------------------|----------------------| +| `clusterName` | Sets the `k8s.cluster.name` field in all collected data. The cluster name is automatically detected for `EKS/GKE/AKS` clusters, but it might be useful for other environments. When monitoring multiple Kubernetes clusters, ensure that the cluster name is properly set in all your environments.

Refer to the [resourcedetection processor](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/processor/resourcedetectionprocessor/README.md#cluster-name) for more details about cluster name detection. | +| `collectors.cluster.resources` | Configures `CPU` and `memory` `requests` and `limits` applied to the `Deployment` EDOT Collector responsible for cluster-level metrics.
This setting follows the standard [Kubernetes resources syntax](https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/#requests-and-limits) for specifying requests and limits. | +| `collectors.daemon.resources` | Configures `CPU` and `memory` `requests` and `limits` applied to the `DaemonSet` EDOT Collector responsible for node-level metrics and application traces.
This setting follows the standard [Kubernetes resources syntax](https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/#requests-and-limits) for specifying requests and limits. | +| `certManager.enabled` | Defaults to `false`.
Refer to [cert-manager integrated installation](#cert-manager-integrated-installation) for more details. | + +For more information on all available parameters and their meaning, refer to: + +* The provided `values.yaml`, which includes the default settings for the EDOT installation. +* The official OpenTelemetry `kube-stack` Helm chart [values file](https://github.com/elastic/elastic-agent/blob/main/deploy/helm/edot-collector/kube-stack/values.yaml), with explanations of all parameters. + +## Cert-manager integrated installation + +In Kubernetes, for the API server to communicate with the webhook component created by the operator, the webhook requires a TLS certificate that the API server is configured to trust. The default provided configuration sets the Helm chart to auto generate the required certificate as a self-signed certificate with an expiration policy of 365 days. + +These certificates won't be renewed if the Helm chart's release is not manually updated. For production environments, use a certificate manager like [cert-manager](https://cert-manager.io/docs/installation/). + +Integrating the operator with [cert-manager](https://cert-manager.io/) turns on automatic generation and renewal of the TLS certificate. Make sure that cert-manager and its CRDs are already installed in your Kubernetes environment. If that's not the case, refer to the [cert-manager installation guide](https://cert-manager.io/docs/installation/) before continuing. + +Follow any of the following options to install the `opentelemetry-kube-stack` Helm chart integrated with `cert-manager`. + +### Install using the CLI + +Add `--set opentelemetry-operator.admissionWebhooks.certManager.enabled=true --set opentelemetry-operator.admissionWebhooks.autoGenerateCert=null` to the installation command. + +For example: + +```bash subs=true +helm upgrade --install --namespace opentelemetry-operator-system opentelemetry-kube-stack open-telemetry/opentelemetry-kube-stack \ +--values https://raw.githubusercontent.com/elastic/elastic-agent/refs/tags/v{{version.edot_collector}}/deploy/helm/edot-collector/kube-stack/values.yaml --version {{kube-stack-version}} \ +--set opentelemetry-operator.admissionWebhooks.certManager.enabled=true --set opentelemetry-operator.admissionWebhooks.autoGenerateCert=null +``` + +### Install using values.yaml + +Keep an updated copy of the `values.yaml` file by following these steps: + + 1. **Update** the `values.yaml` file with the following changes: + + - **Enable cert-manager integration for admission webhooks.** + + ```yaml + opentelemetry-operator: + admissionWebhooks: + certManager: + enabled: true # Change from `false` to `true` + ``` + + - **Remove the generation of a self-signed certificate.** + + ```yaml + # Remove the following lines: + autoGenerateCert: + enabled: true + recreate: true + ``` + + 2. Run the installation (or upgrade) command pointing to the updated file. For example, assuming that the updated file has been saved as `values_cert-manager.yaml`: + + ```bash + helm upgrade --install --namespace opentelemetry-operator-system opentelemetry-kube-stack open-telemetry/opentelemetry-kube-stack \ + --values ./resources/kubernetes/operator/helm/values_cert-manager.yaml --version {{kube-stack-version}} + ``` diff --git a/solutions/observability/get-started/opentelemetry/use-cases/kubernetes/deployment.md b/solutions/observability/get-started/opentelemetry/use-cases/kubernetes/deployment.md new file mode 100644 index 0000000000..0735cde60a --- /dev/null +++ b/solutions/observability/get-started/opentelemetry/use-cases/kubernetes/deployment.md @@ -0,0 +1,118 @@ +--- +navigation_title: Deployment +description: Instructions for deploying EDOT components for Kubernetes monitoring, using guided onboarding or manual steps. +applies_to: + stack: + serverless: + observability: + product: + edot_collector: ga +products: + - id: cloud-serverless + - id: observability + - id: edot-collector +--- + +# Deployment + +You can use the [guided onboarding](#deploy-using-the-guided-onboarding) or [deploy all components manually](#manual-deployment) + +## Deploy using the guided onboarding + +The guided onboarding simplifies deploying your Kubernetes components by setting up an [API Key](/deploy-manage/api-keys/elasticsearch-api-keys.md) and the needed [Integrations](integration-docs://reference/index.md) in the background. + +Follow these steps to use the guided onboarding: + +1. In {{kib}}, navigate to **Observability** → **Add data**. +2. Select **Kubernetes**, then choose **Kubernetes monitoring with EDOT Collector**. +3. Follow the instructions to install the OpenTelemetry Operator using the Helm chart and the provided `values.yaml`. + +When installing the OpenTelemetry Operator: + +- Make sure the `elastic_endpoint` shown in the installation command is valid for your environment. If not, replace it with the correct {{es}} endpoint. +- The `elastic_api_key` shown in the installation command corresponds to an API key created by {{kib}} when the onboarding process is initiated. + +:::{note} +The default installation deploys an OpenTelemetry Operator with a self-signed TLS certificate. +To automatically generate and renew certificates, refer to [cert-manager integrated installation](/solutions/observability/get-started/opentelemetry/use-cases/kubernetes/customization.md#cert-manager-integrated-installation) for instructions on customizing the `values.yaml` file before running the `helm install` command. +::: + +## Manual deployment + +Follow these steps for a manual deployment of all components. + +### Elastic Stack preparations + +Before installing the operator do the following: + +1. Create an [API Key](/deploy-manage/api-keys/elasticsearch-api-keys.md). + +2. Install the following integrations in {{kib}}: + - `System` + - `Kubernetes` + - `Kubernetes OpenTelemetry Assets` + +When using the [{{kib}} onboarding UX](#deploy-using-the-guided-onboarding), the previous actions are automatically handled by {{kib}}. + +### Operator installation + +Follow these steps to install the operator: + +1. Create the `opentelemetry-operator-system` Kubernetes namespace: + + ```bash + $ kubectl create namespace opentelemetry-operator-system + ``` + +2. Create a secret in the new namespace with the following command: + + ```bash + kubectl create -n opentelemetry-operator-system secret generic elastic-secret-otel \ + --from-literal=elastic_endpoint='YOUR_ELASTICSEARCH_ENDPOINT' \ + --from-literal=elastic_api_key='YOUR_ELASTICSEARCH_API_KEY' + ``` + + Don't forget to replace: + + - `YOUR_ELASTICSEARCH_ENDPOINT`: {{es}} endpoint (**with `https://` prefix**). For example: `https://1234567.us-west2.gcp.elastic-cloud.com:443`. + - `YOUR_ELASTICSEARCH_API_KEY`: {{es}} API Key created in the previous step. + +3. If you need to [customize the configuration](/solutions/observability/get-started/opentelemetry/use-cases/kubernetes/customization.md), copy the `values.yaml` file and adapt it to your needs. Refer to the [compatibility matrix](/solutions/observability/get-started/opentelemetry/use-cases/kubernetes/prerequisites-compatibility.md#compatibility-matrix) for a complete list of available manifests in the `release branches`. + +4. Run the following commands to deploy the `opentelemetry-kube-stack` Helm chart, using the appropriate values file: + + ```bash subs=true + helm repo add open-telemetry https://open-telemetry.github.io/opentelemetry-helm-charts + helm repo update + helm upgrade --install --namespace opentelemetry-operator-system opentelemetry-kube-stack open-telemetry/opentelemetry-kube-stack \ + --values 'https://raw.githubusercontent.com/elastic/elastic-agent/refs/tags/v{{version.edot_collector}}/deploy/helm/edot-collector/kube-stack/values.yaml' \ + --version {{kube-stack-version}} + ``` + +## Verify the installation + +Perform the following checks to verify that everything is running properly: + +### Check Pods status + +Ensure the following components are running without errors: + + - Operator Pod + - DaemonSet Collector Pod + - Deployment Collector Pod + +### Validate instrumentation object + +Confirm that the Instrumentation object is deployed and configured with a valid endpoint. + +### Kibana dashboard check + +Verify that the **[OTEL][Metrics Kubernetes] Cluster Overview** dashboard in {{kib}} is displaying data correctly. + +### Log data availability in Kibana + +In **{{kib}} Discover**, confirm the availability of data under the `__logs-*__` data view. + +### Metrics data availability in Kibana + +In **{{kib}} Discover**, ensure data is available under the `__metrics-*__` data view. diff --git a/solutions/observability/get-started/opentelemetry/use-cases/kubernetes/index.md b/solutions/observability/get-started/opentelemetry/use-cases/kubernetes/index.md new file mode 100644 index 0000000000..083f3e9508 --- /dev/null +++ b/solutions/observability/get-started/opentelemetry/use-cases/kubernetes/index.md @@ -0,0 +1,35 @@ +--- +navigation_title: Kubernetes observability +description: Detailed description of the Kubernetes setup for EDOT, including components and customization guidance. +applies_to: + stack: + serverless: + observability: + product: + edot_collector: ga +products: + - id: cloud-serverless + - id: observability + - id: edot-collector +--- + +# Kubernetes observability with EDOT + +The [quickstart guides]/solutions/observability/get-started/opentelemetry/quickstart/index.md) for Kubernetes install a set of different EDOT Collectors and EDOT SDKs to cover collection of OpenTelemetry data for infrastructure monitoring, logs collection and application monitoring. + +The Kubernetes setup relies on the OpenTelemetry Operator, configured to automate orchestration of EDOT as follows: + +* EDOT Collector Cluster: Collection of cluster metrics. +* EDOT Collector Daemon: Collection of node metrics, logs, and application telemetry. +* EDOT Collector Gateway: Preprocessing, aggregation, and ingestion of data into Elastic. +* EDOT SDKs: Annotated applications are auto-instrumented with [EDOT SDKs](opentelemetry://reference/edot-sdks/index.md). + +The following diagram summarizes the previous components and how they interact with Elastic: + +![K8s-architecture](../../images/EDOT-K8s-architecture.png) + +Read on to learn how to: + +- Install the [OpenTelemetry Operator](https://github.com/open-telemetry/opentelemetry-operator/) using the [kube-stack Helm chart](https://github.com/open-telemetry/opentelemetry-helm-charts/tree/main/charts/opentelemetry-kube-stack). +- Use the EDOT Collectors to send Kubernetes logs, metrics, and application traces to an {{es}} cluster. +- Use the operator for applications [auto-instrumentation](https://opentelemetry.io/docs/kubernetes/operator/automatic/) in all supported languages. diff --git a/solutions/observability/get-started/opentelemetry/use-cases/kubernetes/instrumenting-applications.md b/solutions/observability/get-started/opentelemetry/use-cases/kubernetes/instrumenting-applications.md new file mode 100644 index 0000000000..38f7214156 --- /dev/null +++ b/solutions/observability/get-started/opentelemetry/use-cases/kubernetes/instrumenting-applications.md @@ -0,0 +1,336 @@ +--- +navigation_title: Instrumenting Applications +description: Guidance on instrumenting applications with EDOT SDKs on Kubernetes using automatic or manual instrumentation. +applies_to: + stack: + serverless: + observability: + product: + edot_collector: ga +products: + - id: cloud-serverless + - id: observability + - id: edot-collector +--- + +# Instrument applications with EDOT SDKs on Kubernetes + +The [Elastic Distributions of OpenTelemetry (EDOT) SDKs](opentelemetry://reference/edot-sdks/index.md) cover multiple languages. Read on to learn how to instrument applications for APM in a Kubernetes environment for all supported languages. + +In Kubernetes environments with the OpenTelemetry Operator, [automatic or zero-code instrumentation](https://opentelemetry.io/docs/kubernetes/operator/automatic/) simplifies the instrumentation process by injecting and configuring instrumentation libraries into the targeted Pods. + +On the other hand, manual instrumentation with OpenTelemetry allows you to customize trace spans, metrics, and logging directly in your application’s code. This approach provides more granular control over what and how data is captured. + +## Prerequisites + +Before starting with application auto-instrumentation, ensure the following prerequisites are in place for proper setup: + +- Install the OpenTelemetry operator and EDOT collectors following the [quickstart guide]/solutions/observability/get-started/opentelemetry/quickstart/index.md). +- Ensure a valid `kind: Instrumentation` object exists in the cluster. + +## Auto-instrumentation basics + +Zero-code instrumentation is handled by the operator through `Instrumentation` objects, used to automatically inject the necessary SDKs and configuration into application workloads. + +If you followed the [quickstart guide]/solutions/observability/get-started/opentelemetry/quickstart/index.md) to install the operator, there should be an `Instrumentation` object with name `elastic-instrumentation` in namespace `opentelemetry-operator-system`: + +```bash +kubectl get instrumentation -A +NAMESPACE NAME AGE ENDPOINT SAMPLER SAMPLER ARG +opentelemetry-operator-system elastic-instrumentation 5d20h http://opentelemetry-kube-stack-daemon-collector.opentelemetry-operator-system.svc.cluster.local:4318 parentbased_traceidratio 1.0 +``` + +The `Instrumentation` object stores important parameters: + +- The exporter endpoint represents the destination for the traces, in this case the HTTP receiver configured in the EDOT DaemonSet Collector. That endpoint has to be reachable by the Pods being instrumented. + + ```yaml + exporter: + endpoint: http://opentelemetry-kube-stack-daemon-collector.opentelemetry-operator-system.svc.cluster.local:4318 + ``` + +- Language-specific images used by the operator to inject the appropriate library into each Pod. + + ```yaml subs=true + dotnet: + image: docker.elastic.co/observability/elastic-otel-dotnet:{{version.edot_dotnet}} + java: + image: docker.elastic.co/observability/elastic-otel-javaagent:{{version.edot_java}} + nodejs: + image: docker.elastic.co/observability/elastic-otel-node:{{version.edot_node}} + python: + image: docker.elastic.co/observability/elastic-otel-python:{{version.edot_python}} + ``` + +## Configure auto-instrumentation + +To turn on auto-instrumentation, add the corresponding language annotation to the Pods template in your Deployment or relevant workload object, like StatefulSet, Job, CronJob, and so on. + +For example, the template would contain the following under `spec.template.metadata.annotations`: + +```yaml +apiVersion: apps/v1 +kind: Deployment +metadata: + name: myapp +spec: + # ... + template: + metadata: + annotations: + instrumentation.opentelemetry.io/inject-: "opentelemetry-operator-system/elastic-instrumentation" + # ... + spec: + containers: + - image: myapplication-image + name: app + # ... +``` + +Where ```` is one of: `go` , `java`, `nodejs`, `python`, `dotnet` + +::::{note} +Make sure you add the annotations at Pod level and not directly at the workload `spec` level.Annotation values must point to an existing `Instrumentation` object. +:::: + +You can also turn on auto-instrumentation by adding the annotation at namespace level. This approach automatically applies instrumentation to all Pods within the specified namespace. + +```yaml +apiVersion: v1 +kind: Namespace +metadata: + name: mynamespace + annotations: + instrumentation.opentelemetry.io/inject-: "opentelemetry-operator-system/elastic-instrumentation" +``` + +After adding annotations to Pods or Namespaces, restart the applications for the instrumentation injection to take effect: + +```bash +kubectl rollout restart deployment/my-deployment +``` + +In case you have multiple Instrumentation objects with different settings or images, make sure you point your Pods to the desired `Instrumentation` objects in the annotations. + +The possible values for the annotation are detailed in the [Operator documentation](https://opentelemetry.io/docs/kubernetes/operator/automatic/#add-annotations-to-existing-deployments). For reference purposes, the values are: + +- `"true"`: to inject Instrumentation instance with `default` name from the current namespace. +- `"my-instrumentation"`: to inject Instrumentation instance with name `"my-instrumentation"` in the current namespace. +- `"my-other-namespace/my-instrumentation"`: to inject Instrumentation instance with name `"my-instrumentation"` from another namespace `"my-other-namespace"`. +- `"false"`: do not inject. + +For details on instrumenting specific languages, refer to: + +- [Instrumenting Java](elastic-otel-java://reference/edot-java/setup/k8s.md) +- [Instrumenting Python](elastic-otel-python://reference/edot-python/setup/index.md) +- [Instrumenting Node.js](elastic-otel-node://reference/edot-node/setup/index.md) + +### Namespace based annotations example + +The following example creates a namespace with an annotation to instrument all Pods of the namespace with `java` libraries. + +```bash +kubectl create namespace java-apps + +# Annotate app namespace +kubectl annotate namespace java-apps instrumentation.opentelemetry.io/inject-java="opentelemetry-operator-system/elastic-instrumentation" + +# Run a java example application in the namespace +kubectl run otel-test -n java-apps --env OTEL_INSTRUMENTATION_METHODS_INCLUDE="test.Testing[methodB]" --image docker.elastic.co/demos/apm/k8s-webhook-test +``` + +## Verify auto-instrumentation + +After adding the annotation and restarting the Pods, run `kubectl describe` on your application Pod to verify the SDK has been properly attached. + +Make sure that the `init container`, `volume`, and `environment variables` described in [how auto-instrumentation works](#how-auto-instrumentation-works) have been successfully injected into the Pod. + +## How auto-instrumentation works + +The OpenTelemetry Operator automates the process of instrumenting applications by injecting the necessary libraries and configuration into the application Pods. + +The process may vary slightly depending on the language, but it generally involves the following steps: + +- Creating a shared volume: The operator declares an `emptyDir` shared volume within the Pod, and mounts it the app container and a new init container. This volume serves as the medium for sharing the instrumentation library between the new init container and the application container. + +- Adding an init container: The operator adds an init container into the Pod. This container is responsible for copying the OpenTelemetry instrumentation library to the shared volume. + +- Configuring the main container: The operator injects environment variables into the main application container to configure OpenTelemetry settings (for example, `OTEL_EXPORTER_OTLP_ENDPOINT` or `OTEL_TRACES_SAMPLER`). Additionally, it links the instrumentation library to the application using mechanisms specific to the language runtime, such as: + + - Java: The library is linked through the `javaagent` option using the JAVA_TOOL_OPTIONS environment variable. + - Node.js: The library is linked through the `NODE_OPTIONS` environment variable. + - Python: The operator uses the `PYTHONPATH` environment variable to load the library [sitecustomize](https://docs.python.org/es/dev/library/site.html#module-sitecustomize) module. + +## Advanced configuration + +You can apply OTel-specific configuration to your applications at two different levels: + +- At Pod/container level, by using OTel-related environment variables. +- At `Instrumentation` object level, for example configuring different settings per language. + +Use cases include the following: + +- Change the library to be injected. +- Change the exporter endpoint. +- Apply certain logging level settings (`OTEL_LOG_LEVEL`). + +### Additional Instrumentation objects + +Consider the creation of different `Instrumentation` objects for different purposes, such as: + +- Different configuration options for certain languages. +- Trying out different versions of the SDKs. + +## Troubleshooting auto-instrumentation + +When troubleshooting auto-instrumentation issues in a Kubernetes environment with the OpenTelemetry Operator, follow these steps: + +1. Check that the operator is running. For example: + + ```bash + $ kubectl get pods -n opentelemetry-operator-system + NAME READY STATUS RESTARTS AGE + opentelemetry-kube-stack-opentelemetry-operator-7b8684cfbdbv4hj 2/2 Running 0 58s + ... + ``` + +2. Check that the `Instrumentation` object has been deployed. For example: + + ```bash + $ kubectl describe Instrumentation -n opentelemetry-operator-system + Name: elastic-instrumentation + Namespace: opentelemetry-operator-system + ... + Kind: Instrumentation + Metadata: + ... + Spec: + Dotnet: + Image: docker.elastic.co/observability/elastic-otel-dotnet:edge + Go: + Image: ghcr.io/open-telemetry/opentelemetry-go-instrumentation/autoinstrumentation-go:v0.14.0-alpha + Java: + Image: docker.elastic.co/observability/elastic-otel-javaagent:1.0.0 + Nodejs: + Image: docker.elastic.co/observability/elastic-otel-node:edge + Python: + Image: docker.elastic.co/observability/elastic-otel-python:edge + ... + ``` + +3. Check that your pod is running. The following example checks the `banana` namespace: + + ```bash + $ kubectl get pods -n banana + NAME READY STATUS RESTARTS AGE + example-otel-app 1/1 Running 0 104s + ``` + +4. Check that the pod has the instrumentation `initcontainer` installed, or `container` for Go.The events must show that the Docker image was successfully pulled and containers started: + + ```bash + $ kubectl describe pod/example-otel-app -n banana + Name: example-otel-app + Namespace: banana + ... + Annotations: instrumentation.opentelemetry.io/inject-java: opentelemetry-operator-system/elastic-instrumentation + Init Containers: + opentelemetry-auto-instrumentation-java: + Container ID: docker://7ecdf3954263d591b994ed1c0519d16322479b1515b58c1fbbe51d3066210d99 + Image: docker.elastic.co/observability/elastic-otel-javaagent:1.0.0 + Image ID: docker-pullable://docker.elastic.co/observability/elastic-otel-javaagent@sha256:28d65d04a329c8d5545ed579d6c17f0d74800b7b1c5875e75e0efd29e210566a + ... + Containers: + example-otel-app: + ... + Events: + Type Reason Age From Message + ---- ------ ---- ---- ------- + Normal Scheduled 5m3s default-scheduler Successfully assigned banana/example-otel-app to docker-desktop + Normal Pulled 5m3s kubelet Container image "docker.elastic.co/observability/elastic-otel-javaagent:1.0.0" already present on machine + Normal Created 5m3s kubelet Created container opentelemetry-auto-instrumentation-java + Normal Started 5m3s kubelet Started container opentelemetry-auto-instrumentation-java + Normal Pulling 5m2s kubelet Pulling image "docker.elastic.co/demos/apm/k8s-webhook-test" + Normal Pulled 5m1s kubelet Successfully pulled image "docker.elastic.co/demos/apm/k8s-webhook-test" in 1.139s (1.139s including waiting). Image size: 406961626 bytes. + Normal Created 5m1s kubelet Created container example-otel-app + Normal Started 5m1s kubelet Started container example-otel-app + ``` + +5. Check your pod log for agent output. For example: + + ```bash + $ kubectl logs example-otel-app -n banana + ... + [otel.javaagent 2024-10-11 13:32:44:127 +0000] [main] INFO io.opentelemetry.javaagent.tooling.VersionLogger - opentelemetry-javaagent - version: 1.0.0 + ... + ``` + +6. If there is no obvious agent log output, restart the pod with agent log level set to debug and look for agent debug output. Setting the agent to debug is different for the different language agents. Add or set the `OTEL_LOG_LEVEL` environment variable to `debug`. For example: + + ```yaml + env: + - name: OTEL_LOG_LEVEL + value: "debug" + ``` + + For Java, you can set the `OTEL_JAVAAGENT_DEBUG` environment variable to `true`. + +## Migrate from the Elastic APM Attacher for Kubernetes + +While the Elastic APM Attacher for Kubernetes only supports the Elastic APM application agents, the OpenTelemetry operator can support both the Elastic APM application agents and the EDOT agents. The OpenTelemetry operator has more features and is being actively developed. + +Migrating from the Elastic APM Attacher for Kubernetes consists of the following steps: + +1. Install the OpenTelemetry operator. +2. Define and install Instrumentation CRDs which correspond to the currently defined values in the existing Elastic APM Attacher for Kubernetes deployment. +3. Change the annotations in deployment definitions from the `co.elastic.apm/attach: ...` to the `instrumentation.opentelemetry.io/inject-...` equivalents. +4. Deploy the new deployment definitions. + +### Add instrumentation CRDs + +For example, if you have Elastic APM Java agents version 1.53.0 and Elastic APM Nodejs agents version 4.11.2 defined for your values, +and additionally have set the `ELASTIC_APM_LOG_SENDING` environment variable to `true` for the Java agent by default, then the corresponding +instrumentation would look as in the following snippet. Make sure to replace both `` and `` +with your credentials: + +```yaml +# +# Filename: elastic-apm-instrumentation.yaml +# +apiVersion: opentelemetry.io/v1alpha1 +kind: Instrumentation +metadata: + name: elastic-apm-instrumentation + namespace: opentelemetry-operator-system +spec: + java: + image: docker.elastic.co/observability/apm-agent-java:1.53.0 + env: + - name: ELASTIC_APM_SERVER_URL + value: "" + - name: ELASTIC_APM_API_KEY + value: "" + - name: ELASTIC_APM_LOG_SENDING + value: "true" + nodejs: + image: docker.elastic.co/observability/apm-agent-nodejs:4.11.2 + env: + - name: ELASTIC_APM_SERVER_URL + value: "" + - name: ELASTIC_APM_API_KEY + value: "" +``` + +To add the instrumentation to your cluster, run the following command: + +```bash +kubectl apply -f elastic-apm-instrumentation.yaml +``` + +### Migrate your pods to the new Instrumentation + +After you've defined the new instrumentations, migrate your pods by replacing the annotation that currently applies to them with the new annotation for the instrumentation. + +For example if you had a Java application deployment definition which specified the Elastic APM Attacher for Kubernetes annotation of `co.elastic.apm/attach: java`, you can replace that annotation with the equivalent annotation for the OpenTelemetry operator using the new instrumentation names you have defined. For the previous example Instrumentation, the annotation would be `instrumentation.opentelemetry.io/inject-java: "opentelemetry-operator-system/elastic-apm-instrumentation"` + +Subsequent deployments will use the new instrumentation. The pods themselves will be auto-instrumented in the same way as they were already being instrumented. Any `ELASTIC_*` environment variables and configuration options will continue to apply. diff --git a/solutions/observability/get-started/opentelemetry/use-cases/kubernetes/prerequisites-compatibility.md b/solutions/observability/get-started/opentelemetry/use-cases/kubernetes/prerequisites-compatibility.md new file mode 100644 index 0000000000..5e1d750f52 --- /dev/null +++ b/solutions/observability/get-started/opentelemetry/use-cases/kubernetes/prerequisites-compatibility.md @@ -0,0 +1,44 @@ +--- +navigation_title: Prerequisites and compatibility +description: Prerequisites and compatibility information for monitoring Kubernetes with EDOT. +applies_to: + stack: + serverless: + observability: + product: + edot_collector: ga +products: + - id: cloud-serverless + - id: observability + - id: edot-collector +--- + +# Prerequisites + +Before setting up observability for Kubernetes, make sure you have the following: + +- Elastic Stack (self-managed or [Elastic Cloud](https://www.elastic.co/cloud)) version 8.16.0 or higher, or an [{{es}} serverless](/solutions/search/serverless-elasticsearch-get-started.md) project. + +- A Kubernetes version supported by the OpenTelemetry Operator. Refer to the operator's [compatibility matrix](https://github.com/open-telemetry/opentelemetry-operator/blob/main/docs/compatibility.md#compatibility-matrix) for more details. + +- If you opt for automatic certificate generation and renewal on the OpenTelemetry Operator, install [cert-manager](https://cert-manager.io/docs/installation/) in the Kubernetes cluster. By default, the operator uses a self-signed certificate and doesn't require cert-manager. + +## Compatibility matrix + +The minimum supported version of the Elastic Stack for OpenTelemetry-based monitoring on Kubernetes is `8.16.0`. Different Elastic Stack releases support specific versions of the [kube-stack Helm chart](https://github.com/open-telemetry/opentelemetry-helm-charts/tree/main/charts/opentelemetry-kube-stack). + +You can download the values file for a specific {{stack}} version from the following URL: + +``` +https://raw.githubusercontent.com/elastic/elastic-agent/refs/tags/v/deploy/helm/edot-collector/kube-stack/values.yaml +``` + +Where `` is the version of the Elastic Stack you are using, for example `9.1.2`. + +For Serverless, use the [latest version of the values file](https://raw.githubusercontent.com/elastic/elastic-agent/refs/tags/v{{version.edot_collector}}/deploy/helm/edot-collector/kube-stack/managed_otlp/values.yaml). For version 8.16.0, use [this chart](https://raw.githubusercontent.com/elastic/opentelemetry/refs/heads/8.16/resources/kubernetes/operator/helm/values.yaml). + +:::{important} +When [installing the release](/solutions/observability/get-started/opentelemetry/use-cases/kubernetes/deployment.md), make sure you use the right `--version` and `-f ` parameters. + +The latest Helm chart version is {{kube-stack-version}}. +::: diff --git a/solutions/observability/get-started/opentelemetry/use-cases/kubernetes/upgrade.md b/solutions/observability/get-started/opentelemetry/use-cases/kubernetes/upgrade.md new file mode 100644 index 0000000000..f873cecfa5 --- /dev/null +++ b/solutions/observability/get-started/opentelemetry/use-cases/kubernetes/upgrade.md @@ -0,0 +1,33 @@ +--- +navigation_title: Upgrade +description: Instructions for upgrading the EDOT Helm chart release for Kubernetes monitoring. +applies_to: + stack: + serverless: + observability: + product: + edot_collector: ga +products: + - id: cloud-serverless + - id: observability + - id: edot-collector +--- + +# Upgrade + +:::{note} +Before upgrading or updating the release configuration, refer to the [compatibility matrix](/solutions/observability/get-started/opentelemetry/use-cases/kubernetes/prerequisites-compatibility.md#compatibility-matrix) for a list of supported versions and [customizing configuration](/solutions/observability/get-started/opentelemetry/use-cases/kubernetes/customization.md) for a list of supported configurable parameters. +::: + +To upgrade an installed release, run a `helm upgrade` command providing the desired chart version and using the correct `values.yaml` for your environment. For example: + +```bash subs=true +helm repo update open-telemetry # update information of available charts locally +helm search repo open-telemetry/opentelemetry-kube-stack --versions # list available versions of the chart + +helm upgrade --namespace opentelemetry-operator-system opentelemetry-kube-stack open-telemetry/opentelemetry-kube-stack \ +--values 'https://raw.githubusercontent.com/elastic/elastic-agent/refs/tags/v{{version.edot_collector}}/deploy/helm/edot-collector/kube-stack/values.yaml' \ +--version {{kube-stack-version}} +``` + +If [cert-manager integration](/solutions/observability/get-started/opentelemetry/use-cases/kubernetes/customization.md#cert-manager-integrated-installation) is disabled, Helm generates a new self-signed TLS certificate with every update, even if there are no actual changes to apply. \ No newline at end of file diff --git a/solutions/observability/get-started/opentelemetry/use-cases/llms/index.md b/solutions/observability/get-started/opentelemetry/use-cases/llms/index.md new file mode 100644 index 0000000000..a7b68d607e --- /dev/null +++ b/solutions/observability/get-started/opentelemetry/use-cases/llms/index.md @@ -0,0 +1,46 @@ +--- +navigation_title: LLM observability +description: Overview of LLM observability with Elastic, including supported technologies and quickstart instructions. +applies_to: + stack: + serverless: + observability: + product: + edot_collector: preview +products: + - id: cloud-serverless + - id: observability + - id: edot-collector +--- + +# LLM observability with EDOT + +Applications make more and more use of Generative Artificial Intelligence (GenAI). Telemetry data in terms of spans, metrics and logs when communicating with the GenAI APIs becomes more and more important to operate the application in production and understand the application's behavior and health state. + +Elastic currently supports LLM observability through the Elastic Distributions of Opentelemetry (EDOT). The EDOT Java, EDOT Node.js, and EDOT Python distributions support LLM observability as a tech preview. + +## Supported technologies + +The following LLM platforms are supported: + +| Technology | [EDOT Java](elastic-otel-java://reference/edot-java/supported-technologies.md#openai-client-instrumentation) | [EDOT Node.js](elastic-otel-node://reference/edot-node/supported-technologies.md#llm-instrumentations) | [EDOT Python](elastic-otel-python://reference/edot-python/supported-technologies.md#llm-instrumentations) | +|:-----------|:----------|:-------------|:------------| +| OpenAI Client | ✅ | ✅ | ✅ | +| AWS Bedrock | ❌ | ❌ | ✅ | +| Google Vertex AI | ❌ | ❌ | ✅ | + +See the [Supported Technologies section in the corresponding EDOT SDK](opentelemetry://reference/edot-sdks/index.md) for detailed information on supported versions. + +## Quickstart + +Follow these steps to instrument LLMs using EDOT. + +### Instrument your LLM application + +Select the environment and target system from the [quick start overview](/solutions/observability/get-started/opentelemetry/quickstart/index.md) and follow the setup instructions to instrument your LLM application. Instrumentation for the supported technologies is enabled by default. + +### Configuration + +See the [Configuration section in the corresponding EDOT SDK](opentelemetry://reference/edot-sdks/index.md) to turn on or off specific instrumentations and check which instrumentation is active by default. + +When you complete the setup and configuration of the EDOT SDK and there is a workload on your application, start checking for telemetry data in {{kib}}. If there's no data showing up, see the troubleshooting of the corresponding EDOT SDK. diff --git a/solutions/observability/get-started/quickstart-monitor-your-application-performance.md b/solutions/observability/get-started/quickstart-monitor-your-application-performance.md index d34b245135..459d15f534 100644 --- a/solutions/observability/get-started/quickstart-monitor-your-application-performance.md +++ b/solutions/observability/get-started/quickstart-monitor-your-application-performance.md @@ -56,13 +56,13 @@ Follow these steps to collect application telemetry data using the EDOT SDKs: 2. Under **What do you want to monitor?** select **Application**, and then select **OpenTelemetry**. 3. Follow the instructions to install the EDOT SDK for your application: - - [Android](opentelemetry://reference/edot-sdks/android/index.md) - - [.NET](opentelemetry://reference/edot-sdks/dotnet/setup/index.md) - - [iOS](opentelemetry://reference/edot-sdks/ios/index.md) - - [Java](opentelemetry://reference/edot-sdks/java/setup/index.md) - - [Node.js](opentelemetry://reference/edot-sdks/nodejs/setup/index.md) - - [PHP](opentelemetry://reference/edot-sdks/php/setup/index.md) - - [Python](opentelemetry://reference/edot-sdks/python/setup/index.md) + - [Android](apm-agent-android://reference/edot-android/index.md) + - [.NET](elastic-otel-dotnet://reference/edot-dotnet/setup/index.md) + - [iOS](apm-agent-ios://reference/edot-ios/index.md) + - [Java](elastic-otel-java://reference/edot-java/setup/index.md) + - [Node.js](elastic-otel-node://reference/edot-node/setup/index.md) + - [PHP](elastic-otel-php://reference/edot-php/setup/index.md) + - [Python](elastic-otel-python://reference/edot-python/setup/index.md) 4. Configure your EDOT SDK to send data to the APM endpoint. The **OpenTelemetry** tab provides the required configuration values. :::{note} @@ -78,13 +78,13 @@ If your application runs on Kubernetes, the OpenTelemetry operator automatically 2. Under **What do you want to monitor?** select **Application**, and then select **OpenTelemetry**. 3. Follow the instructions to install the EDOT SDK for your application: - - [Android](opentelemetry://reference/edot-sdks/android/index.md) - - [.NET](opentelemetry://reference/edot-sdks/dotnet/setup/index.md) - - [iOS](opentelemetry://reference/edot-sdks/ios/index.md) - - [Java](opentelemetry://reference/edot-sdks/java/setup/index.md) - - [Node.js](opentelemetry://reference/edot-sdks/nodejs/setup/index.md) - - [PHP](opentelemetry://reference/edot-sdks/php/setup/index.md) - - [Python](opentelemetry://reference/edot-sdks/python/setup/index.md) + - [Android](apm-agent-android://reference/edot-android/index.md) + - [.NET](elastic-otel-dotnet://reference/edot-dotnet/setup/index.md) + - [iOS](apm-agent-ios://reference/edot-ios/index.md) + - [Java](elastic-otel-java://reference/edot-java/setup/index.md) + - [Node.js](elastic-otel-node://reference/edot-node/setup/index.md) + - [PHP](elastic-otel-php://reference/edot-php/setup/index.md) + - [Python](elastic-otel-python://reference/edot-python/setup/index.md) 4. Configure your EDOT SDK to send data to the [Managed OTLP endpoint](opentelemetry://reference/motlp.md). The **OpenTelemetry** tab provides the required configuration values. :::{note} diff --git a/solutions/observability/get-started/quickstart-unified-kubernetes-observability-with-elastic-distributions-of-opentelemetry-edot.md b/solutions/observability/get-started/quickstart-unified-kubernetes-observability-with-elastic-distributions-of-opentelemetry-edot.md index f4629c086c..6ea6ea4fdb 100644 --- a/solutions/observability/get-started/quickstart-unified-kubernetes-observability-with-elastic-distributions-of-opentelemetry-edot.md +++ b/solutions/observability/get-started/quickstart-unified-kubernetes-observability-with-elastic-distributions-of-opentelemetry-edot.md @@ -72,7 +72,7 @@ For a more detailed description of the components and advanced configuration, re 3. Follow the on-screen instructions to install all needed components. ::::{note} - The default installation deploys the OpenTelemetry Operator with a self-signed TLS certificate valid for 365 days. This certificate **won’t be renewed** unless the Helm Chart release is manually updated. Refer to the [cert-manager integrated installation](opentelemetry://reference/use-cases/kubernetes/customization.md#cert-manager-integrated-installation) guide to enable automatic certificate generation and renewal using [cert-manager](https://cert-manager.io/docs/installation/). + The default installation deploys the OpenTelemetry Operator with a self-signed TLS certificate valid for 365 days. This certificate **won’t be renewed** unless the Helm Chart release is manually updated. Refer to the [cert-manager integrated installation](/solutions/observability/get-started/opentelemetry/use-cases/kubernetes/customization.md#cert-manager-integrated-installation) guide to enable automatic certificate generation and renewal using [cert-manager](https://cert-manager.io/docs/installation/). :::: @@ -102,7 +102,7 @@ For a more detailed description of the components and advanced configuration, re 4. Follow the on-screen instructions to install all needed components. ::::{note} - The default installation deploys the OpenTelemetry Operator with a self-signed TLS certificate valid for 365 days. This certificate **won’t be renewed** unless the Helm Chart release is manually updated. Refer to the [cert-manager integrated installation](opentelemetry://reference/use-cases/kubernetes/customization.md#cert-manager-integrated-installation) guide to enable automatic certificate generation and renewal using [cert-manager](https://cert-manager.io/docs/installation/). + The default installation deploys the OpenTelemetry Operator with a self-signed TLS certificate valid for 365 days. This certificate **won’t be renewed** unless the Helm Chart release is manually updated. Refer to the [cert-manager integrated installation](/solutions/observability/get-started/opentelemetry/use-cases/kubernetes/customization.md#cert-manager-integrated-installation) guide to enable automatic certificate generation and renewal using [cert-manager](https://cert-manager.io/docs/installation/). :::: @@ -164,7 +164,7 @@ Find **Machine Learning** in the main menu or use the [global search field](/exp ## Troubleshooting and more [_troubleshooting_and_more] -* To troubleshoot deployment and installation, refer to [installation verification](opentelemetry://reference/use-cases/kubernetes/deployment.md#verify-the-installation). -* For application instrumentation details, refer to [Instrumenting applications with EDOT SDKs on Kubernetes](opentelemetry://reference/use-cases/kubernetes/instrumenting-applications.md). -* To customize the configuration, refer to [custom configuration](opentelemetry://reference/use-cases/kubernetes/customization.md). +* To troubleshoot deployment and installation, refer to [installation verification](/solutions/observability/get-started/opentelemetry/use-cases/kubernetes/deployment.md#verify-the-installation). +* For application instrumentation details, refer to [Instrumenting applications with EDOT SDKs on Kubernetes](/solutions/observability/get-started/opentelemetry/use-cases/kubernetes/instrumenting-applications.md). +* To customize the configuration, refer to [custom configuration](/solutions/observability/get-started/opentelemetry/use-cases/kubernetes/customization.md). * Refer to [Observability overview](/solutions/observability/get-started/what-is-elastic-observability.md) for a description of other useful features. \ No newline at end of file diff --git a/solutions/observability/logs/stream-application-logs.md b/solutions/observability/logs/stream-application-logs.md index 2a640ebb2d..5ec31d7c75 100644 --- a/solutions/observability/logs/stream-application-logs.md +++ b/solutions/observability/logs/stream-application-logs.md @@ -104,11 +104,11 @@ Learn more about log correlation in the agent-specific ingestion guides: The {{edot}} (EDOT) provides SDKs for multiple programming languages with built-in support for log correlation: -* [Java](opentelemetry://reference/edot-sdks/java/index.md) -* [.NET](opentelemetry://reference/edot-sdks/dotnet/index.md) -* [Node.js](opentelemetry://reference/edot-sdks/nodejs/index.md) -* [PHP](opentelemetry://reference/edot-sdks/php/index.md) -* [Python](opentelemetry://reference/edot-sdks/python/index.md) +* [Java](elastic-otel-java://reference/edot-java/index.md) +* [.NET](elastic-otel-dotnet://reference/edot-dotnet/index.md) +* [Node.js](elastic-otel-node://reference/edot-node/index.md) +* [PHP](elastic-otel-php://reference/edot-php/index.md) +* [Python](elastic-otel-python://reference/edot-python/index.md) For more information about EDOT, refer to [Elastic Distribution of OpenTelemetry (EDOT)](opentelemetry://reference/index.md). diff --git a/solutions/toc.yml b/solutions/toc.yml index 971d790238..d7e204f1b1 100644 --- a/solutions/toc.yml +++ b/solutions/toc.yml @@ -100,6 +100,34 @@ toc: - file: observability/get-started/quickstart-monitor-hosts-with-elastic-agent.md - file: observability/get-started/quickstart-monitor-kubernetes-cluster-with-elastic-agent.md - file: observability/get-started/quickstart-collect-data-with-aws-firehose.md + - file: observability/get-started/opentelemetry/quickstart/index.md + children: + - file: observability/get-started/opentelemetry/quickstart/self-managed/index.md + children: + - file: observability/get-started/opentelemetry/quickstart/self-managed/k8s.md + - file: observability/get-started/opentelemetry/quickstart/self-managed/hosts_vms.md + - file: observability/get-started/opentelemetry/quickstart/self-managed/docker.md + - file: observability/get-started/opentelemetry/quickstart/serverless/index.md + children: + - file: observability/get-started/opentelemetry/quickstart/serverless/k8s.md + - file: observability/get-started/opentelemetry/quickstart/serverless/hosts_vms.md + - file: observability/get-started/opentelemetry/quickstart/serverless/docker.md + - file: observability/get-started/opentelemetry/quickstart/ech/index.md + children: + - file: observability/get-started/opentelemetry/quickstart/ech/k8s.md + - file: observability/get-started/opentelemetry/quickstart/ech/hosts_vms.md + - file: observability/get-started/opentelemetry/quickstart/ech/docker.md + - file: observability/get-started/opentelemetry/use-cases/index.md + children: + - file: observability/get-started/opentelemetry/use-cases/kubernetes/index.md + children: + - file: observability/get-started/opentelemetry/use-cases/kubernetes/prerequisites-compatibility.md + - file: observability/get-started/opentelemetry/use-cases/kubernetes/components.md + - file: observability/get-started/opentelemetry/use-cases/kubernetes/deployment.md + - file: observability/get-started/opentelemetry/use-cases/kubernetes/instrumenting-applications.md + - file: observability/get-started/opentelemetry/use-cases/kubernetes/upgrade.md + - file: observability/get-started/opentelemetry/use-cases/kubernetes/customization.md + - folder: observability/get-started/opentelemetry/use-cases/llms - file: observability/get-started/other-tutorials/index.md children: - file: observability/get-started/other-tutorials/tutorial-monitor-java-application.md diff --git a/troubleshoot/ingest/opentelemetry/contact-support.md b/troubleshoot/ingest/opentelemetry/contact-support.md index b79928939c..57a4a779ec 100644 --- a/troubleshoot/ingest/opentelemetry/contact-support.md +++ b/troubleshoot/ingest/opentelemetry/contact-support.md @@ -48,7 +48,7 @@ To help Elastic Support investigate the problem efficiently, please include the ### Deployment context -* Are you using a [standalone EDOT Collector](opentelemetry://reference/edot-collector/config/default-config-standalone.md) or [Kubernetes](opentelemetry://reference/edot-collector/config/default-config-k8s.md)? +* Are you using a [standalone EDOT Collector](elastic-agent://reference/edot-collector/config/default-config-standalone.md) or [Kubernetes](elastic-agent://reference/edot-collector/config/default-config-k8s.md)? * If applicable, include: * Helm chart version and values (for Kubernetes) * Container image version diff --git a/troubleshoot/ingest/opentelemetry/edot-collector/collector-not-starting.md b/troubleshoot/ingest/opentelemetry/edot-collector/collector-not-starting.md index c3d7636011..3fee510ead 100644 --- a/troubleshoot/ingest/opentelemetry/edot-collector/collector-not-starting.md +++ b/troubleshoot/ingest/opentelemetry/edot-collector/collector-not-starting.md @@ -79,7 +79,7 @@ If you're deploying the EDOT Collector in a standalone configuration, try to: * **For managed OTLP endpoint**: use relevant receivers and export using the `otlp` exporter - Refer to [Default configuration of the EDOT Collector (Standalone)](opentelemetry://reference/edot-collector/config/default-config-standalone.md) for full examples for each use case. + Refer to [Default configuration of the EDOT Collector (Standalone)](elastic-agent://reference/edot-collector/config/default-config-standalone.md) for full examples for each use case. * Check for port conflicts diff --git a/troubleshoot/ingest/opentelemetry/edot-sdks/android/index.md b/troubleshoot/ingest/opentelemetry/edot-sdks/android/index.md index 8f60c94161..95bc31be76 100644 --- a/troubleshoot/ingest/opentelemetry/edot-sdks/android/index.md +++ b/troubleshoot/ingest/opentelemetry/edot-sdks/android/index.md @@ -17,7 +17,7 @@ mapped_pages: # Troubleshooting the EDOT Android SDK -Use the information in this section to troubleshoot common problems. As a first step, make sure your stack is compatible with the [supported technologies](opentelemetry://reference/edot-sdks/android/getting-started.md#requirements) for EDOT Android and the OpenTelemetry SDK. +Use the information in this section to troubleshoot common problems. As a first step, make sure your stack is compatible with the [supported technologies](apm-agent-android://reference/edot-android/getting-started.md#requirements) for EDOT Android and the OpenTelemetry SDK. If you have an Elastic support contract, create a ticket in the [Elastic Support portal](https://support.elastic.co/customers/s/login/). If you don't, post in the [APM discuss forum](https://discuss.elastic.co/c/apm) or [open a GitHub issue](https://github.com/elastic/apm-agent-android/issues). @@ -25,11 +25,11 @@ If you have an Elastic support contract, create a ticket in the [Elastic Support The SDK creates logs that allow you to see what it's working on and what might have failed at some point. You can find the logs in [logcat](https://developer.android.com/studio/debug/logcat), filtered by the tag `ELASTIC_AGENT`. -For more information about the SDK's internal logs, as well as how to configure them, refer to the [internal logging policy](opentelemetry://reference/edot-sdks/android/configuration.md#internal-logging-policy) configuration. +For more information about the SDK's internal logs, as well as how to configure them, refer to the [internal logging policy](apm-agent-android://reference/edot-android/configuration.md#internal-logging-policy) configuration. ## Connectivity to the {{stack}} -If after following the [getting started](opentelemetry://reference/edot-sdks/android/getting-started.md) guide and configuring your {{stack}} [endpoint parameters](opentelemetry://reference/edot-sdks/android/configuration.md#export-connectivity), you can't see your application's data in {{kib}}, you can follow the following tips to try and figure out what could be wrong. +If after following the [getting started](apm-agent-android://reference/edot-android/getting-started.md) guide and configuring your {{stack}} [endpoint parameters](apm-agent-android://reference/edot-android/configuration.md#export-connectivity), you can't see your application's data in {{kib}}, you can follow the following tips to try and figure out what could be wrong. ### Check out the logs @@ -151,7 +151,7 @@ Your export endpoint URL is the value for the `OTEL_EXPORTER_OTLP_ENDPOINT` conf ### Cloud hosted and self-managed deployments -For Elastic Cloud Hosted (ECH) and self-managed deployments, the export endpoint, also known as [EDOT Collector](opentelemetry://reference/edot-collector/index.md), is not available out of the box at the moment. You can still create your own service by [downloading](opentelemetry://reference/edot-collector/download.md) and [configuring an EDOT Collector](opentelemetry://reference/edot-collector/config/default-config-standalone.md#gateway-mode). +For Elastic Cloud Hosted (ECH) and self-managed deployments, the export endpoint, also known as [EDOT Collector](opentelemetry://reference/edot-collector/index.md), is not available out of the box at the moment. You can still create your own service by [downloading](elastic-agent://reference/edot-collector/download.md) and [configuring an EDOT Collector](elastic-agent://reference/edot-collector/config/default-config-standalone.md#gateway-mode). ### Local testing deployment diff --git a/troubleshoot/ingest/opentelemetry/edot-sdks/dotnet/index.md b/troubleshoot/ingest/opentelemetry/edot-sdks/dotnet/index.md index 555add1760..e8884141e5 100644 --- a/troubleshoot/ingest/opentelemetry/edot-sdks/dotnet/index.md +++ b/troubleshoot/ingest/opentelemetry/edot-sdks/dotnet/index.md @@ -15,7 +15,7 @@ products: # Troubleshooting the EDOT .NET SDK -Use the information in this section to troubleshoot common problems. As a first step, make sure your stack is compatible with the [supported technologies](opentelemetry://reference/edot-sdks/dotnet/supported-technologies.md) for EDOT .NET and the OpenTelemetry SDK. +Use the information in this section to troubleshoot common problems. As a first step, make sure your stack is compatible with the [supported technologies](elastic-otel-dotnet://reference/edot-dotnet/supported-technologies.md) for EDOT .NET and the OpenTelemetry SDK. If you have an Elastic support contract, create a ticket in the [Elastic Support portal](https://support.elastic.co/customers/s/login/). If you don't, post in the [APM discuss forum](https://discuss.elastic.co/c/apm) or [open a GitHub issue](https://github.com/elastic/elastic-otel-dotnet/issues). diff --git a/troubleshoot/ingest/opentelemetry/edot-sdks/enable-debug-logging.md b/troubleshoot/ingest/opentelemetry/edot-sdks/enable-debug-logging.md index 02aa946cbf..de6b4475af 100644 --- a/troubleshoot/ingest/opentelemetry/edot-sdks/enable-debug-logging.md +++ b/troubleshoot/ingest/opentelemetry/edot-sdks/enable-debug-logging.md @@ -141,7 +141,7 @@ While the EDOT PHP agent supports the standard `OTEL_LOG_LEVEL` variable, you mu * `ELASTIC_OTEL_LOG_LEVEL_STDERR` * `ELASTIC_OTEL_LOG_LEVEL_SYSLOG` -Refer to [Logging configuration](opentelemetry://reference/edot-sdks/php/configuration.md#logging-configuration) for more details. +Refer to [Logging configuration](elastic-otel-php://reference/edot-php/configuration.md#logging-configuration) for more details. For deeper troubleshooting, you can also enable diagnostic data collection. For example: diff --git a/troubleshoot/ingest/opentelemetry/edot-sdks/java/index.md b/troubleshoot/ingest/opentelemetry/edot-sdks/java/index.md index 0ec73cdbfa..1ae64b09e3 100644 --- a/troubleshoot/ingest/opentelemetry/edot-sdks/java/index.md +++ b/troubleshoot/ingest/opentelemetry/edot-sdks/java/index.md @@ -15,7 +15,7 @@ products: # Troubleshooting the EDOT Java Agent -Use the information in this section to troubleshoot common problems. As a first step, make sure your stack is compatible with the [supported technologies](opentelemetry://reference/edot-sdks/java/supported-technologies.md) for EDOT Java and the OpenTelemetry SDK. +Use the information in this section to troubleshoot common problems. As a first step, make sure your stack is compatible with the [supported technologies](elastic-otel-java://reference/edot-java/supported-technologies.md) for EDOT Java and the OpenTelemetry SDK. If you need help and you're an existing Elastic customer with a support contract, create a ticket in the [Elastic Support portal](https://support.elastic.co/customers/s/login/). Other users can post in the [APM discuss forum](https://discuss.elastic.co/c/apm) or [open a GitHub issue](https://github.com/elastic/elastic-otel-node/issues) @@ -52,7 +52,7 @@ As debugging output is verbose and might produce noticeable overhead on the appl - In case of a technical issue or exception with the agent, use [agent debugging](#agent-debugging). - If you need details on the captured data, use [per-signal debugging](#per-signal-debugging). -In case of missing data, check first that the technology used in the application is supported in [OpenTelemetry Java Instrumentation](https://github.com/open-telemetry/opentelemetry-java-instrumentation/blob/main/docs/supported-libraries.md) and in [EDOT Java](opentelemetry://reference/edot-sdks/java/supported-technologies.md). +In case of missing data, check first that the technology used in the application is supported in [OpenTelemetry Java Instrumentation](https://github.com/open-telemetry/opentelemetry-java-instrumentation/blob/main/docs/supported-libraries.md) and in [EDOT Java](elastic-otel-java://reference/edot-java/supported-technologies.md). ### Agent debugging @@ -162,4 +162,4 @@ Updates of the OpenTelemetry API/SDK in the application and the EDOT Java agent ### How to update -Updating EDOT Java agent is done by replacing the agent binary `.jar` that has been [added during setup](opentelemetry://reference/edot-sdks/java/setup/index.md). +Updating EDOT Java agent is done by replacing the agent binary `.jar` that has been [added during setup](elastic-otel-java://reference/edot-java/setup/index.md). diff --git a/troubleshoot/ingest/opentelemetry/edot-sdks/nodejs/index.md b/troubleshoot/ingest/opentelemetry/edot-sdks/nodejs/index.md index 64ae7815ff..c59cb3a001 100644 --- a/troubleshoot/ingest/opentelemetry/edot-sdks/nodejs/index.md +++ b/troubleshoot/ingest/opentelemetry/edot-sdks/nodejs/index.md @@ -19,7 +19,7 @@ Use the information on this page to troubleshoot issues using EDOT Node.js. If you need help and you're an existing Elastic customer with a support contract, create a ticket in the [Elastic Support portal](https://support.elastic.co/customers/s/login/). Other users can post in the [APM discuss forum](https://discuss.elastic.co/c/apm) or [open a GitHub issue](https://github.com/elastic/elastic-otel-node/issues). -As a first step, review the [supported technologies](opentelemetry://reference/edot-sdks/nodejs/supported-technologies.md) to ensure your application is supported by the SDK. Are you using a Node.js version that the SDK supports? Are the versions of your dependencies in the [supported version range](opentelemetry://reference/edot-sdks/nodejs/supported-technologies.md#instrumentations) to be instrumented? +As a first step, review the [supported technologies](elastic-otel-node://reference/edot-node/supported-technologies.md) to ensure your application is supported by the SDK. Are you using a Node.js version that the SDK supports? Are the versions of your dependencies in the [supported version range](elastic-otel-node://reference/edot-node/supported-technologies.md#instrumentations) to be instrumented? ## Set a service name @@ -35,7 +35,7 @@ curl -i $ELASTIC_OTLP_ENDPOINT \ -H "Authorization: ApiKey $ELASTIC_API_KEY" ``` -For example, if you [configured](opentelemetry://reference/edot-sdks/nodejs/configuration.md#basic-configuration) EDOT Node.js with: +For example, if you [configured](elastic-otel-node://reference/edot-node/configuration.md#basic-configuration) EDOT Node.js with: ```bash export OTEL_EXPORTER_OTLP_ENDPOINT="https://my-deployment-abc123.ingest.us-west-2.aws.elastic.cloud" @@ -108,7 +108,7 @@ Look for warnings (`"level":40`) or errors (`"level":50`) in the log output that ## Deactivate an instrumentation -To deactivate an instrumentation, set the [`OTEL_NODE_DISABLED_INSTRUMENTATIONS`](opentelemetry://reference/edot-sdks/nodejs/configuration.md#otel_node_disabledenabled_instrumentations-details) environment variable. +To deactivate an instrumentation, set the [`OTEL_NODE_DISABLED_INSTRUMENTATIONS`](elastic-otel-node://reference/edot-node/configuration.md#otel_node_disabledenabled_instrumentations-details) environment variable. For example, to deactivate `@opentelemetry/instrumentation-net` and `@opentelemetry/instrumentation-dns` run the following commands: diff --git a/troubleshoot/ingest/opentelemetry/edot-sdks/php/index.md b/troubleshoot/ingest/opentelemetry/edot-sdks/php/index.md index b5c67b8469..c2c7eaf282 100644 --- a/troubleshoot/ingest/opentelemetry/edot-sdks/php/index.md +++ b/troubleshoot/ingest/opentelemetry/edot-sdks/php/index.md @@ -19,11 +19,11 @@ Use the information on this page to troubleshoot issues using EDOT PHP. If you need help and you're an existing Elastic customer with a support contract, create a ticket in the [Elastic Support portal](https://support.elastic.co/customers/s/login/). Other users can post in the [APM discuss forum](https://discuss.elastic.co/c/apm) or [open a GitHub issue](https://github.com/elastic/elastic-otel-node/issues). -As a first step, review the [supported technologies](opentelemetry://reference/edot-sdks/php/supported-technologies.md) to ensure your application is supported by the agent. Are you using a PHP version that EDOT PHP supports? Are the versions of your dependencies in the supported version range to be instrumented? +As a first step, review the [supported technologies](elastic-otel-php://reference/edot-php/supported-technologies.md) to ensure your application is supported by the agent. Are you using a PHP version that EDOT PHP supports? Are the versions of your dependencies in the supported version range to be instrumented? ## Turn on logging -When diagnosing issues with the agent's operation, logs play a key role. You can find a detailed explanation of the logging configuration options in [Configuration](opentelemetry://reference/edot-sdks/php/configuration.md#logging-configuration). +When diagnosing issues with the agent's operation, logs play a key role. You can find a detailed explanation of the logging configuration options in [Configuration](elastic-otel-php://reference/edot-php/configuration.md#logging-configuration). In most cases, setting the logging level to `debug` is sufficient. You can also use `trace` can be used, but keep in mind that the amount of generated data might be significant. @@ -42,7 +42,7 @@ Upload your complete debug logs to a service like [GitHub Gist](https://gist.git If you suspect that the agent might be causing disruptions to a production application, you can deactivate the agent while you troubleshoot. -To deactivate the agent, set the [`elastic_otel.enabled`](opentelemetry://reference/edot-sdks/php/configuration.md#general-configuration) setting to `false`. +To deactivate the agent, set the [`elastic_otel.enabled`](elastic-otel-php://reference/edot-php/configuration.md#general-configuration) setting to `false`. :::{note} You need to restart your application for the changes to apply. @@ -75,7 +75,7 @@ When turned off, the agent falls back to a PHP-based serializer, which has lower ### `open_basedir` PHP configuration option -If you see a similar entry in the agent log, this indicates an incorrect `open_basedir` configuration. For more details, refer to [Limitations](opentelemetry://reference/edot-sdks/php/setup/limitations.md#open_basedir-php-configuration-option). +If you see a similar entry in the agent log, this indicates an incorrect `open_basedir` configuration. For more details, refer to [Limitations](elastic-otel-php://reference/edot-php/setup/limitations.md#open_basedir-php-configuration-option). ``` EDOT PHP bootstrap file (...php/bootstrap_php_part.php) is located outside of paths allowed by open_basedir ini setting. diff --git a/troubleshoot/ingest/opentelemetry/edot-sdks/proxy.md b/troubleshoot/ingest/opentelemetry/edot-sdks/proxy.md new file mode 100644 index 0000000000..dee09ba411 --- /dev/null +++ b/troubleshoot/ingest/opentelemetry/edot-sdks/proxy.md @@ -0,0 +1,45 @@ +--- +navigation_title: Proxy settings +description: Configuration of the EDOT SDKs proxy settings. +applies_to: + stack: + serverless: + observability: + product: + edot_collector: ga +products: + - id: cloud-serverless + - id: observability + - id: edot-collector +--- + +# Proxy settings for EDOT SDKs + +EDOT SDKs generally use the standard proxy environment variables. However, there are exceptions and limitations depending on the language and exporter type. + +## Python SDK + +The Python SDK honors `HTTP_PROXY` and `HTTPS_PROXY` only when using the `http/protobuf` exporter. + +If you use the default gRPC-based exporter, proxy settings are ignored. To enable proxy support, you can either: + +* Switch to the `http/protobuf` exporter, which supports standard proxy environment variables. + +* Route telemetry through a local EDOT Collector configured with proxy settings. + +## Node.js SDK + +The Node.js SDK does not currently support `HTTP_PROXY`, `HTTPS_PROXY`, or `NO_PROXY` by default. You can route telemetry through an EDOT Collector. + +## Java SDK + +If you’re using Java SDK, you must configure Java system properties using the Java Virtual Machine (JVM). Refer to [Troubleshooting Java SDK proxy issues](/troubleshoot/ingest/opentelemetry/edot-sdks/java/proxy-issues.md) for more information. + +## Other SDKs + +Other EDOT SDKs honor standard proxy environment variables with no additional setup required. For example: + +``` +export HTTP_PROXY=http://: +export HTTPS_PROXY=http://: +``` \ No newline at end of file diff --git a/troubleshoot/ingest/opentelemetry/edot-sdks/python/index.md b/troubleshoot/ingest/opentelemetry/edot-sdks/python/index.md index c59f206a39..c09b799c02 100644 --- a/troubleshoot/ingest/opentelemetry/edot-sdks/python/index.md +++ b/troubleshoot/ingest/opentelemetry/edot-sdks/python/index.md @@ -19,7 +19,7 @@ Use the information on this page to troubleshoot issues using EDOT Python. If you need help and you're an existing Elastic customer with a support contract, create a ticket in the [Elastic Support portal](https://support.elastic.co/customers/s/login/). Other users can post in the [APM discuss forum](https://discuss.elastic.co/c/apm) or [open a GitHub issue](https://github.com/elastic/elastic-otel-node/issues). -As a first step, review the [supported technologies](opentelemetry://reference/edot-sdks/python/supported-technologies.md) to ensure your application is supported by the agent. Are you using a Python version that EDOT Python supports? Are the versions of your dependencies in the supported version range to be instrumented? +As a first step, review the [supported technologies](elastic-otel-python://reference/edot-python/supported-technologies.md) to ensure your application is supported by the agent. Are you using a Python version that EDOT Python supports? Are the versions of your dependencies in the supported version range to be instrumented? ## General troubleshooting diff --git a/troubleshoot/ingest/opentelemetry/toc.yml b/troubleshoot/ingest/opentelemetry/toc.yml index 7cee5d7992..56d4490681 100644 --- a/troubleshoot/ingest/opentelemetry/toc.yml +++ b/troubleshoot/ingest/opentelemetry/toc.yml @@ -20,5 +20,6 @@ toc: - file: edot-sdks/php/index.md - file: edot-sdks/python/index.md - file: edot-sdks/enable-debug-logging.md + - file: edot-sdks/proxy.md - file: no-data-in-kibana.md - file: contact-support.md diff --git a/troubleshoot/observability/apm-agent-swift/apm-ios-agent.md b/troubleshoot/observability/apm-agent-swift/apm-ios-agent.md index a6f1a5c900..3de6326cdc 100644 --- a/troubleshoot/observability/apm-agent-swift/apm-ios-agent.md +++ b/troubleshoot/observability/apm-agent-swift/apm-ios-agent.md @@ -10,7 +10,7 @@ products: # Troubleshoot APM iOS Agent [troubleshooting] -Is something not working as expected? Don’t worry if you can’t figure out what the problem is; we’re here to help! As a first step, ensure your app is compatible with the agent’s [supported technologies](opentelemetry://reference/edot-sdks/ios/supported-technologies.md). +Is something not working as expected? Don’t worry if you can’t figure out what the problem is; we’re here to help! As a first step, ensure your app is compatible with the agent’s [supported technologies](apm-agent-ios://reference/edot-ios/supported-technologies.md). If you’re an existing Elastic customer with a support contract, create a ticket in the [Elastic Support portal](https://support.elastic.co/customers/s/login/). Other users can post in the [APM discuss forum](https://discuss.elastic.co/c/apm).