Skip to content

Commit 27fc926

Browse files
Merge branch 'main' into 2493-add-info-agent-logging-level
2 parents 9c5ddbb + 53a7eb2 commit 27fc926

File tree

8 files changed

+206
-20
lines changed

8 files changed

+206
-20
lines changed
Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,6 @@
11
:::{dropdown} Can I use Cloud Connect to connect my {{ech}} clusters to AutoOps?
22

33
This path is not supported. Currently, we only support using Cloud Connect to connect ECE, ECK, and self-managed clusters to AutoOps.
4+
5+
For {{ech}} clusters, AutoOps is set up and enabled automatically in all supported [regions](/deploy-manage/monitor/autoops/ec-autoops-regions.md), and can be [accessed](/deploy-manage/monitor/autoops/ec-autoops-how-to-access.md) from the deployment overview page.
46
:::

deploy-manage/monitor/autoops.md

Lines changed: 0 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -12,8 +12,6 @@ products:
1212

1313
AutoOps diagnoses issues in {{es}} by analyzing hundreds of metrics, providing root-cause analysis and accurate resolution paths. With AutoOps, customers can prevent and resolve issues, cut down administration time, and optimize resource utilization.
1414

15-
AutoOps is currently only available for [{{ech}} deployments](/deploy-manage/deploy/elastic-cloud/cloud-hosted.md).
16-
1715
:::{image} /deploy-manage/images/cloud-autoops-overview-page.png
1816
:alt: The Overview page
1917
:::

deploy-manage/monitor/autoops/cc-autoops-as-cloud-connected.md

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -17,7 +17,8 @@ Cloud Connect enables users of ECE, ECK, and self-managed clusters to use {{eclo
1717

1818
In this section, you'll find the following information:
1919

20-
* How to [connect your cluster to AutoOps](/deploy-manage/monitor/autoops/cc-connect-self-managed-to-autoops.md)
20+
* How to [connect your self-managed cluster to AutoOps](/deploy-manage/monitor/autoops/cc-connect-self-managed-to-autoops.md)
21+
* How to [connect your local development cluster to AutoOps](/deploy-manage/monitor/autoops/cc-connect-local-dev-to-autoops.md)
2122
* How to [manage users of your connected clusters](/deploy-manage/monitor/autoops/cc-manage-users.md)
2223
* [FAQs](/deploy-manage/monitor/autoops/cc-cloud-connect-autoops-faq.md) about AutoOps for your clusters
2324
* A [troubleshooting guide](/deploy-manage/monitor/autoops/cc-cloud-connect-autoops-troubleshooting.md) to help you with any issues you may encounter
Lines changed: 59 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,59 @@
1+
---
2+
applies_to:
3+
deployment:
4+
self:
5+
navigation_title: Connect your local development cluster
6+
---
7+
8+
# Connect your local development cluster to AutoOps
9+
10+
If you have an {{es}} cluster set up for local development or testing, you can connect it to AutoOps using Docker.
11+
12+
## Prerequisites
13+
14+
Ensure your system meets the following requirements before proceeding:
15+
16+
* You have set up [{{es}} for local development](/deploy-manage/deploy/self-managed/local-development-installation-quickstart.md).
17+
* You have installed [Docker Desktop](https://www.docker.com/products/docker-desktop).
18+
* You have an {{ecloud}} account with the [Organization owner role](/deploy-manage/monitor/autoops/cc-manage-users.md#assign-roles).
19+
20+
## Connect your local development cluster to AutoOps
21+
22+
Complete the following steps to connect your local development cluster to AutoOps.
23+
24+
1. Run the following command in your terminal to open the `/etc/hosts` file in a text editor with administrator privileges:
25+
26+
```sh
27+
vim /etc/hosts
28+
```
29+
2. On a new line in the `/etc/hosts` file, add an entry to map the {{es}} cluster URL to the IP address representing the local development cluster.
30+
31+
The entry should be formatted as `127.0.0.1 {{hostname}}`.
32+
3. Save the changes.
33+
4. In your terminal, run the following command to reload the hostname service:
34+
* For Linux:
35+
```sh
36+
/bin/systemctl restart systemd-hostnamed
37+
```
38+
* For macOS:
39+
```sh
40+
sudo dscacheutil -flushcache
41+
```
42+
5. Follow the instructions to [Connect to AutoOps](/deploy-manage/monitor/autoops/cc-connect-self-managed-to-autoops.md#connect-to-autoops) with the following differences:
43+
* In the [Select installation method](/deploy-manage/monitor/autoops/cc-connect-self-managed-to-autoops.md#select-installation-method) step, select **Docker**.
44+
* In the [Configure agent](/deploy-manage/monitor/autoops/cc-connect-self-managed-to-autoops.md#configure-agent) step, when prompted to enter your **{{es}} endpoint URL**, enter the name of your local development cluster or enter the following:
45+
```sh
46+
http://localhost:9200
47+
```
48+
* In the [Install agent](/deploy-manage/monitor/autoops/cc-connect-self-managed-to-autoops.md#install-agent) step, paste the command into the text editor and replace `docker run -d \` with:
49+
```sh
50+
docker run -d --network host \
51+
```
52+
53+
This replacement is also required if your cluster is running on macOS.
54+
55+
After completing all the steps, you can [Access AutoOps](/deploy-manage/monitor/autoops/cc-connect-self-managed-to-autoops.md#access-autoops).
56+
57+
58+
59+

deploy-manage/monitor/autoops/cc-connect-self-managed-to-autoops.md

Lines changed: 21 additions & 17 deletions
Original file line numberDiff line numberDiff line change
@@ -4,15 +4,19 @@ applies_to:
44
self:
55
ece:
66
eck:
7-
navigation_title: Connect your cluster
7+
navigation_title: Connect your self-managed cluster
88
---
99

10-
# Connect your cluster to AutoOps
10+
# Connect your self-managed cluster to AutoOps
1111

1212
To use AutoOps with your ECE, ECK, or self-managed cluster, you first need to create an {{ecloud}} account or log in to your existing account. An installation wizard will then guide you through the steps of installing {{agent}} to send metrics from your cluster to AutoOps in {{ecloud}}.
1313

1414
Complete the steps in the following subsections to connect your cluster to AutoOps. The connection process takes about 10 minutes.
1515

16+
:::{note}
17+
If you have an {{es}} cluster set up for local development or testing, you can connect it to AutoOps using Docker. Refer to [](/deploy-manage/monitor/autoops/cc-connect-local-dev-to-autoops.md).
18+
:::
19+
1620
## Prerequisites
1721

1822
Ensure your system meets the following requirements before proceeding:
@@ -21,7 +25,7 @@ Ensure your system meets the following requirements before proceeding:
2125
* Your cluster is on an [Enterprise self-managed license](https://www.elastic.co/subscriptions) or an active self-managed [trial](https://cloud.elastic.co/registration).
2226
* The agent you install for the connection is allowed to send metrics to {{ecloud}}.
2327

24-
## Connect to AutoOps
28+
## Connect to AutoOps (private preview) [connect-to-autoops]
2529

2630
:::{note}
2731
:::{include} /deploy-manage/monitor/_snippets/single-cloud-org.md
@@ -86,8 +90,8 @@ This is the first step of the installation wizard. Your cluster ships metrics to
8690

8791
Select one of the following methods to install {{agent}}:
8892

89-
* Kubernetes
90-
* Docker
93+
* **Kubernetes**
94+
* **Docker**
9195
<!-- Not applicable for private preview
9296
* Linux
9397
* Windows
@@ -226,18 +230,18 @@ Complete the following steps to run the command:
226230
1. Copy the command.
227231
2. Paste it into a text editor and update the placeholder values in the following environment variables:
228232

229-
| Environment variable | Description |
230-
| --- | --- |
231-
| `AUTOOPS_OTEL_URL` | The {{ecloud}} URL to which {{agent}} ships data. The URL is generated based on the CSP and region you pick. <br> This URL shouldn't be edited. |
232-
| `AUTOOPS_ES_URL` | The URL {{agent}} uses to communicate with {{es}}. |
233-
| `ELASTICSEARCH_READ_API_KEY` | The API key for API key authentication to access the cluster. It combines the `${id}:${api_key}` values. <br> This variable shouldn't be used with `ELASTICSEARCH_READ_USERNAME` and `ELASTICSEARCH_READ_PASSWORD`. |
234-
| `ELASTICSEARCH_READ_USERNAME` | The username for basic authentication to access the cluster. <br> This variable should be used with `ELASTICSEARCH_READ_PASSWORD`. |
235-
| `ELASTICSEARCH_READ_PASSWORD` | The password for basic authentication to access the cluster. <br> This variable should be used with `ELASTICSEARCH_READ_USERNAME`. |
236-
| `ELASTIC_CLOUD_CONNECTED_MODE_API_KEY` | The {{ecloud}} API Key used to register the cluster. <br> This key shouldn't be edited. |
237-
| `AUTOOPS_TEMP_RESOURCE_ID` | The temporary ID for the current installation wizard. |
238-
239-
4. Run the command from the machine where you want to install the agent.
240-
5. Return to the wizard and select **I have run the command**.
233+
| Environment variable | Description |
234+
| --- | --- |
235+
| `AUTOOPS_OTEL_URL` | The {{ecloud}} URL to which {{agent}} ships data. The URL is generated based on the CSP and region you pick. <br> This URL shouldn't be edited. |
236+
| `AUTOOPS_ES_URL` | The URL {{agent}} uses to communicate with {{es}}. |
237+
| `ELASTICSEARCH_READ_API_KEY` | The API key for API key authentication to access the cluster. It combines the `${id}:${api_key}` values. <br> This variable shouldn't be used with `ELASTICSEARCH_READ_USERNAME` and `ELASTICSEARCH_READ_PASSWORD`. |
238+
| `ELASTICSEARCH_READ_USERNAME` | The username for basic authentication to access the cluster. <br> This variable should be used with `ELASTICSEARCH_READ_PASSWORD`. |
239+
| `ELASTICSEARCH_READ_PASSWORD` | The password for basic authentication to access the cluster. <br> This variable should be used with `ELASTICSEARCH_READ_USERNAME`. |
240+
| `ELASTIC_CLOUD_CONNECTED_MODE_API_KEY` | The {{ecloud}} API Key used to register the cluster. <br> This key shouldn't be edited. |
241+
| `AUTOOPS_TEMP_RESOURCE_ID` | The temporary ID for the current installation wizard. |
242+
243+
3. Run the command from the machine where you want to install the agent.
244+
4. Return to the wizard and select **I have run the command**.
241245

242246
It might take a few minutes for your cluster details to be validated and the first metrics to be shipped to AutoOps.
243247

deploy-manage/toc.yml

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -676,6 +676,7 @@ toc:
676676
- file: monitor/autoops/ec-autoops-faq.md
677677
- hidden: monitor/autoops/cc-autoops-as-cloud-connected.md
678678
- hidden: monitor/autoops/cc-connect-self-managed-to-autoops.md
679+
- hidden: monitor/autoops/cc-connect-local-dev-to-autoops.md
679680
- hidden: monitor/autoops/cc-manage-users.md
680681
- hidden: monitor/autoops/cc-cloud-connect-autoops-faq.md
681682
- hidden: monitor/autoops/cc-cloud-connect-autoops-troubleshooting.md
Lines changed: 120 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,120 @@
1+
---
2+
navigation_title: Insufficient resources with Kube-Stack chart
3+
description: Learn what to do when the Kube-Stack chart is deployed with insufficient resources.
4+
applies_to:
5+
stack:
6+
serverless:
7+
observability:
8+
product:
9+
edot_collector: ga
10+
products:
11+
- id: cloud-serverless
12+
- id: observability
13+
- id: edot-collector
14+
---
15+
16+
# Insufficient resources issue with Kube-Stack Helm Chart
17+
18+
The OpenTelemetry Kube-Stack Helm Chart deploys multiple EDOT collectors with varying configurations based on the selected architecture and deployment mode. On larger clusters, the default Kubernetes resource limits might be insufficient.
19+
20+
## Symptoms
21+
22+
These symptoms are common when the Kube-Stack chart is deployed with insufficient resources:
23+
24+
- Collector Pods in a `CrashLoopBackOff`/`OOMKilled` state.
25+
- Cluster or Daemon pods are unable to export data to the Gateway collector due being `OOMKilled` (high memory usage).
26+
- Pods have logs similar to: `error internal/queue_sender.go:128 Exporting failed. Dropping data.`
27+
28+
## Resolution
29+
30+
Follow these steps to resolve the issue.
31+
32+
:::::{stepper}
33+
34+
::::{step} Check for OOMKilled Pods
35+
Run the following command to check the Pods:
36+
37+
```bash
38+
kubectl get pods -n opentelemetry-operator-system
39+
```
40+
41+
Look for any Pods in the `OOMKilled` state:
42+
43+
```
44+
NAME READY STATUS RESTARTS AGE
45+
opentelemetry-kube-stack-cluster-stats-collector-7cd88c77drvj76 1/1 Running 0 49s
46+
opentelemetry-kube-stack-daemon-collector-pn4qj 1/1 Running 0 47s
47+
opentelemetry-kube-stack-gateway-collector-85795c7965-wxqls 0/1 OOMKilled 3 (34s ago) 49s
48+
opentelemetry-kube-stack-gateway-collector-8cfdb59df-lgpbr 0/1 OOMKilled 3 (30s ago) 49s
49+
opentelemetry-kube-stack-gateway-collector-8cfdb59df-s7plz 0/1 CrashLoopBackOff 2 (17s ago) 34s
50+
opentelemetry-kube-stack-opentelemetry-operator-77d46bc4dbv2h6k 2/2 Running 0 3m14s
51+
```
52+
::::
53+
54+
::::{step} Verify the Pod last status
55+
56+
Run the following command to verify the last status of the Pod:
57+
58+
```bash
59+
kubectl describe pod -n opentelemetry-operator-system opentelemetry-kube-stack-gateway-collector-85795c7965-wxqls
60+
61+
State: Waiting
62+
Reason: CrashLoopBackOff
63+
Last State: Terminated
64+
Reason: OOMKilled
65+
Exit Code: 137
66+
```
67+
::::
68+
69+
::::{step} Increase resource limits
70+
71+
Edit the `values.yaml` file used to deploy the corresponding Helm release. For the Gateway collector, ensure horitzontal Pod autoscaling is turned on. The Gateway collector configuration should be similar to this:
72+
73+
```yaml
74+
gateway:
75+
fullnameOverride: "opentelemetry-kube-stack-gateway"
76+
suffix: gateway
77+
replicas: 2
78+
autoscaler:
79+
minReplicas: 2 # Start with at least 2 replicas for better availability.
80+
maxReplicas: 5 # Allow more scale-out if needed.
81+
targetCPUUtilization: 70 # Scale when CPU usage exceeds 70%.
82+
targetMemoryUtilization: 75 # Scale when memory usage exceeds 75%.
83+
```
84+
85+
If the autoscaler configuration is already available, or another Collector type is running out of memory, increase the resource limits in the corresponding Collector configuration section:
86+
87+
```yaml
88+
gateway:
89+
fullnameOverride: "opentelemetry-kube-stack-gateway"
90+
...
91+
resources:
92+
limits:
93+
cpu: 500m
94+
memory: 20Mi
95+
requests:
96+
cpu: 100m
97+
memory: 10Mi
98+
```
99+
100+
Make sure to update the resource limits within the correct Collector type section. Available types are: `gateway`, `daemon`, `cluster`, and `opentelemetry-operator`.
101+
::::
102+
103+
::::{step} Update the Helm release
104+
105+
Run the following command to update the Helm release:
106+
107+
```bash
108+
$ helm upgrade opentelemetry-kube-stack open-telemetry/opentelemetry-kube-stack --namespace opentelemetry-operator-system --values values.yaml --version '0.6.3'
109+
```
110+
111+
:::{note}
112+
The hard memory limit should be around 2GB.
113+
:::
114+
::::
115+
:::::
116+
117+
## Resources
118+
119+
* [Elastic Kube-stack Helm chart](https://github.com/open-telemetry/opentelemetry-helm-charts/tree/main/charts/opentelemetry-kube-stack)
120+
* [Elastic stack Kubernetes Helm charts](https://github.com/elastic/helm-charts)

troubleshoot/ingest/opentelemetry/toc.yml

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -5,6 +5,7 @@ toc:
55
- file: edot-collector/index.md
66
children:
77
- file: edot-collector/collector-oomkilled.md
8+
- file: edot-collector/insufficient-resources-kubestack.md
89
- file: edot-collector/metadata.md
910
- file: edot-collector/enable-debug-logging.md
1011
- file: edot-collector/collector-not-starting.md

0 commit comments

Comments
 (0)