notes on OpAmp

Author: gingerwizard

docs/use-cases/observability/clickstack/deployment/docker-compose.md

@@ -89,16 +89,17 @@ Users can modify settings for the stack, such as the version used, through the e
 ```bash
 user@example-host hyperdx % cat .env
 # Used by docker-compose.yml
+# Used by docker-compose.yml
+HDX_IMAGE_REPO=docker.hyperdx.io
 IMAGE_NAME=ghcr.io/hyperdxio/hyperdx
 IMAGE_NAME_DOCKERHUB=hyperdx/hyperdx
-IMAGE_NAME_HDX=docker.hyperdx.io/hyperdx/hyperdx
 LOCAL_IMAGE_NAME=ghcr.io/hyperdxio/hyperdx-local
 LOCAL_IMAGE_NAME_DOCKERHUB=hyperdx/hyperdx-local
 ALL_IN_ONE_IMAGE_NAME=ghcr.io/hyperdxio/hyperdx-all-in-one
 ALL_IN_ONE_IMAGE_NAME_DOCKERHUB=hyperdx/hyperdx-all-in-one
 OTEL_COLLECTOR_IMAGE_NAME=ghcr.io/hyperdxio/hyperdx-otel-collector
 OTEL_COLLECTOR_IMAGE_NAME_DOCKERHUB=hyperdx/hyperdx-otel-collector
-CHANGESET_TAG=2.0.0-beta.16
+CODE_VERSION=2.0.0-beta.16
 IMAGE_VERSION_SUB_TAG=.16
 IMAGE_VERSION=2-beta
 IMAGE_NIGHTLY_TAG=2-nightly
@@ -108,6 +109,10 @@ HYPERDX_API_PORT=8000 #optional (should not be taken by other services)
 HYPERDX_APP_PORT=8080
 HYPERDX_APP_URL=http://localhost
 HYPERDX_LOG_LEVEL=debug
+HYPERDX_OPAMP_PORT=4320
+
+# Otel/Clickhouse config
+HYPERDX_OTEL_EXPORTER_CLICKHOUSE_DATABASE=default
 ```
 
 ### Configuring the OpenTelemetry collector {#configuring-collector}
@@ -128,7 +133,9 @@ This distribution can be used with ClickHouse Cloud. Users should:
           CLICKHOUSE_ENDPOINT: '<CLICKHOUSE_ENDPOINT>' # https endpoint here
           CLICKHOUSE_USER: '<CLICKHOUSE_USER>'
           CLICKHOUSE_PASSWORD: '<CLICKHOUSE_PASSWORD>'
+          HYPERDX_OTEL_EXPORTER_CLICKHOUSE_DATABASE: ${HYPERDX_OTEL_EXPORTER_CLICKHOUSE_DATABASE}
           HYPERDX_LOG_LEVEL: ${HYPERDX_LOG_LEVEL}
+          OPAMP_SERVER_URL: 'http://app:${HYPERDX_OPAMP_PORT}'
         ports:
           - '13133:13133' # health_check extension
           - '24225:24225' # fluentd receiver

docs/use-cases/observability/clickstack/deployment/hyperdx-only.md

@@ -70,21 +70,4 @@ Users can modify the [Docker Compose configuration](/use-cases/observability/cli
 
 Even if you are managing your own OpenTelemetry collector, independent of the other components in the stack, we still recommend using the ClickStack distribution of the collector. This ensures the default schema is used and best practices for ingestion are applied.
 
-This image configuration can be found [here](https://github.com/hyperdxio/hyperdx/tree/v2/docker/otel-collector) and can be deployed with the following command:
-
-```bash
-docker run -e OPAMP_SERVER_URL=${OPAMP_SERVER_URL} -e CLICKHOUSE_ENDPOINT=CLICKHOUSE_ENDPOINT -e CLICKHOUSE_USER=USER -e CLICKHOUSE_PASSWORD=PASSWORD -p 4317:4127 -p 4318:4318 docker.hyperdx.io/hyperdx/hyperdx-otel-collector:2-nightly
-```
-
-For example,
-
-```bash
-docker run -e OPAMP_SERVER_URL=http://localhost:4320 -e CLICKHOUSE_ENDPOINT=myhost:9000 -e CLICKHOUSE_USER=default -e CLICKHOUSE_PASSWORD=password -p 4317:4127 -p 4318:4318 docker.hyperdx.io/hyperdx/hyperdx-otel-collector:2-nightly
-```
-
-The `OPAMP_SERVER_URL` variable should point to your HyperDX deployment e.g. `http://localhost:4320`. This runs an OpAMP (Open Agent Management Protocol) server endpoint at `/v1/opamp` on port 4320 by default. This ensures the collectors OTLP interface is secured with the ingestion API key. See [Securing the collector](/use-cases/observability/clickstack/ingesting-data/otel-collector#securing-the-collector). The `CLICKHOUSE_ENDPOINT` here should be the full ClickHouse endpoint including the protocol and port e.g. `http://localhost:8123`.
-
-
-This command exposes an OTLP endpoint on ports 4317 (HTTP) and 4318 (gRPC) for users to send OTel events.
-
-For further details on deploying and configuring the collector see ["Ingesting with OpenTelemetry"](/use-cases/observability/clickstack/ingesting-data/opentelemetry) and ["ClickStack OpenTelemetry Collector"](/use-cases/observability/clickstack/ingesting-data/otel-collector).
+For details on deploying and configuring a standalone collector see ["Ingesting with OpenTelemetry"](/use-cases/observability/clickstack/ingesting-data/otel-collector#modifying-otel-collector-configuration).

docs/use-cases/observability/clickstack/ingesting-data/collector.md

@@ -47,6 +47,8 @@ export CLICKHOUSE_USER=<CLICKHOUSE_USER>
 export CLICKHOUSE_PASSWORD=<CLICKHOUSE_PASSWORD>
 ```
 
+The `CLICKHOUSE_ENDPOINT` should be the full ClickHouse HTTP endpoint, including the protocol and port—for example, `http://localhost:8123`.
+
 For example the all-in-one image:
 
 ```bash
@@ -59,7 +61,11 @@ Or the standalone image:
 docker run -e OPAMP_SERVER_URL=${OPAMP_SERVER_URL} -e CLICKHOUSE_ENDPOINT=${CLICKHOUSE_ENDPOINT} -e CLICKHOUSE_USER=default -e CLICKHOUSE_PASSWORD=${CLICKHOUSE_PASSWORD} -p 8080:8080 -p 4317:4317 -p 4318:4318 docker.hyperdx.io/hyperdx/hyperdx-otel-collector:2-nightly
 ```
 
-The `OPAMP_SERVER_URL` variable should point to your HyperDX deployment e.g. `http://localhost:4320`. This runs an OpAMP (Open Agent Management Protocol) server endpoint at `/v1/opamp` on port 4320 by default. This ensures the collectors OTLP interface is secured with the ingestion API key. See [Securing the collector](#securing-the-collector).
+The `OPAMP_SERVER_URL` should point to your HyperDX deployment - for example, `http://localhost:4320`. HyperDX exposes an OpAMP (Open Agent Management Protocol) server at `/v1/opamp` on port `4320` by default, ensuring the collector is configured securely to use the ingestion API key. For details, see [Securing the collector](/use-cases/observability/clickstack/ingesting-data/otel-collector#securing-the-collector).
+
+:::note Connecting to and exposing the OpAMP port
+For the collector to connect to the OpAMP port it must be exposed by the HyperDX container e.g. `-p 4320:4320`. For local testing, OSX users can then set `OPAMP_SERVER_URL=http://host.docker.internal:4320`. Linux users can start the collector container with `--network=host`.
+:::
 
 #### Docker Compose {#docker-compose-otel}
 

docs/use-cases/observability/clickstack/ingesting-data/opentelemetry.md

@@ -40,10 +40,11 @@ To deploy the connector, run the following docker command:
 docker run -e OPAMP_SERVER_URL=${OPAMP_SERVER_URL} -e CLICKHOUSE_ENDPOINT=${CLICKHOUSE_ENDPOINT} -e CLICKHOUSE_USER=default -e CLICKHOUSE_PASSWORD=${CLICKHOUSE_PASSWORD} -p 8080:8080 -p 4317:4317 -p 4318:4318 docker.hyperdx.io/hyperdx/hyperdx-otel-collector:2-nightly
 ```
 
-Note that we can overwrite the target ClickHouse instance with environment variables for `CLICKHOUSE_ENDPOINT`, `CLICKHOUSE_USERNAME`, and `CLICKHOUSE_PASSWORD`. **These environment variables can be used with any of the docker distributions which include the connector.** The `CLICKHOUSE_ENDPOINT` here should be the full ClickHouse endpoint including the protocol and port e.g. `http://localhost:8123`.
+Note that we can overwrite the target ClickHouse instance with environment variables for `CLICKHOUSE_ENDPOINT`, `CLICKHOUSE_USERNAME`, and `CLICKHOUSE_PASSWORD`. The `CLICKHOUSE_ENDPOINT` should be the full ClickHouse HTTP endpoint, including the protocol and port—for example, `http://localhost:8123`.
 
-The `OPAMP_SERVER_URL` variable should point to your HyperDX deployment e.g. `http://localhost:4320`. This runs an OpAMP (Open Agent Management Protocol) server endpoint at `/v1/opamp` on port 4320 by default. This ensures the collectors OTLP interface is secured with the ingestion API key.
+**These environment variables can be used with any of the docker distributions which include the connector.** 
 
+The `OPAMP_SERVER_URL` should point to your HyperDX deployment - for example, `http://localhost:4320`. HyperDX exposes an OpAMP (Open Agent Management Protocol) server at `/v1/opamp` on port `4320` by default. Make sure to expose this port from the container running HyperDX (e.g., using `-p 4320:4320`).
 
 Users should use a user with the [appropriate credentials](/use-cases/observability/clickstack/ingesting-data/otel-collector#creating-an-ingestion-user) in production.
 
@@ -65,7 +66,7 @@ In addition, an authorization header containing the API ingestion key is require
 <Image img={ingestion_key} alt="Ingestion keys" size="lg"/>
 
 
-For language SDKs, this can either be set by an `init` function or via an`OTEL_EXPORTER_OTLP_HEADERS` environment variable:
+For language SDKs, this can either be set by an `init` function or via an`OTEL_EXPORTER_OTLP_HEADERS` environment variable e.g.:
 
 ```bash
 OTEL_EXPORTER_OTLP_HEADERS='authorization=<YOUR_INGESTION_API_KEY>'