Merge pull request #4033 from ClickHouse/clickstack-json

Docs for JSON in ClickStack

Author: Blargian

docs/use-cases/observability/clickstack/config.md

@@ -336,6 +336,12 @@ For example, below is the Logs source configured with correlated sources:
     - Useful for development and testing
     - Disable in production
 
+- `BETA_CH_OTEL_JSON_SCHEMA_ENABLED`
+    - **Default:** `false`
+    - **Description:** Enables Beta support for the JSON type in HyperDX. See also [`OTEL_AGENT_FEATURE_GATE_ARG`](#otel-collector) to enable JSON support in the OTel collector.
+    - **Guidance:**
+    - Set to `true` to enable JSON support in ClickStack.
+
 
 ## OpenTelemetry collector {#otel-collector}
 
@@ -383,6 +389,12 @@ See ["ClickStack OpenTelemetry Collector"](/use-cases/observability/clickstack/i
     - Set if using a custom database name  
     - Ensure the specified user has access to this database  
 
+- `OTEL_AGENT_FEATURE_GATE_ARG`
+    - **Default:** `<empty string>`
+    - **Description:** Enables feature flags to enabled in the collector. If set to `--feature-gates=clickhouse.json` enables Beta support for the JSON type in collector, ensuring schemas are created with the type. See also [`BETA_CH_OTEL_JSON_SCHEMA_ENABLED`](#hyperdx) to enable JSON support in HyperDX.
+    - **Guidance:**
+    - Set to `true` to enable JSON support in ClickStack.
+
 ## ClickHouse {#clickhouse}
 
 ClickStack ships with a default ClickHouse configuration designed for multi-terabyte scale, but users are free to modify and optimize it to suit their workload.

docs/use-cases/observability/clickstack/deployment/_snippets/_json_support.md

@@ -0,0 +1,15 @@
+import BetaBadge from '@theme/badges/BetaBadge';
+
+## JSON type support {#json-type-support}
+
+<BetaBadge/>
+
+ClickStack has beta support for the [JSON type](/interfaces/formats/JSON) from version `2.0.4`.
+
+For the benefits of this type see [Benefits of the JSON type](/use-cases/observability/clickstack/ingesting-data/otel-collector#benefits-json-type).
+
+In order to enable support for the JSON type users must set the following environment variables:
+
+- `OTEL_AGENT_FEATURE_GATE_ARG='--feature-gates=clickhouse.json'` - enables support in the OTel collector, ensuring schemas are created using the JSON type.
+- `BETA_CH_OTEL_JSON_SCHEMA_ENABLED=true` - enables support in the HyperDX application, allowing JSON data to be queried.
+

docs/use-cases/observability/clickstack/deployment/all-in-one.md

@@ -7,6 +7,7 @@ sidebar_position: 0
 description: 'Deploying ClickStack with All In One - The ClickHouse Observability Stack'
 ---
 
+import JSONSupport from '@site/docs/use-cases/observability/clickstack/deployment/_snippets/_json_support.md';
 import Image from '@theme/IdealImage';
 import hyperdx_login from '@site/static/images/use-cases/observability/hyperdx-login.png';
 import hyperdx_logs from '@site/static/images/use-cases/observability/hyperdx-logs.png';
@@ -112,3 +113,11 @@ On connecting to the HyperDX UI, navigate to [`Team Settings`](http://localhost:
 ## Configuring the OpenTelemetry collector {#configuring-collector}
 
 The OTel collector configuration can be modified if required - see ["Modifying configuration"](/use-cases/observability/clickstack/ingesting-data/otel-collector#modifying-otel-collector-configuration).
+
+<JSONSupport/>
+
+For example:
+
+```shell
+docker run -e OTEL_AGENT_FEATURE_GATE_ARG='--feature-gates=clickhouse.json' -e BETA_CH_OTEL_JSON_SCHEMA_ENABLED=true -p 8080:8080 -p 4317:4317 -p 4318:4318 docker.hyperdx.io/hyperdx/hyperdx-all-in-one
+```

docs/use-cases/observability/clickstack/deployment/docker-compose.md

@@ -7,9 +7,11 @@ sidebar_position: 2
 description: 'Deploying ClickStack with Docker Compose - The ClickHouse Observability Stack'
 ---
 
+
 import Image from '@theme/IdealImage';
 import hyperdx_login from '@site/static/images/use-cases/observability/hyperdx-login.png';
 import hyperdx_logs from '@site/static/images/use-cases/observability/hyperdx-logs.png';
+import JSONSupport from '@site/docs/use-cases/observability/clickstack/deployment/_snippets/_json_support.md';
 
 All ClickStack components are distributed separately as individual Docker images:
 
@@ -150,3 +152,29 @@ This distribution can be used with ClickHouse Cloud. Users should:
     The `CLICKHOUSE_ENDPOINT` should be the ClickHouse Cloud HTTPS endpoint, including the port `8443` e.g. `https://mxl4k3ul6a.us-east-2.aws.clickhouse.com:8443`
 
 - On connecting to the HyperDX UI and creating a connection to ClickHouse, use your Cloud credentials.
+
+<JSONSupport/>
+
+To set these, modify the relevant services in the `docker-compose.yaml`:
+
+
+```yaml
+  app:
+    image: ${HDX_IMAGE_REPO}/${IMAGE_NAME_DOCKERHUB}:${IMAGE_VERSION}
+    ports:
+      - ${HYPERDX_API_PORT}:${HYPERDX_API_PORT}
+      - ${HYPERDX_APP_PORT}:${HYPERDX_APP_PORT}
+    environment:
+      BETA_CH_OTEL_JSON_SCHEMA_ENABLED: true # enable JSON
+      FRONTEND_URL: ${HYPERDX_APP_URL}:${HYPERDX_APP_PORT}
+      HYPERDX_API_KEY: ${HYPERDX_API_KEY}
+      HYPERDX_API_PORT: ${HYPERDX_API_PORT}
+    # truncated for brevity
+
+    otel-collector:
+    image: ${HDX_IMAGE_REPO}/${OTEL_COLLECTOR_IMAGE_NAME_DOCKERHUB}:${IMAGE_VERSION}
+    environment:
+      OTEL_AGENT_FEATURE_GATE_ARG: '--feature-gates=clickhouse.json' # enable JSON
+      CLICKHOUSE_ENDPOINT: 'tcp://ch-server:9000?dial_timeout=10s' 
+      # truncated for brevity
+```

docs/use-cases/observability/clickstack/deployment/helm.md

@@ -10,6 +10,7 @@ description: 'Deploying ClickStack with Helm - The ClickHouse Observability Stac
 import Image from '@theme/IdealImage';
 import hyperdx_24 from '@site/static/images/use-cases/observability/hyperdx-24.png';
 import hyperdx_login from '@site/static/images/use-cases/observability/hyperdx-login.png';
+import JSONSupport from '@site/docs/use-cases/observability/clickstack/deployment/_snippets/_json_support.md';
 
 The helm chart for HyperDX can be found [here](https://github.com/hyperdxio/helm-charts) and is the **recommended** method for production deployments.
 
@@ -219,6 +220,18 @@ clickhouse:
 
 otel:
   clickhouseEndpoint: ${CLICKHOUSE_URL}
+
+hyperdx:
+  defaultConnections: |
+    [
+      {
+        "name": "External ClickHouse",
+        "host": "http://your-clickhouse-server:8123",
+        "port": 8123,
+        "username": "your-username",
+        "password": "your-password"
+      }
+    ]
 ```
 
 ```shell
@@ -291,3 +304,33 @@ helm install my-hyperdx hyperdx/hdx-oss-v2 --debug --dry-run
 ```shell
 kubectl get pods -l app.kubernetes.io/name=hdx-oss-v2
 ```
+
+<JSONSupport/>
+
+Users can set these environment variables via either parameters or the `values.yaml` e.g.
+
+
+*values.yaml*
+
+```yaml
+hyperdx:
+  ...
+  env:
+    - name: BETA_CH_OTEL_JSON_SCHEMA_ENABLED
+      value: "true"
+
+otel:
+  ...
+  env:
+    - name: OTEL_AGENT_FEATURE_GATE_ARG
+      value: "--feature-gates=clickhouse.json"
+```
+
+or via `--set`:
+
+```shell
+helm install myrelease hyperdx-helm --set "hyperdx.env[0].name=BETA_CH_OTEL_JSON_SCHEMA_ENABLED" \
+  --set "hyperdx.env[0].value=true" \
+  --set "otel.env[0].name=OTEL_AGENT_FEATURE_GATE_ARG" \
+  --set "otel.env[0].value=--feature-gates=clickhouse.json"
+```

docs/use-cases/observability/clickstack/deployment/hyperdx-only.md

@@ -11,6 +11,7 @@ import Image from '@theme/IdealImage';
 import hyperdx_login from '@site/static/images/use-cases/observability/hyperdx-login.png';
 import hyperdx_logs from '@site/static/images/use-cases/observability/hyperdx-logs.png';
 import hyperdx_2 from '@site/static/images/use-cases/observability/hyperdx-2.png';
+import JSONSupport from '@site/docs/use-cases/observability/clickstack/deployment/_snippets/_json_support.md';
 
 This option is designed for users who already have a running ClickHouse instance populated with observability or event data.
 
@@ -71,3 +72,11 @@ Users can modify the [Docker Compose configuration](/use-cases/observability/cli
 Even if you are managing your own OpenTelemetry collector, independent of the other components in the stack, we still recommend using the ClickStack distribution of the collector. This ensures the default schema is used and best practices for ingestion are applied.
 
 For details on deploying and configuring a standalone collector see ["Ingesting with OpenTelemetry"](/use-cases/observability/clickstack/ingesting-data/otel-collector#modifying-otel-collector-configuration).
+
+<JSONSupport/>
+
+For the HyperDX-only image, users only need to set the `BETA_CH_OTEL_JSON_SCHEMA_ENABLED=true` parameter e.g.
+
+```shell
+docker run -e BETA_CH_OTEL_JSON_SCHEMA_ENABLED=true -e MONGO_URI=mongodb://YOUR_MONGODB_URI -p 8080:8080 docker.hyperdx.io/hyperdx/hyperdx
+```

docs/use-cases/observability/clickstack/deployment/local-mode-only.md

@@ -10,7 +10,7 @@ description: 'Deploying ClickStack with Local Mode Only - The ClickHouse Observa
 import Image from '@theme/IdealImage';
 import hyperdx_logs from '@site/static/images/use-cases/observability/hyperdx-logs.png';
 import hyperdx_2 from '@site/static/images/use-cases/observability/hyperdx-2.png';
-
+import JSONSupport from '@site/docs/use-cases/observability/clickstack/deployment/_snippets/_json_support.md';
 
 Similar to the [all-in-one image](/use-cases/observability/clickstack/deployment/docker-compose), this comprehensive Docker image bundles all ClickStack components:
 
@@ -55,3 +55,12 @@ Create a source, retain all default values, and complete the `Table` field with
 <Image img={hyperdx_logs} alt="Create logs source" size="md"/>
 
 </VerticalStepper>
+
+
+<JSONSupport/>
+
+For the local mode only image, users only need to set the `BETA_CH_OTEL_JSON_SCHEMA_ENABLED=true` parameter e.g.
+
+```shell
+docker run -e BETA_CH_OTEL_JSON_SCHEMA_ENABLED=true -p 8080:8080 docker.hyperdx.io/hyperdx/hyperdx-local
+```

docs/use-cases/observability/clickstack/example-datasets/remote-demo-data.md

@@ -39,9 +39,9 @@ import demo_sources from '@site/static/images/use-cases/observability/hyperdx-de
 import edit_connection from '@site/static/images/use-cases/observability/edit_connection.png';
 
 
-**The following guide assumes you have deployed ClickStack using the [instructions for the all-in-one image](/use-cases/observability/clickstack/getting-started), or [Local Mode Only](/use-cases/observability/clickstack/deployment/local-mode-only) and completed initial user creation.**
+**The following guide assumes you have deployed ClickStack using the [instructions for the all-in-one image](/use-cases/observability/clickstack/getting-started), or [Local Mode Only](/use-cases/observability/clickstack/deployment/local-mode-only) and completed initial user creation. Alternatively, users can skip all local setup and simply connect to our ClickStack hosted demo [play-clickstack.clickhouse.com](https://play-clickstack.clickhouse.com) which uses this dataset.**
 
-This getting started guide uses a dataset available on the demo server that users can access when first deploying HyperDX. The dataset is hosted on the public ClickHouse instance at [sql.clickhouse.com](https://sql.clickhouse.com).
+This guide uses a sample dataset hosted on the public ClickHouse playground at [sql.clickhouse.com](https://sql.clickhpouse.com), which you can connect to from your local ClickStack deployment.
 
 It contains approximately 40 hours of data captured from the ClickHouse version of the official OpenTelemetry (OTel) demo. The data is replayed nightly with timestamps adjusted to the current time window, allowing users to explore system behavior using HyperDX's integrated logs, traces, and metrics.
 

docs/use-cases/observability/clickstack/ingesting-data/collector.md

@@ -8,6 +8,7 @@ title: 'ClickStack OpenTelemetry Collector'
 ---
 
 import Image from '@theme/IdealImage';
+import BetaBadge from '@theme/badges/BetaBadge';
 import observability_6 from '@site/static/images/use-cases/observability/observability-6.png';
 import observability_8 from '@site/static/images/use-cases/observability/observability-8.png';
 import clickstack_with_gateways from '@site/static/images/use-cases/observability/clickstack-with-gateways.png';
@@ -291,3 +292,88 @@ For agent instances responsible for shipping events to a gateway, and only setti
 | 1k/second    | 0.2CPU, 0.2GiB              |
 | 5k/second    | 0.5 CPU, 0.5GiB             |
 | 10k/second   | 1 CPU, 1GiB                 |
+
+## JSON support {#json-support}
+
+<BetaBadge/>
+
+ClickStack has beta support for the [JSON type](/interfaces/formats/JSON) from version `2.0.4`.
+
+### Benefits of the JSON type {#benefits-json-type}
+
+The JSON type offers the following benefits to ClickStack users:
+
+- **Type preservation** - Numbers stay numbers, booleans stay booleans—no more flattening everything into strings. This means fewer casts, simpler queries, and more accurate aggregations.
+- **Path-level columns** - Each JSON path becomes its own sub-column, reducing I/O. Queries only read the fields they need, unlocking major performance gains over the old Map type which required the entire column to be read in order to query a specific field.
+- **Deep nesting just works** - Naturally handle complex, deeply nested structures without manual flattening (as required by the Map type) and subsequent awkward JSONExtract functions.
+- **Dynamic, evolving schemas** - Perfect for observability data where teams add new tags and attributes over time. JSON handles these changes automatically, without schema migrations. 
+- **Faster queries, lower memory** - Typical aggregations over attributes like `LogAttributes` see 5-10x less data read and dramatic speedups, cutting both query time and peak memory usage.
+- **Simple management** - No need to pre-materialize columns for performance. Each field becomes its own sub-column, delivering the same speed as native ClickHouse columns.
+
+### Enabling JSON support {#enabling-json-support}
+
+To enable this support for the collector, set the environment variable `OTEL_AGENT_FEATURE_GATE_ARG='--feature-gates=clickhouse.json'` on any deployment that includes the collector. This ensures the schemas are created in ClickHouse using the JSON type.
+
+:::note HyperDX support
+In order to query the JSON type, support must also be enabled in the HyperDX application layer via the environment variable `BETA_CH_OTEL_JSON_SCHEMA_ENABLED=true`.
+:::
+
+For example:
+
+```shell
+docker run -e OTEL_AGENT_FEATURE_GATE_ARG='--feature-gates=clickhouse.json' -e OPAMP_SERVER_URL=${OPAMP_SERVER_URL} -e CLICKHOUSE_ENDPOINT=${CLICKHOUSE_ENDPOINT} -e CLICKHOUSE_USER=default -e CLICKHOUSE_PASSWORD=${CLICKHOUSE_PASSWORD} -p 8080:8080 -p 4317:4317 -p 4318:4318 docker.hyperdx.io/hyperdx/hyperdx-otel-collector
+```
+
+### Migrating from map-based schemas to the JSON type {#migrating-from-map-based-schemas-to-json}
+
+:::important Backwards compatibility
+The [JSON type](/interfaces/formats/JSON) type is not backwards compatible with existing map-based schemas. New tables will be created using the `JSON` type.
+:::
+
+
+To migrate from the Map-based schemas, follow these steps:
+
+<VerticalStepper headerLevel="h4">
+
+
+#### Stop the OTel collector {#stop-the-collector}
+
+#### Rename existing tables and update sources {#rename-existing-tables-sources}
+
+Rename existing tables and update data sources in HyperDX. 
+
+For example:
+
+```sql
+RENAME TABLE otel_logs TO otel_logs_map;
+RENAME TABLE otel_metrics TO otel_metrics_map;
+```
+
+#### Deploy the collector  {#deploy-the-collector}
+
+Deploy the collector with `OTEL_AGENT_FEATURE_GATE_ARG` set.
+
+#### Restart the HyperDX container with JSON schema support {#restart-the-hyperdx-container}
+
+```shell
+export BETA_CH_OTEL_JSON_SCHEMA_ENABLED=true
+```
+
+#### Create new data sources {#create-new-data-sources}
+
+Create new data sources in HyperDX pointing to the JSON tables.
+
+</VerticalStepper>
+
+#### Migrating existing data (optional) {#migrating-existing-data}
+
+To move old data into the new JSON tables:
+
+```sql
+INSERT INTO otel_logs SELECT * FROM otel_logs_map;
+INSERT INTO otel_metrics SELECT * FROM otel_metrics_map;
+```
+
+:::warning
+Recommended only for datasets smaller than ~10 billion rows. Data previously stored with the Map type did not preserve type precision (all values were strings). As a result, this old data will appear as strings in the new schema until it ages out, requiring some casting on the frontend. Type for new data will be preserved with the JSON type.
+:::