feat(presto-clp): Add Docker compose setup for Presto cluster that can connect to clp-json.

docs/src/user-guide/guides-overview.md

@@ -12,6 +12,13 @@ Using object storage
 Using CLP to ingest logs from object storage and store archives on object storage.
 :::
 
+:::{grid-item-card}
+:link: guides-using-presto
+Using Presto with CLP
+^^^
+How to use Presto to query compressed logs in CLP.
+:::
+
 :::{grid-item-card}
 :link: guides-multi-node
 Multi-node deployment

docs/src/user-guide/guides-using-presto.md

@@ -0,0 +1,154 @@
+# Using Presto with CLP
+
+[Presto] is a distributed SQL query engine that can be used to query data stored in CLP (using SQL).
+This guide describes how to set up and use Presto with CLP.
+
+:::{warning}
+Currently, only the [clp-json](quick-start/clp-json.md) flavor of CLP supports queries through
+Presto.
+:::
+
+:::{note}
+This integration with Presto is under development and may change in the future. It is also being
+maintained in a [fork][yscope-presto] of the Presto project. At some point, these changes will have
+been merged into the main Presto repository so that you can use official Presto releases with CLP.
+:::
+
+## Requirements
+
+* [CLP][clp-releases] (clp-json) v0.4.0 or higher
+* [Docker] v28 or higher
+* [Docker Compose][docker-compose] v2.20.2 or higher
+* Python
+* python3-venv (for the version of Python installed)
+
+## Set up
+
+Using Presto with CLP requires:
+
+* [Setting up CLP](#setting-up-clp) and compressing some logs.
+* [Setting up Presto](#setting-up-presto) to query CLP's metadata database and archives.
+
+### Setting up CLP
+
+Follow the [quick-start](./quick-start/index.md) guide to set up CLP and compress your logs. A
+sample dataset that works well with Presto is [postgresql].
+
+### Setting up Presto
+
+1. Clone the CLP repository:
+
+    ```bash
+    git clone https://github.com/y-scope/clp.git
+    ```
+
+2. Navigate to the `tools/deployment/presto-clp` directory in your terminal.
+3. Generate the necessary config for Presto to work with CLP:
+
+    ```bash
+    scripts/set-up-config.sh <clp-json-dir>
+    ```
+
+    * Replace `<clp-json-dir>` with the location of the clp-json package you set up in the previous
+      section.
+
+4. Configure Presto to use CLP's metadata database as follows:
+
+    * Open and edit `coordinator/config-template/metadata-filter.json`.
+    * For each dataset you want to query, add a filter config of the form:
+
+      ```json
+      {
+        "clp.default.<dataset>": [
+          {
+            "columnName": "<timestamp-key>",
+            "rangeMapping": {
+              "lowerBound": "begin_timestamp",
+              "upperBound": "end_timestamp"
+            },
+            "required": false
+          }
+        ]
+      }
+      ```
+      
+      * Replace `<dataset>` with the name of the dataset you want to query. (If you didn't specify a
+        dataset when compressing your logs, they would be compressed into the `default` dataset.)
+      * Replace `<timestamp-key>` with the timestamp key you specified when compressing logs for
+        this particular dataset.
+    * The complete syntax for this file is [here][clp-connector-docs].
+
+5. Start a Presto cluster by running:
+
+    ```bash
+    docker compose up
+    ```
+
+    * To use more than Presto worker, you can use the `--scale` option as follows:
+
+      ```bash
+      docker compose up --scale presto-worker=<num-workers>
+      ```
+
+      * Replace `<num-workers>` with the number of Presto worker nodes you want to run.
+
+### Stopping the Presto cluster
+
+To stop the Presto cluster, use CTRL + C.
+
+To clean up the Presto cluster entirely:
+
+```bash
+docker compose rm
+```
+
+## Querying your logs through Presto
+
+To query your logs through Presto, you can use the Presto CLI:
+
+```bash
+docker compose exec presto-coordinator \
+  presto-cli \
+    --catalog clp \
+    --schema default
+```
+
+Each dataset in CLP shows up as a table in Presto. To show all available datasets:
+
+```sql
+SHOW TABLES;
+```
+
+If you didn't specify a dataset when compressing your logs in CLP, your logs will have been stored
+in the `default` dataset. To query the logs in this dataset:
+
+```sql
+SELECT * FROM default LIMIT 1;
+```
+
+All kv-pairs in each log event can be queried directly using dot-notation. For example, if your logs
+contain the field `foo.bar`, you can query it using:
+
+```sql
+SELECT foo.bar FROM default LIMIT 1;
+```
+
+## Limitations
+
+The Presto CLP integration has the following limitations at present:
+
+* Nested fields containing special characters cannot be queried (see [y-scope/presto#8]). Allowed
+  characters are alphanumeric characters and underscores. To get around this limitation, you'll
+  need to preprocess your logs to remove any special characters.
+* Only logs stored on the filesystem, rather than S3, can be queried through Presto.
+
+These limitations will be addressed in a future release of the Presto integration.
+
+[clp-connector-docs]: https://docs.yscope.com/presto/connector/clp.html#metadata-filter-config-file
+[clp-releases]: https://github.com/y-scope/clp/releases
+[docker-compose]: https://docs.docker.com/compose/install/
+[Docker]: https://docs.docker.com/engine/install/
+[postgresql]: https://zenodo.org/records/10516401
+[Presto]: https://prestodb.io/
+[y-scope/presto#8]: https://github.com/y-scope/presto/issues/8
+[yscope-presto]: https://github.com/y-scope/presto

docs/src/user-guide/index.md

@@ -62,6 +62,7 @@ quick-start/clp-text
 guides-overview
 guides-using-object-storage/index
 guides-multi-node
+guides-using-presto
 :::
 
 :::{toctree}

taskfiles/lint.yaml

@@ -103,7 +103,8 @@ tasks:
           components/package-template/src/etc \
           docs \
           taskfile.yaml \
-          taskfiles
+          taskfiles \
+          tools/deployment
 
   check-cpp-format:
     sources: &cpp_source_files
@@ -772,6 +773,7 @@ tasks:
           - "components/clp-py-utils/clp_py_utils"
           - "components/core/tools/scripts/utils"
           - "components/job-orchestration/job_orchestration"
+          - "tools/deployment"
           - "tools/scripts"
           - "docs/conf"
         cmd: |-

tools/deployment/presto-clp/coordinator-common.env

@@ -0,0 +1,5 @@
+PRESTO_COORDINATOR_HTTPPORT=8080
+PRESTO_COORDINATOR_SERVICENAME=presto-coordinator
+
+# node.properties
+PRESTO_COORDINATOR_NODEPROPERTIES_ENVIRONMENT=production

tools/deployment/presto-clp/coordinator.env

@@ -0,0 +1,14 @@
+# clp.properties
+PRESTO_COORDINATOR_CLPPROPERTIES_METADATA_PROVIDER_TYPE=mysql
+PRESTO_COORDINATOR_CLPPROPERTIES_SPLIT_PROVIDER=mysql
+
+# config.properties
+PRESTO_COORDINATOR_CONFIGPROPERTIES_QUERY_MAX_MEMORY=1GB
+PRESTO_COORDINATOR_CONFIGPROPERTIES_QUERY_MAX_MEMORY_PER_NODE=1GB
+
+# jvm.config
+PRESTO_COORDINATOR_CONFIG_JVMCONFIG_MAXHEAPSIZE=4G
+PRESTO_COORDINATOR_JVMCONFIG_G1HEAPREGIONSIZE=32M
+
+# log.properties
+PRESTO_COORDINATOR_LOGPROPERTIES_LEVEL=INFO

tools/deployment/presto-clp/coordinator/config-template/clp.properties

@@ -0,0 +1,9 @@
+connector.name=clp
+clp.metadata-provider-type=${PRESTO_COORDINATOR_CLPPROPERTIES_METADATA_PROVIDER_TYPE}
+clp.metadata-db-url=${PRESTO_COORDINATOR_CLPPROPERTIES_METADATA_DATABASE_URL}
+clp.metadata-db-name=${PRESTO_COORDINATOR_CLPPROPERTIES_METADATA_DATABASE_NAME}
+clp.metadata-db-user=${PRESTO_COORDINATOR_CLPPROPERTIES_METADATA_DATABASE_USER}
+clp.metadata-db-password=${PRESTO_COORDINATOR_CLPPROPERTIES_METADATA_DATABASE_PASSWORD}
+clp.metadata-table-prefix=${PRESTO_COORDINATOR_CLPPROPERTIES_METADATA_TABLE_PREFIX}
+clp.split-provider-type=${PRESTO_COORDINATOR_CLPPROPERTIES_SPLIT_PROVIDER}
+clp.metadata-filter-config=/opt/presto-server/etc/metadata-filter.json

tools/deployment/presto-clp/coordinator/config-template/config.properties

@@ -0,0 +1,13 @@
+coordinator=true
+node-scheduler.include-coordinator=false
+http-server.http.port=${PRESTO_COORDINATOR_HTTPPORT}
+query.max-memory=${PRESTO_COORDINATOR_CONFIGPROPERTIES_QUERY_MAX_MEMORY}
+query.max-memory-per-node=${PRESTO_COORDINATOR_CONFIGPROPERTIES_QUERY_MAX_MEMORY_PER_NODE}
+discovery-server.enabled=true
+discovery.uri=${PRESTO_COORDINATOR_CONFIGPROPERTIES_DISCOVERY_URI}
+optimizer.optimize-hash-generation=false
+regex-library=RE2J
+use-alternative-function-signatures=true
+inline-sql-functions=false
+nested-data-serialization-enabled=false
+native-execution-enabled=true

tools/deployment/presto-clp/coordinator/config-template/jvm.config

@@ -0,0 +1,9 @@
+-server
+-Xmx${PRESTO_COORDINATOR_CONFIG_JVMCONFIG_MAXHEAPSIZE}
+-XX:+UseG1GC
+-XX:G1HeapRegionSize=${PRESTO_COORDINATOR_JVMCONFIG_G1HEAPREGIONSIZE}
+-XX:+UseGCOverheadLimit
+-XX:+ExplicitGCInvokesConcurrent
+-XX:+HeapDumpOnOutOfMemoryError
+-XX:+ExitOnOutOfMemoryError
+-Djdk.attach.allowAttachSelf=true

tools/deployment/presto-clp/coordinator/config-template/log.properties

@@ -0,0 +1 @@
+com.facebook.presto=${PRESTO_COORDINATOR_LOGPROPERTIES_LEVEL}

tools/deployment/presto-clp/coordinator/config-template/metadata-filter.json

@@ -0,0 +1,2 @@
+{
+}

tools/deployment/presto-clp/coordinator/config-template/node.properties

@@ -0,0 +1,2 @@
+node.environment=${PRESTO_COORDINATOR_NODEPROPERTIES_ENVIRONMENT}
+node.id=${PRESTO_COORDINATOR_SERVICENAME}

tools/deployment/presto-clp/coordinator/scripts/generate-configs.sh

@@ -0,0 +1,19 @@
+#!/usr/bin/env bash
+
+set -eu
+set -o pipefail
+
+readonly PRESTO_CONFIG_DIR="/opt/presto-server/etc"
+
+# Substitute environment variables in config template
+find /configs -type f | while read -r f; do
+    (
+        echo "cat <<EOF"
+        cat "$f"
+        echo "EOF"
+    ) | sh >"${PRESTO_CONFIG_DIR}/$(basename "$f")"
+done
+
+# Remove existing catalog files that exist in the image and add the CLP catalog
+rm -f "${PRESTO_CONFIG_DIR}/catalog/"*
+mv "${PRESTO_CONFIG_DIR}/clp.properties" "${PRESTO_CONFIG_DIR}/catalog"

tools/deployment/presto-clp/docker-compose.yaml

@@ -0,0 +1,48 @@
+services:
+  presto-coordinator:
+    image: "ghcr.io/y-scope/presto/coordinator:dev"
+    entrypoint: ["/bin/bash", "-c", "/scripts/generate-configs.sh && /opt/entrypoint.sh"]
+    env_file:
+      - ".env"
+      - "coordinator-common.env"
+      - "coordinator.env"
+    volumes:
+      - "./coordinator/config-template:/configs:ro"
+      - "./coordinator/scripts:/scripts:ro"
+      - "coordinator-config:/opt/presto-server/etc"
+    networks:
+      - "presto"
+    healthcheck:
+      test:
+        - "CMD"
+        - "curl"
+        - "-f"
+        - "${PRESTO_COORDINATOR_CONFIGPROPERTIES_DISCOVERY_URI}/v1/info"
+      interval: "10s"
+      retries: 30
+
+  presto-worker:
+    image: "ghcr.io/y-scope/presto/prestissimo-worker:dev"
+    depends_on:
+      presto-coordinator:
+        condition: "service_healthy"
+    entrypoint: ["/bin/bash", "-c", "/scripts/generate-configs.sh && /opt/entrypoint.sh"]
+    env_file:
+      - ".env"
+      - "coordinator-common.env"
+      - "worker.env"
+    volumes:
+      - "./worker/config-template:/configs:ro"
+      - "./worker/scripts:/scripts:ro"
+      - "${CLP_ARCHIVES_DIR}:${CLP_ARCHIVES_DIR}"
+      - "worker-config:/opt/presto-server/etc"
+    networks:
+      - "presto"
+
+volumes:
+  coordinator-config:
+  worker-config:
+
+networks:
+  presto:
+    driver: "bridge"

tools/deployment/presto-clp/scripts/.gitignore

@@ -0,0 +1 @@
+/.venv/