-
Notifications
You must be signed in to change notification settings - Fork 83
docs(presto-clp): Add instructions for connecting CLP's UI to Presto; Simplify some set up instructions. #1310
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Changes from all commits
2e336cb
828f55c
d38cbe7
83c9016
2156ce1
da3942e
3bab531
6b4c82a
832ffb2
f3043fe
d8ad955
f9ec9a6
d495f97
43a39cc
e09ef68
51ded2a
b61b55d
9d82ad9
e394803
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -33,27 +33,60 @@ Using Presto with CLP requires: | |
|
|
||
| 1. Follow the [quick-start](quick-start/index.md) guide to download and extract the CLP package, | ||
| but don't start the package just yet. | ||
| 2. Before starting the package, update the package's config as follows: | ||
| 2. Before starting the package, update the package's config file (`etc/clp-config.yml`) as follows: | ||
|
|
||
| * Open `etc/clp-config.yml` located within the package. | ||
| * Uncomment the `database` section. | ||
| * Change `database.host` value to a non-localhost hostname/IP. | ||
| * After the change, the `database` section should look something like this: | ||
| * Set the `package.query_engine` key to `"presto"`. | ||
|
|
||
| ```yaml | ||
| database: | ||
| type: "mariadb" # "mariadb" or "mysql" | ||
| host: "<new-IP-address>" | ||
| port: 3306 | ||
| name: "clp-db" | ||
| package: | ||
| storage_engine: "clp-s" | ||
| query_engine: "presto" | ||
| ``` | ||
|
|
||
| :::{note} | ||
| This change is necessary since the Presto containers run on a Docker network, whereas CLP's | ||
| database runs on the host network. So `localhost` refers to two different entities in those | ||
| networks. This limitation will be addressed in the future when we unify Presto and CLP's | ||
| deployment infrastructure. | ||
| ::: | ||
| * Set the `database.host` key to a non-localhost hostname/IP. | ||
|
|
||
| ```yaml | ||
| database: | ||
| # type: "mariadb" | ||
| host: "<non-local-ip-address>" | ||
| # port: 3306 | ||
| # name: "clp-db" | ||
| ``` | ||
|
|
||
| :::{note} | ||
| This change is necessary because the Presto containers run on a Docker network, and CLP's | ||
| database runs on the host network. `localhost` will refer to a different entity in each of | ||
| those contexts. This limitation will be addressed in the future when we unify Presto and CLP's | ||
| deployment infrastructure. | ||
| ::: | ||
|
|
||
| * Set the `results_cache.retention_period` key to `null` since the CLP + Presto integration | ||
| doesn't yet support garbage collection. | ||
|
|
||
| ```yaml | ||
| results_cache: | ||
| # host: "localhost" | ||
| # port: 27017 | ||
| # db_name: "clp-query-results" | ||
| # stream_collection_name: "stream-files" | ||
| # | ||
| # # Retention period for search results, in minutes. Set to null to disable automatic deletion. | ||
| retention_period: null | ||
| ``` | ||
davemarco marked this conversation as resolved.
Show resolved
Hide resolved
|
||
|
|
||
| * Update the `presto` key with the host and port of the Presto cluster. If you follow the | ||
| [Setting up Presto](#setting-up-presto) section, the host is `localhost` and the port is | ||
| `8889`. | ||
|
|
||
| ```yaml | ||
| presto: | ||
| host: "<ip-address>" | ||
| port: <port> | ||
| ``` | ||
davemarco marked this conversation as resolved.
Show resolved
Hide resolved
|
||
|
|
||
| :::{note} | ||
| Presto doesn't need to be running before you start CLP. | ||
| ::: | ||
davemarco marked this conversation as resolved.
Show resolved
Hide resolved
|
||
|
|
||
| 3. If you'd like to store your compressed logs on S3, follow the | ||
| [using object storage](guides-using-object-storage/index.md) guide. | ||
|
|
@@ -115,7 +148,7 @@ Using Presto with CLP requires: | |
| 5. Start a Presto cluster by running: | ||
|
|
||
| ```bash | ||
| docker compose up | ||
| docker compose up --detach | ||
|
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. it might be better to also add the @coderabbitai add an issue to track
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. 🧩 Analysis chain🏁 Script executed: Length of output: 42 @junhaoliao, I've created issue #1495 to track adding the |
||
| ``` | ||
davemarco marked this conversation as resolved.
Show resolved
Hide resolved
|
||
|
|
||
| * To use more than one Presto worker, you can use the `--scale` option as follows: | ||
|
|
@@ -128,24 +161,22 @@ Using Presto with CLP requires: | |
|
|
||
| ### Stopping the Presto cluster | ||
|
|
||
| To stop the Presto cluster, use CTRL + C. | ||
| To stop the Presto cluster: | ||
|
|
||
| ```bash | ||
| docker compose stop | ||
| ``` | ||
|
|
||
| To clean up the Presto cluster entirely: | ||
|
|
||
| ```bash | ||
| docker compose rm | ||
| docker compose down | ||
| ``` | ||
|
|
||
| ## Querying your logs through Presto | ||
|
|
||
| To query your logs through Presto, you can use the Presto CLI: | ||
|
|
||
| ```bash | ||
| docker compose exec presto-coordinator \ | ||
| presto-cli \ | ||
| --catalog clp \ | ||
| --schema default | ||
| ``` | ||
| You can query your compressed logs in your browser from [CLP's UI](#querying-from-clps-ui), or | ||
| from the command line using the [Presto CLI](#querying-from-the-presto-cli). | ||
|
|
||
quinntaylormitchell marked this conversation as resolved.
Show resolved
Hide resolved
|
||
| Each dataset in CLP shows up as a table in Presto. To show all available datasets: | ||
|
|
||
|
|
@@ -179,6 +210,26 @@ contain the field `foo.bar`, you can query it using: | |
| SELECT foo.bar FROM default LIMIT 1; | ||
| ``` | ||
|
|
||
| ### Querying from CLP's UI | ||
|
|
||
| CLP's UI should be available at [http://localhost:4000](http://localhost:4000) (if you changed | ||
| `webui.host` or `webui.port` in `etc/clp-config.yml`, use the new values). | ||
|
|
||
| :::{note} | ||
| The UI can only run one query at a time, and queries must not end with a `;`. | ||
| ::: | ||
|
|
||
| ### Querying from the Presto CLI | ||
|
|
||
| To access the Presto CLI, navigate to the `tools/deployment/presto-clp` directory and run: | ||
|
|
||
| ```bash | ||
| docker compose exec presto-coordinator \ | ||
| presto-cli \ | ||
| --catalog clp \ | ||
| --schema default | ||
| ``` | ||
|
|
||
| ## Limitations | ||
|
|
||
| The Presto CLP integration has the following limitations at present: | ||
|
|
||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
(i figure this docs PR should provide better context, although the code for the Presto Docker Compose project was originally added in #1132.)
The mitigation documented above is practical but may be inconvenient for users operating in diverse network environments.
In the long term, we plan to integrate the Presto services into the CLP Package Docker Compose project introduced in #1178. Running
du -h tools/deployment/presto-clpshows the directory is only 132 KB, indicating that the integration would not significantly increase the package size while substantially improving user convenience.@coderabbitai please file an issue to track this long-term integration plan.
Since implementing and reviewing this integration may take time, a short-term solution is needed before the next release. After #1178, the
webuiservice may also encounter connectivity issues with the Presto connector, as host network mode was removed in that update.A likely mitigation is to allow the Presto project to attach to an external Docker network defined in the CLP Package Compose project.
@coderabbitai please create a separate issue to track this short-term fix.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
🧩 Analysis chain
🏁 Script executed:
Length of output: 229
@junhaoliao, I've created both issues as requested:
Long-term integration: #1497 - Integrate Presto services into CLP Package Docker Compose project
Short-term fix: #1498 - Allow Presto project to attach to external Docker network for CLP Package Compose
Both issues have been assigned to you and include references to PR #1310 and the relevant context.
🧠 Learnings used