@@ -46,42 +46,43 @@ This demo will
46
46
47
47
== List the deployed Stackable services
48
48
49
- To list the installed installed Stackable services run the following command:
49
+ To list the installed Stackable services run the following command:
50
50
51
51
[source,console]
52
52
----
53
53
$ stackablectl stacklet list
54
- ┌─────────┬──────────────┬───────────┬──────────────────────────────────────────────┐
55
- │ PRODUCT ┆ NAME ┆ NAMESPACE ┆ ENDPOINTS │
56
- ╞═════════╪══════════════╪═══════════╪══════════════════════════════════════════════╡
57
- │ hive ┆ hive-iceberg ┆ default ┆ hive 172.18.0.4:30637 │
58
- │ ┆ ┆ ┆ metrics 172.18.0.4:30176 │
59
- ├╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┤
60
- │ opa ┆ opa ┆ default ┆ http http://172.18.0.2:32470 │
61
- ├╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┤
62
- │ trino ┆ trino ┆ default ┆ coordinator-metrics 172.18.0.2:32402 │
63
- │ ┆ ┆ ┆ coordinator-https https://172.18.0.2:31605 │
64
- ├╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┤
65
- │ minio ┆ minio ┆ default ┆ http http://172.18.0.2:30357 │
66
- │ ┆ ┆ ┆ console-http http://172.18.0.2:30310 │
67
- └─────────┴──────────────┴───────────┴──────────────────────────────────────────────┘
54
+
55
+ ┌─────────┬───────────────┬───────────┬───────────────────────────────────────────────┬─────────────────────────────────┐
56
+ │ PRODUCT ┆ NAME ┆ NAMESPACE ┆ ENDPOINTS ┆ CONDITIONS │
57
+ ╞═════════╪═══════════════╪═══════════╪═══════════════════════════════════════════════╪═════════════════════════════════╡
58
+ │ hive ┆ hive-iceberg ┆ default ┆ ┆ Available, Reconciling, Running │
59
+ ├╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌ ┼╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼ ╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┤
60
+ │ opa ┆ opa ┆ default ┆ ┆ Available, Reconciling, Running │
61
+ ├╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌ ┼╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼ ╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┤
62
+ │ trino ┆ trino ┆ default ┆ coordinator-metrics 172.18.0.2:32612 ┆ Available, Reconciling, Running │
63
+ │ ┆ ┆ ┆ coordinator-https https://172.18.0.2:30856 ┆ │
64
+ ├╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌ ┼╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼ ╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┤
65
+ │ minio ┆ minio-console ┆ default ┆ http http://172.18.0.2:32489 ┆ │
66
+ └─────────┴───────────────┴───────────┴───────────────────────────────────────────────┴─────────────────────────────────┘
67
+
68
68
----
69
69
70
70
include::partial$instance-hint.adoc[]
71
71
72
72
== MinIO
73
73
74
74
You can view the available buckets and objects (think of files) described in the
75
- xref:data-lakehouse-iceberg-trino-spark.adoc#_minio[data-lakehouse-iceberg-trino-spark] demo.
75
+ xref:data-lakehouse-iceberg-trino-spark.adoc#_minio[data-lakehouse-iceberg-trino-spark] demo. Currently, the bucket `lakehouse`
76
+ is still empty, but will be filled during the next steps.
76
77
77
78
== Connect to Trino
78
79
79
- Have a look at the xref:data-lakehouse-iceberg- trino-spark.adoc #_connect_with_dbeaver[documentation] on how to
80
+ Have a look at the https://docs.stackable.tech/home/stable/ trino/usage-guide/connect_to_trino #_connect_with_dbeaver[documentation] on how to
80
81
connect with DBeaver. As an alternative, you can use https://trino.io/download.html[trino-cli] by running:
81
82
82
83
[source,console]
83
84
----
84
- $ java -jar ~/Downloads/trino-cli-396 -executable.jar --user admin --insecure --password --server https://172.18.0.3:31250
85
+ $ java -jar ~/Downloads/trino-cli-451 -executable.jar --user admin --insecure --password --server https://172.18.0.2:30856
85
86
----
86
87
87
88
Make sure to replace the server endpoint with the endpoint listed in the `stackablectl stacklet list` output. When
@@ -131,7 +132,7 @@ show tables in tpch.sf5;
131
132
(8 rows)
132
133
----
133
134
134
- The dataset comes with different scale factors. This demo is intended to run on a Laptop , so it starts with a scale
135
+ The dataset comes with different scale factors. This demo is intended to run on a laptop , so it starts with a scale
135
136
factor of 5 (hence the `sf5` in the above query). If you have a sufficiently large S3 and Trino deployed, you can easily
136
137
re-run the statements below with a different scale factor. This demo has been tested up to a scale factor of 10000, but
137
138
you can choose any scale in between or even more if desired.
@@ -498,91 +499,6 @@ try to spin up 8 Trino workers with 16GB RAM each, chances are high that Pods wi
498
499
resources required can't be fulfilled.
499
500
====
500
501
501
- === Scale S3
502
-
503
- That should be the preferred option if you can access a managed S3, for example, from a Cloud provider with an excellent
504
- network interconnection.
505
-
506
- You can change the endpoint of the S3 by running `kubectl edit s3connection minio -o yaml` and `kubectl edit secret
507
- minio-s3-credentials`. Please note that the credentials need to be base64 encoded.
508
-
509
- .Example IONOS configuration
510
- [%collapsible]
511
- ====
512
- [source,sql]
513
- ----
514
- apiVersion: s3.stackable.tech/v1alpha1
515
- kind: S3Connection
516
- metadata:
517
- name: ionos-sbernauer
518
- spec:
519
- host: s3-eu-central-1.ionoscloud.com
520
- port: 443
521
- tls:
522
- verification:
523
- server:
524
- caCert:
525
- webPki: {}
526
- credentials:
527
- secretClass: ionos-sbernauer-s3-credentials
528
- ---
529
- apiVersion: secrets.stackable.tech/v1alpha1
530
- kind: SecretClass
531
- metadata:
532
- name: ionos-sbernauer-s3-credentials
533
- spec:
534
- backend:
535
- k8sSearch:
536
- searchNamespace:
537
- pod: {}
538
- ---
539
- apiVersion: v1
540
- kind: Secret
541
- metadata:
542
- name: ionos-sbernauer-s3-credentials
543
- labels:
544
- secrets.stackable.tech/class: ionos-sbernauer-s3-credentials
545
- stringData:
546
- accessKey: "<username>"
547
- secretKey: "<password>"
548
- ----
549
- ====
550
-
551
- If you don't have access to a managed S3 or don't want to use it, you can also scale up the MinIO cluster. You can see
552
- the available replicas using the following command:
553
-
554
- [source,console]
555
- ----
556
- $ kubectl get statefulsets.apps minio
557
-
558
- NAME READY AGE
559
- minio 2/2 4m16s
560
- ----
561
-
562
- You can edit the MinIO cluster using `kubectl edit statefulsets.apps minio`.
563
- Especially interesting are the following options:
564
-
565
- [source,yaml]
566
- ----
567
- apiVersion: apps/v1
568
- kind: StatefulSet
569
- metadata:
570
- name: minio
571
- spec:
572
- replicas: 5 # Number of MinIO nodes
573
- template:
574
- spec:
575
- containers:
576
- - name: minio
577
- resources:
578
- requests:
579
- cpu: 1000m # Guaranteed CPU available (one core in this case)
580
- memory: 4Gi # RAM available
581
- ----
582
-
583
- For example, set `spec.replicas` to `5` and save the changes. You can re-run `kubectl get statefulsets.apps minio` to see the
584
- effect.
585
-
586
502
=== Scale Trino
587
503
588
504
Run `kubectl edit trinocluster trino`. Modify the following settings to your needs:
@@ -603,8 +519,8 @@ spec:
603
519
limit: 6Gi # Available RAM
604
520
workers:
605
521
config:
606
- # This limit can't be to big as otherwise the workers won't start.
607
- # I suggest setting it to half of spec.coordinators.config.resources.memory.limit
522
+ # This limit can't be too big as otherwise the workers won't start.
523
+ # We suggest setting it to half of spec.coordinators.config.resources.memory.limit
608
524
queryMaxMemoryPerNode: 6GB
609
525
resources:
610
526
cpu:
0 commit comments