Skip to content

fix(demo/spark-k8s-anomaly-detection-taxi-data): Use TLS to load data into Minio #179

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 6 commits into from
Mar 18, 2025
Merged

fix(demo/spark-k8s-anomaly-detection-taxi-data): Use TLS to load data into Minio #179

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
a386fa9
wip
razvan Mar 18, 2025
bbf350d
Merge remote-tracking branch 'origin/main' into fix/anomaly-trino-s3-tls
maltesander Mar 18, 2025
c2cda1c
Merge remote-tracking branch 'origin/main' into fix/anomaly-trino-s3-tls
maltesander Mar 18, 2025
be9081f
use cert for load test data, remove unsecure call
maltesander Mar 18, 2025
01fe9b6
remove local reference
maltesander Mar 18, 2025
dffc817
fix linter
maltesander Mar 18, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
29 changes: 28 additions & 1 deletion demos/spark-k8s-anomaly-detection-taxi-data/load-test-data.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,13 +10,40 @@ spec:
- name: load-ny-taxi-data
image: "bitnami/minio:2022-debian-10"
# yamllint disable-line rule:line-length
command: ["bash", "-c", "cd /tmp && for month in 2020-09 2020-10 2020-11 2020-12; do curl -O https://repo.stackable.tech/repository/misc/ny-taxi-data/fhvhv_tripdata_$month.parquet && mc --insecure alias set minio http://minio:9000/ $(cat /minio-s3-credentials/accessKey) $(cat /minio-s3-credentials/secretKey) && mc cp fhvhv_tripdata_$month.parquet minio/demo/ny-taxi-data/raw/ && mc mb --ignore-existing minio/prediction; done"]
command:
- "bash"
- "-ce"
- |
cd /tmp
for month in 2020-09 2020-10 2020-11 2020-12; do
curl -O https://repo.stackable.tech/repository/misc/ny-taxi-data/fhvhv_tripdata_$month.parquet
mc alias set minio https://minio.default.svc.cluster.local:9000/ $(cat /minio-s3-credentials/accessKey) $(cat /minio-s3-credentials/secretKey)
mc cp fhvhv_tripdata_$month.parquet minio/demo/ny-taxi-data/raw/
mc mb --ignore-existing minio/prediction;
done
volumeMounts:
- name: minio-s3-credentials
mountPath: /minio-s3-credentials
- name: tls
mountPath: /.mc/certs/CAs
volumes:
- name: minio-s3-credentials
secret:
secretName: minio-s3-credentials
- name: tls
ephemeral:
volumeClaimTemplate:
metadata:
annotations:
secrets.stackable.tech/class: tls
secrets.stackable.tech/scope: |-
service=minio
spec:
storageClassName: secrets.stackable.tech
accessModes:
- ReadWriteOnce
resources:
requests:
storage: "1"
restartPolicy: OnFailure
backoffLimit: 50
2 changes: 1 addition & 1 deletion stacks/stacks-v2.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -254,7 +254,7 @@ stacks:
- minio
- s3
manifests:
- helmChart: https://raw.githubusercontent.com/stackabletech/demos/main/stacks/_templates/minio.yaml
- plainYaml: https://raw.githubusercontent.com/stackabletech/demos/main/stacks/_templates/minio-tls/rendered-chart.yaml
- helmChart: https://raw.githubusercontent.com/stackabletech/demos/main/stacks/_templates/postgresql-hive.yaml
- helmChart: https://raw.githubusercontent.com/stackabletech/demos/main/stacks/_templates/postgresql-hive-iceberg.yaml
- helmChart: https://raw.githubusercontent.com/stackabletech/demos/main/stacks/_templates/postgresql-superset.yaml
Expand Down