Skip to content

Commit d3d11a0

Browse files
xeniapexeniape
authored
ci: Add ny-tlc-report workflow, update READMEs (#474)
* ci: Add ny-tlc-report workflow, update READMEs * add new line * adjust headings in README * linter: modify links
1 parent 0a4923c commit d3d11a0

File tree

3 files changed

+97
-12
lines changed

3 files changed

+97
-12
lines changed

‎.github/workflows/dev_ny-tlc-report.yaml‎

Lines changed: 89 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,89 @@
1+
---
2+
name: Build and publish ny-tlc-report
3+
4+
env:
5+
IMAGE_NAME: ny-tlc-report
6+
IMAGE_VERSION: 0.2.0
7+
REGISTRY_PATH: stackable
8+
DOCKERFILE_PATH: "apps/docker/Dockerfile"
9+
10+
on:
11+
workflow_dispatch:
12+
push:
13+
branches:
14+
- main
15+
paths:
16+
- apps/docker/Dockerfile
17+
- apps/ny_tlc_report.py
18+
- .github/workflows/dev_ny-tlc-report.yaml
19+
20+
jobs:
21+
build:
22+
name: Build/Publish ${{ matrix.runner.arch }} Image
23+
permissions:
24+
id-token: write
25+
runs-on: ${{ matrix.runner.name }}
26+
strategy:
27+
matrix:
28+
runner:
29+
- {name: "ubuntu-latest", arch: "amd64"}
30+
- {name: "ubicloud-standard-8-arm", arch: "arm64"}
31+
steps:
32+
- name: Checkout Repository
33+
uses: actions/checkout@692973e3d937129bcbf40652eb9f2f61becf3332 # v4.1.7
34+
35+
- name: Build image
36+
id: build
37+
uses: stackabletech/actions/build-container-image@013e6482fbc0edf2d38cf9220fc931f6a81336fb # v0.0.6
38+
with:
39+
image-name: ${{ env.IMAGE_NAME }}
40+
image-index-manifest-tag: ${{ env.IMAGE_VERSION }}
41+
container-file: ${{ env.DOCKERFILE_PATH }}
42+
43+
- name: Publish Container Image on docker.stackable.tech
44+
uses: stackabletech/actions/publish-image@013e6482fbc0edf2d38cf9220fc931f6a81336fb # v0.0.6
45+
with:
46+
image-registry-uri: docker.stackable.tech
47+
image-registry-username: github
48+
image-registry-password: ${{ secrets.NEXUS_PASSWORD }}
49+
image-repository: stackable/${{ env.IMAGE_NAME }}
50+
image-manifest-tag: ${{ steps.build.outputs.image-manifest-tag }}
51+
source-image-uri: ${{ steps.build.outputs.image-manifest-uri }}
52+
53+
- name: Publish Container Image on oci.stackable.tech
54+
uses: stackabletech/actions/publish-image@013e6482fbc0edf2d38cf9220fc931f6a81336fb # v0.0.6
55+
with:
56+
image-registry-uri: oci.stackable.tech
57+
image-registry-username: robot$stackable+github-action-build
58+
image-registry-password: ${{ secrets.HARBOR_ROBOT_STACKABLE_GITHUB_ACTION_BUILD_SECRET }}
59+
image-repository: ${{ env.REGISTRY_PATH }}/${{ env.IMAGE_NAME }}
60+
image-manifest-tag: ${{ steps.build.outputs.image-manifest-tag }}
61+
source-image-uri: ${{ steps.build.outputs.image-manifest-uri }}
62+
63+
publish_manifests:
64+
name: Build/Publish Manifest
65+
needs: [build]
66+
permissions:
67+
id-token: write
68+
runs-on: ubuntu-latest
69+
steps:
70+
- name: Checkout Repository
71+
uses: actions/checkout@692973e3d937129bcbf40652eb9f2f61becf3332 # v4.1.7
72+
73+
- name: Publish and Sign Image Index Manifest to docker.stackable.tech
74+
uses: stackabletech/actions/publish-index-manifest@013e6482fbc0edf2d38cf9220fc931f6a81336fb # v0.0.6
75+
with:
76+
image-registry-uri: docker.stackable.tech
77+
image-registry-username: github
78+
image-registry-password: ${{ secrets.NEXUS_PASSWORD }}
79+
image-repository: stackable/${{ env.IMAGE_NAME }}
80+
image-index-manifest-tag: ${{ env.IMAGE_VERSION }}
81+
82+
- name: Publish and Sign Image Index Manifest to oci.stackable.tech
83+
uses: stackabletech/actions/publish-index-manifest@013e6482fbc0edf2d38cf9220fc931f6a81336fb # v0.0.6
84+
with:
85+
image-registry-uri: oci.stackable.tech
86+
image-registry-username: robot$stackable+github-action-build
87+
image-registry-password: ${{ secrets.HARBOR_ROBOT_STACKABLE_GITHUB_ACTION_BUILD_SECRET }}
88+
image-repository: ${{ env.REGISTRY_PATH }}/${{ env.IMAGE_NAME }}
89+
image-index-manifest-tag: ${{ env.IMAGE_VERSION }}

‎apps/README.md‎

Lines changed: 5 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -1,9 +1,5 @@
11

2-
## Build job image
3-
4-
docker build -t docker.stackable.tech/stackable/ny-tlc-report:0.2.0 -t docker.stackable.tech/stackable/ny-tlc-report:latest -f apps/docker/Dockerfile apps/
5-
6-
## Generate report from the public data set
2+
# Generate report from the public data set
73

84
spark-submit --conf spark.hadoop.fs.s3a.aws.credentials.provider=org.apache.hadoop.fs.s3a.AnonymousAWSCredentialsProvider --packages org.apache.hadoop:hadoop-aws:3.2.0,com.amazonaws:aws-java-sdk-s3:1.12.180,com.amazonaws:aws-java-sdk-core:1.12.180 ny_tlc_report.py --input 's3a://nyc-tlc/trip data/yellow_tripdata_2021-07.csv'
95

@@ -14,8 +10,8 @@ Notes:
1410
* Only one file is used for reporting in this example `yellow_tripdata_2021-07.csv`
1511
* This example works with `spark-3.1.3-bin-hadoop3.2` and Python 3.9.7. Other versions may require different dependency versions or even complete different dependencies altogether.
1612

17-
### Links
13+
## Links
1814

19-
[0] TLC trip data set https://www1.nyc.gov/site/tlc/about/tlc-trip-record-data.page
20-
[1] AWS Java SDK https://github.com/aws/aws-sdk-java
21-
[2] Hadoop AWS module https://hadoop.apache.org/docs/current/hadoop-aws/tools/hadoop-aws/index.html
15+
[0] [TLC trip data set](https://www1.nyc.gov/site/tlc/about/tlc-trip-record-data.page)
16+
[1] [AWS Java SDK](https://github.com/aws/aws-sdk-java)
17+
[2] [Hadoop AWS module](https://hadoop.apache.org/docs/current/hadoop-aws/tools/hadoop-aws/index.html)

‎examples/README-examples.md‎

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -9,11 +9,11 @@ This note outlines a few things that are needed to run these examples on a local
99
Create a new local cluster (e.g. with [Kind](https://kind.sigs.k8s.io/docs/user/quick-start/) and the [stackablectl tool](https://github.com/stackabletech/stackablectl)). This creates a cluster named `stackable-data-platform`.
1010

1111
````text
12-
kind delete clusters --all
13-
stackablectl operator install spark-k8s commons secret -k
12+
kind create cluster --name stackable-data-platform
13+
stackablectl operator install spark-k8s commons secret
1414
````
1515

16-
Build the `ny-tlc-report` image from the Dockerfile in this repository (apps/docker/Dockerfile) and then load it to the cluster:
16+
Load the `ny-tlc-report` image to the cluster:
1717

1818
````text
1919
kind load docker-image docker.stackable.tech/stackable/ny-tlc-report:0.2.0 --name stackable-data-platform

0 commit comments

Comments
 (0)