Add z3fdb-web service #218
+991
−3
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,123 @@ | ||
| name: Build Z3FDB Web Service Container | ||
|
|
||
| on: | ||
| workflow_dispatch: | ||
| push: | ||
| branches: | ||
| - develop | ||
| tags: | ||
| - '[0-9]+.[0-9]+.[0-9]+' | ||
| pull_request: | ||
|
|
||
| env: | ||
| REGISTRY: eccr.ecmwf.int | ||
| IMAGE_NAME: z3fdb-web-service | ||
| PROJECT_NAME: kkratz | ||
|
|
||
| jobs: | ||
| build-packages: | ||
| name: Build Packages | ||
| runs-on: [self-hosted, Linux, platform-builder-docker] | ||
| container: | ||
| image: eccr.ecmwf.int/ci-utils/ubuntu_rolling_fdb_build_env:latest | ||
| credentials: | ||
| username: ${{ secrets.ECMWF_DOCKER_REGISTRY_USERNAME }} | ||
| password: ${{ secrets.ECMWF_DOCKER_REGISTRY_ACCESS_TOKEN }} | ||
| steps: | ||
| - name: Get ecbuild | ||
| uses: actions/checkout@v5 | ||
| with: | ||
| repository: ecmwf/ecbuild | ||
| ref: develop | ||
| path: ecbuild | ||
| - name: Get cxx-dependencies | ||
| uses: actions/checkout@v5 | ||
| with: | ||
| repository: ecmwf/cxx-dependencies | ||
| ref: master | ||
| path: cxx-dependencies | ||
| token: ${{ secrets.GH_REPO_READ_TOKEN }} | ||
| submodules: recursive | ||
| - name: Get eccodes | ||
| uses: actions/checkout@v5 | ||
| with: | ||
| repository: ecmwf/eccodes | ||
| ref: develop | ||
| path: eccodes | ||
| - name: Get eckit | ||
| uses: actions/checkout@v5 | ||
| with: | ||
| repository: ecmwf/eckit | ||
| ref: develop | ||
| path: eckit | ||
| - name: Get metkit | ||
| uses: actions/checkout@v5 | ||
| with: | ||
| repository: ecmwf/metkit | ||
| ref: develop | ||
| path: metkit | ||
| - name: Get fdb5 | ||
| uses: actions/checkout@v5 | ||
| with: | ||
| repository: ecmwf/fdb | ||
| path: fdb | ||
| - name: Compile, test and pack | ||
| run: | | ||
| ./fdb/containerfiles/z3fdb_web_service/build_contents.sh | ||
| rm -rf stage | ||
| mkdir stage | ||
| mv *.zst *.whl stage | ||
| - name: Upload stage archive | ||
| uses: actions/upload-artifact@v4 | ||
| with: | ||
| name: stage | ||
| path: stage/ | ||
|
|
||
|
|
||
| build-container: | ||
| name: Build Image | ||
| runs-on: [self-hosted, Linux, platform-builder-docker] | ||
| needs: [build-packages] | ||
| steps: | ||
| - name: Get fdb5 | ||
| uses: actions/checkout@v5 | ||
| with: | ||
| repository: ecmwf/fdb | ||
| - name: Download stage archive | ||
| uses: actions/download-artifact@v4 | ||
| with: | ||
| name: stage | ||
| path: stage/ | ||
| - name: Log in to ECCR | ||
| uses: docker/login-action@v3 | ||
| with: | ||
| registry: ${{ env.REGISTRY }} | ||
| username: ${{ secrets.KKRATZ_ECCR_USER }} | ||
|
Contributor
There was a problem hiding this comment. Those should probably also be the DMST/FDB_ECCR credentials?
Member
Author
There was a problem hiding this comment. No, this will change later when the service gets integrated. |
||
| password: ${{ secrets.KKRATZ_ECCR_TOKEN }} | ||
| - name: Extract metadata (tags, labels) | ||
| id: meta | ||
| uses: docker/metadata-action@v5 | ||
| with: | ||
| images: ${{ env.REGISTRY }}/${{ env.PROJECT_NAME }}/${{ env.IMAGE_NAME }} | ||
| tags: | | ||
| # SHA tag for all builds | ||
| type=sha | ||
| # Semver tag: use git tag as-is (e.g., 1.2.3) | ||
| type=semver,pattern={{version}} | ||
| # latest tag only for semver releases | ||
| type=raw,value=latest,enable=${{ github.ref_type == 'tag' }} | ||
| # latest-develop for develop branch | ||
| type=raw,value=latest-develop,enable=${{ github.ref == 'refs/heads/develop' }} | ||
| # latest-pr-<number> for pull requests | ||
| type=raw,value=latest-pr-${{ github.event.pull_request.number }},enable=${{ github.event_name == 'pull_request' }} | ||
| # latest-manual for workflow_dispatch | ||
| type=raw,value=latest-manual,enable=${{ github.event_name == 'workflow_dispatch' }} | ||
| - name: Set up Docker Buildx | ||
| uses: docker/setup-buildx-action@v3 | ||
| - name: Build and push Docker image | ||
| uses: docker/build-push-action@v5 | ||
| with: | ||
| context: ./stage | ||
| file: ./containerfiles/z3fdb_web_service/Dockerfile | ||
| push: true | ||
| tags: ${{ steps.meta.outputs.tags }} | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -16,3 +16,4 @@ compile_commands.json | |
| __pycache__/ | ||
|
|
||
| *.swp | ||
| **/*.egg-info/ | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,23 @@ | ||
| # Patterns to ignore when building packages. | ||
| # This supports shell glob matching, relative path matching, and | ||
| # negation (prefixed with !). Only one pattern per line. | ||
| .DS_Store | ||
| # Common VCS dirs | ||
| .git/ | ||
| .gitignore | ||
| .bzr/ | ||
| .bzrignore | ||
| .hg/ | ||
| .hgignore | ||
| .svn/ | ||
| # Common backup files | ||
| *.swp | ||
| *.bak | ||
| *.tmp | ||
| *.orig | ||
| *~ | ||
| # Various IDEs | ||
| .project | ||
| .idea/ | ||
| *.tmproj | ||
| .vscode/ |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,15 @@ | ||
| apiVersion: v2 | ||
| name: z3fdb-web | ||
| description: Remote access to a z3fdb store. | ||
| type: application | ||
|
|
||
| # This is the chart version. This version number should be incremented each time you make changes | ||
| # to the chart and its templates, including the app version. | ||
| # Versions are expected to follow Semantic Versioning (https://semver.org/) | ||
| version: 0.1.0 | ||
|
|
||
| # This is the version number of the application being deployed. This version number should be | ||
| # incremented each time you make changes to the application. Versions are not expected to | ||
| # follow Semantic Versioning. They should reflect the version the application is using. | ||
| # It is recommended to use it with quotes. | ||
| appVersion: "0.1.0" |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,62 @@ | ||
| {{/* | ||
| Expand the name of the chart. | ||
| */}} | ||
| {{- define "z3fdb-web.name" -}} | ||
| {{- default .Chart.Name .Values.nameOverride | trunc 63 | trimSuffix "-" }} | ||
| {{- end }} | ||
|
|
||
| {{/* | ||
| Create a default fully qualified app name. | ||
| We truncate at 63 chars because some Kubernetes name fields are limited to this (by the DNS naming spec). | ||
| If release name contains chart name it will be used as a full name. | ||
| */}} | ||
| {{- define "z3fdb-web.fullname" -}} | ||
| {{- if .Values.fullnameOverride }} | ||
| {{- .Values.fullnameOverride | trunc 63 | trimSuffix "-" }} | ||
| {{- else }} | ||
| {{- $name := default .Chart.Name .Values.nameOverride }} | ||
| {{- if contains $name .Release.Name }} | ||
| {{- .Release.Name | trunc 63 | trimSuffix "-" }} | ||
| {{- else }} | ||
| {{- printf "%s-%s" .Release.Name $name | trunc 63 | trimSuffix "-" }} | ||
| {{- end }} | ||
| {{- end }} | ||
| {{- end }} | ||
|
|
||
| {{/* | ||
| Create chart name and version as used by the chart label. | ||
| */}} | ||
| {{- define "z3fdb-web.chart" -}} | ||
| {{- printf "%s-%s" .Chart.Name .Chart.Version | replace "+" "_" | trunc 63 | trimSuffix "-" }} | ||
| {{- end }} | ||
|
|
||
| {{/* | ||
| Common labels | ||
| */}} | ||
| {{- define "z3fdb-web.labels" -}} | ||
| helm.sh/chart: {{ include "z3fdb-web.chart" . }} | ||
| {{ include "z3fdb-web.selectorLabels" . }} | ||
| {{- if .Chart.AppVersion }} | ||
| app.kubernetes.io/version: {{ .Chart.AppVersion | quote }} | ||
| {{- end }} | ||
| app.kubernetes.io/managed-by: {{ .Release.Service }} | ||
| {{- end }} | ||
|
|
||
| {{/* | ||
| Selector labels | ||
| */}} | ||
| {{- define "z3fdb-web.selectorLabels" -}} | ||
| app.kubernetes.io/name: {{ include "z3fdb-web.name" . }} | ||
| app.kubernetes.io/instance: {{ .Release.Name }} | ||
| {{- end }} | ||
|
|
||
| {{/* | ||
| Create the name of the service account to use | ||
| */}} | ||
| {{- define "z3fdb-web.serviceAccountName" -}} | ||
| {{- if .Values.serviceAccount.create }} | ||
| {{- default (include "z3fdb-web.fullname" .) .Values.serviceAccount.name }} | ||
| {{- else }} | ||
| {{- default "default" .Values.serviceAccount.name }} | ||
| {{- end }} | ||
| {{- end }} |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,20 @@ | ||
| apiVersion: v1 | ||
| kind: ConfigMap | ||
| metadata: | ||
| name: caddy-config | ||
| data: | ||
| Caddyfile: | | ||
| :80 { | ||
| log { | ||
| level DEBUG | ||
| output stdout | ||
| format console | ||
| } | ||
| encode { | ||
| match { | ||
| header Content-Type application/octet-stream | ||
| } | ||
| zstd best | ||
| } | ||
| reverse_proxy localhost:5000 | ||
| } |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,7 @@ | ||
| apiVersion: v1 | ||
| kind: ConfigMap | ||
| metadata: | ||
| name: fdb-config | ||
| data: | ||
| fdb-config: | | ||
| {{- required ".Values.config.fdb_config is required " .Values.config.fdb_config | toYaml | nindent 4 }} |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,39 @@ | ||
| apiVersion: apps/v1 | ||
| kind: Deployment | ||
| metadata: | ||
| name: z3fdb-deployment | ||
| spec: | ||
| replicas: 1 | ||
| selector: | ||
| matchLabels: | ||
| app: z3fdb | ||
| template: | ||
| metadata: | ||
| labels: | ||
| app: z3fdb | ||
| spec: | ||
| containers: | ||
| - name: {{ .Values.image.z3fdb.name }} | ||
| image: {{ .Values.image.z3fdb.repository }}:{{ .Values.image.z3fdb.tag }} | ||
| args: ["--fdb-config=/etc/fdb/fdb.config"] | ||
| ports: | ||
| - containerPort: 5000 | ||
| volumeMounts: | ||
| - name: z3fdb-web-volume | ||
| mountPath: /etc/fdb/fdb.config | ||
| subPath: fdb-config # Mounts the ConfigMap data as a single file | ||
| - name: {{ .Values.image.caddy.name }} | ||
| image: {{ .Values.image.caddy.repository }}:{{ .Values.image.caddy.tag }} | ||
| ports: | ||
| - containerPort: 80 | ||
| volumeMounts: | ||
| - name: caddyfile | ||
| mountPath: /etc/caddy/Caddyfile | ||
| subPath: Caddyfile # Mounts the ConfigMap data as a single file | ||
| volumes: | ||
| - name: z3fdb-web-volume | ||
| configMap: | ||
| name: fdb-config | ||
| - name: caddyfile | ||
| configMap: | ||
| name: caddy-config |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,36 @@ | ||
| apiVersion: networking.k8s.io/v1 | ||
| kind: Ingress | ||
| metadata: | ||
| name: z3fdb-ingress | ||
| annotations: | ||
| dns.operators.ecmwf.int/on-transport-server: vs-transport-https | ||
| cert-manager.io/cluster-issuer: | ||
| nginx.ingress.kubernetes.io/proxy-body-size: "100m" | ||
| nginx.ingress.kubernetes.io/limit-rps: "500" | ||
| nginx.ingress.kubernetes.io/limit-req-status-code: "429" | ||
|
|
||
| nginx.org/redirect-to-https: "True" | ||
| ingress.kubernetes.io/ssl-redirect: "True" | ||
|
|
||
| spec: | ||
| ingressClassName: nginx | ||
| rules: | ||
| - host: "{{ .Release.Namespace }}.lumi.apps.dte.destination-earth.eu" | ||
| http: | ||
| paths: | ||
| - path: / | ||
| pathType: Prefix | ||
| backend: | ||
| service: | ||
| name: z3fdb-service | ||
| port: | ||
| number: 80 | ||
| defaultBackend: | ||
| service: | ||
| name: z3fdb-service | ||
| port: | ||
| number: 80 | ||
| tls: | ||
| - hosts: | ||
| - "{{ .Release.Namespace }}.lumi.apps.dte.destination-earth.eu" | ||
| secretName: z3fdb-wildcard-tls |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,12 @@ | ||
| apiVersion: v1 | ||
| kind: Service | ||
| metadata: | ||
| name: z3fdb-service | ||
| spec: | ||
| type: NodePort | ||
| selector: | ||
| app: z3fdb | ||
| ports: | ||
| - port: 80 | ||
| targetPort: 80 | ||
| name: http |
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Should this still be in this path or do we have a dmst project name?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This will change later when we the service gets integrated.