Example Docker-based Sematic Pipelines

This repo contains examples of how to build and submit a Sematic pipelines using the native Python+Docker build system, as opposed to using Bazel.

In order to make this work with your own container registry, replace the occurrences of <image push registry> and <image push repository> in the configuration yaml with your own values.

See the Sematic documentation for more details.

Usage

All these examples require a running Sematic Server:

$ sematic start

Basic

To run a pipeline directly locally, without building or submitting a cloud execution:

$ sematic run basic/main.py

Running pipelines locally is only supported when using LocalResolver or SilentResolver.

Intermediate

This command will build a Docker image starting from the Sematic worker base image, and run a remote pipeline in the cloud (given all steps in the docs have been completed):

$ sematic run intermediate/main.py --build -- /intermediate/data/message.txt

This example also shows how to package data files and pass arguments to the pipeline execution.

Advanced

This command will build a Docker image from scratch using a build script and Dockerfile, and run a remote pipeline in the cloud (given all steps in the docs have been completed):

$ sematic run advanced/main.py --build -- /advanced/data/message.txt

This example also shows how to package data files and pass arguments to the pipeline execution, and how to configure a custom Docker client connection to the Docker server.