Build a Pipeline

This page describes how to create a pipeline for deployment in Kubeflow, using the provided sample pipelines.

Create a python3 environment

Python 3.5 or above is required. If you don't have Python3 set up, we suggest the following steps to install Miniconda.

In a Debian/Ubuntu/Cloud shell environment:

apt-get update; apt-get install -y wget bzip2
wget https://repo.continuum.io/miniconda/Miniconda3-latest-Linux-x86_64.sh
bash Miniconda3-latest-Linux-x86_64.sh

In a Windows environment, download the installer and make sure you select "Add Miniconda to my PATH environment variable" option during the installation.

In a Mac environment, download the installer and run the following command:

bash Miniconda3-latest-MacOSX-x86_64.sh

Create a clean python3 environment:

conda create --name mlpipeline python=3.6
source activate mlpipeline

If the conda command is not found, be sure to add the Miniconda path:

export PATH=MINICONDA_PATH/bin:$PATH

Install the Kubeflow Pipelines SDK

Go to the release page. Choose the latest version of the Kubeflow Pipelines SDK (dsl and dsl-compiler). Download the source code zip and decompress it.

Run the following:

cd $EXTRACTED_DIRECTORY
pip install ./dsl/ --upgrade # The library to specify pipelines with Python.
pip install ./dsl-compiler/ --upgrade # The compiler that converts pipeline code into the form required by the pipeline system.

Note: if you prefer adding "--user" in installation of dsl-compiler, please also run "export PATH=~/.local/bin:$PATH". Note: if you are feeling adventurous and want the latest, just git clone the repo and install dsl and dsl-compiler from there.

After successful installation the command "dsl-compile" should be added to your PATH.

Compile the samples

The pipelines are written in Python, but they must be compiled to an intermediate representation before submitting to the Kubeflow pipelines service.

dsl-compile --py [path/to/py/file] --output [path/to/output/tar.gz]

For example:

dsl-compile --py [ML_REPO_DIRECTORY]/samples/basic/sequential.py --output [ML_REPO_DIRECTORY]/samples/basic/sequential.tar.gz

Deploy the samples

Upload the generated .tar.gz file through the Kubeflow pipelines UI.

Advanced: Build your own components

See how to build your own pipeline components.