Python first API and dataflow composition

[heyong4725]

@haixuanTao currently our dataflow composition is using declarative YAML file. I am wondering in future if we can support Python first SDK such that developer can directly compose the dataflow graph (Python DSL).

Take a example of this very successful project: https://flyte.org/ , https://www.youtube.com/watch?v=OLD5-G9R9fw, https://github.com/flyteorg/flyte, https://github.com/flyteorg/flytekit:

from flytekit import task, workflow

@task(cache=True, cache_version="1", retries=3)
def sum(x: int, y: int) -> int:
return x + y

@task(cache=True, cache_version="1", retries=3)
def square(z: int) -> int:
return z*z

@workflow
def my_workflow(x: int, y: int) -> int:
return sum(x=square(z=x), y=square(z=y))

I am not sure if this type of API style is suitable for Dora, but let's explore it.

[haixuanTao]

There is definitely ideas that we can borrow from this API. Thanks! But, we might have to be cautious to not mix too much imperative programming and declarative YAML programming.

But I think that decorators API for parametrization is going to be a huge plus for developers.