Skip to content

Commit aae0267

Browse files
RyanLettieriberndverst
authored and
acroca
committed
Creation of pipeline to publish dts python package to pypi (microsoft#43)
* Creating of pipeline to publish dts python package to pypi Signed-off-by: Ryan Lettieri <[email protected]> * Upgrading version of durabletask-azuremanaged from 0.1b1 to 0.1 Signed-off-by: Ryan Lettieri <[email protected]> * Updating versioning on packages Signed-off-by: Ryan Lettieri <[email protected]> * Incrementing version to allign with pypi Signed-off-by: Ryan Lettieri <[email protected]> * Adressing majority of first round of feedback Signed-off-by: Ryan Lettieri <[email protected]> * Updating pipeline to have linting Signed-off-by: Ryan Lettieri <[email protected]> * Updating versions in pyproject.toml Signed-off-by: Ryan Lettieri <[email protected]> * Updating working dirs in yml Signed-off-by: Ryan Lettieri <[email protected]> * Adding requirements.txt Signed-off-by: Ryan Lettieri <[email protected]> * Moving durabletask tests into specific dir and more Signed-off-by: Ryan Lettieri <[email protected]> * Fixing more paths Signed-off-by: Ryan Lettieri <[email protected]> * ATtemptign to ignore durabletask-azuremanaged folder Signed-off-by: Ryan Lettieri <[email protected]> * installing dts dependencies Signed-off-by: Ryan Lettieri <[email protected]> * Changing path for requirements.txt Signed-off-by: Ryan Lettieri <[email protected]> * Moving init.py Signed-off-by: Ryan Lettieri <[email protected]> * Updating readme and some tests Signed-off-by: Ryan Lettieri <[email protected]> * Running all dts tests in publish pipeline Signed-off-by: Ryan Lettieri <[email protected]> * Removing PYTHONPATH and installing regular deps Signed-off-by: Ryan Lettieri <[email protected]> * Adding timeout to dts orchestration e2e test Signed-off-by: Ryan Lettieri <[email protected]> * Removing suspend and continue as new tests from dts Signed-off-by: Ryan Lettieri <[email protected]> * Removing raise event timeout tests Signed-off-by: Ryan Lettieri <[email protected]> * Only runnign publish on tag push Signed-off-by: Ryan Lettieri <[email protected]> * Changing dts action to run on tag creation Signed-off-by: Ryan Lettieri <[email protected]> * Updating tag name Signed-off-by: Ryan Lettieri <[email protected]> * Adressing review feedback Signed-off-by: Ryan Lettieri <[email protected]> * Fixing run requirements in actions and adding exit-zero Signed-off-by: Ryan Lettieri <[email protected]> * Update .github/workflows/publish-dts-sdk.yml --------- Signed-off-by: Ryan Lettieri <[email protected]> Co-authored-by: Bernd Verst <[email protected]> Signed-off-by: Albert Callarisa <[email protected]>
1 parent 75f573b commit aae0267

File tree

13 files changed

+777
-54
lines changed

13 files changed

+777
-54
lines changed

‎.github/workflows/pr-validation.yml‎

Lines changed: 58 additions & 51 deletions
Original file line numberDiff line numberDiff line change
@@ -1,51 +1,58 @@
1-
# This workflow will install Python dependencies, run tests and lint with a variety of Python versions
2-
# For more information see: https://docs.github.com/en/actions/automating-builds-and-tests/building-and-testing-python
3-
4-
name: Build Validation
5-
6-
on:
7-
push:
8-
branches: [ "main" ]
9-
pull_request:
10-
branches: [ "main" ]
11-
merge_group:
12-
13-
jobs:
14-
build:
15-
16-
runs-on: ubuntu-latest
17-
strategy:
18-
fail-fast: false
19-
matrix:
20-
python-version: ["3.9", "3.10", "3.11", "3.12", "3.13"]
21-
22-
steps:
23-
- uses: actions/checkout@v4
24-
- name: Set up Python ${{ matrix.python-version }}
25-
uses: actions/setup-python@v5
26-
with:
27-
python-version: ${{ matrix.python-version }}
28-
- name: Install dependencies
29-
run: |
30-
python -m pip install --upgrade pip
31-
pip install flake8 pytest
32-
pip install -r requirements.txt
33-
- name: Lint with flake8
34-
run: |
35-
flake8 . --count --show-source --statistics --exit-zero
36-
- name: Pytest unit tests
37-
run: |
38-
pytest -m "not e2e" --verbose
39-
40-
# Sidecar for running e2e tests requires Go SDK
41-
- name: Install Go SDK
42-
uses: actions/setup-go@v5
43-
with:
44-
go-version: 'stable'
45-
46-
# Install and run the durabletask-go sidecar for running e2e tests
47-
- name: Pytest e2e tests
48-
run: |
49-
go install github.com/microsoft/durabletask-go@main
50-
durabletask-go --port 4001 &
51-
pytest -m "e2e" --verbose
1+
# This workflow will install Python dependencies, run tests and lint with a variety of Python versions
2+
# For more information see: https://docs.github.com/en/actions/automating-builds-and-tests/building-and-testing-python
3+
4+
name: Build Validation
5+
6+
on:
7+
push:
8+
branches: [ "main" ]
9+
pull_request:
10+
branches: [ "main" ]
11+
merge_group:
12+
13+
jobs:
14+
build:
15+
16+
runs-on: ubuntu-latest
17+
strategy:
18+
fail-fast: false
19+
matrix:
20+
python-version: ["3.9", "3.10", "3.11", "3.12", "3.13"]
21+
22+
steps:
23+
- uses: actions/checkout@v4
24+
- name: Set up Python ${{ matrix.python-version }}
25+
uses: actions/setup-python@v5
26+
with:
27+
python-version: ${{ matrix.python-version }}
28+
- name: Install durabletask dependencies
29+
run: |
30+
python -m pip install --upgrade pip
31+
pip install flake8 pytest
32+
pip install -r requirements.txt
33+
- name: Install durabletask-azuremanaged dependencies
34+
working-directory: examples/dts
35+
run: |
36+
python -m pip install --upgrade pip
37+
pip install -r requirements.txt
38+
- name: Lint with flake8
39+
run: |
40+
flake8 . --count --show-source --statistics --exit-zero
41+
- name: Pytest unit tests
42+
working-directory: tests/durabletask
43+
run: |
44+
pytest -m "not e2e and not dts" --verbose
45+
46+
# Sidecar for running e2e tests requires Go SDK
47+
- name: Install Go SDK
48+
uses: actions/setup-go@v5
49+
with:
50+
go-version: 'stable'
51+
52+
# Install and run the durabletask-go sidecar for running e2e tests
53+
- name: Pytest e2e tests
54+
working-directory: tests/durabletask
55+
run: |
56+
go install github.com/microsoft/durabletask-go@main
57+
durabletask-go --port 4001 &
58+
pytest -m "e2e and not dts" --verbose

‎.github/workflows/publish-dts-sdk.yml‎

Lines changed: 110 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,110 @@
1+
name: Publish Durable Task Scheduler to PyPI
2+
3+
on:
4+
push:
5+
branches:
6+
- "main"
7+
tags:
8+
- "azuremanaged-v*" # Only run for tags starting with "azuremanaged-v"
9+
pull_request:
10+
branches:
11+
- "main"
12+
13+
jobs:
14+
lint:
15+
runs-on: ubuntu-latest
16+
steps:
17+
- uses: actions/checkout@v4
18+
- name: Set up Python 3.12
19+
uses: actions/setup-python@v5
20+
with:
21+
python-version: 3.12
22+
- name: Install dependencies
23+
working-directory: durabletask-azuremanaged
24+
run: |
25+
python -m pip install --upgrade pip
26+
pip install setuptools wheel tox
27+
pip install flake8
28+
- name: Run flake8 Linter
29+
working-directory: durabletask-azuremanaged
30+
run: flake8 .
31+
32+
run-docker-tests:
33+
env:
34+
EMULATOR_VERSION: "v0.0.5" # Define the variable
35+
needs: lint
36+
runs-on: ubuntu-latest
37+
steps:
38+
- name: Checkout repository
39+
uses: actions/checkout@v4
40+
41+
- name: Pull Docker image
42+
run: docker pull mcr.microsoft.com/dts/dts-emulator:$EMULATOR_VERSION
43+
44+
- name: Run Docker container
45+
run: |
46+
docker run --name dtsemulator -d -p 8080:8080 mcr.microsoft.com/dts/dts-emulator:$EMULATOR_VERSION
47+
48+
- name: Wait for container to be ready
49+
run: sleep 10 # Adjust if your service needs more time to start
50+
51+
- name: Set environment variables
52+
run: |
53+
echo "TASKHUB=default" >> $GITHUB_ENV
54+
echo "ENDPOINT=http://localhost:8080" >> $GITHUB_ENV
55+
56+
- name: Install durabletask dependencies
57+
run: |
58+
python -m pip install --upgrade pip
59+
pip install flake8 pytest
60+
pip install -r requirements.txt
61+
62+
- name: Install durabletask-azuremanaged dependencies
63+
working-directory: examples/dts
64+
run: |
65+
python -m pip install --upgrade pip
66+
pip install -r requirements.txt
67+
68+
- name: Run the tests
69+
working-directory: tests/durabletask-azuremanaged
70+
run: |
71+
pytest -m "dts" --verbose
72+
73+
publish:
74+
if: startsWith(github.ref, 'refs/tags/azuremanaged-v') # Only run if a matching tag is pushed
75+
needs: run-docker-tests
76+
runs-on: ubuntu-latest
77+
steps:
78+
- name: Checkout code
79+
uses: actions/checkout@v4
80+
81+
- name: Extract version from tag
82+
run: echo "VERSION=${GITHUB_REF#refs/tags/azuremanaged-v}" >> $GITHUB_ENV # Extract version from the tag
83+
84+
- name: Set up Python
85+
uses: actions/setup-python@v5
86+
with:
87+
python-version: "3.12" # Adjust Python version as needed
88+
89+
- name: Install dependencies
90+
run: |
91+
python -m pip install --upgrade pip
92+
pip install build twine
93+
94+
- name: Build package from directory durabletask-azuremanaged
95+
working-directory: durabletask-azuremanaged
96+
run: |
97+
python -m build
98+
99+
- name: Check package
100+
working-directory: durabletask-azuremanaged
101+
run: |
102+
twine check dist/*
103+
104+
- name: Publish package to PyPI
105+
env:
106+
TWINE_USERNAME: __token__
107+
TWINE_PASSWORD: ${{ secrets.PYPI_API_TOKEN_AZUREMANAGED }} # Store your PyPI API token in GitHub Secrets
108+
working-directory: durabletask-azuremanaged
109+
run: |
110+
twine upload dist/*

‎durabletask-azuremanaged/pyproject.toml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -9,7 +9,7 @@ build-backend = "setuptools.build_meta"
99

1010
[project]
1111
name = "durabletask.azuremanaged"
12-
version = "0.1b1"
12+
version = "0.1.2"
1313
description = "Extensions for the Durable Task Python SDK for integrating with the Durable Task Scheduler in Azure"
1414
keywords = [
1515
"durable",

‎examples/dts/README.md‎

Lines changed: 29 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -4,8 +4,13 @@ This directory contains examples of how to author durable orchestrations using t
44

55
## Prerequisites
66

7-
All the examples assume that you have a Durable Task Scheduler taskhub created.
7+
There are 2 separate ways to run an example:
8+
1. Using the emulator.
9+
2. Using a real scheduler and taskhub.
810

11+
All the examples by defualt assume that you have a Durable Task Scheduler taskhub created.
12+
13+
## Running with a scheduler and taskhub resource
914
The simplest way to create a taskhub is by using the az cli commands:
1015

1116
1. Create a scheduler:
@@ -46,6 +51,29 @@ The simplest way to create a taskhub is by using the az cli commands:
4651

4752
1. Grant yourself the `Durable Task Data Contributor` role over your scheduler
4853

54+
## Running with the emulator
55+
The emulator is a simulation of a scheduler and taskhub. It is the 'backend' of the durabletask-azuremanaged system packaged up into an easy to use docker container. For these steps, it is assumed that you are using port 8080.
56+
57+
In order to use the emulator for the examples, perform the following steps:
58+
1. Install docker if it is not already installed.
59+
60+
2. Pull down the docker image for the emulator:
61+
`docker pull mcr.microsoft.com/dts/dts-emulator:v0.0.4`
62+
63+
3. Run the emulator and wait a few seconds for the container to be ready:
64+
`docker run --name dtsemulator -d -p 8080:8080 mcr.microsoft.com/dts/dts-emulator:v0.0.4`
65+
66+
4. Set the environment variables that are referenced and used in the examples:
67+
1. If you are using windows powershell:
68+
`$env:TASKHUB="default"`
69+
`$env:ENDPOINT="http://localhost:8080"`
70+
2. If you are using bash:
71+
`export TASKHUB=default`
72+
`export ENDPOINT=http://localhost:8080`
73+
74+
5. Finally, edit the examples to change the `token_credential` input of both the `DurableTaskSchedulerWorker` and `DurableTaskSchedulerClient` to a value of `None`
75+
76+
4977
## Running the examples
5078

5179
Now, you can simply execute any of the examples in this directory using `python3`:

‎examples/dts/requirements.txt‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,6 @@
1+
autopep8
2+
grpcio>=1.60.0 # 1.60.0 is the version introducing protobuf 1.25.X support, newer versions are backwards compatible
3+
protobuf
4+
azure-identity
5+
durabletask-azuremanaged
6+
durabletask

‎pyproject.toml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -9,7 +9,7 @@ build-backend = "setuptools.build_meta"
99

1010
[project]
1111
name = "durabletask"
12-
version = "0.2b1"
12+
version = "0.2.0"
1313
description = "A Durable Task Client SDK for Python"
1414
keywords = [
1515
"durable",

‎tests/durabletask-azuremanaged/__init__.py‎

Whitespace-only changes.

‎tests/durabletask-azuremanaged/test_dts_activity_sequence.py‎

Lines changed: 69 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,69 @@
1+
"""End-to-end sample that demonstrates how to configure an orchestrator
2+
that calls an activity function in a sequence and prints the outputs."""
3+
import os
4+
5+
from durabletask import client, task
6+
from durabletask.azuremanaged.client import DurableTaskSchedulerClient
7+
from durabletask.azuremanaged.worker import DurableTaskSchedulerWorker
8+
9+
import pytest
10+
11+
12+
pytestmark = pytest.mark.dts
13+
14+
def hello(ctx: task.ActivityContext, name: str) -> str:
15+
"""Activity function that returns a greeting"""
16+
return f'Hello {name}!'
17+
18+
19+
def sequence(ctx: task.OrchestrationContext, _):
20+
"""Orchestrator function that calls the 'hello' activity function in a sequence"""
21+
# call "hello" activity function in a sequence
22+
result1 = yield ctx.call_activity(hello, input='Tokyo')
23+
result2 = yield ctx.call_activity(hello, input='Seattle')
24+
result3 = yield ctx.call_activity(hello, input='London')
25+
26+
# return an array of results
27+
return [result1, result2, result3]
28+
29+
30+
# Read the environment variable
31+
taskhub_name = os.getenv("TASKHUB")
32+
33+
# Check if the variable exists
34+
if taskhub_name:
35+
print(f"The value of TASKHUB is: {taskhub_name}")
36+
else:
37+
print("TASKHUB is not set. Please set the TASKHUB environment variable to the name of the taskhub you wish to use")
38+
print("If you are using windows powershell, run the following: $env:TASKHUB=\"<taskhubname>\"")
39+
print("If you are using bash, run the following: export TASKHUB=\"<taskhubname>\"")
40+
exit()
41+
42+
# Read the environment variable
43+
endpoint = os.getenv("ENDPOINT")
44+
45+
# Check if the variable exists
46+
if endpoint:
47+
print(f"The value of ENDPOINT is: {endpoint}")
48+
else:
49+
print("ENDPOINT is not set. Please set the ENDPOINT environment variable to the endpoint of the scheduler")
50+
print("If you are using windows powershell, run the following: $env:ENDPOINT=\"<schedulerEndpoint>\"")
51+
print("If you are using bash, run the following: export ENDPOINT=\"<schedulerEndpoint>\"")
52+
exit()
53+
54+
# configure and start the worker
55+
with DurableTaskSchedulerWorker(host_address=endpoint, secure_channel=True,
56+
taskhub=taskhub_name, token_credential=None) as w:
57+
w.add_orchestrator(sequence)
58+
w.add_activity(hello)
59+
w.start()
60+
61+
# Construct the client and run the orchestrations
62+
c = DurableTaskSchedulerClient(host_address=endpoint, secure_channel=True,
63+
taskhub=taskhub_name, token_credential=None)
64+
instance_id = c.schedule_new_orchestration(sequence)
65+
state = c.wait_for_orchestration_completion(instance_id, timeout=60)
66+
if state and state.runtime_status == client.OrchestrationStatus.COMPLETED:
67+
print(f'Orchestration completed! Result: {state.serialized_output}')
68+
elif state:
69+
print(f'Orchestration failed: {state.failure_details}')

0 commit comments

Comments
 (0)