Skip to content

Commit 59032d7

Browse files
davidmrdavidscgbearShervyna Ruanpriyaananthasankarasedighi
authored
Promote to master (#130)
* task_utility docstrings * ActivityType docstrings Contains refactoring to conform to python naming conventions to * Doc strings for Actions Call Activity and CallActivity with Retry * Doc Strings History Included refactor for enum naming conventions in python * Final docstring bits docstrings for tasks and json utilities * add waitforexternalevent basic files * fix bugs to make waitForExternalEvent working * add waitforexternalevent samples * remove explicit binding_info imp defined in conftest as a fixture, explicit import not required. * demo sample for function chaining with docstring and json changes * demo sample for function chaining with docstring and json changes * added dev pipeline status in README * minor fixes(variable name, delete comment) * flake8 fixes * add docstrings * implement task_any function * change pip install library and add docstrings to samples * change pip install library and add docstrings to samples * unittest for waitforexternalevents * fix bugs after merging dev * fix flake8 * Base implementation of tests test initial call test get activity and build of task set * parrot values success * test full complete flow * test failed scenario add missing bits in task_all to account for a failed task * docstring to numpy format * minor changes (rename, remove logging) * unittest for task_any, added tasks_test_utils * add class __eq__ function for Waitforexternalevent actions * add samples readme doc * fix flake8 * Refactoring HistoryEvent Get rid of all of those hardcoded property value references * add docstrings for HistoryEvent class * Refactor json conversion refactor the classes that are parsing json strings to remove all of the hardcoded property names also allows for the classes dangle additional attributes that may be present but not explicitly used for construction * simple Fan out fan in sample * Fix flake errors * Remove local debugging bits * remove state in task_any * add handle faulted task_any case +unittest * Undo De Morgan's Law cause it's really hard to read * replace filters with list comprehension more concise/readable * Add documentation for tracking API implementation * move datetime format string to azure package * replace filter with list comprehension more concise method * remove extra zimezone from format causing parsing warnings * Push context initialization our of handle method have this dependency injected instead of built within * able to pass in tasksets to task_any and task_all * update unittest for adding timestamp to taskset, add unittest for passing taskset to task_any as args * fix bugs in task_all(when all tasks fail), and fix unittest for that scenario * fix flake8 * test from orchestrator level(draft) * Remove IFunctionContext abstraction unneeded layer of abstraction with a DUCK typing language like Python * Starting of schema validation bits * createCheckStatusResponse() * wire up schema validation into the orchestrator tests * Test commit * fix flake 8 issues * fix pytest, remove task_any_tests from orchestrator level * fix flake8 * implement raise_event api, fix docstring * add unittest, create separate methods, fix naming style, handle placeholder in url validation * Fan Out Fan In sample uses tensorflow to predict whether images from bing are dogs or cats * fix flake8 * add aiohttp to requirement.txt and setup.py * add async await to start_new and raise_event api, flake8 * update api_implementation_status * fix variable naming style in docstring * update sample for external events, update readme in sample/external_events * Refactoring and docstrings removed the get generation data function. With the configurability, we can increase the volume of images to simulate the high CPU scenario. No need to fake it now. refactored some of the larger, multi task functions to create better readability added some docstrings on the functions to aid in understanding what is going on without the visual aid of powerpoint * Continue as new implementation * new_guid implementation * Fix flake8 issues * update sample to simple version * add func.httpresponse in durableorchestratorClient.py * update docstring for createcheckstatusresponse api * fix flake8 * update sample for create_check_status_response fixes * update pytest for the changes in createcheckstatusresponse api, add azure-function to requirement.txt for pytest * Implementation of call_http * Remove traceback print statement * remove url validator * Updates from demo feedback rename guid to uuid attach additional context attributes to an attribute of the durable context, not directly to the durable context * Fix flake8 * copy paste error giving a more descriptive orchestrator name * remove abcd from sample * Update API_IMPLEMENTATION_STATUS.md Sprint 7 update * Unit tests for call_http found a couple of bugs with these * merged shervyna changes * Fix the squiggles mostly formatting/inspection surpession * Move iAction moving and renaming IAction to Action * Update actions to implement action base class * remove the squiggles some minor formatting updates * Fix format of docstring * update to use rpc_base_url for start new and raise event urls * minor updates docstring on new function add Action to __init__ for actions module * flake8 bits * Add action_type property of base class * update docstrings per convention * docstring convention updates made sure docstrings conformed to the same format and indentions * Base class implementations for get_status * refactor massive string into a Dict[str, Any] object makes for a more readable bit of code * Move duplicate test rpc url constants to centralized location * adding updated contributing and getting started guides * get_status implementation includes refactoring of aiohttp calls to be able to test the rest of the bits without this server requirement tests for GetStatusOptions included base test for get_status_success * test for DurableOrchestrationStatus parsing * Add non ok message tests * add raises exception test * get_status_by and get_status_all implementation * flake8 fixes * fix docstring for get_status_by parameters copy paste error * addig nox to the project - updated requirements file * Updated noxfile to include flake, move getting started to investigations folder * typo in the commit from Jscript to python * PR recommendations remove PyCharm # noinspection tags move get and post async functions to util module replace abc123 stand-in test value with a uuid stand-in * add docstrings for new public functions * Continued Client API implementation purge_instance_history purge_instance_history_by terminate wait_for_completion_or_create_check_status_response * bug fixed for retry function. * refactor tests remove the redundant function declarations * unit tests for purge history and terminate * unit tests for wait or create check response includes bug fixes found from tests * update implementation status with features included in branch * update name match the name of the class under test * change to use asynio sleep don't want a blocking sleep call here * Use azure-functions>=1.2.0 * Make DurableOrchestrationContext available for import * Update start_new type hints to return str * Add get_input to DurableOrchestrationContext * Allow passing OrchestrationContext to create - maintain backwards compatibility when JSON body is passed * Change str to Awaitable[str] * Add get_input tests, get_input always returns str * Fix flake8 errors * add furl to setup.py * Fix merge error * # This is a combination of 3 commits. (#58) Fix samples to use the latest func.OrchestrationContext annotation Use new syntax to replace main endpoint Fix old python_durable_bindings samples * Replace badge with the new Azure Functions Python pipeline * edited contributor guide * furl is required for this project - added it to the requirements.doc * fixed pytest-asyncio in requirements.txt * fixed pytest-asyncio in requirements.txt * fixed pytest-asyncio in requirements.txt * Remove static method tag (#72) Remove static method tag (#72): residual bits left over from when a part of the client class. Causing failures in current state. Also removing unnecessary wrapper of http helper calls * Remove furl from RpcManagementOptions only durable class that is requiring this package, and is only using it to construct a url query string. Lighter requirements list to just use string joining * release artifacts * Added DateUtil and Furl to install_requires * changelog and setup.py changes * Update azure-pipelines.yml for Azure Pipelines * Update azure-pipelines.yml for Azure Pipelines * Update azure-pipelines.yml for Azure Pipelines * Update azure-pipelines.yml for Azure Pipelines * Update azure-pipelines.yml for Azure Pipelines * Update azure-pipelines.yml for Azure Pipelines * Update azure-pipelines.yml for Azure Pipelines * Update azure-pipelines.yml for Azure Pipelines * tested version of pipeline yaml * Update azure-pipelines.yml for Azure Pipelines * Update azure-pipelines.yml for Azure Pipelines * Update azure-pipelines.yml for Azure Pipelines * Update azure-pipelines.yml for Azure Pipeline * Update azure-pipelines.yml for Azure Pipelines * Update azure-pipelines.yml for Azure Pipelines * Update azure-pipelines.yml for Azure Pipelines * Update azure-pipelines.yml for Azure Pipelines * Update azure-pipelines.yml for Azure Pipelines * Update azure-pipelines.yml for Azure Pipelines * setup.py CD testing * setup.py CD testing * PyPI testing * added info logging to the external event sample * Correctness and documentation of the samples (#85) * validated all samples except tensorflow * deleted non-samples, tested tensorflow sample, added types and comments * new readmes for fan-out-fan-in and function-chaining * renamed some functions so they are easier to reason about * disable aiohttp's input validation... potentially unsafe. ExternalEvents now shows no warnings! * Fixed Contributor Guide Issues * Fixed Contributor Guide Issues * Input value not required for start_new * input values not required for start_new * Add limitations, link to quickstart, update samples * Update example * Add link * Update words * Add versions * Update links * versioning-via-tags is enabled (#101) * updated readme for external events (#103) * Readme file for FanOutFanIn (#104) * Readme file for FanOutFanIn * Readme file for FanOutFanIn * Readme file for FanOutFanIn * Add activity trigger return type sample (#105) * Add sample for activity trigger type checks * Remove extensions.csproj Co-authored-by: Priya Ananthasankar <[email protected]> * misc contributions (#108) * Set Custom Status (#110) * custom status * custom status * custom status * custom status * custom status * custom status * custom status * custom status * set custom status rework comments * Enabled nox to recognize docstring formatting errors (#122) * enables trivial orchestrators * added flake8-docstring, fixed arg to flake8 ,fixed formatting on noxfile * removed accidental commit from other PR * enables trivial orchestrators (#121) * removed grpc folder (#120) * [WIP] Semi-automatic type-serialization (#109) * Promote to master (#99) (#126) Co-authored-by: Shawn Gaul <[email protected]> Co-authored-by: Shervyna Ruan <[email protected]> Co-authored-by: Shawn Gaul <[email protected]> Co-authored-by: asedighi <[email protected]> Co-authored-by: Anthony <[email protected]> Co-authored-by: Anthony Chu <[email protected]> Co-authored-by: Hanzhang Zeng (Roger) <[email protected]> Co-authored-by: Hanzhang Zeng (Roger) <[email protected]> Co-authored-by: David Justo <[email protected]> Co-authored-by: Priya Ananthasankar <[email protected]> Co-authored-by: Shawn Gaul <[email protected]> Co-authored-by: Shervyna Ruan <[email protected]> Co-authored-by: Shawn Gaul <[email protected]> Co-authored-by: asedighi <[email protected]> Co-authored-by: Anthony <[email protected]> Co-authored-by: Anthony Chu <[email protected]> Co-authored-by: Hanzhang Zeng (Roger) <[email protected]> Co-authored-by: Hanzhang Zeng (Roger) <[email protected]> * Promote to master (#99) (#127) * Monitoring Feature (#119) * updated CHANGELOG (#128) * Changelog includes links (#129) * updated CHANGELOG * improved changelog descriptions w/ issue links Co-authored-by: Shawn Gaul <[email protected]> Co-authored-by: Shervyna Ruan <[email protected]> Co-authored-by: Priya Ananthasankar <[email protected]> Co-authored-by: Shawn Gaul <[email protected]> Co-authored-by: asedighi <[email protected]> Co-authored-by: Anthony <[email protected]> Co-authored-by: Anthony Chu <[email protected]> Co-authored-by: Hanzhang Zeng (Roger) <[email protected]> Co-authored-by: Hanzhang Zeng (Roger) <[email protected]>
1 parent fc15a55 commit 59032d7

24 files changed

+527
-3
lines changed

CHANGELOG.md

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -2,6 +2,15 @@
22

33
All notable changes to this project will be documented in this file.
44

5+
## 1.0.0b6
6+
7+
- [Create timer](https://github.com/Azure/azure-functions-durable-python/issues/35) functionality available
8+
9+
## 1.0.0b5
10+
11+
- [Object serialization](https://github.com/Azure/azure-functions-durable-python/issues/90) made available
12+
- [Can set custom status](https://github.com/Azure/azure-functions-durable-python/issues/117) of orchestration
13+
514
## 1.0.0b3-b4
615
- Release to test CD pipeline with push to PyPI
716

azure/durable_functions/models/DurableOrchestrationContext.py

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -9,7 +9,7 @@
99
from ..models.Task import Task
1010
from ..models.TokenSource import TokenSource
1111
from ..tasks import call_activity_task, task_all, task_any, call_activity_with_retry_task, \
12-
wait_for_external_event_task, continue_as_new, new_uuid, call_http
12+
wait_for_external_event_task, continue_as_new, new_uuid, call_http, create_timer_task
1313
from azure.functions._durable_functions import _deserialize_custom_object
1414

1515

@@ -51,6 +51,7 @@ def __init__(self,
5151
self.continue_as_new = lambda i: continue_as_new(input_=i)
5252
self.task_any = lambda t: task_any(tasks=t)
5353
self.task_all = lambda t: task_all(tasks=t)
54+
self.create_timer = lambda d: create_timer_task(state=self.histories, fire_at=d)
5455
self.decision_started_event: HistoryEvent = \
5556
[e_ for e_ in self.histories
5657
if e_.event_type == HistoryEventType.ORCHESTRATOR_STARTED][0]
Lines changed: 42 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,42 @@
1+
from typing import Any, Dict
2+
3+
from .ActionType import ActionType
4+
from ..utils.json_utils import add_attrib, add_datetime_attrib
5+
import datetime
6+
7+
8+
class CreateTimerAction:
9+
"""Defines the structure of the Create Timer object.
10+
11+
Returns
12+
-------
13+
Information needed by durable extension to schedule the activity
14+
15+
Raises
16+
------
17+
ValueError
18+
if the event fired is not of valid datetime object
19+
"""
20+
21+
def __init__(self, fire_at: datetime, is_cancelled: bool = False):
22+
self.action_type: ActionType = ActionType.CREATE_TIMER
23+
self.fire_at: datetime = fire_at
24+
self.is_cancelled: bool = is_cancelled
25+
26+
if not isinstance(self.fire_at, datetime.date):
27+
raise ValueError("fireAt: Expected valid datetime object but got ", self.fire_at)
28+
29+
def to_json(self) -> Dict[str, Any]:
30+
"""
31+
Convert object into a json dictionary.
32+
33+
Returns
34+
-------
35+
Dict[str, Any]
36+
The instance of the class converted into a json dictionary
37+
"""
38+
json_dict = {}
39+
add_attrib(json_dict, self, 'action_type', 'actionType')
40+
add_datetime_attrib(json_dict, self, 'fire_at', 'fireAt')
41+
add_attrib(json_dict, self, 'is_cancelled', 'isCanceled')
42+
return json_dict

azure/durable_functions/models/actions/__init__.py

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -5,12 +5,14 @@
55
from .CallActivityWithRetryAction import CallActivityWithRetryAction
66
from .WaitForExternalEventAction import WaitForExternalEventAction
77
from .CallHttpAction import CallHttpAction
8+
from .CreateTimerAction import CreateTimerAction
89

910
__all__ = [
1011
'Action',
1112
'ActionType',
1213
'CallActivityAction',
1314
'CallActivityWithRetryAction',
1415
'CallHttpAction',
15-
'WaitForExternalEventAction'
16+
'WaitForExternalEventAction',
17+
'CreateTimerAction'
1618
]

azure/durable_functions/tasks/__init__.py

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -8,6 +8,7 @@
88
from .continue_as_new import continue_as_new
99
from .new_uuid import new_uuid
1010
from .call_http import call_http
11+
from .create_timer import create_timer_task
1112

1213
__all__ = [
1314
'call_activity_task',
@@ -18,5 +19,6 @@
1819
'task_all',
1920
'task_any',
2021
'should_suspend',
21-
'wait_for_external_event_task'
22+
'wait_for_external_event_task',
23+
'create_timer_task'
2224
]
Lines changed: 41 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,41 @@
1+
from typing import List
2+
from ..models.actions.CreateTimerAction import CreateTimerAction
3+
from ..models.history import HistoryEvent
4+
from .task_utilities import find_task_timer_created, find_task_retry_timer_fired, set_processed
5+
import datetime
6+
from .timer_task import TimerTask
7+
8+
9+
def create_timer_task(state: List[HistoryEvent],
10+
fire_at: datetime) -> TimerTask:
11+
"""Durable Timers are used in orchestrator function to implement delays.
12+
13+
Parameters
14+
----------
15+
state : List[HistoryEvent]
16+
The list of history events to search to determine the current state of the activity
17+
fire_at : datetime
18+
The time interval to fire the timer trigger
19+
20+
Returns
21+
-------
22+
TimerTask
23+
A Durable Timer Task that schedules the timer to wake up the activity
24+
"""
25+
new_action = CreateTimerAction(fire_at)
26+
27+
timer_created = find_task_timer_created(state, fire_at)
28+
timer_fired = find_task_retry_timer_fired(state, timer_created)
29+
30+
set_processed([timer_created, timer_fired])
31+
32+
if timer_fired:
33+
return TimerTask(
34+
is_completed=True, action=new_action,
35+
timestamp=timer_fired.timestamp,
36+
id_=timer_fired.event_id)
37+
else:
38+
return TimerTask(
39+
is_completed=False, action=new_action,
40+
timestamp=None,
41+
id_=None)

azure/durable_functions/tasks/task_utilities.py

Lines changed: 24 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,6 @@
11
import json
22
from ..models.history import HistoryEventType
3+
from ..constants import DATETIME_STRING_FORMAT
34
from azure.functions._durable_functions import _deserialize_custom_object
45

56

@@ -118,6 +119,29 @@ def find_task_failed(state, scheduled_task):
118119
return tasks[0]
119120

120121

122+
def find_task_timer_created(state, fire_at):
123+
"""Locate the Timer Created Task.
124+
125+
Within the state passed, search for an event that has hasn't been processed,
126+
is a timer created task type,
127+
and has the an event id that is one higher then Scheduled Id of the provided
128+
failed task provided.
129+
"""
130+
if fire_at is None:
131+
return None
132+
133+
tasks = []
134+
for e in state:
135+
if e.event_type == HistoryEventType.TIMER_CREATED and hasattr(e, "FireAt"):
136+
if e.FireAt == fire_at.strftime(DATETIME_STRING_FORMAT):
137+
tasks.append(e)
138+
139+
if len(tasks) == 0:
140+
return None
141+
142+
return tasks[0]
143+
144+
121145
def find_task_retry_timer_created(state, failed_task):
122146
"""Locate the Timer Created Task.
123147
Lines changed: 47 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,47 @@
1+
from ..models.Task import Task
2+
3+
4+
class TimerTask(Task):
5+
"""Represents a pending timer.
6+
7+
All pending timers must be completed or canceled for an orchestration to complete.
8+
9+
Example: Cancel a timer
10+
```
11+
timeout_task = context.df.create_timer(expiration_date)
12+
if not timeout_task.is_completed():
13+
timeout_task.cancel()
14+
```
15+
"""
16+
17+
def __init__(self, action, is_completed, timestamp, id_):
18+
self._action = action
19+
self._is_completed = is_completed
20+
self._timestamp = timestamp
21+
self._id = id_
22+
23+
super().__init__(self._is_completed, False,
24+
self._action, None, self._timestamp, self._id, None)
25+
26+
def is_cancelled(self) -> bool:
27+
"""Check of a timer is cancelled.
28+
29+
Returns
30+
-------
31+
bool
32+
Returns whether a timer has been cancelled or not
33+
"""
34+
return self._action.is_cancelled
35+
36+
def cancel(self):
37+
"""Cancel a timer.
38+
39+
Raises
40+
------
41+
ValueError
42+
Raises an error if the task is already completed and an attempt is made to cancel it
43+
"""
44+
if not self._is_completed:
45+
self._action.is_cancelled = True
46+
else:
47+
raise ValueError("Cannot cancel a completed task.")

samples/aml_monitoring/.funcignore

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,5 @@
1+
.git*
2+
.vscode
3+
local.settings.json
4+
test
5+
py36

samples/aml_monitoring/.gitignore

Lines changed: 44 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,44 @@
1+
bin
2+
obj
3+
csx
4+
.vs
5+
edge
6+
Publish
7+
8+
*.user
9+
*.suo
10+
*.cscfg
11+
*.Cache
12+
project.lock.json
13+
14+
/packages
15+
/TestResults
16+
17+
/tools/NuGet.exe
18+
/App_Data
19+
/secrets
20+
/data
21+
.secrets
22+
appsettings.json
23+
local.settings.json
24+
25+
node_modules
26+
dist
27+
28+
# Local python packages
29+
.python_packages/
30+
31+
# Python Environments
32+
.env
33+
.venv
34+
env/
35+
venv/
36+
ENV/
37+
env.bak/
38+
venv.bak/
39+
40+
# Byte-compiled / optimized / DLL files
41+
__pycache__/
42+
*.py[cod]
43+
*$py.class
44+
py36

0 commit comments

Comments
 (0)