Skip to content

Commit 9946f25

Browse files
alanbchristieAlan Christiekaliif
authored
Initial 2.0 feature set (#40)
* feat: kind-version now 2025.2 BREAKING CHANGE * feat: Initial schema for replicate step declaration * feat: Add get_generated_outputs_for_step_output() to API adapter * feat: Rename new API method * feat: Add replica to step creation (and step-by-name query) * feat: Removed 'as' from workflow mapping output declaration * feat: Add get_running_workflow_step_output_values_for_output() to API * fix: Removed rogue 'print' statement * test: Add mock of step outputs * fix: it's a mess * docs: Add instance-directory comment * feat: Creating instances now adds instance-directory property to step record * fix: get_instance() response type * test: Add tests for instance_directory * build: Add experimental json copy of the schema * style: Removed unnecessary json schema file * fix: Better rfc1035-label-name pattern * fix: Better rfc1035-label-name regex * test: Fix decoder tests * test: Fix test module name (for consistency) * Remove unnecessary logic * fix: stashing * fix: stashing * refactor: Major refactor (new variable-mapping schema) * refactor: from-workflow-variable becomes from-workflow * refactor: from-workflow-variable becomes from-workflow * refactor: Replicate now uses variable not input * fix: No longer need realise-outputs * build: Add devcontainer * build: No need for DinD * docs: Docs on devcontainer * docs: Doc tweak * feat: Some work on the refactored engine * fix: More fixes for engine * fix: More work on the decoder * fix: Variable mapping now exposed as a Translation dataclass * fix: Major refactoring of logic (for new launch/workflow API) * feat: First successful replicating workflow test * feat: Use of decoder 2.4.0 (traits) * refactor: Switch away from workflow replicate property * refactor: Refactored using decoder 2.5.0 * docs: Doc tweak * refactor: variable-map is now 'plumbing' and Translation is a 'Connector' * refactor: Better function and variable naming (plumbing) * feat: new _prepare_step_variables function * feat: Refactoring * refactor: More combiner logic * feat: refactor definition of an output * fix: Add get_status_of_all_step_instances_by_name implementation (and fix step replicas) * docs: Doc tweak * fix: Typo in YAML * feat: Minor work on combiner logic * fix: First very basic combiner run * docs: Doc tweak * fix: replica always starts at 0 * docs: Doc tweak * feat: Add from-link-prefix variables * feat: Switch to pre-defined variables (rather then link) * fix: Fix input output hanes in decoder * fix: Add from-project support * refactor: lin-prefix becomes link-glob * fix: Return kwargs to launch() * feat: Add dependent_instances to LaunchParameters * feat: Populate dependent_instances * refactor: link-glob is now instance-link-glob * refactor: Remove to/from project schema * feat: Add outputs to step prepration response * test: Attempt to fix tests * test: Fix instance creation * fix: Fix test execution * docs: Significant level of module documentation * docs: More docs * docs: Doc tweak * docs: More doc and typo corrections * docs: Code typos corrected * dev: Logs end of step and end of running workflow * dev: Engine now collects inputs and sets LP inputs & outputs * style: Log inputs * dev: Fix handling of launch() result * fix: Better handling of get_running_workflow_step_by_name() response * feat: Engine now attempts to prefix inputs (from prior steps) * fix: Fix prior-step prefix * fix: Launch parameter sets are now lists * dev: fix build * docs: Doc tweak * dev: Fix instance prefix * fix: Engine now handles lack of job better * fix: Validator now ensures jobs exist (run-level test) * fix: Better testing of validation (job must be present) * feat: Better validation error message * test: Fix tests * fix: Better handling of lack of outputs * docs: Doc tweak * fix: Fix iteration input path * test: Disable unit test * fix: Support for dirsGlob * fix: Better step preparation initial log (combiners) * fix: Combiners no longer get built-in variables automatically (#39) --------- Co-authored-by: Alan Christie <[email protected]> Co-authored-by: Kalev Takkis <[email protected]>
1 parent 0a4d554 commit 9946f25

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

41 files changed

+2228
-1806
lines changed

.cz.yaml

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -9,8 +9,8 @@
99
commitizen:
1010
name: cz_customize
1111
customize:
12-
schema_pattern: "^(?P<change_type>feat|fix|perf|refactor|remove|style|test|build|docs|chore|ci|BREAKING CHANGE)(?:\\((?P<scope>[^()\\r\\n]*)\\)|\\()?(?P<breaking>!)?:\\s(?P<message>.*)?"
13-
commit_parser: "^(?P<change_type>feat|fix|perf|refactor|remove|style|test|build|docs|chore|ci|BREAKING CHANGE)(?:\\((?P<scope>[^()\\r\\n]*)\\)|\\()?(?P<breaking>!)?:\\s(?P<message>.*)?"
12+
schema_pattern: "^(?P<change_type>feat|fix|perf|refactor|remove|style|test|build|docs|chore|ci|dev|BREAKING CHANGE)(?:\\((?P<scope>[^()\\r\\n]*)\\)|\\()?(?P<breaking>!)?:\\s(?P<message>.*)?"
13+
commit_parser: "^(?P<change_type>feat|fix|perf|refactor|remove|style|test|build|docs|chore|ci|dev|BREAKING CHANGE)(?:\\((?P<scope>[^()\\r\\n]*)\\)|\\()?(?P<breaking>!)?:\\s(?P<message>.*)?"
1414
# The changelog_pattern identifies the commit types
1515
# that will be included.
1616
# Build the changelog with 'cz ch' on the staging or production branches.

.devcontainer/devcontainer.json

Lines changed: 41 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,41 @@
1+
// For format details, see https://aka.ms/devcontainer.json. For config options, see the
2+
// README at: https://github.com/devcontainers/templates/tree/main/src/python
3+
{
4+
"name": "WorkflowEngine Python 3.13",
5+
"image": "mcr.microsoft.com/devcontainers/python:1-3.13-bullseye",
6+
"features": {
7+
"ghcr.io/devcontainers/features/git:1": {
8+
"ppa": true,
9+
"version": "os-provided"
10+
}
11+
},
12+
// We mount bash history in an attempt to preserver history
13+
// between container restarts
14+
// (see https://code.visualstudio.com/remote/advancedcontainers/persist-bash-history)
15+
"mounts": [
16+
"source=projectname-bashhistory,target=/commandhistory,type=volume"
17+
],
18+
"customizations": {
19+
"vscode": {
20+
"extensions": [
21+
"codezombiech.gitignore",
22+
"donjayamanne.githistory",
23+
"donjayamanne.git-extension-pack",
24+
"eamodio.gitlens",
25+
"github.vscode-github-actions",
26+
"ms-kubernetes-tools.vscode-kubernetes-tools",
27+
"ms-python.vscode-pylance",
28+
"sourcery.sourcery",
29+
"streetsidesoftware.code-spell-checker",
30+
"trond-snekvik.simple-rst",
31+
"vivaxy.vscode-conventional-commits",
32+
"yzhang.markdown-all-in-one"
33+
]
34+
}
35+
},
36+
"postCreateCommand": {
37+
"Install Python requirements": "pip3 install --user -r requirements.txt",
38+
"Fix Volume Permissions": "sudo chown -R $(whoami): /commandhistory"
39+
},
40+
"forwardPorts": []
41+
}

.gitignore

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -3,6 +3,7 @@ dist/
33
**/__pycache__/
44
**/*.pickle
55
tests/project-root/project-*/
6+
**/.DS_Store
67

78
# temp files
89
*~

README.rst

Lines changed: 5 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -38,10 +38,9 @@ The project's written in Python and uses `Poetry`_ for dependency and package
3838
management. We also use `pre-commit`_ to manage our pre-commit hooks, which
3939
rely on `black`_, `mypy`_, `pylint`_, amongst others.
4040

41-
Create your environment::
41+
From within a VS Code `devcontainer`_ environment (recommended)::
4242

43-
poetry shell
44-
poetry install --with dev
43+
poetry install --with dev --sync
4544
pre-commit install -t commit-msg -t pre-commit
4645

4746
And then start by running the pre-commit hooks to ensure you're stating with a
@@ -51,9 +50,10 @@ _clean_ project::
5150

5251
And then run the tests::
5352

54-
coverage run -m pytest
55-
coverage report
53+
poetry run coverage run -m pytest
54+
poetry run coverage report
5655

56+
.. _devcontainer: https://code.visualstudio.com/docs/devcontainers/containers
5757
.. _Poetry: https://python-poetry.org
5858
.. _pre-commit: https://pre-commit.com
5959
.. _black: https://github.com/psf/black

poetry.lock

Lines changed: 306 additions & 237 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

pyproject.toml

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -14,7 +14,7 @@ packages = [
1414
[tool.poetry.dependencies]
1515
python = "^3.12"
1616
im-protobuf = "^8.2.0"
17-
im-data-manager-job-decoder = "^2.1.0"
17+
im-data-manager-job-decoder = "^2.5.0"
1818
jsonschema = "^4.21.1"
1919
pyyaml = ">= 5.3.1, < 7.0"
2020

requirements.txt

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,2 @@
1+
poetry == 1.8.5
2+
pre-commit == 4.2.0

tests/instance_launcher.py

Lines changed: 40 additions & 14 deletions
Original file line numberDiff line numberDiff line change
@@ -68,39 +68,64 @@ def __init__(
6868
elif os.path.isdir(file_path):
6969
shutil.rmtree(file_path)
7070

71-
def launch(self, launch_parameters: LaunchParameters) -> LaunchResult:
71+
def launch(self, *, launch_parameters: LaunchParameters) -> LaunchResult:
7272
assert launch_parameters
7373
assert launch_parameters.project_id == TEST_PROJECT_ID
7474
assert launch_parameters.specification
7575
assert isinstance(launch_parameters.specification, dict)
7676

7777
os.makedirs(EXECUTION_DIRECTORY, exist_ok=True)
7878

79-
# We're passed a RunningWorkflowStep ID but a record is expected to have been
80-
# created bt the caller, we simply create instance records.
81-
response, _ = self._api_adapter.get_running_workflow_step(
82-
running_workflow_step_id=launch_parameters.running_workflow_step_id
83-
)
84-
# Now simulate the creation of a Task and Instance record
85-
response = self._api_adapter.create_instance(
86-
running_workflow_step_id=launch_parameters.running_workflow_step_id
87-
)
79+
if launch_parameters.step_replication_number:
80+
assert (
81+
launch_parameters.step_replication_number
82+
<= launch_parameters.total_number_of_replicas
83+
)
84+
85+
# Create an Instance record (and dummy Task ID)
86+
response = self._api_adapter.create_instance()
8887
instance_id = response["id"]
8988
task_id = "task-00000000-0000-0000-0000-000000000001"
9089

91-
# Apply variables to the step's Job command.
90+
# Create a running workflow step
91+
assert launch_parameters.running_workflow_id
92+
assert launch_parameters.step_name
93+
response, _ = self._api_adapter.create_running_workflow_step(
94+
running_workflow_id=launch_parameters.running_workflow_id,
95+
step=launch_parameters.step_name,
96+
instance_id=instance_id,
97+
replica=launch_parameters.step_replication_number,
98+
replicas=launch_parameters.total_number_of_replicas,
99+
)
100+
assert "id" in response
101+
rwfs_id: str = response["id"]
102+
# And add the variables we've been provided with
103+
if launch_parameters.variables:
104+
_ = self._api_adapter.set_running_workflow_step_variables(
105+
running_workflow_step_id=rwfs_id, variables=launch_parameters.variables
106+
)
107+
108+
# Now add the running workflow ID ot the instance record.
109+
self._api_adapter.set_instance_running_workflow_step_id(
110+
instance_id=instance_id,
111+
running_workflow_step_id=rwfs_id,
112+
)
113+
114+
# Get the job defitnion.
115+
# This is expected to exist in the tests/job-definitions directory.
92116
job, _ = self._api_adapter.get_job(
93117
collection=launch_parameters.specification["collection"],
94118
job=launch_parameters.specification["job"],
95119
version="do-not-care",
96120
)
97121
assert job
98122

99-
# Now apply the variables to the command
123+
# Now apply the provided variables to the command.
124+
# The command may not need any, but we do the decoding anyway.
100125
decoded_command, status = job_decoder.decode(
101126
job["command"],
102-
launch_parameters.specification_variables,
103-
launch_parameters.running_workflow_step_id,
127+
launch_parameters.variables,
128+
rwfs_id,
104129
TextEncoding.JINJA2_3_0,
105130
)
106131
print(f"Decoded command: {decoded_command}")
@@ -132,6 +157,7 @@ def launch(self, launch_parameters: LaunchParameters) -> LaunchResult:
132157
self._msg_dispatcher.send(pod_message)
133158

134159
return LaunchResult(
160+
running_workflow_step_id=rwfs_id,
135161
instance_id=instance_id,
136162
task_id=task_id,
137163
command=" ".join(subprocess_cmd),

tests/job-definitions/job-definitions.yaml

Lines changed: 33 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -131,4 +131,36 @@ jobs:
131131
132132
concatenate:
133133
command: >-
134-
concatenate.py {% for ifile in inputFile %}{{ ifile }} {% endfor %} --outputFile {{ outputFile }}
134+
concatenate.py --inputFile {{ inputFile }} --outputFile {{ outputFile }}
135+
# Simulate a multiple input files Job (combiner)...
136+
variables:
137+
inputs:
138+
properties:
139+
inputFile:
140+
type: files
141+
options:
142+
type: object
143+
properties:
144+
inputDirPrefix:
145+
title: Optional inoput directory prefix
146+
type: string
147+
outputs:
148+
properties:
149+
outputBase:
150+
creates: '{{ outputFile }}'
151+
type: file
152+
153+
splitsmiles:
154+
command: >-
155+
copyf.py {{ inputFile }}
156+
# Simulate a multiple output files Job (splitetr)...
157+
variables:
158+
inputs:
159+
properties:
160+
inputFile:
161+
type: file
162+
outputs:
163+
properties:
164+
outputBase:
165+
creates: '{{ outputBase }}_*.smi'
166+
type: files

tests/jobs/concatenate.py

Lines changed: 7 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -2,13 +2,16 @@
22

33
parser = argparse.ArgumentParser(
44
prog="addcol",
5-
description="Takes a list of files and writes them into single outputfile",
5+
description="Takes an optional directory prefix and a file,"
6+
" and combines all the input files that are found"
7+
" into single outputfile",
68
)
7-
parser.add_argument("inputFile", nargs="+", type=argparse.FileType("r"))
9+
parser.add_argument("--inputDirPrefix")
10+
parser.add_argument("--inputFile", required=True)
811
parser.add_argument("-o", "--outputFile", required=True)
912
args = parser.parse_args()
1013

1114

1215
with open(args.outputFile, "wt", encoding="utf8") as ofile:
13-
for f in args.inputFile:
14-
ofile.write(f.read())
16+
with open(args.inputFile, "rt", encoding="utf8") as ifile:
17+
ofile.write(ifile.read())

0 commit comments

Comments
 (0)