Skip to content

Commit e9a0d84

Browse files
authored
Arch doc updates (#992)
1 parent d34c1d6 commit e9a0d84

File tree

5 files changed

+354
-49
lines changed

5 files changed

+354
-49
lines changed

ARCHITECTURE.md

Lines changed: 44 additions & 48 deletions
Original file line numberDiff line numberDiff line change
@@ -1,32 +1,31 @@
11
Core SDK Architecture
22
===
33

4-
## High level description
4+
If you're newer to SDKs in general, first check out the [SDKs Intro](./arch_docs/sdks_intro.md)!
55

6-
The below diagram depicts how future SDKs are split into two parts. The `sdk-core` common code,
6+
## High-level description
7+
8+
The below diagram depicts how Core-based SDKs are split into two parts. The `sdk-core` common code,
79
which is written in Rust, and a `sdk-lang` package specific to the language the user is writing
810
their workflow/activity in. For example a user writing workflows in Rust would be pulling in (at
911
least) two crates - `temporal-sdk-core` and `temporal-sdk-rust`.
1012

1113
![Arch Diagram](https://lucid.app/publicSegments/view/7872bb33-d2b9-4b90-8aa1-bac111136aa5/image.png)
1214

13-
The `core` communicates with the Temporal service in the same way that existing SDKs today do, via
14-
gRPC. It's responsible for polling for tasks, processing those tasks according to our state machine
15-
logic, and then driving the language-specific code and shuttling events to it and commands back.
15+
Core communicates with the Temporal service in the same way that existing SDKs today do, via
16+
gRPC. It's responsible for polling for tasks (Workflow, Activity, and Nexus), processing those tasks
17+
and updating internal state as necessary, and then delivering them to the lang-layer which is
18+
polling for them (and handling the responses).
1619

17-
The `sdk-lang` side communicates with `sdk-core` via either C bindings, IPC, or (later) bindings to
18-
a WASM interface. Many languages already have nice support for calling into Rust code - generally
19-
speaking these implementations are using C bindings under the hood. For example, we
20-
use [neon](https://neon-bindings.com/) to support the TS/JS sdk, and we will likely
21-
use [PyO3](https://github.com/PyO3/pyo3) for Python. It is expected that such usages will layer
22-
another crate on top of `core` which brings in these language specific libraries to expose `core` to
23-
that language in an ergonomic manner. IPC will exist as a thin layer on top of the C bindings. Care
24-
should be taken here to avoid unnecessary copying and [de]serialization. Then `sdk-lang` is
25-
responsible for dispatching tasks to the appropriate user code (to whatever extent parts of this can
26-
be reasonably put in the core code, we desire that to make lang-specific SDKs as small as possible).
20+
The `sdk-lang` side communicates with `sdk-core` via C bindings. Many languages already have nice
21+
support for calling into Rust code - generally speaking these implementations are using C bindings
22+
under the hood. For example, we use [neon](https://neon-bindings.com/) to support the TS/JS SDK, and
23+
we use [PyO3](https://github.com/PyO3/pyo3) for Python. Languages using libraries like that layer
24+
another crate on top of `core` which brings in these language-specific helpers to expose `core` to
25+
that language in an ergonomic manner.
2726

28-
As a general note, the more we can push from `sdk-lang` into `sdk-core`, the easier our ecosystem is
29-
to maintain in the long run as we will have less semantically identical code.
27+
Other SDKs (ex: .NET) directly use C bindings available in this repo
28+
at [core-c-bridge](./core-c-bridge).
3029

3130
### Glossary of terms
3231

@@ -38,9 +37,9 @@ the code. This list should help to disambiguate:
3837
represent the history of the workflow. They are defined in the protobuf definitions for the
3938
Temporal service itself.
4039
* `Command`: These are the commands defined in the temporal service protobufs that are returned by
41-
clients upon completing a `WorkflowTask`. For example, starting a timer or an activity.
42-
* `WorkflowTask`: These are how the server represents the need to run user workflow code (or the
43-
result of it executing). See the `HistoryEvent` proto definition for more.
40+
workers upon completing a `WorkflowTask`. For example, starting a timer or an activity.
41+
* `WorkflowTask`: These are how the server represents the need to run user workflow code, and the
42+
results of that execution. See the `HistoryEvent` proto definition for more.
4443
* `WorkflowActivation`: These are produced by the Core SDK when the lang sdk needs to "activate" the
4544
user's workflow code, either running it from the beginning or resuming a cached workflow.
4645
* `WorkflowActivationJob` (shorthand: `Job`s): These are included in `WorkflowActivation`s and
@@ -57,50 +56,51 @@ Additional clarifications that are internal to Core:
5756
* `StateMachine`s also handle events and produce commands, which often map directly to the above
5857
`HistoryEvent`s and `Command`s, but are distinct types. The state machine library is Temporal
5958
agnostic - but all interactions with the machines pass through a `TemporalStateMachine` trait,
60-
which accepts `HistoryEvent`s, and produces `WorkflowTrigger`s.
61-
* `WorkflowTrigger`: These allow the state machines to trigger things to happen to the workflow.
62-
Including pushing new `WfActivationJob`s, or otherwise advancing workflow state.
59+
which accepts `HistoryEvent`s, and produces `MachineResponse`s.
60+
* `MachineResponse`: These allow the state machines to trigger things to happen to the workflow.
61+
Including pushing new Activation Jobs, or otherwise advancing workflow state.
6362

6463
### Core SDK Responsibilities
6564

6665
- Communication with Temporal service using a generated gRPC client, which is wrapped with somewhat
6766
more ergonomic traits.
6867
- Provide interface for language-specific SDK to drive event loop and handle returned commands. The
6968
lang sdk will continuously call/poll the core SDK to receive new tasks, which either represent
70-
workflows being started or awoken (`WorkflowActivation`) or activities to execute (
71-
`ActivityTask`). It will then call its workflow/activity functions with the provided information
72-
as appropriate, and will then push completed tasks back into the core SDK.
69+
workflows being started or awoken (`WorkflowActivation`), activities to execute (`ActivityTask`),
70+
or Nexus Operations to invoke (`NexusTask`). It will then call its workflow/activity/nexus
71+
functions with the provided information as appropriate, and will then push completed tasks back
72+
into the core SDK.
7373
- Advance state machines and report back to the temporal server as appropriate when handling events
7474
and commands
7575

7676
### Language Specific SDK Responsibilities
7777

7878
- Periodically poll Core SDK for tasks
79-
- Call workflow and activity functions as appropriate, using information in events it received from
80-
Core SDK
81-
- Return results of workflows/activities to Core SDK
79+
- Call workflow, activity, and nexus functions as appropriate, using information in events it
80+
received from Core SDK
81+
- Return results of workflows/activities/nexus ops to Core SDK
8282
- Manage concurrency using language appropriate primitives. For example, it is up to the language
8383
side to decide how frequently to poll, and whether or not to execute worklows and activities in
8484
separate threads or coroutines, etc.
8585

8686
### Example Sequence Diagrams
8787

8888
Here we consider what the sequence of API calls would look like for a simple workflow executing a
89-
happy path. The hello-world workflow & activity in imaginary Rust (don't pay too much attention to
90-
the specifics, just an example) is below. It is meant to be like our most basic hello world samples.
91-
92-
```rust
93-
#[workflow]
94-
async fn hello_world_workflow(name: &str) -> Result<String, Error> {
95-
info!("Hello world workflow started! Name: {}", name);
96-
// Very much TBD how this would actually work in rust sdk. Many options here.
97-
activity!(hello_activity(name), timeout: 2s).await
98-
}
99-
100-
#[activity]
101-
async fn hello_activity(name: &str) -> String {
102-
format!("Hello {}!", name)
103-
}
89+
happy path. The hello-world workflow & activity in Python is below.
90+
91+
```python
92+
@workflow.defn
93+
class SayHello:
94+
@workflow.run
95+
async def run(self, name: str) -> str:
96+
return await workflow.execute_activity(
97+
say_hello, name, schedule_to_close_timeout=timedelta(seconds=5)
98+
)
99+
100+
101+
@activity.defn
102+
async def say_hello(name: str) -> str:
103+
return f"Hello, {name}!"
104104
```
105105

106106
[![](https://mermaid.ink/img/pako:eNptk81O6zAQhV9l5AWr8gIRqoQCC0QXQJDuJpvBnrZWbY-vf9rbi3h3bJqE0JKVlfP5zMyx_S4kKxKNiPQ3k5N0p3ET0PYOyucxJC21R5egA4zwStZzQAMdhb2WdIm1FWs5EHR3j5fyqsordJuTfAJWcL1cQtvAg9NJo9H_CQ4cdhTmetfAJnjZQJeK4Z-irw0f7v-RzEmzG80Ms__aXVVIGHfgA0uKUbvNCWl_-j2xMaPda-GfM-Vhsg6uh9aqEOEKtjomDseizb0KcOu9OU5yYogJE4FFudWO4ometUh7KnnU5VkItR1YcwCUSe-xzhanWpVZNTC2ezshc5MCvGQ3hbCoAahcIgDJ1qJTcaKHmpd-LVtvqK5u3sJSskuoXUnwzOJ7fLXHcn1-nZqcGk_nLPoXip6dmqc_FCZ1sXdKqNvmpPhQfouFsBQsalWu8HvFepG2ZKkXTVkqDLte9O6jcNmr0tm90uV8RLNGE2khMCfujk6KJoVMIzS8gYH6-ASOZQf0)](https://mermaid-js.github.io/mermaid-live-editor/edit#pako:eNptk81O6zAQhV9l5AWr8gIRqoQCC0QXQJDuJpvBnrZWbY-vf9rbi3h3bJqE0JKVlfP5zMyx_S4kKxKNiPQ3k5N0p3ET0PYOyucxJC21R5egA4zwStZzQAMdhb2WdIm1FWs5EHR3j5fyqsordJuTfAJWcL1cQtvAg9NJo9H_CQ4cdhTmetfAJnjZQJeK4Z-irw0f7v-RzEmzG80Ms__aXVVIGHfgA0uKUbvNCWl_-j2xMaPda-GfM-Vhsg6uh9aqEOEKtjomDseizb0KcOu9OU5yYogJE4FFudWO4ometUh7KnnU5VkItR1YcwCUSe-xzhanWpVZNTC2ezshc5MCvGQ3hbCoAahcIgDJ1qJTcaKHmpd-LVtvqK5u3sJSskuoXUnwzOJ7fLXHcn1-nZqcGk_nLPoXip6dmqc_FCZ1sXdKqNvmpPhQfouFsBQsalWu8HvFepG2ZKkXTVkqDLte9O6jcNmr0tm90uV8RLNGE2khMCfujk6KJoVMIzS8gYH6-ASOZQf0)
@@ -118,7 +118,3 @@ definition [here](https://github.com/temporalio/sdk-core/tree/master/sdk-core-pr
118118

119119
- [Sticky task queues](arch_docs/sticky_queues.md)
120120
- [Workflow task chunking](arch_docs/workflow_task_chunking.md)
121-
122-
## Workflow Processing Internals
123-
124-
![Workflow Internals](arch_docs/diagrams/workflow_internals.svg)
Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,14 @@
1+
@startuml
2+
[*] --> Created
3+
Created -[#blue]-> StartCommandCreated: Schedule
4+
StartCommandCreated -[#blue]-> StartCommandCreated: CommandStartTimer
5+
StartCommandCreated -[#blue]-> StartCommandRecorded: TimerStarted
6+
StartCommandCreated -[#blue]-> Canceled: Cancel
7+
StartCommandRecorded -[#blue]-> Fired: TimerFired
8+
StartCommandRecorded -[#blue]-> CancelTimerCommandCreated: Cancel
9+
CancelTimerCommandCreated --> CancelTimerCommandCreated: Cancel
10+
CancelTimerCommandCreated -[#blue]-> CancelTimerCommandSent: CommandCancelTimer
11+
CancelTimerCommandSent --> Canceled: TimerCanceled
12+
Canceled -[#blue]-> Canceled: Cancel
13+
Fired -[#blue]-> Fired: Cancel
14+
@enduml
48.2 KB
Loading

arch_docs/diagrams/workflow_internals.svg

Lines changed: 0 additions & 1 deletion
This file was deleted.

0 commit comments

Comments
 (0)