Skip to content

Commit e08f45a

Browse files
zambrovskii-am-not-giving-my-name-to-a-machineemaarco
authored
Provide idempotency (#214)
* add idempotency registry and provided in-mem version of it * fix some comments * idempotent repo now stores converted result instead of raw, which simplifies serialization * feat: rework `IdempotencyRegistry` [WIP] * refactor: remove unused imports * feat: add integration test for transactional and non-transactional worker with idempotency * feat: implement `JpaIdempotencyRegistry` [WIP] * chore: refactor integration test structure to conform to general conventions * complete refactoring of itest setup, fix jpa repository being run outside of tx, add docs * improve profile handling, killed warnings * make sure itests are run by failsafe * ongoing improvement * minor fixes * remove liquibase --------- Co-authored-by: Enno Thieleke <enno.thieleke@gmail.com> Co-authored-by: Marco Schaeck <marco.schaeck@outlook.de>
1 parent 8c3bd7e commit e08f45a

File tree

46 files changed

+1147
-292
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

46 files changed

+1147
-292
lines changed

‎.github/workflows/development.yml‎

Lines changed: 0 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -45,11 +45,6 @@ jobs:
4545
- name: Build with Maven
4646
run: ./mvnw clean verify -U -B -ntp -T4
4747

48-
# itest
49-
- name: Run itest
50-
run: ./mvnw integration-test failsafe:verify -Pitest -ntp -U -B -T4
51-
52-
5348
# - name: Upload coverage to Codecov
5449
# if: github.event_name == 'push' && github.actor != 'dependabot[bot]'
5550
# uses: codecov/codecov-action@v1.0.2

‎.github/workflows/master.yml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -45,7 +45,7 @@ jobs:
4545

4646
# Publish release
4747
- name: Deploy a new release version to Maven Central
48-
run: ./mvnw clean deploy -B -ntp -DskipTests -DskipExamples -Prelease -Dgpg.keyname="${{ secrets.GPG_KEYNAME }}"
48+
run: ./mvnw clean deploy -B -ntp -DskipTests -DskipITests -DskipExamples -Prelease -Dgpg.keyname="${{ secrets.GPG_KEYNAME }}"
4949
env:
5050
OSS_CENTRAL_USERNAME: ${{ secrets.SONATYPE_USERNAME }}
5151
OSS_CENTRAL_PASSWORD: ${{ secrets.SONATYPE_PASSWORD }}

‎docs/idempotency.md‎

Lines changed: 98 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,98 @@
1+
---
2+
title: Idempotency Registry
3+
---
4+
5+
The Idempotency Registry is a feature designed to prevent duplicate worker invocations. It ensures that if a task is processed multiple times (e.g., due to network issues or retries in the process engine), the worker logic is only executed once, and the previous result is returned for subsequent calls.
6+
7+
## How it Works
8+
9+
When a worker is triggered, the process engine worker:
10+
1. Checks the `IdempotencyRegistry` if a result already exists for the given `taskId`.
11+
2. If a result exists, it skips the worker execution and returns the stored result.
12+
3. If no result exists, it executes the worker.
13+
4. After successful execution, it registers the result in the `IdempotencyRegistry`.
14+
15+
## Implementations
16+
17+
There are three available implementations of the `IdempotencyRegistry`:
18+
19+
| Implementation | Description | Recommended Use |
20+
| --- | --- | --- |
21+
| `NoOpIdempotencyRegistry` | Does nothing. No results are stored or retrieved. | Default, use if idempotency is handled elsewhere. |
22+
| `InMemoryIdempotencyRegistry` | Stores results in a local `ConcurrentHashMap`. | Testing or non-clustered environments. |
23+
| `JpaIdempotencyRegistry` | Stores results in a database using JPA. | Production, clustered environments. |
24+
25+
## Setup Procedures
26+
27+
### In-Memory Registry
28+
29+
To use the in-memory registry, you need to provide a bean of type `IdempotencyRegistry` in your Spring configuration:
30+
31+
```kotlin
32+
@Configuration
33+
class IdempotencyConfiguration {
34+
35+
@Bean
36+
fun idempotencyRegistry(): IdempotencyRegistry = InMemoryIdempotencyRegistry()
37+
38+
}
39+
```
40+
41+
> **Warning:** The `InMemoryIdempotencyRegistry` is not suitable for clustered environments as the state is not shared between nodes.
42+
43+
### JPA-based Registry
44+
45+
The JPA-based registry is suitable for production environments. It persists the results in the database, allowing multiple instances of the worker to share the same idempotency state.
46+
47+
#### 1. Add Dependency
48+
49+
Add the following dependency to your `pom.xml`:
50+
51+
```xml
52+
<dependency>
53+
<groupId>dev.bpm-crafters.process-engine-worker</groupId>
54+
<artifactId>process-engine-worker-spring-boot-idempotency-registry-jpa</artifactId>
55+
<version>${process-engine-worker.version}</version>
56+
</dependency>
57+
```
58+
59+
The `JpaIdempotencyAutoConfiguration` will automatically register the `JpaIdempotencyRegistry` if an `EntityManager` is present and no other `IdempotencyRegistry` bean is defined.
60+
61+
#### 2. Database Schema (Liquibase)
62+
63+
The JPA registry requires a table named `task_log_entry_`. You can use the following Liquibase changeSet to create it:
64+
65+
```yaml
66+
databaseChangeLog:
67+
- changeSet:
68+
id: create-idempotency-table
69+
author: bpm-crafters
70+
changes:
71+
- createTable:
72+
tableName: task_log_entry_
73+
columns:
74+
- column:
75+
name: task_id_
76+
type: varchar(100)
77+
constraints:
78+
nullable: false
79+
primaryKey: true
80+
primaryKeyName: task_log_entry_pk_
81+
- column:
82+
name: process_instance_id_
83+
type: varchar(100)
84+
constraints:
85+
nullable: false
86+
- column:
87+
name: created_at_
88+
type: timestamp
89+
constraints:
90+
nullable: false
91+
- column:
92+
name: result_
93+
type: blob # or bytea for PostgreSQL
94+
constraints:
95+
nullable: false
96+
```
97+
98+
> **Note:** The `result_` column type should be suitable for storing binary data (e.g., `blob` for most databases, `bytea` for PostgreSQL).

‎docs/process-engine-worker.md‎

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -37,6 +37,15 @@ public class MySmartWorker {
3737
}
3838
```
3939

40+
The `@ProcessEngineWorker` annotation supports the following properties:
41+
42+
| Property | Type | Default | Description |
43+
|----------------|------------|------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
44+
| `topic` | `String` | `""` | Topic name to subscribe this worker for. Alias for `value`. |
45+
| `autoComplete` | `boolean` | `true` | Flag indicating if the task should be automatically completed after the worker execution. If the return type is `Map<String, Any>`, it will overrule this setting and auto-complete with the returned payload. |
46+
| `completion` | `enum` | `DEFAULT` | Configures when the worker completes a task if `autoComplete` is active. Possible values are `DEFAULT`, `BEFORE_COMMIT`, and `AFTER_COMMIT`. Has no effect if the worker is not transactional. |
47+
| `lockDuration` | `long` | `-1` | Optional lock duration in seconds for this worker. If set to `-1` (default), the global configuration of the process engine adapter will be used. (Available since `0.8.0`) |
48+
4049
## Method parameter resolution
4150

4251
Parameter resolution of the method annotated with `ProcessEngineWorker` is based on a set of strategies

‎itest/pom.xml‎

Lines changed: 43 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,43 @@
1+
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
2+
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
3+
<modelVersion>4.0.0</modelVersion>
4+
5+
<parent>
6+
<groupId>dev.bpm-crafters.process-engine-worker</groupId>
7+
<artifactId>process-engine-worker-root</artifactId>
8+
<version>0.7.2-SNAPSHOT</version>
9+
<relativePath>../pom.xml</relativePath>
10+
</parent>
11+
12+
<artifactId>process-engine-worker-itest</artifactId>
13+
<packaging>pom</packaging>
14+
<description>ITest: Parent for the integration testing.</description>
15+
16+
<properties>
17+
<!-- Never deploy integration testing -->
18+
<deploy.skip>true</deploy.skip>
19+
<gpg.skip>true</gpg.skip>
20+
<dokka.skip>true</dokka.skip>
21+
</properties>
22+
23+
<modules>
24+
<module>spring-boot-integration-testing</module>
25+
<module>spring-boot-starter-integration-test</module>
26+
</modules>
27+
28+
<dependencyManagement>
29+
<dependencies>
30+
<dependency>
31+
<groupId>dev.bpm-crafters.process-engine-worker</groupId>
32+
<artifactId>process-engine-worker-spring-boot-starter</artifactId>
33+
<version>${project.version}</version>
34+
</dependency>
35+
<dependency>
36+
<groupId>dev.bpm-crafters.process-engine-worker</groupId>
37+
<artifactId>process-engine-worker-spring-boot-idempotency-registry-jpa</artifactId>
38+
<version>${project.version}</version>
39+
</dependency>
40+
</dependencies>
41+
</dependencyManagement>
42+
43+
</project>

‎itest/spring-boot-integration-testing/pom.xml‎

Lines changed: 149 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,149 @@
1+
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
2+
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
3+
<modelVersion>4.0.0</modelVersion>
4+
<parent>
5+
<groupId>dev.bpm-crafters.process-engine-worker</groupId>
6+
<artifactId>process-engine-worker-itest</artifactId>
7+
<version>0.7.2-SNAPSHOT</version>
8+
<relativePath>../pom.xml</relativePath>
9+
</parent>
10+
11+
<artifactId>process-engine-worker-integration-testing</artifactId>
12+
<description>ITest: Testing utilities for integration testing of the library.</description>
13+
14+
<dependencyManagement>
15+
<dependencies>
16+
<dependency>
17+
<groupId>org.testcontainers</groupId>
18+
<artifactId>testcontainers-bom</artifactId>
19+
<version>2.0.3</version>
20+
<scope>import</scope>
21+
<type>pom</type>
22+
</dependency>
23+
24+
<dependency>
25+
<groupId>org.springframework.boot</groupId>
26+
<artifactId>spring-boot-dependencies</artifactId>
27+
<version>${spring-boot.version}</version>
28+
<scope>import</scope>
29+
<type>pom</type>
30+
</dependency>
31+
32+
<dependency>
33+
<groupId>org.mockito.kotlin</groupId>
34+
<artifactId>mockito-kotlin</artifactId>
35+
<version>${mockito.version}</version>
36+
</dependency>
37+
</dependencies>
38+
</dependencyManagement>
39+
40+
<dependencies>
41+
<dependency>
42+
<groupId>${project.groupId}</groupId>
43+
<artifactId>process-engine-worker-spring-boot-starter</artifactId>
44+
<version>${project.version}</version>
45+
</dependency>
46+
<dependency>
47+
<groupId>com.fasterxml.jackson.core</groupId>
48+
<artifactId>jackson-databind</artifactId>
49+
</dependency>
50+
<dependency>
51+
<groupId>com.fasterxml.jackson.datatype</groupId>
52+
<artifactId>jackson-datatype-jdk8</artifactId>
53+
</dependency>
54+
<dependency>
55+
<groupId>com.fasterxml.jackson.datatype</groupId>
56+
<artifactId>jackson-datatype-jsr310</artifactId>
57+
</dependency>
58+
<dependency>
59+
<groupId>com.fasterxml.jackson.module</groupId>
60+
<artifactId>jackson-module-kotlin</artifactId>
61+
</dependency>
62+
<dependency>
63+
<groupId>dev.bpm-crafters.process-engine-adapters</groupId>
64+
<artifactId>process-engine-adapter-camunda-platform-c7-remote-spring-boot-starter</artifactId>
65+
<version>${process-engine-adapters-c7.version}</version>
66+
</dependency>
67+
<dependency>
68+
<groupId>io.holunda.c7</groupId>
69+
<artifactId>c7-rest-client-spring-boot-starter-feign</artifactId>
70+
<version>${c7.version}</version>
71+
</dependency>
72+
<dependency>
73+
<groupId>org.camunda.bpm.springboot</groupId>
74+
<artifactId>camunda-bpm-spring-boot-starter-external-task-client</artifactId>
75+
<version>${camunda-bpm-spring-boot-starter-external-task-client.version}</version>
76+
</dependency>
77+
78+
<dependency>
79+
<groupId>io.micrometer</groupId>
80+
<artifactId>micrometer-core</artifactId>
81+
</dependency>
82+
<dependency>
83+
<groupId>org.assertj</groupId>
84+
<artifactId>assertj-core</artifactId>
85+
</dependency>
86+
<dependency>
87+
<groupId>org.awaitility</groupId>
88+
<artifactId>awaitility-kotlin</artifactId>
89+
</dependency>
90+
<dependency>
91+
<groupId>org.jetbrains.kotlin</groupId>
92+
<artifactId>kotlin-test-junit5</artifactId>
93+
</dependency>
94+
<dependency>
95+
<groupId>org.junit.jupiter</groupId>
96+
<artifactId>junit-jupiter</artifactId>
97+
</dependency>
98+
<dependency>
99+
<groupId>org.junit.jupiter</groupId>
100+
<artifactId>junit-jupiter-params</artifactId>
101+
</dependency>
102+
<dependency>
103+
<groupId>org.mockito.kotlin</groupId>
104+
<artifactId>mockito-kotlin</artifactId>
105+
</dependency>
106+
<dependency>
107+
<groupId>org.postgresql</groupId>
108+
<artifactId>postgresql</artifactId>
109+
</dependency>
110+
<dependency>
111+
<groupId>org.springframework</groupId>
112+
<artifactId>spring-beans</artifactId>
113+
</dependency>
114+
<dependency>
115+
<groupId>org.springframework</groupId>
116+
<artifactId>spring-context</artifactId>
117+
</dependency>
118+
<dependency>
119+
<groupId>org.springframework</groupId>
120+
<artifactId>spring-tx</artifactId>
121+
</dependency>
122+
<dependency>
123+
<groupId>org.springframework.boot</groupId>
124+
<artifactId>spring-boot-autoconfigure</artifactId>
125+
</dependency>
126+
<dependency>
127+
<groupId>org.springframework.boot</groupId>
128+
<artifactId>spring-boot-configuration-processor</artifactId>
129+
<optional>true</optional>
130+
</dependency>
131+
<dependency>
132+
<groupId>org.springframework.boot</groupId>
133+
<artifactId>spring-boot-starter-data-jpa</artifactId>
134+
</dependency>
135+
<dependency>
136+
<groupId>org.springframework.boot</groupId>
137+
<artifactId>spring-boot-starter-test</artifactId>
138+
</dependency>
139+
<dependency>
140+
<groupId>org.testcontainers</groupId>
141+
<artifactId>junit-jupiter</artifactId>
142+
</dependency>
143+
<dependency>
144+
<groupId>org.testcontainers</groupId>
145+
<artifactId>postgresql</artifactId>
146+
</dependency>
147+
</dependencies>
148+
149+
</project>

0 commit comments

Comments
 (0)