Jbtm 3987 starting LRAs via a cluster of coordinators by mmusgrov · Pull Request #50 · jbosstm/lra

mmusgrov · 2025-06-16T16:54:56Z

https://issues.redhat.com/browse/JBTM-3987

The depends on pr has been merged.

!JACOCO !JDK21

This PR adds support for starting LRAs using a cluster of coordinators. This version supports a static list of coordinator endpoints via the config property "lra.coordinator.urls". The change uses the SmallRye Stork service discovery and load balancer framework. The discovery of coordinators is integrated into NarayanaLRAClient, ie client side, and currently uses the "lra.coordinator.urls" property to configure a static list of coordinators, a simple change can be made for supporting the many other discovery mechanisms such as Consul, Kubernetes etc provided by Stork but I thought it best to start out with a static list. Note that only the creation of new LRAs is load balanced and we don't support shared lifecycle management of LRAs yet so, in particular, it won't support failover.

I need the PR integration tests to run with the latest code so I included a "TEST CHANGE ONLY: enable the option to use war from Narayana when run commit that does that using the feature/fix that @arjantijms recently added.

I updated LRATest to use a cluster of two coordinators. They share the same store and even though such a config would not be viable for users it works well in the context of our test suite since the issues around sharing stores only becomes apparent when coordinators re-read the store and start managing each others transactions (I may add a test that shows the problem at a future date). I'll also follow up with a doc update and a jbossts blogspot article.

marcosgopen · 2025-06-17T14:48:34Z

Thanks @mmusgrov for your PR! It is great to have the 'support for starting LRAs using a cluster of coordinators'! I can see that the TCK failed, that is probably because the Stork module needs to be added in the lra-participant module of WildFly first[1].

[1]WELD-001474: Class io.narayana.lra.client.internal.NarayanaLRAClient is on the classpath, but was ignored because a class it references was not found: io.smallrye.stork.api.config.ConfigWithType from [Module \"org.jboss.narayana.lra.lra-participant\" version 1.0.2.Final-SNAPSHOT from local module loader @12591ac8 (finder: local module finder @5a7fe64f (roots: /home/runner/work/lra/lra/jboss-as/build/target/wildfly-37.0.0.Beta1-SNAPSHOT/modules,/home/runner/work/lra/lra/jboss-as/build/target/wildfly-37.0.0.Beta1-SNAPSHOT/modules/system/layers/base,/home/runner/work/lra/lra/jboss-as/testsuite/integration/microprofile-tck/lra/target/modules))].

mmusgrov · 2025-06-18T07:12:05Z

Thanks Marco, I've added the HOLD label while we investigate. I'll try out your recommendation and will report back later today.

mmusgrov · 2025-06-18T09:11:03Z

~~FYI On a clean check out and build of WildFly followed by a run of the microprofile-tck testsuite [1], (after building my PR 1.0.2.Final-SNAPSHOT), I get a pass.~~

Since the PR runs with our version of WildFly (ie https://github.com/jbosstm/jboss-as.git) the failure must be related to our clone, although our clone was updated on Monday and only contains some unrelated testsuite changes, sha d46d20f9d1

~~I'll try building a standard WildFly using our script (./scripts/hudson/narayana.sh) to see if that is producing/doing something odd.~~

~~[1] ./build.sh -f testsuite/integration/microprofile-tck/lra/pom.xml -fae -B -Dversion.org.jboss.narayana.lra=1.0.2.Final-SNAPSHOT clean test~~

[Edit] Martin has given me a patch for our jboss-as clone that enables the change to run against the WildFly MP LRA testsuite. I'll update that with his patch and then re-trigger PR.

mmusgrov · 2025-06-19T15:40:03Z

Thanks @xstefank - adding your commit fixes the WildFly testsuite failure ie `Test Suite: Integration - MicroProfile TCK - LRA' passes.

Since the fix was to add a "Depends on jbosstm/jboss-as#97" to the PR description I had to manually triggered the LRA-CORE / LRA github action for JDK 17 and 21 (https://github.com/jbosstm/lra/actions/runs/15759919429 shows them passing) and therefore the errors in the "Some checks were not successful" panel below remain - I wonder if there is a GH Action trigger corresponding to "Depends on ...".

Will you review the PR. @marcosgopen I'll remove the HOLD label.

jbosstm-bot · 2025-06-19T16:46:28Z

Started testing this pull request with LRA profile: https://jenkins-csb-narayana-ci.dno.corp.redhat.com/job/btny-pulls-lra/PROFILE=LRA,jdk=openJDK21,label=jnlp-agent/145/

jbosstm-bot · 2025-06-19T16:46:40Z

Started testing this pull request with LRA profile: https://jenkins-csb-narayana-ci.dno.corp.redhat.com/job/btny-pulls-lra/PROFILE=LRA,jdk=openJDK17,label=jnlp-agent/145/

marcosgopen · 2025-06-19T16:48:52Z

Thanks Martin and Mike, I started the jobs on the CI to test this PR against our jboss-as repo.
The PR is great! I am going to approve it but let's wait for the tests to pass please. Thanks
BTW could we use a specific PR of jboss-as to test lra PRs with github actions? cc @tomjenkinson @jmfinelli

jbosstm-bot · 2025-06-19T17:21:11Z

LRA profile tests failed (https://jenkins-csb-narayana-ci.dno.corp.redhat.com/job/btny-pulls-lra/PROFILE=LRA,jdk=openJDK17,label=jnlp-agent/145/): LRA AS Test failed

jbosstm-bot · 2025-06-19T17:22:38Z

LRA profile tests failed (https://jenkins-csb-narayana-ci.dno.corp.redhat.com/job/btny-pulls-lra/PROFILE=LRA,jdk=openJDK21,label=jnlp-agent/145/): LRA AS Test failed

mmusgrov · 2025-06-20T08:53:43Z

Thanks Martin and Mike, I started the jobs on the CI to test this PR against our jboss-as repo. The PR is great! I am going to approve it but let's wait for the tests to pass please. Thanks BTW could we use a specific PR of jboss-as to test lra PRs with github actions? cc @tomjenkinson @jmfinelli

They PR did run with with the specific PR because of the "Depends on jbosstm/jboss-as#97" in the pull description (see https://github.com/jbosstm/lra/actions/runs/15759919429 for the successful runs).

I thought we are using github actions for this repo, are we maintaining both build systems now on this repo?

marcosgopen · 2025-06-20T09:06:56Z

Thanks Martin and Mike, I started the jobs on the CI to test this PR against our jboss-as repo. The PR is great! I am going to approve it but let's wait for the tests to pass please. Thanks BTW could we use a specific PR of jboss-as to test lra PRs with github actions? cc @tomjenkinson @jmfinelli

They PR did run with with the specific PR because of the "Depends on jbosstm/jboss-as#97" in the pull description (see https://github.com/jbosstm/lra/actions/runs/15759919429 for the successful runs).

I thought we are using github actions for this repo, are we maintaining both build systems now on this repo?

@mmusgrov the successful run you are pointing at (https://github.com/jbosstm/lra/actions/runs/15759919429) used the lra source code from main branch.

I thought we are using github actions for this repo, are we maintaining both build systems now on this repo?

Yes we are using github actions for this repo, but I can see the runs (https://github.com/jbosstm/lra/actions/runs/15689158243/job/44199809265?pr=50) are not using the correct jboss-as (jbosstm/jboss-as#97)

jbosstm-bot · 2025-06-20T09:35:29Z

Started testing this pull request with LRA profile: https://jenkins-csb-narayana-ci.dno.corp.redhat.com/job/btny-pulls-lra/PROFILE=LRA,jdk=openJDK21,label=jnlp-agent/146/

marcosgopen · 2025-06-20T09:40:16Z

I had to add explicitly 'jbosstm/jboss-as#97' in the description instead of 'jbosstm/jboss-as#97' in order to use that PR branch for testing cc @mmusgrov (this is something we need to remember/document)

jbosstm-bot · 2025-06-20T09:42:19Z

Started testing this pull request with LRA profile: https://jenkins-csb-narayana-ci.dno.corp.redhat.com/job/btny-pulls-lra/PROFILE=LRA,jdk=openJDK17,label=jnlp-agent/146/

mmusgrov · 2025-06-20T09:49:09Z

I had to add explicitly 'jbosstm/jboss-as#97' in the description instead of 'jbosstm/jboss-as#97' in order to use that PR branch for testing cc @mmusgrov (this is something we need to remember/document)

'jbosstm/jboss-as#97' is in the description twice now. What do we need to remember/document?

tomjenkinson · 2025-06-20T10:05:10Z

I had to add explicitly 'jbosstm/jboss-as#97' in the description instead of 'jbosstm/jboss-as#97' in order to use that PR branch for testing cc @mmusgrov (this is something we need to remember/document)

'jbosstm/jboss-as#97' is in the description twice now. What do we need to remember/document?

I think the one that is not a full URL can be deleted from the description. I have proposed some text to help clarify the expectations (https://github.com/jbosstm/narayana/pull/2379/files)

jbosstm-bot · 2025-06-20T10:10:02Z

LRA profile tests passed - Job complete https://jenkins-csb-narayana-ci.dno.corp.redhat.com/job/btny-pulls-lra/PROFILE=LRA,jdk=openJDK17,label=jnlp-agent/146/

mmusgrov · 2025-06-20T10:14:12Z

I had to add explicitly 'jbosstm/jboss-as#97' in the description instead of 'jbosstm/jboss-as#97' in order to use that PR branch for testing cc @mmusgrov (this is something we need to remember/document)

'jbosstm/jboss-as#97' is in the description twice now. What do we need to remember/document?

I think the one that is not a full URL can be deleted from the description. I have proposed some text to help clarify the expectations (https://github.com/jbosstm/narayana/pull/2379/files)

~~I'm not following you, the two links are identical (jbosstm/jboss-as#97 vs jbosstm/jboss-as#97).~~

[Edit] @tomjenkinson Ah I see what you mean: jbosstm/jboss-as#97 vs https://github.com/jbosstm/jboss-as/pull/97
I'll delete the first one.

jbosstm-bot · 2025-06-20T10:29:17Z

LRA profile tests passed - Job complete https://jenkins-csb-narayana-ci.dno.corp.redhat.com/job/btny-pulls-lra/PROFILE=LRA,jdk=openJDK21,label=jnlp-agent/146/

mmusgrov · 2025-06-20T13:21:13Z

client/src/main/java/io/narayana/lra/client/internal/NarayanaLRAClient.java

     */
    public static final String LRA_COORDINATOR_URL_KEY = "lra.coordinator.url";
+    public static final String COORDINATOR_URLS_KEY = "lra.coordinator.urls";
+    public static final String USE_STORK_KEY = "lra.coordinator.discoverable";


Delete this since it isn't used.

mmusgrov · 2025-06-20T13:23:26Z

coordinator/src/test/java/io/narayana/lra/coordinator/domain/model/LRATest.java

-            fail(testName + ": current thread should not be associated with any LRAs");
-        }
+            if (lraClient.getCurrent() != null) {
+                lraClient.clearCurrent(false); // otherwise it will interfere with subsequent tests


Clear this at the end otherwise the assert on line 203 is always going to pass!

@marcosgopen I'll push an update to these two after your review in case you want to request other changes.

marcosgopen

Thanks @mmusgrov for your PR, together with the changes you mentioned I am approving this great PR! The CI tests passed now

…ning via CI

jbosstm-bot · 2025-06-20T15:15:03Z

Started testing this pull request with LRA profile: https://jenkins-csb-narayana-ci.dno.corp.redhat.com/job/btny-pulls-lra/PROFILE=LRA,jdk=openJDK17,label=jnlp-agent/147/

jbosstm-bot · 2025-06-20T16:26:56Z

LRA profile tests passed - Job complete https://jenkins-csb-narayana-ci.dno.corp.redhat.com/job/btny-pulls-lra/PROFILE=LRA,jdk=openJDK17,label=jnlp-agent/147/

mmusgrov · 2025-06-23T11:10:50Z

I will merge this but note that github actions will fail until we have merged the jboss-as changes because the GH Actions do not have the ability to use that jboss-as clone/.

[edit] I have merged the jboss-as changes now so GH Actions should be working again,.

mmusgrov changed the title ~~Jbtm 3987~~ Jbtm 3987 starting LRAs via a cluster of coordinators Jun 16, 2025

mmusgrov added the Hold label Jun 18, 2025

mmusgrov mentioned this pull request Jun 19, 2025

Add stork module for LRA (required for JBTM-3987) jbosstm/jboss-as#97

Merged

mmusgrov removed the Hold label Jun 19, 2025

mmusgrov commented Jun 20, 2025

View reviewed changes

marcosgopen approved these changes Jun 20, 2025

View reviewed changes

mmusgrov added 4 commits June 20, 2025 15:56

JBTM-3987 clustered coordinators for starting LRAs

6f23ec4

JBTM-3987 update Deployer for module lra-test-basic

55c9a08

TEST CHANGE ONLY: enable the option to use war from Narayana when run…

dcb1485

…ning via CI

JBTM-3987 automated formatting change

7e159d5

mmusgrov force-pushed the JBTM-3987 branch from e8361fe to 7e159d5 Compare June 20, 2025 14:58

marcosgopen approved these changes Jun 20, 2025

View reviewed changes

mmusgrov closed this Jun 23, 2025

mmusgrov reopened this Jun 23, 2025

mmusgrov merged commit 93cfe88 into jbosstm:main Jun 23, 2025
2 of 8 checks passed

mmusgrov deleted the JBTM-3987 branch June 23, 2025 11:49

marcosgopen mentioned this pull request Jun 30, 2025

JBTM-3987 configuring coordinators jbosstm/documentation#76

Merged

mmusgrov mentioned this pull request Jul 6, 2025

Jbtm-3987 Add support for various load balancing algorithms #53

Merged

tomjenkinson mentioned this pull request Nov 14, 2025

Lra client is started based on config instead of parameter value #175

Closed

Conversation

mmusgrov commented Jun 16, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

marcosgopen commented Jun 17, 2025

Uh oh!

mmusgrov commented Jun 18, 2025

Uh oh!

mmusgrov commented Jun 18, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

mmusgrov commented Jun 19, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jbosstm-bot commented Jun 19, 2025

Uh oh!

jbosstm-bot commented Jun 19, 2025

Uh oh!

marcosgopen commented Jun 19, 2025

Uh oh!

jbosstm-bot commented Jun 19, 2025

Uh oh!

jbosstm-bot commented Jun 19, 2025

Uh oh!

mmusgrov commented Jun 20, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

marcosgopen commented Jun 20, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jbosstm-bot commented Jun 20, 2025

Uh oh!

marcosgopen commented Jun 20, 2025

Uh oh!

jbosstm-bot commented Jun 20, 2025

Uh oh!

mmusgrov commented Jun 20, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

tomjenkinson commented Jun 20, 2025

Uh oh!

jbosstm-bot commented Jun 20, 2025

Uh oh!

mmusgrov commented Jun 20, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jbosstm-bot commented Jun 20, 2025

Uh oh!

mmusgrov Jun 20, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mmusgrov Jun 20, 2025

Choose a reason for hiding this comment

Uh oh!

mmusgrov Jun 20, 2025

Choose a reason for hiding this comment

Uh oh!

marcosgopen left a comment

Choose a reason for hiding this comment

Uh oh!

jbosstm-bot commented Jun 20, 2025

Uh oh!

jbosstm-bot commented Jun 20, 2025

Uh oh!

mmusgrov commented Jun 23, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

mmusgrov commented Jun 16, 2025 •

edited

Loading

mmusgrov commented Jun 18, 2025 •

edited

Loading

mmusgrov commented Jun 19, 2025 •

edited

Loading

mmusgrov commented Jun 20, 2025 •

edited

Loading

marcosgopen commented Jun 20, 2025 •

edited

Loading

mmusgrov commented Jun 20, 2025 •

edited

Loading

mmusgrov commented Jun 20, 2025 •

edited

Loading

mmusgrov Jun 20, 2025 •

edited

Loading

mmusgrov commented Jun 23, 2025 •

edited

Loading