UNDERTOW-1794 DefaultAccessLogReceiver violates Closeable contract #1041

baranowb · 2021-02-16T09:11:53Z

Issue: https://issues.redhat.com/browse/UNDERTOW-1794
main: #1041
2.3.x: #1825
2.2.x: #1593

baranowb · 2021-02-16T09:29:35Z

core/src/main/java/io/undertow/server/handlers/accesslog/DefaultAccessLogReceiver.java

-                if (stateUpdater.compareAndSet(this, 3, 1)) {
-                    logWriteExecutor.execute(this);
+            for (int i = 0; i < 1000 && !this.closed && !pendingMessages.isEmpty(); ++i) {
+                final String msg = pendingMessages.poll();


It might be better to use peek()+remove() here. Though Id assume follow up might be better than to alter PRs ad hoc.

@baranowb because this class has concurrent access, poll is better than peek() + remove(). Between the moment we invoke peek() and the moment we invoke remove, we risk having another thread executing flushAndTerminate. This is why we need to run poll() here.

There is trade off. It is possible that XNIO thread will be terminated between poll() and write(), leaking one message.
If its peek()+remove(), it might double entry, but not loose it. Its a trade off.

You're right about it. My question is now is hot to prevent both scenarios. Is there some way we can synchronize just a little bit to prevent a duplicate entry in the log? I think we must prevent an output that will be considered buggy by users if someone finds a duplicate entry after shutting down the server.

Not unless we control threat termination, which we dont.

core/src/main/java/io/undertow/server/handlers/accesslog/DefaultAccessLogReceiver.java

fl4via · 2021-02-16T10:01:58Z

core/src/main/java/io/undertow/server/handlers/accesslog/DefaultAccessLogReceiver.java

-        closed = true;
-        if (stateUpdater.compareAndSet(this, 0, 1)) {
-            logWriteExecutor.execute(this);
+        this.closed = true;


method must take into account that the state could change between one if and the other. So, there is a possibility that all stateUpdater checks return false.

core/src/main/java/io/undertow/server/handlers/accesslog/DefaultAccessLogReceiver.java

fl4via · 2021-02-16T10:08:05Z

core/src/main/java/io/undertow/server/handlers/accesslog/DefaultAccessLogReceiver.java

+        try {
+            while (!this.pendingMessages.isEmpty()) {
+                final String msg = this.pendingMessages.poll();
+                // TODO: clarify this, how is this possible?


It might not be possible. Lets wait for the final version of the code of run() and close() to decide this

I think it is very unlikely, but still somethiing to safeguard against. Because we are on a best effort basis in graceful shutdown when closing, we might leave the loop in line 372 still in state 2. If that occurs, we have a race when trying to write the last messages. So, it could happens that poll returns null (and it could also result in a duplicate write of the same message).

core/src/main/java/io/undertow/server/handlers/accesslog/DefaultAccessLogReceiver.java

fl4via · 2021-02-16T10:10:15Z

as this is a new feature, merging of this PR will require synchronization with the RFE process. I'll assist in having all the process followed as fast as we can to have this merged the soonest.

core/src/main/java/io/undertow/server/handlers/accesslog/DefaultAccessLogReceiver.java

fl4via · 2023-03-21T20:31:22Z

core/src/main/java/io/undertow/server/handlers/accesslog/DefaultAccessLogReceiver.java

-            doRotate();
+    private void writeMessage(final String message) {
+        //NOTE: is there a need to rotate on write?
+        //if (System.currentTimeMillis() > changeOverPoint) {


Yes, but only once per run. Now that this method is called per message it is no longer the adequate place for that.

Probably we should add the limit to 1000 here? I think it would perform a bit better.

I still stand my ground, I think the granularity of checking 1000 here performs better.

I don't know what I meant by limit of 1000. I think it looks good the way it is because we are checking for rotation in run already, and, as I said, checking per message is not the ideal place for this.

core/src/main/java/io/undertow/server/handlers/accesslog/DefaultAccessLogReceiver.java

baranowb · 2023-04-26T08:33:02Z

core/src/main/java/io/undertow/UndertowLogger.java

+    void accessLogWorkerFailureOnTransition();
+
+    @LogMessage(level = DEBUG)
+    @Message(id = 6000, value = "Access Log Worker failed to reschedule.")


Big brain math....

On the other hand - no one noticed :)

I cant find this one to mark it as outdated, lets reset.

fl4via

There are some pending issues here to consider.

core/src/main/java/io/undertow/server/handlers/accesslog/DefaultAccessLogReceiver.java

fl4via · 2024-04-30T08:02:07Z

core/src/main/java/io/undertow/server/handlers/accesslog/DefaultAccessLogReceiver.java

-                    logWriteExecutor.execute(this);
+            for (int i = 0; i < 1000 && !this.closed && !pendingMessages.isEmpty(); ++i) {
+                final String msg = pendingMessages.peek();
+                if (msg == null) {


it is not supposed to be null, because you are checking it is not empty

fl4via · 2024-04-30T08:08:19Z

core/src/main/java/io/undertow/server/handlers/accesslog/DefaultAccessLogReceiver.java

-                if (stateUpdater.compareAndSet(this, 3, 1)) {
-                    logWriteExecutor.execute(this);
+            for (int i = 0; i < 1000 && !this.closed && !pendingMessages.isEmpty(); ++i) {
+                final String msg = pendingMessages.peek();


I might be looking too much into details here, but why peeking and removing instead of just polling?

in case there is some intermitent IO failure, it should not purge entries without writing them first. Though, in such case, some overflow mechanism should be in place as well?

Also "i < 1000 && !this.closed &&" checking this in loop might not be best, should be enough to loop through and check at the end IMHO

core/src/main/java/io/undertow/server/handlers/accesslog/DefaultAccessLogReceiver.java

fl4via · 2024-04-30T08:32:01Z

core/src/main/java/io/undertow/server/handlers/accesslog/DefaultAccessLogReceiver.java

+                }
+                return;
+            } else {
+                if (!pendingMessages.isEmpty() || forceLogRotation) {


I have been thinking... we still need a state for being at the finally block, as it prevents us from having more than one thread scheduling a new execution of run at the same time

how? if there is nothing in pendingMessages, it does not need scheduling, if someone writes, it will be able to schedule. In reality, pendingMessages are filled before state change, so its just a race between logMessage thread and run() to schedule new execution.

as a second thought, I because we verify if we can go from 1 to 2 in the beginning of the method, we can get rid of finally block because only one finally will be running

fl4via · 2024-04-30T08:35:39Z

core/src/main/java/io/undertow/server/handlers/accesslog/DefaultAccessLogReceiver.java

    //0 = not running
    //1 = queued
    //2 = running
-    //3 = final state of running (inside finally of run())


I added a comment down below that we will still need state 3 to be used to keep tracking of being at finally of run)
That being said what are we trying to achieve with the new state 3? (It will have to be moved to 4 if we decide to keep it)> Are we trying to prevent run from doing anything after closed is invoked? Is that its purpose?

okay, it is clearly stated in the Jira: we want to make sure that close returns only after it is fully closed. The question that remains is if we need a new state for it, I think we need so we can fully skip the rest of the execution of run() in case it is closing, without having to resort to verifying the value of close.

I dont think so. Its a bit different FSM/CAS that was present before. It relied only on worker threads to dump messages even when handler has been closed and once its done close. This cant be done now since those worker threads may be disabled( upstream issue)

fl4via · 2024-04-30T08:57:45Z

core/src/main/java/io/undertow/server/handlers/accesslog/DefaultAccessLogReceiver.java

-            doRotate();
+    private void writeMessage(final String message) {
+        //NOTE: is there a need to rotate on write?
+        //if (System.currentTimeMillis() > changeOverPoint) {


I still stand my ground, I think the granularity of checking 1000 here performs better.

PrarthonaPaul

Hi @baranowb
Thanks! The PR looks good, I have added some minor comments.

core/src/main/java/io/undertow/UndertowLogger.java

core/src/main/java/io/undertow/server/handlers/accesslog/DefaultAccessLogReceiver.java

ropalka

I approved it by mistake, I oversaw the second commit. Leaving as changes requested until all Flavia's concerns are sorted out.

fl4via

Please rebase the PR against main and I think you can delete the question on how is poll returning null possible. The other notes I think we can leave for future reference, unless you are certain the questions have been fully answered, in that case, the cleaner the better. Thanks for looking into this!

baranowb commented Feb 16, 2021

View reviewed changes

fl4via reviewed Feb 16, 2021

View reviewed changes

core/src/main/java/io/undertow/server/handlers/accesslog/DefaultAccessLogReceiver.java Outdated Show resolved Hide resolved

fl4via reviewed Feb 16, 2021

View reviewed changes

core/src/main/java/io/undertow/server/handlers/accesslog/DefaultAccessLogReceiver.java Outdated Show resolved Hide resolved

fl4via reviewed Feb 16, 2021

View reviewed changes

core/src/main/java/io/undertow/server/handlers/accesslog/DefaultAccessLogReceiver.java Outdated Show resolved Hide resolved

fl4via reviewed Feb 16, 2021

View reviewed changes

fl4via requested changes Feb 16, 2021

View reviewed changes

core/src/main/java/io/undertow/server/handlers/accesslog/DefaultAccessLogReceiver.java Show resolved Hide resolved

fl4via added the new feature/API change New feature to be introduced or a change to the API (non suitable to minor releases) label Feb 16, 2021

baranowb force-pushed the UNDERTOW-1794 branch from febb374 to 7beb8be Compare February 17, 2021 08:45

baranowb force-pushed the UNDERTOW-1794 branch 2 times, most recently from bb266f8 to 8407e8d Compare March 8, 2021 09:58

baranowb force-pushed the UNDERTOW-1794 branch from 8407e8d to 00a68be Compare April 27, 2021 11:14

fl4via previously requested changes Mar 21, 2023

View reviewed changes

fl4via added the waiting PR update Awaiting PR update(s) from contributor before merging label Mar 21, 2023

baranowb force-pushed the UNDERTOW-1794 branch from 00a68be to d782713 Compare April 14, 2023 11:26

baranowb requested a review from fl4via April 14, 2023 11:31

baranowb commented Apr 26, 2023

View reviewed changes

baranowb force-pushed the UNDERTOW-1794 branch from d782713 to 8ff8540 Compare April 26, 2023 08:51

baranowb force-pushed the UNDERTOW-1794 branch from 8ff8540 to 77cb16d Compare March 12, 2024 08:47

fl4via requested changes Apr 30, 2024

View reviewed changes

baranowb force-pushed the UNDERTOW-1794 branch 4 times, most recently from 3b217e9 to 65e2b2b Compare May 14, 2024 08:06

baranowb mentioned this pull request May 14, 2024

UNDERTOW-1794 DefaultAccessLogReceiver violates Closeable contract #1593

Open

PrarthonaPaul reviewed Jun 11, 2024

View reviewed changes

baranowb force-pushed the UNDERTOW-1794 branch from 65e2b2b to 823c558 Compare July 9, 2024 08:34

ropalka approved these changes Sep 11, 2024

View reviewed changes

ropalka requested changes Sep 11, 2024

View reviewed changes

baranowb added 2 commits October 21, 2024 08:20

[UNDERTOW-1794] improve DefaultAccessLogReceiver to obey contract!

fc12159

[UNDERTOW-1794] Add methods to configure retry/delay

62e7cf4

fl4via force-pushed the UNDERTOW-1794 branch from 823c558 to 62e7cf4 Compare October 21, 2024 11:21

fl4via added WildFly RFE This PR is tied to a WildFly RFE process. and removed waiting peer review PRs that edit core classes might require an extra review labels Oct 15, 2025

fl4via requested changes Oct 15, 2025

View reviewed changes

fl4via added waiting PR update Awaiting PR update(s) from contributor before merging and removed under verification Currently being verified (running tests, reviewing) before posting a review to contributor labels Oct 15, 2025

baranowb mentioned this pull request Oct 22, 2025

Undertow 1794 2.3.x #1825

Open

UNDERTOW-1794 DefaultAccessLogReceiver violates Closeable contract #1041

Are you sure you want to change the base?

UNDERTOW-1794 DefaultAccessLogReceiver violates Closeable contract #1041

Uh oh!

Conversation

baranowb commented Feb 16, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

baranowb Feb 16, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

fl4via commented Feb 16, 2021

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

fl4via left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

PrarthonaPaul left a comment

Choose a reason for hiding this comment

Uh oh!

baranowb commented Feb 16, 2021 •

edited

Loading

baranowb Feb 16, 2021 •

edited

Loading