Many requests break after 4.2.0 upgrade for unclear reasons

[jamesforwardnwboxed]

Describe the bug
Using spring-boot-starter-gateway with 3.3.5 spring boot version and 2023.0.3 spring cloud version, everything works as expected and all of our requests are routed.
We have discovered upgrading to 3.4.1 spring boot version with 2024.0.0, a number of external requests being made are failing. We have attempted to debug the reason and it is pretty unclear. We know it's something to do with the upgrade as the requests made have been identical, and reverting solves the issue.
We checked the changes between 4.1.5 and 4.2.0 of this dependency (which is what the starter provides) and they are very substantial. It may also be down to a transitive dependency - at the moment it is unclear.

Debugging using our observability tool, we find that some of the external requests seem to match /** under Spring Webflux rather than the normal route. This is odd, as other requests coming in match the route correctly.

The only things different between the requests that break and those that don't are certain headers which should be unrelated to routing.

Sample
Cannot provide samples due to proprietary code unfortunately.

[spencergibb]

Without more information it's going to be difficult, if not impossible to diagnose anything. We don't expect proprietary code to be submitted, but taking some time to create a minimal standalone sample that recreates the problem is essential.

[jamesforwardnwboxed]

Without more information it's going to be difficult, if not impossible to diagnose anything. We don't expect proprietary code to be submitted, but taking some time to create a minimal standalone sample that recreates the problem is essential.

As a fellow dev I totally appreciate that. The problem is I can't recreate this in integration tests (for some really unclear reason) but we can recreate it in a deployed environment. So sample code isn't going to be possible. It might be a multithreading issue, who knows...
What we can confirm is reverting the upgrade completely fixes the problem, so it's something in the Gateway.

This is the exact error:

org.springframework.web.reactive.resource.noresourcefoundexception: 404 NOT_FOUND "No static resource my/custom/endpoint"

Maybe other users will run into it, and they will be able to comment on this issue if they have more info and it can help the debugging. At the very least raising this acts as a flag that something is off.

[manojs-kr]

for my case whenever request coming through another nginx having header X-Forwarded-Prefix then its try to find the predict with prefix and uri. witch is causing 404. I am able to reproduce this with passing header X-Forwarded-Prefix and server.forward-headers-strategy: framework in my local setup. and kubernetes it all fail.

[jamesforwardnwboxed]

for my case whenever request coming through another nginx having header X-Forwarded-Prefix then its try to find the predict with prefix and uri. witch is causing 404. I am able to reproduce this with passing header X-Forwarded-Prefix and server.forward-headers-strategy: framework in my local setup. and kubernetes it all fail.

Interestingly, it turns out the requests being made which fail did include X-Forwarded-Prefix headers, however I'm unable to find any config we set which adds server.forward-headers-strategy at all. I can also only reproduce this in my integration tests with server.forward-headers-strategy, but with this not being configured (which as far as I can tell is whatever the default is) the integration tests pass but the requests in the environment fail.

@spencergibb Do you know if there have been any changes in this area between 4.1.5 and 4.2.0, and if so is there a way to get previous behaviour working?

[Airidas36]

@jamesforwardnwboxed
We are experiencing the same issue when application is sitting behind a load balancer. This issue was discovered only after the deployment since unit/integration tests / running it on docker does not entirely replicate such an infrastructure setup. I'm not 100% sure but because of the prefixed path looks like it also causes problems when trying to open Swagger UI when querying with /swagger-ui.htmlSo far I identified that this may be caused by these changes in reactor-netty as well as spring-framework:

• Support X-Forwarded-Prefix in DefaultHttpForwardedHeaderHandler reactor/reactor-netty#3432
• Handle X-Forwarded-Prefix parsed by Reactor-Netty in ReactorServerHttpRequest spring-projects/spring-framework#33638

Issue is also mentioned here spring-projects/spring-boot#43577

Currently, this can be worked around using configuration server.forward-headers-strategy: none, although this breaks things if you have some BFF gateway running with Spring Security and CORS, so far we have not managed to work around that and downgraded the spring boot version for such services.

[SlinToWin]

We have a similiar issue and could also track it down to the SB / Cloud Gateway Upgrade to 2024.
Our API Gateway sits behind another Gateway.

Whenever the hosts header is send via the other gateway, SCG can not find the route and responds with a 404.
Removing the Host header (from our Insomnia tests) resolved the issue.

The issue is not existing on previous Spring Cloud Gateway version

[capnrotbart]

We had the same issue, and I managed to write a reproduction in #3677. I too think that there are two changes in Spring Framework & Reactor Netty, that have unintended effects for Gateway.

[tai-yi]

After working on it two days, resolved this issue.
Change below items

• server.forward-headers-strategy=framework
• add Bean

  @Bean
   public ForwardedHeaderTransformer forwardedHeaderTransformer() {
       ForwardedHeaderTransformer transformer = new ForwardedHeaderTransformer();
       transformer.setRemoveOnly(true);
       return transformer;
   }