Replies: 1 comment 1 reply
-
I finally managed to create a reproducer! Issue coming soon. |
Beta Was this translation helpful? Give feedback.
1 reply
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Uh oh!
There was an error while loading. Please reload this page.
-
I've been debugging the weirdest problem in a long time in a Micronaut 3.7.2 service.
The service (A) main purpose is to listen for Kafka events, make a HTTP GET request to service B, process data, then POST result to service C.
Getting data from service B is done with a declarative HTTP client and the client method contains Header annotations.
Posting result to service C is done with another declarative HTTP client and the client method contains same Header annotations.
Micronaut multitenancy dependency is installed because other parts of the service uses tenant propagation. It reads/writes the X-Tenant header.
Micrometer Prometheus dependency is installed and the /prometheus endpoint is scraped regularly.
We noticed in the logs a number of
uploadResult
requests got unauthorized errors. To my surprise I found that sometimes the value bound to the X-Tenant header when posting result to service C was the tenant that only prometheus uses!.If I disable this endpoint in prometheus. The problem never appear. But in theory, if this is Micronaut or Netty leaking header values, then we should see this problem with other tenants. But I only see the Prometheus tenant in the error logs.
Yesterday I made a small application to reproduce this problem. But so far that application works as expected. I've not been able to trigger the issue.
My question is: Where should I focus my debugging? I can see that the
tenant
value I pass to the declarative client method is correct. But when TRACE logging the headers I can see that the wrong value appear in the X-Tenant header of the POST request. I want to see where the wrong Header value is bound to the request, but I didn't find the right spot when debugging.My next step is to remove the micronaut-multitenancy dependency to see if it's related to that.
Beta Was this translation helpful? Give feedback.
All reactions