fix(http/prom): record bodies when eos reached #3856
Merged
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
this commit fixes a bug discovered by @alpeb, which was introduced in proxy
v2.288.0.
in #3086 we introduced a suite of route- and backend-level metrics. that
subsystem contains a body middleware that will report itself as having
reached the end-of-stream by delegating directly down to its inner
body's
is_end_stream()hint.this is roughly correct, but is slightly distinct from the actual invariant: a
linkerd_http_prom::record_response::ResponseBody<B>must call itsend_streamhelper to classify the outcome and increment the correspondingtime series in the
outbound_http_route_request_statuses_totalmetric family.in #3504 we upgraded our hyper dependency. while doing so, we neglected to
include a call to
end_streamif a data frame is yielded and the inner bodyreports itself as having reached the end-of-stream.
this meant that instrumented bodies would be polled until the end is reached,
but were being dropped before a
Nonewas encountered.this commit fixes this issue in two ways, to be defensive:
end_stream()if a non-trailers frame is yielded, and the inner bodynow reports itself as having ended. this restores the behavior in place prior
to chore(deps)!: upgrade to hyper 1.x #3504. see the relevant component of that diff, here:
https://github.com/linkerd/linkerd2-proxy/pull/3504/files#diff-45d0bc344f76c111551a8eaf5d3f0e0c22ee6e6836a626e46402a6ae3cbc0035L262-R274
<B as Body>::is_end_stream()method,report the end-of-stream being reached by inspecting whether or not the inner
response state has been taken. this is the state that directly indicates
whether or not the
ResponseBody<B>middleware is finished.this adds a
linkerd-mock-http-bodydevelopment dependency, so we can use thismock body type in the outbound proxy's unit tests. additional test coverage for
http and grpc routes' metrics is added to demonstrate that this fix works as
expected.
X-ref: #3504
X-ref: #3086
X-ref: linkerd/linkerd2#8733