Tracing has a weird integration with Client Middlewares

[UltimateLobster]

After testing abit with the new client middlewares feature I've noticed something weird about the integration of the feature with tracing.

It seems that the tracing hooks for on_request_start, on_request_end and on_request_exception are called before the client middlewares. It seems weird to me as I would expect tracing hooks to be as close to the actual sent requests as possible.

That means that if I have a middleware that modifies the requests, or retries the requests multiple times, it will not be reflected in the traces. Here I've created a minimal retry middleware and trace configs to reflect the issue.

import asyncio

from types import SimpleNamespace
from aiohttp import (
    ClientSession, 
    TraceConfig, 
    TraceRequestStartParams, 
    TraceRequestEndparams, 
    TraceExceptionParams, 
    ClientRequest, 
    ClientHandlerType, 
    ClientResponse
)

async def on_request_start(session: ClientSession, context: SimpleNamespace, params: TraceRequestStartParams) -> None:
    print("on request start")

async def on_request_end(session: ClientSession, context: SimpleNamespace, params: TraceRequestEndParams) -> None:
    print("on request end")

async def on_request_exception(session: ClientSession, context: SimpleNamespace, params: TraceRequestExceptionParams) -> None:
    print("on request exception")

trace_config = TraceConfig()
trace_config.on_request_start.append(on_request_start)
trace_config.on_request_end.append(on_request_end)
trace_config.on_request_exception.append(on_request_exception)

async def middleware(request: ClientRequest, handler: ClientHandlerType) -> ClientResponse:
    print("middleware start")
    for i in range(3):
        print(f"attempt {i}")
        response = await handler(request)
        
        if response.ok:
            break
    
    print("middleware end")
    return response

async def main():
    async with ClientSession(trace_configs=[trace_config], middlewares=(middleware,)) as session:
        async with session.get("http://something") as response:
            print(response.status)

if __name__ == "__main__":
    asyncio.run(main())

For an endpoint that always returns 500 we will see the following:

on request start
middleware start
attempt 0
attempt 1
attempt 2
middleware end
on request end
500

Which means if I use OpenTelemetry instrumentation, I would see a single request where I would expect to see 3.
Things become even more complicated when you take the automatic retry for idempotent methods into account.
At that point I can have up to 6 attempts to send the request, but tracing-wise, I would only see one which (at least for me) is completly unexpected.

Is this a bug or the intended behavior? I would love to see some documentation about the way client middlewares should be integrated with other features of aiohttp.

[Dreamsorcerer]

Not sure we thought about it during development.