BREAKING-CHANGE Are Chunked Responses Being Handled Wrong?

[dnwiebe]

A Route consists of a Vec of CryptData objects. Most of those CryptData objects, when decrypted, are LiveHops, containing the public key of the destination, a Payer object, and a Component enum telling which actor in the target Node will be handling the data. However, the very last CryptData object in a Route can't be decrypted into a LiveHop, because it's actually just a single u32 that's known as a Return Route ID.

When a ClientResponsePayload_0v1 comes back to the originating Node from the exit Node (refer to ProxyServer::handle_client_response_payload(), we use the Return Route ID at the very end of the Route to look up the return route information in the Proxy Server. This return route information was stored when we sent the request that resulted in the ClientResponsePayload_0v1, so that we'd know what to do when the response came back, and when it was stored, we got an ID back, which was put in the Route at the end where we'd see it.

The return route information contains almost everything we need to pay what we owe to all the Nodes along the return route, including each of those Nodes' RatePacks; but we can't compute an actual amount of money until we know how much data they processed for us, which we find out when the ClientResponsePayload_0v1 arrives. So after we use the Return Route ID in the Route to look up the return route information in the Proxy Server's HashMap (actually a TtlHashMap, more on that later), we can combine the two pieces of information and report the services consumed to the Accountant.

So here's the issue: when should the Proxy Server forget the return route information it has stored? When does the Proxy Server forget the return route information it has stored? Are those two the same?

At a high level, here are the answers to the first question:

1. When we're done receiving the response to the request that resulted in the return route information being stored
2. When the stream over which the request was sent is terminated, even if no response has yet arrived
3. After a decent amount of time has elapsed, in case the server never responded or the response (or request) got lost along the way, so that we don't endlessly accumulate useless data in the Proxy Server

However, #1 is deceptive, in the face of chunked HTTP responses (and possibly things that protocols other than HTTP do). If we remove the RRI immediately after receiving the first chunk of response, then every other chunk will arrive and find no RRI, and so we will have no idea how to pay the return-route bill for it.

We're not sure whether #2 is implemented or not; we've looked briefly at the StreamKey-retirement code and haven't found anything about RRI, but we haven't searched exhaustively.

#3 is implemented by the fact that we're using a TtlHashMap, which keeps its data around for only two minutes, and then removes it. #3 may be taking care of #2 as well.

What we do know is that we're receiving too many logs like this one from NetFlix:

2025-05-25 23:13:02.380 Thd8: ERROR: ProxyServer: Can't report services consumed: received response with bogus return-route ID 2257 for client response. Ignoring
2025-05-25 23:13:02.404 Thd8: ERROR: ProxyServer: Can't report services consumed: received response with bogus return-route ID 2255 for client response. Ignoring
2025-05-25 23:13:02.448 Thd8: ERROR: ProxyServer: Can't report services consumed: received response with bogus return-route ID 2256 for client response. Ignoring
2025-05-25 23:13:02.504 Thd8: ERROR: ProxyServer: Can't report services consumed: received response with bogus return-route ID 2254 for client response. Ignoring
2025-05-25 23:13:03.195 Thd8: ERROR: ProxyServer: Can't report services consumed: received response with bogus return-route ID 2257 for client response. Ignoring
2025-05-25 23:13:03.221 Thd8: ERROR: ProxyServer: Can't report services consumed: received response with bogus return-route ID 2255 for client response. Ignoring
2025-05-25 23:13:03.266 Thd8: ERROR: ProxyServer: Can't report services consumed: received response with bogus return-route ID 2256 for client response. Ignoring
2025-05-25 23:13:03.336 Thd8: ERROR: ProxyServer: Can't report services consumed: received response with bogus return-route ID 2254 for client response. Ignoring

That's what happens when a response arrives with a Return Route ID in its Route, but the Proxy Server has forgotten the corresponding return route information. Notice that in these eight logs, there are four pairs of two. Each ID is being searched for twice in vain, with the two attempts occurring a little less than a second apart. This would tend to indicate that it's not just a random problem with late single responses arriving after the 120-second timeout; there's probably something deeper going on here.

Task: Investigate the ProxyServer and related code and find out how return route information is actually removed from the ProxyServer, and determine how the design should change to eliminate logs like the ones above. We should not use loads and loads of memory remembering RRI forever, but we should also be able to handle HTTP chunked responses and other protocols that send multiple response packets for a single request packet.

Do not fix the code as part of this card; create a card or series of cards to solve whatever problems you find.

[dnwiebe]

I considered enhancing the ProtocolPack to be an instance class rather than (essentially) a static class, attaching an appropriate ProtocolPack instance to each RRI, and running every response packet through the ProtocolPack until it said that was the last packet in the response, and the RRI could be discarded.

However, while that's theoretically possible with HttpProtocolPack, it's not with TlsProtocolPack, because with TLS there's no way for the Node to see the unencrypted contents of the packets.

[dnwiebe]

Observation: the TtlHashMap is quite badly designed. The distinguishing factor of a HashMap as opposed to a Vec is that it's much faster to look up the element you want to find (or to add an additional element) than it is with a Vec. However, TtlHashMap saddles every insert() and get() with a call to remove_expired_entries(), which is a O(n) table scan. That means TtlHashMap is no better than a Vec from a performance standpoint, and would probably be simpler to implement as a Vec.

An idea from @czarte : don't do remove_expired_entries() on insert() or get(); instead, have a Future do it periodically. Every few seconds, or every minute, have the Future wake up and remove_expired_entries(); then have it reschedule itself for another few seconds down the road.

This would mean that the contents of the TtlHashMap would be accessed from multiple threads, which means there'd need to be some kind of synchronization; but it'd probably still be faster than a Vec.

CORRECTION

This comment is invalid. I didn't look closely enough at the remove_expired_entries() method. It does run on every get() and insert(), but the first thing it does is check to see if it's run recently, and if it has, it aborts rather than traversing the data. So it's fine the way it is.

[dnwiebe]

Here's another idea.

This card is mostly about chunked responses in HTTP. The temporal pattern for chunked responses is that there might be a significant delay between the request and the first chunk of the response, because the server might have a fair amount of searching, communicating, and computation to do to figure out what data should be returned; but once that determination is made, the second and succeeding chunks follow with much less delay.

So: make the TtlHashMap solely responsible for getting rid of stored RRI when it times out, but instead of having one TTL for the entire TtlHashMap, have individual TTL values for each element. When an element of RRI is first stored in the ProxyServer's route_ids_to_return_routes, make its TTL two minutes, which is the standard timeout for HTTP requests. But the first time a response arrives and stimulates the lookup of an RRI element, change its TTL from two minutes to something smaller: say, five seconds. This would reduce the size of route_ids_to_return_routes in the face of heavy chunked-response usage.

However: chunked responses are used mostly for streaming, as in the case of audio or video. If most of the traffic through a Node isn't audio or video, then every element of RRI in route_ids_to_return_routes will stay for the whole two minutes, and the structure could still get pretty large. A large TtlHashMap is quite inefficient, because of its inept design.

[dnwiebe]

Half-baked idea: a two-level cache for RRI. Operation much as today, where unused RRI is removed from route_ids_to_return_routes after the standard two minutes; but if the RRI is used, instead of being discarded, it's transferred to a second TtlHashMap where the TTL is shorter (five seconds or so). Then if the main route_ids_to_return_routes is searched for RRI and nothing is found, the second TtlHashMap can be searched before giving up.

If TtlHashMap's performance is improved back essentially to O(1), two TtlHashMap searches might be performed in less time than is required by one today.

Further inspiration: The second-level TtlHashMap, except in cases of streaming, will be full of RRI that is probably never going to be needed again; except in cases of streaming, it will almost always expire rather than be explicitly removed. However, in the case of streaming, every small streaming packet will require a failed search of route_ids_to_return_routes followed by a successful search of the second-level TtlHashMap. So, how about this?

When a response packet comes in, search route_ids_to_return_routes for its RRI. If you find it, use it to process the response packet and move the RRI to the second-level TtlHashMap.

If you don't find it, search the second-level TtlHashMap for the RRI. If you don't find it, give up and log the error mentioned in the card above.

But if you do find it, use it to process the response packet and move the RRI back to route_ids_to_return_routes. That way, finding the RRI for the second chunk in a stream will be a little slower, but finding the RRI for the third and following chunks will be just as fast as before. (Faster, if TtlHashMap is rearchitected not to be so slow.)

In order to do this, there'd have to be a flag added to the RRI to say whether it had been resurrected from the second-level TtlHashMap back into route_ids_to_return_routes, because if it had, then it shouldn't be transferred back to second level if another response packet arrives for it.

[dnwiebe]

We noticed something strange. The return route ID (RRID), which identifies the set of return-route information created just before the request is sent, that waits to be used by the returning response(s), is created in the Neighborhood and shares cardinality with the RouteQueryResponse and the StreamKey that becomes associated with it.

That means that for a given stream S that navigates a route R determined by the neighborhood, every request and every response that travels over that stream carries the same RRID. This probably just happens to work for HTTP 1.1, since that protocol's rules dictate that all of a request's response packets must arrive before the next request is allowed; but it's not the proper way.

The RRID should be generated in the Proxy Server, not the Neighborhood, and every request should have its own RRID. After a Node has been running for an hour, we should be seeing five- or six- or seven-digit RRIDs, not two-digit. RRIDs are eventually reused, but there are four billion available, so as long as we don't cache RRI too long, we shouldn't ever get collisions.