feat(s2n-quic-dc): Reuse handshakes across application processes #2854

mehnazyunus · 2025-10-15T17:08:56Z

Release Summary:

Reuse handshakes across application processes to reduce memory utilization.

Resolved issues:

Description of changes:

This PR introduces a manager server and an application server, which are intended to run in separate processes. A client handshakes with and connects to the manager server. The manager server accepts incoming streams and transfers them via Unix domain sockets to the application server. The application server receives these streams and continues communicating with the client. The same handshake can be reused with multiple manager servers.

Call-outs:

Added in comments.

Testing:

Added happy path integration test. The client handshakes once and connects to two manager servers, reusing the same handshake. It then continues communication with separate application servers.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

dc/s2n-quic-dc/src/stream/server/tokio/tcp/worker.rs

mehnazyunus · 2025-10-15T17:16:34Z

dc/s2n-quic-dc/src/stream/server/tokio/tcp/worker.rs

+            };
+
+            if keys.application.opener.dedup_check().is_err() {
+                return Ok(ControlFlow::Continue(())).into();


is it okay to just continue here? The dedup check implementation does handle emitting events and sending a control packet.

Realised the dedup_check function is behind the testing flag, does it make sense to remove that flag and use it here?

I think we should use the on_decrypt_success function -- and actually decrypt the packet we got.

mehnazyunus · 2025-10-15T17:21:42Z

dc/s2n-quic-dc/src/stream/server/tokio.rs

        let acceptor = udp::Acceptor::new(
            id,
            socket,
            &self.stream_sender,


The Start struct could be fully reused in the manager once we can move this stream sender for UDP to a behavior struct like for TCP. (manager server does not have a stream sender)

mehnazyunus · 2025-10-15T17:44:13Z

dc/s2n-quic-dc/src/stream/server/tokio/tcp/worker.rs

+                    return Err(Some(error)).into();
+                }
+                WorkerState::Sending { future, event_data } => match future.as_mut().poll(cx) {
+                    Poll::Ready(res) => match res {


I tried reusing the poll_send function here, but wasn't able to resolve the borrowing issues when calling the function with the future from the WorkerState, not sure if there is a way around that.

I'm not worried about it for the initial PR, but we can take a look together at doing so.

mehnazyunus · 2025-10-15T17:46:17Z

dc/s2n-quic-dc/src/stream/server/tokio/uds.rs

+        let recv_buffer: either::Either<_, Channel> = Either::A(recv_buffer);
+        let sub = self.env.subscriber();
+
+        let map = Map::new(


Using a placeholder map here to send to accept_stream, afaict it is only used for handling unexpected packets.

I think we'll want to avoid creating a new map for each stream -- that's pretty expensive.

Could we create a single placeholder map and reuse that for all streams?

Possibly? I'm not sure what we're using the map for. Maybe a placeholder would work, or we can edit the signatures to have Option<&Map> if that is better/easier.

mehnazyunus · 2025-10-15T17:48:38Z

dc/s2n-quic-dc/src/stream/server/tokio/tcp/worker.rs

+                    Poll::Ready(Ok(ControlFlow::Continue(())))
+                }
+                Err(err) => {
+                    debug!("Error sending message to socket {:?}", err);


Do we have to close the stream here, something like this?

Yes, that seems right.

dc/s2n-quic-dc/events/acceptor.rs

Mark-Simulacrum · 2025-10-15T18:00:57Z

dc/s2n-quic-dc/events/acceptor.rs

+    /// The ID of the stream
+    stream_id: u64,
+
+    /// The amount of time the TCP stream spent in the queue before being enqueued


Which queue is this? I suspect this is time from TCP accept() to UDS sendmsg()?

Yes, the "fresh" queue. Can change the comment to "before being sent over UDS"

Mark-Simulacrum · 2025-10-15T18:11:37Z

dc/s2n-quic-dc/events/acceptor.rs

+    sojourn_time: core::time::Duration,
+
+    /// The number of times the Unix domain socket was blocked on send
+    #[measure("blocked_count")]


Is this for sending this particular stream? It might make sense to switch this (or add) a #[counter(...)] as well.

Measure will emit histograms.

Mark-Simulacrum · 2025-10-15T18:16:18Z

dc/s2n-quic-dc/src/stream/server/application.rs

+    #[allow(dead_code)]
+    env: Environment<S>,
+    #[allow(dead_code)]
+    acceptor_rt: runtime::Shared<S>,


Why do we have the allow(dead_code) here? Are these only kept for a Drop impl or something?

Removed those fields here, but I'm not sure why we have it in the original server in the first place. fmt/clippy/build do not seem to complain without it.

Mark-Simulacrum · 2025-10-15T18:18:22Z

dc/s2n-quic-dc/src/stream/server/application.rs

+    enable_udp: bool,
+    enable_tcp: bool,
+    accept_flavor: accept::Flavor,
+    socket_path: Option<PathBuf>,


I wouldn't expect accept_flavor, workers, acceptor_addr, and maybe even enable_udp/tcp to be relevant here. Are we actually making use of most of these values?

workers is used in deciding the concurrency of the acceptor runtime. Do we want to use a fixed value instead?
Will remove acceptor_addr.
Wouldn't enable_udp/tcp be required when we add the UDP socket worker?

I think I'm hoping we can get rid of the application builder needing an acceptor runtime.

enable-udp/tcp is a good call out though, keeping those makes sense.

Removed the runtime. Currently not using enable-udp/tcp, but would probably have to use it in accept when UDP logic is added.

dc/s2n-quic-dc/src/stream/server/tokio/tcp/worker.rs

Mark-Simulacrum · 2025-10-15T18:42:14Z

dc/s2n-quic-dc/src/stream/server/tokio/tcp/worker.rs

+                    return Err(Some(error)).into();
+                }
+                WorkerState::Sending { future, event_data } => match future.as_mut().poll(cx) {
+                    Poll::Ready(res) => match res {


I'm not worried about it for the initial PR, but we can take a look together at doing so.

Mark-Simulacrum · 2025-10-15T18:46:41Z

dc/s2n-quic-dc/src/stream/server/tokio/tcp/worker.rs

+            };
+
+            if keys.application.opener.dedup_check().is_err() {
+                return Ok(ControlFlow::Continue(())).into();


I think we should use the on_decrypt_success function -- and actually decrypt the packet we got.

dc/s2n-quic-dc/src/stream/server/tokio/tcp/worker.rs

Mark-Simulacrum · 2025-10-15T18:58:04Z

dc/s2n-quic-dc/src/stream/server/tokio/uds.rs

+        let recv_buffer: either::Either<_, Channel> = Either::A(recv_buffer);
+        let sub = self.env.subscriber();
+
+        let map = Map::new(


I think we'll want to avoid creating a new map for each stream -- that's pretty expensive.

Mark-Simulacrum · 2025-10-17T20:28:28Z

quic/s2n-quic-core/src/time/timestamp.rs

+
+impl EncoderValue for Timestamp {
+    fn encode<E: Encoder>(&self, buffer: &mut E) {
+        buffer.encode(&self.0.get());


I suspect we shouldn't add these impls. Instead, let's capture SystemTime before/after send/recv on the unix socket.

mehnazyunus added 7 commits October 14, 2025 22:01

Move mpmc sender to poll behavior struct

9749b32

Move server builder methods to macros

e4eb9ca

Add Unix domain socket send event

75fde84

Add helper functions to access secret

8f4dcd4

Add manager server functionality

f26c672

Add application server functionality

8a1a9d3

Add shared cache happy path test

7c8b019

mehnazyunus requested a review from a team as a code owner October 15, 2025 17:08

mehnazyunus commented Oct 15, 2025

View reviewed changes

dc/s2n-quic-dc/src/stream/server/tokio/tcp/worker.rs Show resolved Hide resolved

mehnazyunus commented Oct 15, 2025

View reviewed changes

mehnazyunus requested a review from Mark-Simulacrum October 15, 2025 17:49

mehnazyunus force-pushed the shared-cache branch from f29e579 to 1efc867 Compare October 15, 2025 18:07

Fix clippy warnings

ed64761

mehnazyunus force-pushed the shared-cache branch from 1efc867 to ed64761 Compare October 15, 2025 18:16

Mark-Simulacrum reviewed Oct 15, 2025

View reviewed changes

mehnazyunus added 9 commits October 15, 2025 22:48

Add counter for blocked_count

4a75860

Decrypt packet for dedup check

fe198e1

Fix integration test

e7b29b0

Remove acceptor addr from application server

a0fd7f4

Use different socket paths in test

cf206ad

Remove acceptor runtime and mpmc channel from application server

3821caf

Add queue_time to packet

b209b71

Init manager server

8ea84c3

Implement manager server

6777e07

Mark-Simulacrum reviewed Oct 17, 2025

View reviewed changes

Use a single map across streams

4e9967b

Make uds receiver cloneable

d8eca96

feat(s2n-quic-dc): Reuse handshakes across application processes #2854

Are you sure you want to change the base?

feat(s2n-quic-dc): Reuse handshakes across application processes #2854

Uh oh!

Conversation

mehnazyunus commented Oct 15, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Release Summary:

Resolved issues:

Description of changes:

Call-outs:

Testing:

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mehnazyunus Oct 15, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mehnazyunus Oct 15, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

mehnazyunus commented Oct 15, 2025 •

edited

Loading

mehnazyunus Oct 15, 2025 •

edited

Loading

mehnazyunus Oct 15, 2025 •

edited

Loading