Runtime flavours

[RPallas92]

The runtime used by Pingora is based on Tokio. There are two flavours:

1. Tokio standard multi-thread runtime that has work stealing.
2. A custom multi-thread runtime without work stealing.

According to the comments in the code the second runtime is as efficient as the single-threaded runtime while allows the async program to use multiple cores. However, I took a look at the benchmarks and the second flavour has higher latency and handles less reqs/sec.

In which real cases a multi-thread runtime without work stealing could be more efficient?

[eaufavor]

Could you share your test methods and results? Thanks.

This https://github.com/cloudflare/pingora/blob/main/pingora-runtime/benches/hello.rs bench shows a case where the non-stealing runtime could be better. The benchmark is just to show that there is a potential trade-off. The numbers and results could vary based on your exact workload and hardware.

[lpj145]

Hello guy's, I've tested a lot tokio tcp stream, axum, actix, hyper and mio without async just for understanding, and I'm a little worried about the way that you have choosed this flavour, right now the tests using axum, hyper and actix shows a super stability with low latency and the same do not occur with pingora take this examples.

This is a just hello world tested on Intel® Core™ i9-14900HX × 32 32Gb Ram

# This is axum
wrk -t12 -c24 -d30s -R100k http://localhost:3000 --latency
Running 30s test @ http://localhost:3000
  12 threads and 24 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency     1.07ms  567.71us   3.41ms   60.54%
    Req/Sec     8.78k   695.87    11.22k    70.02%
  Latency Distribution (HdrHistogram - Recorded Latency)
 50.000%    1.06ms
 75.000%    1.52ms
 90.000%    1.86ms
 99.000%    2.21ms
 99.900%    2.57ms
 99.990%    2.84ms
 99.999%    3.04ms
100.000%    3.41ms

# This is pingora:
wrk -t12 -c24 -d30s -R100k http://localhost:8080 --latency
Running 30s test @ http://localhost:8080
  12 threads and 24 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency     1.11ms  758.09us  10.55ms   77.58%
    Req/Sec     8.79k   751.08    17.00k    70.57%
  Latency Distribution (HdrHistogram - Recorded Latency)
 50.000%    1.04ms
 75.000%    1.52ms
 90.000%    1.86ms
 99.000%    3.47ms
 99.900%    8.67ms
 99.990%    9.84ms
 99.999%   10.42ms
100.000%   10.56ms