Performance comparison bias

[boazsegev]

Hi,

Lovely start and good luck with the project.

I noticed that both Puma and Iodine seem to be tuned in favor of IO blocking apps rather than CPU bound apps, which creates a certain performance comparison bias.

For example: you'd probably get better throughput on iodine with a single processes and 2 or 1 threads when using a simple "hello world" type app. If the app was using more CPU, you might want a single process with CPU_CORES - 1 threads. Puma might be different / similar, I haven't tested.

Having said that, I'm not sure it really matters and I wish you luck!

B.

[janbiedermann]

Hi Bo, thanks for your comment. In fact, the result Up! achieved, have been due to benchmark tool error.
I have switched to bombardier, iodine shines again. Love iodine!

Up! result are very low now ... working on it.

[janbiedermann]

Well, worked on it. The original benchmark triggered a sweet spot in Opal, so i decided to benchmark a static string, just for Up! to be the fastest again :)

I followed your suggestion and tried variations of -w 1..4 -t 1..4, best results i achieved with -w 4 -t 1.
Results are included in the README.md

[janbiedermann]

I added header support to Up!, updated the env.to_s benchmark to include a header and ran benchmarks again.
Iodine on my machine achieved best result also for this benchmark with -w 4 -t 1 (59116.53 req/s). I updated numbers.
(I did not push yet, work in still progress)

[boazsegev]

I'm not sure if env.to_s adds much to the benchmark.

It is mostly CPU bound work that would have benefited from threads on a concurrent Ruby implementation (probably not on MRI)...

I suspect that in a mixed Ruby implementation comparison it adds a Ruby implementation benchmark bias to the original server tests.

[janbiedermann]

With env.to_s i want to measure rack environment access speed, as this is the way, rack servers communicate with a app. #to_s i have chosen as a simple way to achieve that and generate some content. JSON unfortunately may fail with some objects in the rack env. For Opal the situation has improved with the latest PR merges, numbers show, a slight disadvantage here is ok at the moment. With more features added and things fixed, maybe soon a real app can be measured.