Unix socket listening instead of TCP?

[bbrtj]

I am currently implementing a Thunderhorse benchmark for https://github.com/TechEmpower/FrameworkBenchmarks, and I wanted to keep it the same as Kelp. Server::Starter has a feature of listening to a unix socket instead of TCP, and Kelp benchmark used it. Nginx then communicates with the app through that socket, which is quite elegant.

There are no docs for unix sockets in PAGI::Server. Is it currently supported, and if not, is the support for this planned?

[jjn1056]

It’s on the plan sheet, I was trying to settle down issues with http1.1 first; I think it’s getting stable enough that we can work on that. FWIW http2 is in progress as well.

Feel free to let me know how benching is going. I’ve been mostly stress testing, not optimizing for performance yet.

[bbrtj]

Great thanks for clarifying. I was not sure whether I'm missing some way of setting this up or not.

[bbrtj]

Thunderhorse + pagi-server + nginx reverse proxy through HTTP is at least 3 times slower than Kelp + gazelle + nginx reverse proxy through unix socket. Can't say how much of that difference can be accounted to web server and how much to the framework.

[bbrtj]

I benchmarked a similar minimal PAGI app that does the same thing with just PAGI::Response object. It improved throughput from 3600 reqs per sec to 4300 per sec, so the framework is not the main bottleneck.

I tried installing Future::XS, but that did not change much. I also tried to install EV and use it with pagi-server, but no measurable improvement either.

It is currently running a bit slower than Mojolicious/Hypnotoad (3600 vs 5000 requests per sec). The program I'm looking at in itself is as basic as it can get - it just encodes some json data and returns it. I made sure Cpanel::JSON::XS is available for both frameworks.

If you have any performance tips, I'll be open to try them out.

[jjn1056]

@bbrtj what platform are you running the tests on? Any chance you can share a repo with the test setup?

PAGI::Server hasn't been heavily optimized yet, mostly been working on getting it sturdy, but I think we should be able to get better numbers.

Here's how I run it on my 16 core Mac when I'm testing to check for performance regressions

LIBEV_FLAGS=8 perl -Ilib ./bin/pagi-server -E production --loop EV --timeout 0 --future-xs --workers 16 ...

Yeah its surprising switching to FutureXS doesn't seem to do much, it almost makes me think I'm not using it right

The big ones are -E production which turns off console logging (which currently is blocking and slow, fixing that is on the list) and turning the timeout counter off, that adds a bit of work per request and if you are running behind nginx you don't need it.

I haven't done any testing on Nginx directly yet so will be interested to see how this goes.

[bbrtj]

Here is my benchmark code. It's a docker-based benchmark suite. https://github.com/bbrtj/FrameworkBenchmarks/tree/tmp/thunderhorse/frameworks/Perl/thunderhorse

files of interest would be run.pl, thunderhorse.dockerfile and app.pl. This has Thunderhorse code commented out and only a simple PAGI app rendering required json data. I am benchmarking json using ./tfb --test thunderhorse -m benchmark --type json and comparing against ./tfb --test mojolicious -m benchmark --type json

[jjn1056]

@bbrtj I wonder if its something with the way PAGI::Server behaves when its behind a proxy, honestly your results are very different from what I get when I test locally, hitting the server directly. I will need to spend more time deconstructing this.

[bbrtj]

Oh wow you are right. I'm getting 10k+ reqs per sec when skipping nginx completely, which is like 3 times faster than with it. Maybe nginx is misconfigured in this benchmark, it is an ancient config that I haven't really touched.

[jjn1056]

@bbrtj there's also something wrong with --reuseport, that's also deoptimizing I think. I wasn't testing it much since on my MacBook the kernel really doesn't support it, but my implementation is busted.

I'll play with this some more but there's definitely something wrong in PAGI::Server.

[bbrtj]

I added that reuseport mostly to test if it helps, I don't think I actually need it. But if being ~100% faster than mojo is not its peak then it's in a great state. Very good job :)

Do you think I should use nginx at all in this benchmark? It's not really necessary, I was copying from Kelp benchmark. Mojo does not use anything and exposes Hypnotoad directly. I don't really want to hyper-optimize it, I want the benchmark to be realistic. I guess my question is: is using pagi-server in production on a busy website without any frontend with reverse proxy a realistic scenario for the future?

[jjn1056]

@bbrtj FWIW I think its useful to test these systems without an edge server to get an idea of their absolute limits, but since most production systems will have a setup with something like nginx in front I think having that as well is very useful.

There's a bug in --reuseport that seems to mess up worker mode on some systems, leading to performance reductions, I'd be curious the impact for your system.

[bbrtj]

Well hard to say if --reuseport affects performance in that docker. It appears I'm getting slightly lower numbers, but it may be due to normal runtime deviations.

[jjn1056]

I think it's on Macs and BSD like systems that the current reuse port is broken. glad it doesn't immediately seem so on linux.

Oh, I don't think I answered the question "is using pagi-server in production on a busy website without any frontend..." I think the answer is I'm aiming for that to be true, but it's not true yet since https support isn't working well. When that is done and http2 works I'd say yes.

[jjn1056]

@bbrtj

hey just FYI there's branch #36 with experimental socket support. There's also a bunch of minor fixes there. I still havent' had time to investigate your performance issues with running behind nginx tho. if you get time to test the branch associated with the PR let me know what you think. I never deploy this way so I'm honestly guessing, mostly by just copying what Starman and others do