HAProxy
Replies: 12 comments 32 replies
-
|
I had a bit of a question on it. Do you think the recent addition of Linux's sched-ext and its bpf userland part, the scx project, could potentially help HAProxy performance to some extent, even though the latter does its own Numa nodes, LLC, cpu topology ? |
Beta Was this translation helpful? Give feedback.
1 reply
-
|
Hi David! At the moment I don't think so. The difficulty is not related to scheduling but really to avoid binding too large a cpu-set so that we limit inter-LLC communications to the strict minimum needed. Our thread groups were initially designed as an elegant way to work around the 64 threads limit, but in the end it reveals as a much more powerful tool that allows to group low-latency threads together and isolate them from the resources touched by other groups, effectively limiting the contention. And the difficulty clearly is to do that setup properly. |
Beta Was this translation helpful? Give feedback.
-
|
@awlx @pierrecdn @cpaillet @dclaisse @jvgutierrez @philos @exander77 @jvinolas @JB0925 : this can also be of interest to you based on some of your previous reports indicating you're dealing with large setups. |
Beta Was this translation helpful? Give feedback.
0 replies
-
|
We use it in docker. Do you know if you will build it for docker https://hub.docker.com/_/haproxy/tags?name=3.2-dev8 ? |
Beta Was this translation helpful? Give feedback.
1 reply
-
|
I'm seeing that previous versions were built, so I guess this one will eventually appear as well. I don't know the process there to be honnest, but I'm seeing a user doijanky who seems to be producing them on average a week after they're emitted, so that could appear soon. Thanks for testing BTW! |
Beta Was this translation helpful? Give feedback.
-
|
Hi everybody! @rnsanchez and I tested 3.2-dev8 on our production machines. Just FYI, they are 2 x Intel(R) Xeon(R) Gold 6430. Using our original conf (no Using On this machine specifically, we also tried with: This got different results: Now, with this One thing to notice: We're forcing Please let us know if need us to run any more tests. Happy to help! :-) |
Beta Was this translation helpful? Give feedback.
3 replies
-
|
Hi Felipe! Thank you very much for your tests! As you can see the debugging output is not the most usable yet. If you simply comment out "nbthread" (and "thread-groups" if you have one), it should make a difference. |
Beta Was this translation helpful? Give feedback.
-
|
Hi Willy, Removing all except If we stick to Also, if we happen to deploy on a P/E-core server, adding a |
Beta Was this translation helpful? Give feedback.
-
|
Hi Ricardo!
It's always better not to set anything so that you'll always get the default behavior along version upgades.
Ah great, I guessed some would do that :-) Then you can simply use
Yes that's also the point. If you use
Yes that's exactly the point. In fact I'm not yet aware of P/E-core servers, but since it has long started on high-end desktops, it should eventually happen. But Since your machine has uniform L3 caches and a single group per socket is needed, and you want to use a single socket, you indeed just don't need |
Beta Was this translation helpful? Give feedback.
-
|
Hi wtarreau! This work looks promising in general, but I do have a question. How is the load-balancing handled internally when using the new setup? Right now our go-to solution So the question is, am I correct in thinking that using thread-groups is effectively the same as running Having said that, I'd be willing to test this relatively soon on a massive AMD 9684X cpu. |
Beta Was this translation helpful? Give feedback.
1 reply
-
|
You're right. Thread groups consist in sharing less. For example, queues, scheduling, file descriptors, idle connections, tons of stuff that it under extreme contention are shared only between threads of the same group. This sensibly limits the inter-group communication. There are still some, of course, but much less. This allows for example to manage a single instance to add/remove servers, update maps, add/remove certificates, have a single instance of health checks etc. At the moment I'd say that thread groups provide the best of both worlds: little sharing providing almost process-like performance, with a single management. In any case if you're using an EPYC, you must absolutely use thread groups (or at least make sure that no process crosses a CCX boundary) otherwise the costs can be huge. |
Beta Was this translation helpful? Give feedback.
-
|
Hello @wtarreau 👋, From our perspective, this works well. Here it is: So, from that point of view, it's a win, less toil to configure this, and probably no headache when a new hardware is coming. But now I have a question: is it possible to make use of the computed
|
Beta Was this translation helpful? Give feedback.
4 replies
-
|
Hi Pierre! Thanks for the test and confirming the easier setup, since that was the goal! Hmmm good question regarding capa. The capacity really serves to compare P and E cores in fact. The problem here in fact is that you're keeping 31 threads, so there's no good way to split that evenly into any number. How do you configure your cpu-maps normally ? Maybe you're making fewer but larger groups ? If so maybe we could imagine having something such as "group-by-2clusters" which would pack more threads on two clusters. It could smooth the difference a little bit, you'd have 38 cores and 17 cores. Does this correspond more or less to what you're looking for ? |
Beta Was this translation helpful? Give feedback.
-
Clearly we do this, bundle of 8 cores instead, 7 for the last one. But that's simply because we do not want to waste these 3 cores.
To be honest I'm not sure this is something to plan for others. Most probably people can afford to automate and build their own maps in such cases. I was simply curious if there was some ability to "load-balance according to |
Beta Was this translation helpful? Give feedback.
-
|
In fact the primary purpose is to respect the topology as much as possible. Every cross-CCX communication hurts performance a lot, so for example if you had 31 cores and 62 threads, we could imagine 7 groups of 9 threads and this would not be great due to the poor split of threads over CCX (some would be 8 of one CCX against one of another one which would lose most of the races, others would be 4 against 5 etc). Regarding the approach of doing it "by two", it's not just for you. We've had a 74F3 here, it's 24 cores over 8 CCX, hence 3 cores per CCX. And many times I've configured some workloads to use 6 cores of two adjacent CCX. That's of course not great, but for some workloads it was better than either only 3 per group or having a group cover everything. Thus that's something I had already envisioned. Since now we're two, I think it's not an exception but probably something pragmatic. Let's add it to the todo list! |
Beta Was this translation helpful? Give feedback.
-
|
Hi Pierre! I've just pushed an update. Now there are 3 new policies: "group-by-2-clusters", "group-by-3-clusters" and "group-by-4-clusters". I think it should be enough to offer a wide choice. I've also improved the output dump so that it now includes the mapping of threads to CPUs. For example, if I reserve a CPU to the system on the 64-core machine here, I'm getting this: with with I think it will do the job for what you're looking for. It also helped me notice some inconsistencies in |
Beta Was this translation helpful? Give feedback.
-
|
Just happened the bug in one of our setups and restarting the docker failed again. I've used latest |
Beta Was this translation helpful? Give feedback.
5 replies
-
|
Thanks for your feedback. We've found that there are still subtle cases where the warning/show thread/panic code can still race especially if re-entering some libc function. You might possibly have faced that one. We've worked today on designing a durable solution against this and hopefully we'll write the fix tomorrow. I'm not seeing any backtrace in your output, could you please check in haproxy -vv if USE_BACKTRACE=1 appears and/or what libc/platform you're using ? |
Beta Was this translation helpful? Give feedback.
-
|
HAProxy version 3.1.1-717960d 2024/12/11 - https://haproxy.org/ Feature list : -51DEGREES +ACCEPT4 -BACKTRACE -CLOSEFROM +CPU_AFFINITY +CRYPT_H -DEVICEATLAS +DL -ENGINE +EPOLL -EVPORTS +GETADDRINFO -KQUEUE -LIBATOMIC +LIBCRYPT +LINUX_CAP +LINUX_SPLICE +LINUX_TPROXY +LUA +MATH -MEMORY_PROFILING +NETFILTER +NS -OBSOLETE_LINKER +OPENSSL -OPENSSL_AWSLC -OPENSSL_WOLFSSL -OT -PCRE +PCRE2 +PCRE2_JIT -PCRE_JIT +POLL +PRCTL -PROCCTL +PROMEX -PTHREAD_EMULATION -QUIC -QUIC_OPENSSL_COMPAT +RT +SHM_OPEN +SLZ +SSL -STATIC_PCRE -STATIC_PCRE2 +TFO +THREAD +THREAD_DUMP +TPROXY -WURFL -ZLIB Default settings : Built with multi-threading support (MAX_TGROUPS=16, MAX_THREADS=256, default=64). Available polling systems : Available multiplexer protocols : Available services : prometheus-exporter |
Beta Was this translation helpful? Give feedback.
-
|
Thanks, it confirms what I seemed to remember, i.e. you're on libmusl + x86 and I mentioned that I'd like to enable that one as a working platform for backtrace even without libc support. I'll try to address that for 3.2. BTW, since you're on 3.1.1, there are definitely some races regarding warnings. But all are not yet fully addressed in 3.1.6 anyway. You may prefer to consider that this one is not expected to be frequent for the time being and update at once to 3.1.7 or so which will include all of them, or you may still update to 3.1.6 to benefit from more fixes and have something more stable in any case. If I manage to finish my fixes in time, I'll try to enable backtraces on your platform and see if we can backport it. |
Beta Was this translation helpful? Give feedback.
-
|
Do you mean if we use a libc docker image instead of the alpine we won't have this bugs? |
Beta Was this translation helpful? Give feedback.
-
|
No no it will be the same, the only difference is that you'll have the backtrace at the end that helps locate the problem. |
Beta Was this translation helpful? Give feedback.
-
|
In case it helps: With "cpu-policy group-by-cluster": Without "cpu-policy group-by-cluster": |
Beta Was this translation helpful? Give feedback.
1 reply
-
|
Something is not right tho. Running 3.2-dev11 and just having added: A few minutes later I got an alert from all our LBs running -dev that the stats sockets are / seem unavaiable. Removing the option and doing a reload wasn't enough. Had to restart HAProxy. So it's related to the cpu-policy it seems. |
Beta Was this translation helpful? Give feedback.
-
|
Thanks Christian ,for this feedback. However I want find any reason for the stats socket to stop working, especially in relation with thread groups (since the policy basically does only that, it configures thread groups and cpu maps for you). During our tests we constantly have commands looping over the CLI using socat (like the ones running show threads etc). Do you know if you're having any monitoring scripts connecting and issuing commands ? This makes me think that maybe one command doesn't end well and leaves a connection pending, and this would be serious enough that I'd like to figure what this is :-/ The fact that it happened on multiple LBs is not a coincidence, there's definitely a pattern that triggers this problem. I'll try to issue various debugging commands like show sess etc just in case I spot anything. |
Beta Was this translation helpful? Give feedback.
10 replies
-
|
Thanks. For now I'm doing reloads in loops 10 times a second via the master CLI and "show sess all" on another independent connection, but no issue yet. I'll check if hanging on the connection triggers the problem. |
Beta Was this translation helpful? Give feedback.
-
That's me being in a socat session with "prompt". So not sure what's blocking there but that's not it. |
Beta Was this translation helpful? Give feedback.
-
|
Ok, it's not a reload. I just have one started without any reloads. It's a semi productive, low traffic site. It takes some minutes until it happens. |
Beta Was this translation helpful? Give feedback.
-
|
OK that's not good at all. I'll have to experiment but I have no idea what to look for. Does your monitoring script send any "show sess" or long dumps that may use more than one buffer ? Or does it add/remove servers maybe ? I'm just worried that it could be a recent regression and really want to nail it down! |
Beta Was this translation helpful? Give feedback.
-
|
BTW if you can attach gdb and issue "t a a bt" to show where each thread is when it starts looping at 100%, this might also give me some hints. |
Beta Was this translation helpful? Give feedback.
-
|
I just managed to block a CLI session by sending a command at the same moment the timeout expired. The stream is still there, with no timeout: I'll now try to reproduce this more consistently. |
Beta Was this translation helpful? Give feedback.
3 replies
-
|
That's unrelated, it's the timeout not working anymore since latest changes, that's a post-dev11 thing (in dev11 it's OK). |
Beta Was this translation helpful? Give feedback.
-
|
Christian, could you please open an issue for that one so that we can track it separately ? I sense it will not be an easy one :-/ |
Beta Was this translation helpful? Give feedback.
-
Of course! Here we go: #2951 |
Beta Was this translation helpful? Give feedback.
-
|
Update on the last point, we finally found the root cause thanks to Christian's help, it's indeed thread-groups + stats socket with a certain connection rate that causes incoming connections to be bounced to other listeners. The connection accounting was not transfered in this case, resulting in listeners being blocked after several iterations. Now fixed in mainline. Better pull latest mainline when testing thread groups then. |
Beta Was this translation helpful? Give feedback.
0 replies
-
|
We got another hung with proposed 3.1.6 version. It's docker alpine version running within a KVM virtual machine now: dmesg haproxy docker: |
Beta Was this translation helpful? Give feedback.
3 replies
-
|
With dev13 it seems even more unstable as it hung almost immediately: |
Beta Was this translation helpful? Give feedback.
-
Please try dev14 instead |
Beta Was this translation helpful? Give feedback.
-
|
I agree with @idl0r here. Just to be clear:
For 3.1, please use 3.1.7. For 3.2, please use 3.2-dev14. Sorry for the confusion if any. |
Beta Was this translation helpful? Give feedback.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Uh oh!
There was an error while loading. Please reload this page.
-
In 3.2-dev8 (to come soon), we've finally merged the so-called NUMA series that lasted for almost 2 years, which detects CPU topologies and arranges threads into groups based on nodes, CCX, clusters etc. By default nothing is changed (at least changes are not expected), but you can now adjust the default CPU binding policy in the
globalsection using thecpu-policydirective to one of these:group-by-cluster: will create at least one thread group per cluster, a cluster here having boundaries on CPU types, declared clusters, LLC, nodes and packages; if more than 64 threads are present in the same cluster, more groups will automatically be created.performance: same as above but only uses P-cores on mixed platformsefficiency: same but only uses E-coresresource: only uses the smallest detected cluster, probably just for containers and/or VMs.first-usable-node: that's the current default since 2.5 or so, it uses the first node only, and within the limit of 64 threads.These policies are ignored if "nbthread", "thread-groups" or "cpu-map" are defined, so in order to test them, just use:
And that will be all. You can check with
-dcon the command line what is detected (though that's a bit too developer-oriented at this point).I'd be interested in gathering feedback from those running on:
cpu-mapexcept that this is a PITA on some machines)For now I've tested on various x86 (64-core EPYC, 24- and 8-core Xeon, atoms, skylake, core i7 8th gen, 14th gen), ARM (v7, v8, v9, various models of each, including big.LITTLE, 16-core LX2), dual-core quad-thread MIPS.
The detection only works on Linux (everything is exposed under /sys). On FreeBSD and MacOS we only know the number of CPUs (and the node on FreeBSD) and that's still used to set the number of threads.
For the curious, there's also
cpu-setwhich allows to reset taskset, use only / ignore certain CPUs (by number, by core number, thread number, cluster number, node number etc). That's only used to pre-select the ones the cpu-policy will operate with (e.g. "I want to disable HT0 because I reserve it for network IRQs").The goal if this works well is to change the default policy for 3.3 from
first-usable-nodeto one to determine, probablyperformanceorgroup-by-clusteror any new one that's left to be designed based on the feedback.In particular, @idl0r @jaroslawr @felipewd @rnsanchez @majedrze you've often reported issues related to large setups, if you could run some tests before the final release, that would be great ;-) Others are absolutely welcome of course!
Beta Was this translation helpful? Give feedback.
All reactions