XLibre
Incorporating drivers into Xserver tree #207
Replies: 8 comments 30 replies
-
|
My question was: How loose or tight is the coupling between the Xserver and the drivers? And what would be the benefits of keeping the drivers separate? Historical fun fact: According to DDX - freedesktop DRI Wiki they thought about the same, quote:
|
Beta Was this translation helpful? Give feedback.
-
Beta Was this translation helpful? Give feedback.
-
|
The discussion is happing more over there then here. |
Beta Was this translation helpful? Give feedback.
This comment was marked as off-topic.
This comment was marked as off-topic.
This comment was marked as off-topic.
This comment was marked as off-topic.
-
|
The DDX drivers are not meant to be useful outside an X server. It makes sense to keep them in a same tree. For source-based distros, we can provide flags like There is an extra problem to deal, however. |
Beta Was this translation helpful? Give feedback.
-
|
How tightly coupled will be the DDXes and the DIX ion this new configuration? |
Beta Was this translation helpful? Give feedback.
-
The incorporated one can use more functions/structures that aren't part of the official SDK / public ABI. If we - hypothetically - had all drivers in-tree (and also the ugly Nvidia proprietary would be out of the game), we wouldn't have to care about ABI compatibility at all anymore. The whole reason behind is pretty much the same as why the Linux kernel never had any public driver/module ABI in the first place. |
Beta Was this translation helpful? Give feedback.
-
Exactly, they must be recompiled (at least anything that's directly connected to any kernel structure / function), because they can change anytime, even within minor releases. The kernel team explicitly doesn't have to care about any ABI stability (the only thing that's stable is the UAPI - userland interfaces, eg. syscalls, /proc entries, etc). That is the reason why it could be some massively improved (and still is) so quickly all the time, and why it's so highly efficient (in contrast to proprietary OSes).
Because drivers are deeply interconnected with DIX structures. Some are even highjacking core protocol handlers, eg: https://github.com/X11Libre/xf86-video-amdgpu/blob/master/src/amdgpu_kms.c#L118 |
Beta Was this translation helpful? Give feedback.
-
|
Incorporating the drivers in to the tree for XLibre wouldn't be too different from incorporating them in to the kernel. There are many advantages. The largest of which is no need for ABI matching. Another would be debugging and error detection. They already need to be recompiled every time xserver has an ABI change any way. |
Beta Was this translation helpful? Give feedback.
-
I would recommend using a toolchain on your main PC instead if you are building for a slower system. Building from inside the target is always slower. |
Beta Was this translation helpful? Give feedback.
-
|
I vote to return to the monolithic source tree model used by XFree86. The tradeoff being slightly increased disk usage is negligible by today’s standards, especially when weighed against the performance and latency benefits. Getting more performance gains over wayland is a must to sell using us as a default. Building drivers in-tree rather than as external modules aligns with Wayland’s convention of embedding components at compile time. This approach eliminates dynamic module lookups and linkage jumps, which can reduce runtime overhead and improve input/output latency. It also simplifies packaging and installation. The individual drivers can be enabled or disabled directly via Meson build options without the need to manage separate module files. Keeping drivers as external modules adds unnecessary complexity. Folding them into the xserver tree would streamline scope management and make the build system more coherent. |
Beta Was this translation helpful? Give feedback.
-
|
General question: Was your text generated by "AI"? It seems a bit off.
What do you mean? The amount of code stays the same, be it in-tree or separate.
The idea is about moving source code, not runtime behaviour. The performance stays the same since the drivers will be compiled and executed as loadable modules either way.
Why bother with Wayland conventions? This is XLibre.
To my understanding it does not. Again: We are moving source code, not altering runtime behaviour.
Sorry, read the comments of the team members responsible for packaging XLibre at least on FreeBSD and Gentoo. They all disagree with your statement.
More options and more complexity since to my understanding the top level
Could you please provide at least two to three arguments regarding this statement?
It mixes up separate scopes. For example it would pull in a dependency on the
What do you mean? To give you an example use case: I want to compile a driver only. Then I have to unpack the whole tarball and find the right Meson option out of many and the dependencies of the driver for just that? |
Beta Was this translation helpful? Give feedback.
-
No i have grammarly extension installed i wont use it here if its that offensive. had bad grammar was speaking about how the xserver binary will likely ballon at least a couple mega bytes and its not a huge issue for most systems. it might kill being able to fit on a outdated 300mb hard drive system but you can get SCSI/IDE to sata adapters to overcome this problem.
This is just false also... when you dont have external modules it reduces the amount of wasted time on unneeded jumps in and out of multiple codebases each milisecond. this can sometimes seriously reduce the frametimes/judder in badly coded AAA games when i was using josephs amdgpu fork that has it built in. at the bare minimum the input and common video drivers, amd/intel/nvidia, MUST be monolitic. Furthermore, the number of jumps and wasted cycles from external modules is substantial use perf record and perf report to trace latency hotspots and ftrace or bpftrace to pinpoint costly indirections. there are man pages to help with this.
Wayland isnt all bad ideas some concepts like tighter integration and reduced context switching are genuinely useful. But its refusal to limit external modules is a real flaw. outside of the video drivers, Wayland’s ecosystem is riddled with fragmentation and unnecessary indirection that hurts performance and maintainability. XLibre can learn from the good without inheriting the mess. We can even hack in HDR support faster by reusing same functions it uses for libdrm.
That’s just not true... Moving code in-tree directly alters runtime behavior by eliminating dynamic lookups, reducing indirection, and streamlining symbol resolution. arent you tired of having to fix undefined symbol errors for nvidia and amd driver when metux removes old functions? if we had it all on the same tree it would be far easier to fix these routine errors that kill the gpu module from being able to load at all.
That argument doesn’t hold. having a single integrated package streamlines installation so you dont need 6 different things updated and installed just to use xlibre, and it avoids ABI mismatches between kernel and modules. Fragmented packaging only adds friction and risk. Unfortunately package maintainers will need to learn meson config its just a better way to compile the project they need to suck it up and adapt. the performance losses are too significant to ignore keeping everything separated is bad for gaming performance in general. Wayland feels like lag because they abuse external modules too much we cannot follow same path.
same answer as below. gonna copy and paste here. Thats a solved problem Linux kernel build systems already support fine-grained options like make menuconfig or make nconfig to compile only selected drivers, minimizing bloat without needing to unpack or navigate the entire source tree. we could easily add the same support to meson with a little time and patience.
1 Makes GDB debugging harder due to symbol resolution and external modules completely inhibit your ability to stay in GDB without it randomly stopping whenever it tries to load said external modules forcing you to use KGDB over serial TTY on another pc like steamdeck which is SUPER FUCKING ANNOYING. sorry ive got PTSD from xrdp breaking gdb so much.
Certain external modules might have to be banished to being slower external module for reasons such as these. the QXL driver is unimportant leaving that out could be really smart to keep performance up. I think to start we should bring libinput, wacom, amdgpu, intel, and nvidia inside the xlibre tree but ditch any modules that need a ton of wierd extra libs. those can stay external and people using those unorthodox methods can accept dealing with ABI mismatch and worse gaming performance. when you use wierd shit doing wierd setups is par for the course. God i cant wait for wacom to be built in the input lag in OSU will plummet!
Thats a solved problem Linux kernel build systems already support fine-grained options like make menuconfig or make nconfig to compile only selected drivers, minimizing bloat without needing to unpack or navigate the entire source tree. we could easily add the same support to meson with a little time and patience. |
Beta Was this translation helpful? Give feedback.
-
Putting amdgpu in tree didn't make anything faster. It's still an .so file being linked by xserver at the end of the day. What made it faster was me and metux finding things that were broken and fixing them during the process of moving it in to the xserver source tree. That's the whole reason why this is a good thing. Once it's in tree, we can also make an ACTUAL ABI instead of just a silly pretender that breaks all drivers every time there is a version code change. |
Beta Was this translation helpful? Give feedback.
-
Sorry if this wasn't entirely clear: By the way, we can think about moving some of the extra support modules (eg. exa, int10, etc) directly into the Xserver binary.
Wait, you've observed actual performance increase by using the in-tree driver ? That's surprising.
Monolithic isn't easy, because lots of things depending on the correct load order.
With my packager / distro maintainer background, I can tell that packaging from dozens of separate repos is much more complex than just splitting one source package's build product into multiple binary packages (eg. on deb-based distros the latter one is pretty trivial). The main problem here is taking care of binary compatibility - one at least has to add the right version-range dependencies (ie. driver package version X depends on Xserver binary range A1...A2). Doing the switch indeed needs some extra efforts on packager site, but only once for all.
Well, if you're building the Xserver (w/ qxl enabled), you'll have a build time dependency (for the source package) on spice libs. But the Xserver main binary package won't have a runtime dependency on it (the the qxl driver binary package will have it instead). Keep in mind: source and binary don't necessarily map 1:1.
In fact, it's just one more meson build option. Pretty simple. OTOH, I'd question why one would really want to recompile only one specific driver, without anything else. What's the practical use case (for non-developers) ? |
Beta Was this translation helpful? Give feedback.
-
|
After thinking about the change for some time I would like to add my two cents here: Advantages... as expressed so far in various places.
Neutral
Disadvantages
Open QuestionsBesides the things above:
Social ContextI think the people came here too because they don't want to be forced upon something. Pushing this change without good reasoning and consent could make them leave the project at worst. So we need a clear picture and weigh in pros and cons. For the decision making process I propose Sociocracy - basic concepts and principles - Sociocracy For All, especially Consent Decision Making - Sociocracy For All. In short: address concerns sincerely and seriously so there are no loosers in this quite fundamental question and we have a consent in the core team. We have packagers on board who can assess what the integration of the drivers means for packaging. We should listen to them as well to factor in this aspect since it will also affect the *NIX distributions "out there". Closing CommentsI think we should only ever dare to touch anything if there is someone able to test the things we touched. Like if we say touch the Geode driver there has to be someone testing the correct functionality afterwards. Otherwise we should not touch or move it in my opinion. This project is about software from decades. More than 345.000 SLOCs of the Xserver alone representing wisdom. Or at least a functionality. I think no one is safe to know about all the compromises and ways gone and ways avoided to make this stuff work. I am not against touching and modernizing it, but we should be able to say afterwards: it (still) works. We should keep in mind that one of the unique selling propositions of this project is to keep the oldie but goldie things working. And this has to be ensured. Others - I won‘t mention Wayland here - decided to just kick out all the past working stuff. We have stated that we will act differently, and this is also expected of us. |
Beta Was this translation helpful? Give feedback.
-
|
I have the impression that there is another contradiction:
versus:
External / 3rdparty drivers require a quite, at least per major version stable ABI. Or how would one be able to create such an external driver? |
Beta Was this translation helpful? Give feedback.
-
|
Next possible contradiction:
versus:
Again: External / 3rdparty drivers require a quite, at least per major version stable ABI. Or how would one be able to create such an external driver? |
Beta Was this translation helpful? Give feedback.
-
|
Note that we don't have any stable (and properly documented) ABI at all. It's just a huge bunch of symbols that had been _X_EXPORT'ed ad hoc. And there isn't any clear layering at all - everything twisted and mixed with anything. And driver developers often did weird things (eg. hijacking core request handlers, messing directly with window properties, etc, etc). With the whole process (including the previous Nvidia stabilization work), we're going through all the symbols one by one and carefully document those that really need to be exported. If some external driver team comes around and as a good case another exported API/ABI function, then we can take care of that. But all of that needs to be well thought, no ad hoc hacks. |
Beta Was this translation helpful? Give feedback.
-
preaching to the choir its just how do we get tango on the same page as us its really frustrating to keep seeing him argue with you. |
Beta Was this translation helpful? Give feedback.
-
|
It's not really as contradiction at all. The ones we maintain:
Versus the ones we don't maintain:
This would be great if we actually had a stable ABI:
There is no interface, no standard interaction between these drivers and xserver. Almost every one is different and exports different functions etc willy nilly. What you're missing is that this is what's KEEPING US FROM CREATING a stable, modular structure instead of BEING a stable, modular structure. |
Beta Was this translation helpful? Give feedback.
-
|
This is my elaborated answer to @b-aaz at X11Libre/xserver#449 (comment)
I do not believe this is an unsolvable problem. The GNU Changelog standards solve this by prefixing each entry of changelog by the "modules" it affects. This creates a logical separation that can be grepped later. Further, in a certain sense the project is already following it: https://github.com/X11Libre/xserver/commits/master/
How much, precisely? I believe it will not be so big at all. |
Beta Was this translation helpful? Give feedback.
-
A. 60 repositories give or take a few. |
Beta Was this translation helpful? Give feedback.
-
Oh and if one would want to just build a single driver to package it. It will include 60+ repos of code and an xserver on top just for building that single driver. |
Beta Was this translation helpful? Give feedback.
-
Doing a quick and dirty calculation:
Sum 23165. Multiplying by 60/4, we have an estimate of 347475.
Pushing the drivers inside the server adds approx. 5.5 times, let's take 6 (I don't know how much will be expanded by merging the trees).
They aren't the only monorepos out there. Further, how precisely this is relevant? Why a monorepo is not suitable for X but it is suitable to Linux? |
Beta Was this translation helpful? Give feedback.
-
|
@AndersonTorres we don't intend to import the full history of all the drivers into the Xserver repo. Just copy over the code from a defined commit (eg. last release tag). |
Beta Was this translation helpful? Give feedback.
-
|
This is a bit unusual from the point of code preservation. A buggy commit that lurks on the old code base can't be reached from Nonetheless, this does not change the point: the amount of extra code added is not so big. |
Beta Was this translation helpful? Give feedback.
-
|
Regarding this statement:
I don't buy this. One simply cannot complain about "people did so many weird hacks in their drivers instead of just extending / refactoring the API/ABI" and say "it's just easier to have them inside the Xserver tree, sharing the same lifecycle" in the same sentence. My reasoning: I might not be a C and X11 hacker, but I spent a few years (more!) with doing exactly the opposite of what this idea proposes. I have split up a lot of legacy monoliths that were no longer maintainable due to their lack of separation of concerns and exorbitant complexity and divided them into several well-tailored services. One of the main problems with monoliths was always that unrestricted access to internal methods of “remote” subsystems created such a high degree of dependency within the overall system that no one could make a simple change without causing errors elsewhere (see big ball of mud). One of the first steps was always to identify the responsibilities of the rudimentary subsystems and design an interface (API/ABI) in order to be able to separate parts of the agglomerate code and, for example, make it easy to develop and test again. The next part was to implement the API and connect the newly created service with the old monolith. Once that worked the monolith implementation of the API was removed. As a bonus we gained not only easy testability and per-service scalability but also team-scalability and a clear separation of concerns between the subsystems and the responsible teams. There's more, but I'll spare you the details for now. So given my experience I would go after the, quote "just extending / refactoring the API/ABI" instead of smashing everything together and maybe that way create a big ball of mud. That's exactly what I was referring to in:
|
Beta Was this translation helpful? Give feedback.
-
Yes, that's exactly what I want to do. But that's naturally ABI breaking. It's already really complicated to keep backwards compat with NVidia (even though they're just using a subset). Having to care for all the other (~60) drivers is just a horrible amount of extra work to do. Yes, we can recompile them, but we need to make sure that actually happens in the field. We'd end up with major/abi-breaking release every few weeks/month. And every time the downstreams/distros have extra work to do, in order to make sure they have the matching binaries. In constrast, when we're moving drivers in-tree, this only happens once. It's pretty much the same situation like with the kernel. Nobody's seriously demanding stable in-kernel ABI from them. The Xserver IS a monolith, that just can load extra object files into it's process. |
Beta Was this translation helpful? Give feedback.
-
i get that your background is in building things up, not tearing them down... but it feels like your pushback is rooted more in your personal philosophy than in familiarity with our current codebase. youre not actively coding here and thats fine, but your project lead had the correct path forward with this. We really should not block progress based on what package managers say they arent coders its not their choice... Xlibre is a fresh start and it was legacy convention to make the project monolithic in XFree86 for increased performance. Going back to what worked best is the right way we dont wanna end up like wayland with 20+ external modules making shitloads of input lag do we? Xorg was wrong to even make them external like wayland it was stupid. |
Beta Was this translation helpful? Give feedback.
-
You may be failing to realize that moving the code in to the same source tree doesn't mean the actual .so files will stop being made. In fact, they are made just the same as before in the exact same directories. You can also elect not to build them on a per module basis using compile arguments. The ONLY place where this makes a difference other than to developers (where it's a good thing all the way around) is to packagers and all they have to do is change their build arguments. Driver packages can still be separate just the same as they have always been if the packagers so choose. What it REALLY does for developers is move all the sources in to the same project structure so that our coding tools can find all the places where legacy code is being used so we can strip it out more efficiently without breaking the drivers. It also allows us to put in to place the replacement functions, if any are necessary, for those legacy drivers. |
Beta Was this translation helpful? Give feedback.
Uh oh!
There was an error while loading. Please reload this page.
-
Hi folks,
for quite some time I'm thinking about moving currently external drivers into the Xserver tree, just like we already have some drivers in-tree.
The advantage is not having to care about different Xserver versions within driver code and vice versa. In the longer run, it also makes it easier dropping things from official SDK (that needs to be kept stable) and so making refactoring easier. (of course, only those that aren't used by ugly proprietary Nvidia driver).
Historical note: back in xfree86 times, all drivers been part of the monolithic source tree.
What are your thoughts ?
Beta Was this translation helpful? Give feedback.
All reactions