Create a XLibre Test Drivers program

[callmetango]

Describe the idea

We strive to ensure the correct functionality of XLibre and want to test the software on the many different devices for which drivers exist. But none of our developers have all the hardware for all the drivers we maintain.

On the other hand, there are many contributors out there who have one or maybe two devices that we want to test XLibre on. And they themselves are interested in making XLibre work on their devices as well. We should turn this fact into an advantage for all of us and distribute the device-specific tests to the interested testers.

Therefore we should create a wiki page “XLibre Test Drivers” for the inclusion and listing of driver testers with at least:

• Device manufacturer and name/model
• Contact information like Github handle or e-mail address
• Notes

We should also integrate the following steps into our release process:

• Contact the test drivers before a release of XLibre
• Wait for feedback from the test drivers
• Fix bugs if need be
• Approve the release

It should be done because

See above and to clearly communicate what hardware XLibre will work on. And to not only say this but know it for pretty sure.

What are the alternatives?

Do without testing and make the users unhappy.

Additional context

This idea was sparked by NVIDIA debugging Driver 57x · Issue #158 · X11Libre/xserver, now more general Tracker: Testing Nvidia driver with XLibre on Linux and FreeBSD · Issue #320 · X11Libre/xserver where @hendrack started testing the proprietary Nvidia driver on his own and reported back the results.

The test driver program could integrate well into other plans like creating pre-defined test cases and test runs. Two issue types that will be described in another idea.

Extra fields

• I have checked the existing New Ideas, RFCs Of The Core Team, Ideas Soon To Be Addressed and Someday Ideas
• I have read the Contributing Guidelines
• I'd like to work on this idea

[hendrack]

Sounds good. In addition to hardware there should also be a list of software that the individual testers can check. As seen in X11Libre/xserver#310 (comment), screen recording was broken, for example. I use Nvidia CUDA a lot (pytorch, ollama - which thankfully work right now), these kinds of special use cases in addition to everyday desktop usage.

A list of distributions available to testers would also help.

[callmetango]

Thank you!

there should also be a list of software that the individual testers can check

Yes, that's exactly what I was thinking of. Have some default use cases defined and then run by the test drivers to, well, test the drivers. 😉

A list of distributions available to testers would also help.

Do you mean a selection from this Are We XLibre Yet? · X11Libre/xserver Wiki list of already supported distributions?

[hendrack]

Do you mean a selection from this Are We XLibre Yet? · X11Libre/xserver Wiki list of already supported distributions?

I mean what distros the individual tester has available for testing.

Like ...

Tester	Hardware	Distro	special use case
hendrack	Intel 620, Nvidia (Kepler, Geforce 1080 ti)	Gentoo, Artix, FreeBSD	Nvidia CUDA

[callmetango]

Ah, now I understand. Thank you for providing the draft! It looks really good.

[metux]

I've been thinking about listing the testers right at the supported GPUs page (per model)
https://github.com/X11Libre/xserver/wiki/Graphics-devices-&-drivers

[callmetango]

We could do so, but we would miss the distribution and special use case information which is closer related to the testers than the graphics devices and drivers information. On the Graphics devices & drivers · X11Libre/xserver Wiki page there is more information on for example the kernel drivers and some driver specific remarks which are not that much related to the testers.

Generally speaking, both pages are closely linked, but still have different focuses. The XLibre Test Driver program shall aim to engage the community and inspire active participation, including external promotion of this page. It will also serve as a contact list for the soon-to-be-developed test cases. On the contrary, the Graphics Devices and Drivers page has a more static-documentary character.

[callmetango]

@josephcrowell @mikedld @steven-vd CC

@metux @probonopd What do you think?

@artist4xlibre @b-aaz @stefan11111 I think this program could be communicated to anyone showing up in the discussions or forum threads about using XLibre on distribution X, Y, Z and being interested to test XLibre on his distribution/platform/OS/...

[josephcrowell]

I think this is a great idea. Anything that can move us towards more effectively finding and fixing bugs/regressions is a good thing.

[probonopd]

Have you thought about test automation (using systems with real hardware)?

[callmetango]

Yes, for sure. This is another step toward quality assurance and perfectly complements the Test Drivers program. There are already plans to run automated tests on the Test Drivers' machines. But first things first.

[callmetango]

This idea will be worked on in Create a XLibre Test Drivers program · Issue #302 · X11Libre/misc.

[callmetango]

Resolved by resolving Create a XLibre Test Drivers program · Issue #302 · X11Libre/misc.