From 2ba8da29e7a031b8496d07270a727dcf52008cc1 Mon Sep 17 00:00:00 2001 From: James Graham Date: Tue, 13 Sep 2022 14:41:47 -0700 Subject: [PATCH 1/2] RFC 122: Remove browser specific failures graph --- rfcs/remove_bsf.md | 69 ++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 69 insertions(+) create mode 100644 rfcs/remove_bsf.md diff --git a/rfcs/remove_bsf.md b/rfcs/remove_bsf.md new file mode 100644 index 00000000..089db088 --- /dev/null +++ b/rfcs/remove_bsf.md @@ -0,0 +1,69 @@ +# RFC : Remove browser specific failures from wpt.fyi + +## Summary + +wpt.fyi currently shows a chart of "browser specific failures"; a +score of tests that are failing only in a single browser. This RFC +proposes entirely removing that graph from wpt.fyi. + +## Details + +The original motivation for browser-specific-failures was to provide +browser vendors with insights into tests that might be causing interop +problems and therefore might be especially valuable to spend +engineering effort fixing. On the basis of this hypothesis, a graph +was added to wpt.fyi showing a browser-specific-failures "score" for +each browser engine, so that vendors could track their progress on +fixing these issues. + +However at this point we have identified a number of issues with +"browser specific failures" as a metric, for example: + +* It's hard to correlate a change in the score to a change in browser + behaviour. + +* The way the score is computed biases the score toward missing + features rather than interop failures in already shipping features + (since a missing feature usually causes a large number of failures). + +* The metric doesn't provide any way of controlling for the user + impact of failures; browsers can get a "bad" score from a large + number of failures that in practice don't cause any observed + problems for authors. + +For these reasons and others, we haven't reached a critical mass of +adoption for browser specific failures as a metric to improve interop +on the web platform, and its original function has been largely +replaced by the Interop-20xx project. + +Although browser specific failures isn't providing the initially hoped +for value, having it on the wpt.fyi does create some work as it +encourages people to try to understand the current scores or changes +in the score. + +Removing the graph entirely seems like the simplest way to indicate +that we no longer consider this useful as a metric. + +## Alternatives + +* Keep the graph but move it to a less prominent page. + + Although this would make the graph less obvious, it would still + imply some endorsement of browser specific failures as useful at the + level of a metric. Since Interop scores are a metric that browser + vendors have explicitly committed to, and which solve many of the + problems with browser specific failures, it's better to clearly + commit to one public metric. + +## Risks + +* Browser engineers might be using browser specific failures as a way + to identify good tests to fix. + + This doesn't depend on having a metric / graph of browser-specific + failures, and could be better solved by making it easier to see an + actual list of browser specific failures in a given feature. This is + already possible on wpt.fyi, but is quite complex to write as a + query. Other frontends on wth wpt.fyi data like + https://jgraham.github.io/wptdash/ provide a engineer-focused view + of this data. From df4e330f8d2b174ae5a8d27a88af395b32c67d4d Mon Sep 17 00:00:00 2001 From: jgraham Date: Tue, 13 Sep 2022 22:45:17 +0100 Subject: [PATCH 2/2] fixup summary --- rfcs/remove_bsf.md | 9 +++++---- 1 file changed, 5 insertions(+), 4 deletions(-) diff --git a/rfcs/remove_bsf.md b/rfcs/remove_bsf.md index 089db088..29132ab2 100644 --- a/rfcs/remove_bsf.md +++ b/rfcs/remove_bsf.md @@ -1,10 +1,11 @@ -# RFC : Remove browser specific failures from wpt.fyi +# RFC 122: Remove browser specific failures from wpt.fyi ## Summary -wpt.fyi currently shows a chart of "browser specific failures"; a -score of tests that are failing only in a single browser. This RFC -proposes entirely removing that graph from wpt.fyi. +[wpt.fyi](https://wpt.fyi) currently shows a chart of "browser +specific failures"; a score of tests that are failing only in +a single browser. This RFC proposes entirely removing that graph +from wpt.fyi. ## Details