Significant performance regression in benchmarks using intersection

[jdegenstein]

I don't know the cause of this yet. Sometime between fa8e936 and 5f67a19 the benchmark times became significantly slower for certain benchmarks (ppp-0101 and ppp-0106). Performance is ~33x worse for the former and a whopping 153x for the latter. Also, ttt_24_SPO_06 is about 2x worse after regression.

I took a quick look and all three of these with performance regressions contain an extrude with Mode.INTERSECT

recent result before regression:

------------------------------------------------------------------------------------------ benchmark: 17 tests ------------------------------------------------------------------------------------------
Name (time in ms)                      Min                 Max                Mean             StdDev              Median                 IQR            Outliers       OPS            Rounds  Iterations
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
test_mesher_benchmark[100]          1.6003 (1.0)      185.4864 (1.50)       4.3370 (1.0)      20.0762 (57.65)      2.1848 (1.0)        0.6022 (2.32)          3;3  230.5727 (1.0)         225           1
test_mesher_benchmark[1000]         4.2935 (2.68)     197.4644 (1.60)       9.5580 (2.20)     29.1100 (83.59)      4.5587 (2.09)       0.4576 (1.77)          6;8  104.6249 (0.45)        213           1
test_mesher_benchmark[10000]       30.9104 (19.32)    193.0915 (1.57)      65.1643 (15.03)    64.4008 (184.93)    31.7215 (14.52)      1.6126 (6.22)          7;7   15.3458 (0.07)         32           1
test_ppp_0103                     118.6098 (74.12)    123.3574 (1.0)      120.5924 (27.81)     1.6811 (4.83)     120.1975 (55.01)      2.6855 (10.36)         3;0    8.2924 (0.04)          9           1
test_ppp_0108                     152.9653 (95.58)    155.5350 (1.26)     154.2379 (35.56)     0.8398 (2.41)     154.2055 (70.58)      1.0387 (4.01)          2;0    6.4835 (0.03)          7           1
test_ppp_0106                     176.1107 (110.05)   177.1536 (1.44)     176.7203 (40.75)     0.3482 (1.0)      176.7943 (80.92)      0.2592 (1.0)           2;1    5.6587 (0.02)          6           1
test_ppp_0109                     202.8154 (126.73)   205.6677 (1.67)     204.2451 (47.09)     1.2581 (3.61)     203.7177 (93.24)      2.1526 (8.31)          2;0    4.8961 (0.02)          5           1
test_ppp_0104                     229.2504 (143.25)   231.4830 (1.88)     230.5468 (53.16)     1.0323 (2.96)     230.9739 (105.72)     1.8661 (7.20)          1;0    4.3375 (0.02)          5           1
test_ttt_23_T_24                  276.3915 (172.71)   284.0388 (2.30)     281.7751 (64.97)     3.0860 (8.86)     282.7697 (129.42)     2.8237 (10.90)         1;1    3.5489 (0.02)          5           1
test_ppp_0101                     284.7568 (177.94)   292.5526 (2.37)     289.7015 (66.80)     3.3513 (9.62)     291.2990 (133.33)     5.2475 (20.25)         1;0    3.4518 (0.01)          5           1
test_ttt_24_SPO_06                295.2961 (184.52)   303.0642 (2.46)     298.5478 (68.84)     2.9233 (8.39)     298.7130 (136.72)     3.5012 (13.51)         2;0    3.3495 (0.01)          5           1
test_ppp_0105                     312.5772 (195.32)   314.2039 (2.55)     313.2393 (72.22)     0.6025 (1.73)     313.0754 (143.29)     0.6356 (2.45)          2;0    3.1924 (0.01)          5           1
test_ppp_0102                     451.9085 (282.39)   455.4244 (3.69)     453.9657 (104.67)    1.5761 (4.53)     454.3524 (207.96)     2.8361 (10.94)         1;0    2.2028 (0.01)          5           1
test_mesher_benchmark[100000]     506.5504 (316.53)   688.3405 (5.58)     608.0842 (140.21)   90.6006 (260.16)   662.3386 (303.15)   164.4895 (634.70)        2;0    1.6445 (0.01)          5           1
test_ppp_0110                     514.4487 (321.47)   519.8342 (4.21)     515.9034 (118.95)    2.2237 (6.39)     515.2394 (235.82)     1.7298 (6.67)          1;1    1.9383 (0.01)          5           1
test_ppp_0107                     843.2626 (526.93)   869.7897 (7.05)     849.9931 (195.99)   11.2543 (32.32)    844.3479 (386.46)     9.9900 (38.55)         1;1    1.1765 (0.01)          5           1
test_ttt_23_02_02                 941.3420 (588.22)   967.4651 (7.84)     949.9689 (219.04)   10.0885 (28.97)    947.1579 (433.51)     7.1253 (27.49)         1;1    1.0527 (0.00)          5           1
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

recent result AFTER regression:

------------------------------------------------------------------------------------------------ benchmark: 17 tests -------------------------------------------------------------------------------------------------
Name (time in ms)                         Min                    Max                   Mean              StdDev                 Median                 IQR            Outliers       OPS            Rounds  Iterations
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
test_mesher_benchmark[100]             1.5913 (1.0)         168.6671 (1.40)          4.5529 (1.0)       19.7783 (31.10)         2.1787 (1.0)        0.5866 (7.03)          4;4  219.6422 (1.0)         244           1
test_mesher_benchmark[1000]            4.7512 (2.99)        145.7524 (1.21)          7.0493 (1.55)      16.9462 (26.65)         4.9318 (2.26)       0.0835 (1.0)          3;11  141.8578 (0.65)        194           1
test_mesher_benchmark[10000]          30.9946 (19.48)       179.4609 (1.49)         56.0021 (12.30)     60.4828 (95.12)        31.2669 (14.35)      0.5618 (6.73)          1;1   17.8565 (0.08)          6           1
test_ppp_0103                        118.0134 (74.16)       120.0508 (1.0)         119.0509 (26.15)      0.6359 (1.0)         119.2245 (54.72)      0.8893 (10.65)         2;0    8.3998 (0.04)          9           1
test_ppp_0109                        203.8139 (128.08)      207.0749 (1.72)        205.3113 (45.10)      1.4715 (2.31)        204.7312 (93.97)      2.6287 (31.49)         2;0    4.8707 (0.02)          5           1
test_ppp_0104                        230.5988 (144.92)      233.9408 (1.95)        231.5489 (50.86)      1.3570 (2.13)        231.0394 (106.04)     1.0678 (12.79)         1;1    4.3187 (0.02)          5           1
test_ppp_0108                        261.6871 (164.45)      277.0477 (2.31)        268.1586 (58.90)      6.3048 (9.92)        267.1158 (122.60)    10.2105 (122.32)        2;0    3.7291 (0.02)          5           1
test_ttt_23_T_24                     275.4691 (173.11)      277.8486 (2.31)        277.0598 (60.85)      1.0734 (1.69)        277.7751 (127.49)     1.6238 (19.45)         1;0    3.6093 (0.02)          5           1
test_ppp_0105                        313.5031 (197.02)      316.5948 (2.64)        314.7853 (69.14)      1.2271 (1.93)        314.6930 (144.44)     1.8344 (21.98)         2;0    3.1768 (0.01)          5           1
test_ppp_0102                        451.3410 (283.64)      453.0843 (3.77)        452.1888 (99.32)      0.7295 (1.15)        452.3214 (207.61)     1.2393 (14.85)         2;0    2.2115 (0.01)          5           1
test_mesher_benchmark[100000]        511.2344 (321.28)      657.6413 (5.48)        615.4216 (135.17)    59.5374 (93.63)       636.8229 (292.29)    53.7202 (643.58)        1;1    1.6249 (0.01)          5           1
test_ppp_0110                        571.8256 (359.36)      578.6330 (4.82)        575.5086 (126.41)     2.6990 (4.24)        576.5737 (264.64)     3.9643 (47.49)         2;0    1.7376 (0.01)          5           1
test_ttt_24_SPO_06                   572.8023 (359.97)      647.6379 (5.39)        590.8107 (129.77)    31.8852 (50.14)       578.5744 (265.56)    22.0607 (264.29)        1;1    1.6926 (0.01)          5           1
test_ppp_0107                        842.9902 (529.77)      853.2604 (7.11)        845.3720 (185.68)     4.4264 (6.96)        843.6870 (387.24)     3.1786 (38.08)         1;1    1.1829 (0.01)          5           1
test_ttt_23_02_02                    944.7868 (593.74)      958.1590 (7.98)        950.5092 (208.77)     5.0978 (8.02)        950.7275 (436.37)     6.7531 (80.90)         2;0    1.0521 (0.00)          5           1
test_ppp_0101                      9,706.8569 (>1000.0)   9,732.0902 (81.07)     9,715.1094 (>1000.0)    9.9320 (15.62)     9,713.0243 (>1000.0)    9.9381 (119.06)        1;0    0.1029 (0.00)          5           1
test_ppp_0106                     26,875.1096 (>1000.0)  27,365.1841 (227.95)   27,022.7204 (>1000.0)  211.0890 (331.97)   26,904.6353 (>1000.0)  280.8139 (>1000.0)       1;0    0.0370 (0.00)          5           1
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

[jdegenstein]

OK I can confirm that these performance regressions coincide with this merged PR #1120 pertaining to intersection rework. cc @jwagenet

[jwagenet]

The regression seems to be related to the expand_compound method I added:

build123d/src/build123d/topology/composite.py

Lines 739 to 745 in 7a4f1f7

	def expand_compound(compound: Compound) -> ShapeList:
	shapes = ShapeList(compound.children)
	for shape_type in [Vertex, Edge, Wire, Face, Shell, Solid]:
	shapes.extend(compound.get_type(shape_type))
	return shapes

Name (time in ms)          Min       Max      Mean  StdDev    Median     IQR  Outliers     OPS  Rounds  Iterations
------------------------------------------------------------------------------------------------------------------
before #1120
test_ppp_0106            116.2     120.7     118.2     1.3     118.0     1.5       2;0  8.4571       8           1
------------------------------------------------------------------------------------------------------------------
after #1120
test_ppp_0106         11,585.4  30,286.0  15,343.4 8,353.2  11,596.9 4,720.1       1;1  0.0652       5           1
------------------------------------------------------------------------------------------------------------------
`expand_compound` calls removed
test_ppp_0106            148.1     152.9     149.8     1.6     149.5     2.1       2;0  6.6722       7           1
------------------------------------------------------------------------------------------------------------------

I used this method before and after intersections to dissolve Compounds, but interestingly only partially removing it doesn't notably improve performance (more I need to learn here).

The intended output from intersect is ShapeList[Vertex, Edge, Face, Solid] and the purpose of expand_compound is to both dissolve the Compound to the top level topo objects to meet this output and allow processing the output to remove lower order intersection results which are part of higher order results (eg the Edges of a Face). I need to explore if alternatives are available.

[gumyr]

Could Shape.get_top_level_shapes replace expand_compound?

[jwagenet]

Yes get_top_level_shapes should work (I also tried Compound.__iter__), but no performance change. For the purpose of testing with test_ppp_0106, I called Compound.solids() instead with no performance issue. This leads me to believe something is happening using TopoDS_Iterator or something else specifically related to do this, but exploring further.

[jdegenstein]

Untested here but previous similar issues have pointed significant blame at anytree. cProfile can be a great tool for this, of course. Some discussion at this issue #1030

[jwagenet]

I was too focused on expand_compound/get_top_level_shapes. I think because this method produces a large ShapeList of edges in this case, the many subsequent distance checks in filter_shapes_by_order between the solid from BRepAlgoAPI_Common and edges from BRepAlgoAPI_Section are a probable issue.

[jwagenet]

I got sidetracked from this when I didnt find an immediate solution. A 'quick' fix may be to relax the requirement to filter shapes found on a higher order shape result (eg remove Edges found on a Solid in the intersection results) by removing filter_shapes_by_order which may require more result filtering on the user side.

As far as avoiding regressions like this in the future is concerned, is there a way to track benchmark performance and place regression guards in the GH action?

[gumyr]

There already are test_benchmarks.py that get run but don't automatically fail a PR merge.

[bernhard-42]

I debugged through intersect and I don't claim that I fully understood everything, but I made a few observations:

1 Method filter_shapes_by_order is O(n*m) with expensive distance_to call

I consulted Claude Code and it recommended to not use distance_to since BRepExtrema_DistShapeShape is expensive. Instead, use optimized OCCT functions to check containing of lower shape in higher shape, something like:

        def filter_shapes_by_order(shapes: ShapeList, orders: list) -> ShapeList:
            # Remove lower order shapes from list which *appear* to be part of
            # a higher order shape using containment checks
            t0 = time.perf_counter_ns()

            def is_contained(lo: Shape, hi: Shape) -> bool:
                """Check if lower-order shape is contained in higher-order shape.
                Uses fast specialized checks instead of expensive BRepExtrema."""
                from OCP.BRep import BRep_Tool
                from OCP.BRepClass3d import BRepClass3d_SolidClassifier
                from OCP.BRepClass import BRepClass_FaceClassifier
                from OCP.TopAbs import TopAbs_IN, TopAbs_ON

                # Vertex containment checks
                if isinstance(lo, Vertex):
                    pt = BRep_Tool.Pnt_s(lo.wrapped)
                    if isinstance(hi, Solid):
                        classifier = BRepClass3d_SolidClassifier(hi.wrapped)
                        classifier.Perform(pt, TOLERANCE)
                        state = classifier.State()
                        return state in (TopAbs_IN, TopAbs_ON)
                    elif isinstance(hi, Face):
                        classifier = BRepClass_FaceClassifier()
                        classifier.Perform(hi.wrapped, pt, TOLERANCE)
                        state = classifier.State()
                        return state in (TopAbs_IN, TopAbs_ON)
                    elif isinstance(hi, Edge):
                        # Check if vertex is on edge - use distance (cheap for point-to-curve)
                        return lo.distance_to(hi) <= TOLERANCE

                # Edge containment in Face: check if edge endpoints are on face
                # and edge midpoint is on face (heuristic, fast)
                elif isinstance(lo, Edge) and isinstance(hi, Face):
                    classifier = BRepClass_FaceClassifier()
                    for param in [0.0, 0.5, 1.0]:
                        pt = lo.position_at(param).to_pnt()
                        classifier.Perform(hi.wrapped, pt, TOLERANCE)
                        if classifier.State() not in (TopAbs_IN, TopAbs_ON):
                            return False
                    return True

                # Edge in Solid: check endpoints and midpoint
                elif isinstance(lo, Edge) and isinstance(hi, Solid):
                    classifier = BRepClass3d_SolidClassifier(hi.wrapped)
                    for param in [0.0, 0.5, 1.0]:
                        pt = lo.position_at(param).to_pnt()
                        classifier.Perform(pt, TOLERANCE)
                        if classifier.State() not in (TopAbs_IN, TopAbs_ON):
                            return False
                    return True

                # Face in Solid: check center point
                elif isinstance(lo, Face) and isinstance(hi, Solid):
                    pt = lo.center().to_pnt()
                    classifier = BRepClass3d_SolidClassifier(hi.wrapped)
                    classifier.Perform(pt, TOLERANCE)
                    state = classifier.State()
                    return state in (TopAbs_IN, TopAbs_ON)

                # Fallback to distance check for other combinations
                return lo.distance_to(hi) <= TOLERANCE

            order_groups = []
            for order in orders:
                order_groups.append(
                    ShapeList([s for s in shapes if isinstance(s, order)])
                )

            filtered_shapes = order_groups[-1]
            for i in range(len(order_groups) - 1):
                los = order_groups[i]
                his: list = sum(order_groups[i + 1 :], [])

                for lo in los:
                    # Keep lo only if it's not contained in any higher-order shape
                    if not any(is_contained(lo, hi) for hi in his):
                        filtered_shapes.append(lo)

            print(f"filter_shapes_by_order: {(time.perf_counter_ns() - t0) * 1e-9:.4f}s")
            return filtered_shapes

This change ensures that all intersection tests continue to work, and reduces the time taken by the ppp0106 example in the filter_shapes_by_order function from 8.92 seconds to 0.0166 seconds.

Notes:

• The imports should be at the beginning of the file.
• The code was created by Claude, but it looks reasonable to me. More scrutiny should be applied.
• I haven't checked if these checks are already included in build123d or if they should be included. I was only analysing the intersection case.

2) Simple cases get expanded too much

For the simple case of intersecting two solids, the current code first applies BRepAlgoAPI_Section() which for the ppp0106 example expands the common array to 38 edges and one solid. This leads to filter_shapes_by_order needing to filter 39 shapes, while I think that a solid intersected with a solid can be treated by BRepAlgoAPI_Common() only.

The same should hold for face vs face, which can also be handled by BRepAlgoAPI_Common()

So a quick fix is to handle the special cases (solid vs solid) and (face vs face) first:

            common: list[Vertex | Edge | Wire | Face | Shell | Solid] = []
            result: ShapeList
            for obj in common_set:
                if isinstance(target, Shape):
                    target = ShapeList([target])
                result = ShapeList()
                operation: BRepAlgoAPI_Section | BRepAlgoAPI_Common

                for t in target:
                    if (
                        isinstance(obj, Solid)
                        and isinstance(t, Solid)
                        or (isinstance(obj, Face) and isinstance(t, Face))
                    ):
                        operation = BRepAlgoAPI_Common()
                        result.extend(bool_op((obj,), (t,), operation))
                    else:
                        operation = BRepAlgoAPI_Section()
                        result.extend(bool_op((obj,), (t,), operation))

                        # Face + Edge combinations may produce an intersection
                        # with Common but always with Section.
                        # No easy way to deduplicate
                        # Many Solid + Edge combinations need Common
                        if (
                            not isinstance(obj, Edge | Wire)
                            and not isinstance(t, Edge | Wire)
                        ) or (
                            isinstance(obj, Solid | Compound)
                            or isinstance(t, Solid | Compound)
                        ):
                            operation = BRepAlgoAPI_Common()
                            result.extend(bool_op((obj,), (t,), operation))

                if result:
                    common.extend(result)

With this the time in filter_shapes_by_order goes down to 0.000 sec for ppp0106.

3 Redundant code

I had the impression that the intersect methods have quite some duplication, so I asked Claude to look into it:

Component	one_d.py	two_d.py	three_d.py	composite.py	Duplicate lines per copy
to_vector()	✓	✓	✓	✓	2
to_vertex()	✓	✓	✓	✓	2
bool_op()	✓	✓	✓	✓	12
filter_shapes_by_order()	OLD	OLD	OLD	NEW	23 (OLD)
Target conversion (match)	✓	✓	✓	✓	~20
Post-processing	✓	✓	✓	✓*	~15

I would recommend to clean this up, especially if we agree on filter_shapes_by_order being changed.
I changed all occurrences of filter_shapes_by_order in all files to the new code, and still all intersections tests are successful.

Summary

As mentioned above, I don't have a full understanding of the intersection code, but from my analysis I'd say:

• The proposed changes for observations 1) and 2) reduce the duration in filter_shapes_by_order from 8.92s to 0.000s for the ppp0106 (and also to 0.000 sedc for the ppp1010 example). These changes should improve the intersection code in general from a performance perspective.
• Refactoring for observation 3 will make the code better maintainable

If something is wrong or missing, please let me know.

[bernhard-42]

Just to be clear, is_contained applies heuristics in order to be fast. Typically using start-, mid- and end-point. Given how fast it is we can use more points to be on the safer side. We can also use AABBs to test whether a edge is in a box (should also be fast). The main idea is to move away from the expensive BRepExtrema_DistShapeShape and use some cheaper methods.

[bernhard-42]

I have extended is_contained like that:

                # Edge containment in Face: check if edge endpoints are on face
                # and edge midpoint is on face (heuristic, fast)
                elif isinstance(lo, Edge) and isinstance(hi, Face):
                    classifier = BRepClass_FaceClassifier()
                    for param in [0.0, 0.5, 1.0]:
                        pt = lo.position_at(param).to_pnt()
                        classifier.Perform(hi.wrapped, pt, TOLERANCE)
                        if classifier.State() not in (TopAbs_IN, TopAbs_ON):
                            return False
                    # AABB validation: if edge extends beyond face, it's not contained
                    # (catches false positives where 3 points are on face by coincidence)
                    lo_bb, hi_bb = lo.bounding_box(), hi.bounding_box()
                    if (
                        lo_bb.min.X < hi_bb.min.X - TOLERANCE
                        or lo_bb.min.Y < hi_bb.min.Y - TOLERANCE
                        or lo_bb.min.Z < hi_bb.min.Z - TOLERANCE
                        or lo_bb.max.X > hi_bb.max.X + TOLERANCE
                        or lo_bb.max.Y > hi_bb.max.Y + TOLERANCE
                        or lo_bb.max.Z > hi_bb.max.Z + TOLERANCE
                    ):
                        return False
                    return True

That means,

• Check 3 points of the edge. If they are not on the face, return False (very quick rejection)
• Then calculate the AABB's and compare them (more expensive). If edge BB is not in face BB, reject again.
  This is still fast, but more exact.

When I remove change 2 and force the intersection algorithm to use BRepAlgoAPI_Section() and BRepAlgoAPI_Common() (as in the current implementation), pp0106 needs 0.58 sec instead of 8.9 sec.

Most importantly, change 2 will optimize the very common use cases of intersecting solid vs solid or face vs face, independent of the filter_shapes_by_order and is_contained implementation.

[bernhard-42]

I did another test where I removed the AABs again, but tested startpoint, endpoint + 8 random points on the edge whether they are in the solid. With this test, ppp0106 spends 0.032 sec in filter_shapes_by_order

By using startpoint and endpoint and X random points on the edge, it should be very unlikely, that all points are on face or in solid but the whole edge isn't, I'd say

Sumary: For ppp0106 we have spent time in filter_shapes_by_order

• 3 points: 0.0166 sec
• 10 points: 0.032 sec
• 3 points + AABB: 0.58 sec
• Current: 8.9 sec

[bernhard-42]

I looked a bit further into it. The proposed filter_shapes_by_order can use existing is_inside methods of build123d

• Improved proposal:

      def filter_shapes_by_order(shapes: ShapeList, filter_orders: list) -> ShapeList:
          """Remove lower order shapes contained in higher order shapes."""
  
          def is_inside(lo: Shape, hi: Shape) -> bool:
              if isinstance(lo, Vertex):
                  if isinstance(hi, (Face, Solid)):
                      return hi.is_inside(Vector(lo), TOLERANCE)
                  elif isinstance(hi, Edge):
                      return lo.distance_to(hi) <= TOLERANCE
              elif isinstance(lo, Edge):
                  if isinstance(hi, (Face, Solid)):
                      for p in [0.0, 1.0] + [random.random() for _ in range(8)]:
                          if not hi.is_inside(lo.position_at(p), TOLERANCE):
                              return False
                      return True
              elif isinstance(lo, Face) and isinstance(hi, Solid):
                  # Quick rejection: center and vertices
                  if not hi.is_inside(lo.center(), TOLERANCE):
                      return False
                  for vertex in lo.vertices():
                      if not hi.is_inside(Vector(vertex), TOLERANCE):
                          return False
                  # AABB containment check
                  return lo.bounding_box().is_inside(hi.bounding_box())
              return lo.distance_to(hi) <= TOLERANCE
  ...

• Heuristics:

  • For Edge in Face or Solid, the startpoint, endpoint and 8 random points are checked
  • For Face in Solid, center point and face vertices are checked for quick rejection, if that doesn't reject the more expensive AABB is used.

  The checks are written to end the loop as soon as the first vertex outside is detected

From a pure performance perspective, I think the change to heuristics in filter_shapes_by_order and the special handling of face-vs-face and solid-vs-solid intersections will improve the current situation without imposing too much risk by applying the heuristics. But this is open for discussion.

---

Also, I think I better understand the code redundancy now: filter_shapes_by_order needs Face, Solid, ... in its logic, else shared logic can't be easily moved to Shape (unless one accepts to add runtime imports inside filter_shapes_by_order like from .one_d import Edge, Wire).

I guess that was discussed with the overall PR, so sorry for raising it again. It is not relevant for the performance issue, except that a potential change to filter_shapes_by_order would need to applied at every occurrence.

[jdegenstein]

As far as avoiding regressions like this in the future is concerned, is there a way to track benchmark performance and place regression guards in the GH action?

@jwagenet I am open to suggestions but it is worthwhile to note:

1. runner performance can be highly variable
2. performance across platforms can vary widely (i.e. arm64 vs. x86_64 -- we have seen unusually slow performance with certain OCCT methods on the former platform, which led us to change the build123d codebase to avoid those)
3. RE:(2) makes it difficult to e.g. use only linux x86_64 as the "basis" for any CI failure

My current thought is maybe github-actions bot can comment on any PR where the performance changes significantly vs. the last available benchmark run? I already implemented attaching the benchmark results as a CSV artifact.