Merge remote-tracking branch 'intel/sycl' into HEAD

Author: steffenlarsen

devops/scripts/benchmarks/CONTRIB.md

@@ -67,6 +67,104 @@ The suite is structured around three main components: Suites, Benchmarks, and Re
         * `display_name`: Optional user-friendly name for the benchmark (string). Defaults to `name()`.
         * `explicit_group`: Optional explicit group name for results (string). Used to group results in visualizations.
 
+## Dashboard and Visualization
+
+The benchmark suite generates an interactive HTML dashboard that visualizes `Result` objects and their metadata.
+
+### Data Flow from Results to Dashboard
+
+1. **Collection Phase:**
+    * Benchmarks generate `Result` objects containing performance measurements.
+    * The framework combines these with `BenchmarkMetadata` from benchmarks and suites. The metadata
+    is used for defining charts.
+    * All data is packaged into a `BenchmarkOutput` object containing runs, metadata, and tags for serialization.
+
+2. **Serialization:**
+    * For local viewing (`--output-html local`): Data is written as JavaScript variables in `data.js`.
+    These are directly loaded in the HTML dashboard.
+    * For remote deployment (`--output-html remote`): Data is written as JSON in `data.json`.
+    The `config.js` file contains the URL where the json file is hosted.
+    * Historical runs may be separated into archive files for better dashboard load times.
+
+3. **Dashboard Rendering:**
+    * JavaScript processes the data to create three chart types:
+        * **Historical Results**: Time-series charts showing performance trends over multiple runs. One chart for each unique benchmark scenario.
+        * **Historical Layer Comparisons**: Time-series charts for grouped results. Benchmark scenarios that can be directly compared are grouped either by using `explicit_group()` or matching the beginning of their labels with predefined groups.
+        * **Comparisons**: Bar charts comparing selected runs side-by-side. Again, based on the `explicit_group()` or labels.
+
+### Chart Types and Result Mapping
+
+**Historical Results (Time-series):**
+* One chart per unique `result.label`.
+* X-axis: `BenchmarkRun.date` (time).
+* Y-axis: `result.value` with `result.unit`.
+* Multiple lines for different `BenchmarkRun.name` entries.
+* Points include `result.stddev`, `result.git_hash`, and environment info in tooltips.
+
+**Historical Layer Comparisons:**
+* Groups related results using `benchmark.explicit_group()` or `result.label` prefixes.
+* Useful for comparing different implementations/configurations of the same benchmark.
+* Same time-series format but with grouped data.
+
+**Comparisons (Bar charts):**
+* Compares selected runs side-by-side.
+* X-axis: `BenchmarkRun.name`.
+* Y-axis: `result.value` with `result.unit`.
+* One bar per selected run.
+
+### Dashboard Features Controlled by Results/Metadata
+
+**Visual Properties:**
+* **Chart Title**: `metadata.display_name` or `result.label`.
+* **Y-axis Range**: `metadata.range_min` and `range_max` (when custom ranges enabled).
+* **Direction Indicator**: `result.lower_is_better` (shows "Lower/Higher is better").
+* **Grouping**: `benchmark.explicit_group()` groups related results together.
+
+**Filtering and Organization:**
+* **Suite Filters**: Filter by `result.suite`.
+* **Tag Filters**: Filter by `metadata.tags`.
+* **Regex Search**: Search by `result.label` patterns, `metadata.display_name` patterns are not searchable.
+* **Stability**: Hide/show based on `metadata.unstable`.
+
+**Information Display:**
+* **Description**: `metadata.description` appears prominently above charts.
+* **Notes**: `metadata.notes` provides additional context (toggleable).
+* **Tags**: `metadata.tags` displayed as colored badges with descriptions.
+* **Command Details**: Shows `result.command` and `result.env` in expandable sections.
+* **Git Information**: `result.git_url` and `result.git_hash` for benchmark source tracking.
+
+### Dashboard Interaction
+
+**Run Selection:**
+* Users select which `BenchmarkRun.name` entries to compare.
+* Default selection uses `BenchmarkOutput.default_compare_names`.
+* Changes affect all chart types simultaneously.
+
+**URL State Preservation:**
+* All filters, selections, and options are preserved in URL parameters.
+* Enables sharing specific dashboard views via URL address copy.
+
+### Best Practices for Dashboard-Friendly Results
+
+**Naming:**
+* Use unique `result.label` names that will be most descriptive.
+* Consider `metadata.display_name` for prettier chart titles.
+* Ensure `benchmark.name()` is unique across all suites.
+
+**Grouping:**
+* Use `benchmark.explicit_group()` to group related measurements.
+* Ensure grouped results have the same `result.unit`.
+* Group metadata keys in `Suite.additional_metadata()` should match group prefixes.
+
+**Metadata:**
+* Provide `metadata.description` for user understanding.
+* Use `metadata.notes` for implementation details or caveats.
+* Tag with relevant `metadata.tags` for filtering.
+* Set `metadata.range_min`/`range_max` for consistent comparisons when needed.
+
+**Stability:**
+* Mark unstable benchmarks with `metadata.unstable` to hide them by default.
+
 ## Adding New Benchmarks
 
 1. **Create Benchmark Class:** Implement a new class inheriting from `benches.base.Benchmark`. Implement required methods (`setup`, `run`, `teardown`, `name`) and optional ones (`description`, `get_tags`, etc.) as needed.
@@ -84,6 +182,14 @@ The suite is structured around three main components: Suites, Benchmarks, and Re
 * **Use unique names:** Ensure `benchmark.name()` and `result.label` are descriptive and unique.
 * **Group related results:** Use `benchmark.explicit_group()` consistently for results you want to compare directly in outputs. Ensure units match within a group. If defining group-level metadata in the Suite, ensure the chosen explicit_group name starts with the corresponding key defined in additional_metadata.
 * **Test locally:** Before submitting changes, test with relevant drivers/backends (e.g., using `--compute-runtime --build-igc` for L0). Check the visualization locally if possible (--output-markdown --output-html, then open the generated files).
+* **Test dashboard visualization:** When adding new benchmarks, always generate and review the HTML dashboard to ensure:
+    * Chart titles and labels are clear and readable.
+    * Results are grouped logically using `explicit_group()`.
+    * Metadata (description, notes, tags) displays correctly.
+    * Y-axis ranges are appropriate (consider setting `range_min`/`range_max` if needed).
+    * Filtering by suite and tags works as expected.
+    * Time-series trends make sense for historical data.
+    * **Tip**: Use `--dry-run --output-html local` to regenerate the dashboard without re-running benchmarks. This uses existing historical data and is useful for testing metadata changes, new groupings, or dashboard improvements.
 
 ## Utilities
 

devops/scripts/benchmarks/html/scripts.js

@@ -123,8 +123,10 @@ function createChart(data, containerId, type) {
     }
 
     const ctx = document.getElementById(containerId).getContext('2d');
+
     const options = {
         responsive: true,
+        maintainAspectRatio: false,
         plugins: {
             title: {
                 display: true,
@@ -157,6 +159,13 @@ function createChart(data, containerId, type) {
                     }
                 }
             },
+            legend: {
+                position: 'top',
+                labels: {
+                    boxWidth: 12,
+                    padding: 10,
+                }
+            },
             annotation: type === 'time' ? {
                 annotations: {}
             } : undefined
@@ -227,6 +236,32 @@ function createChart(data, containerId, type) {
     const chart = new Chart(ctx, chartConfig);
     chartInstances.set(containerId, chart);
 
+    // Set explicit canvas size after chart creation to ensure proper sizing
+    const canvas = document.getElementById(containerId);
+    const rect = canvas.getBoundingClientRect();
+    const dpr = window.devicePixelRatio || 1;
+
+    // Calculate dynamic height based on number of legend items
+    const legendItemCount = type === 'time' ?
+        Object.values(data.runs).length :
+        data.datasets.length;
+
+    // Base chart height + legend height (25px per line + padding)
+    const baseChartHeight = 350;
+    const legendHeight = Math.max(legendItemCount * 25, 50); // minimum 50px for legend
+    const totalHeight = baseChartHeight + legendHeight;
+
+    // Set canvas dimensions for crisp rendering
+    canvas.width = rect.width * dpr;
+    canvas.height = totalHeight * dpr;
+
+    // Scale the context to ensure correct drawing operations
+    const context = canvas.getContext('2d');
+    context.scale(dpr, dpr);
+
+    // Force chart to use these exact dimensions
+    chart.resize(rect.width, totalHeight);
+
     // Add annotation interaction handlers for time-series charts
     if (type === 'time') {
         ChartAnnotations.setupAnnotationListeners(chart, ctx, options);
@@ -306,6 +341,10 @@ function createChartContainer(data, canvasId, type) {
     container.setAttribute('data-label', data.label);
     container.setAttribute('data-suite', data.suite);
 
+    // Create header section for metadata
+    const headerSection = document.createElement('div');
+    headerSection.className = 'chart-header';
+
     // Check if this benchmark is marked as unstable
     const metadata = metadataForLabel(data.label, type);
     if (metadata && metadata.unstable) {
@@ -316,15 +355,17 @@ function createChartContainer(data, canvasId, type) {
         unstableWarning.className = 'benchmark-unstable';
         unstableWarning.textContent = metadata.unstable;
         unstableWarning.style.display = isUnstableEnabled() ? 'block' : 'none';
-        container.appendChild(unstableWarning);
+        unstableWarning.style.marginBottom = '5px';
+        headerSection.appendChild(unstableWarning);
     }
 
-    // Add description if present in metadata (moved outside of details)
+    // Add description if present in metadata
     if (metadata && metadata.description) {
         const descElement = document.createElement('div');
         descElement.className = 'benchmark-description';
         descElement.textContent = metadata.description;
-        container.appendChild(descElement);
+        descElement.style.marginBottom = '5px';
+        headerSection.appendChild(descElement);
     }
 
     // Add notes if present
@@ -333,7 +374,7 @@ function createChartContainer(data, canvasId, type) {
         noteElement.className = 'benchmark-note';
         noteElement.textContent = metadata.notes;
         noteElement.style.display = isNotesEnabled() ? 'block' : 'none';
-        container.appendChild(noteElement);
+        headerSection.appendChild(noteElement);
     }
 
     // Add tags if present
@@ -358,12 +399,31 @@ function createChartContainer(data, canvasId, type) {
             tagsContainer.appendChild(tagElement);
         });
 
-        container.appendChild(tagsContainer);
+        headerSection.appendChild(tagsContainer);
     }
 
+    // Add header section to container
+    container.appendChild(headerSection);
+
+    // Create main content section (chart + legend area)
+    const contentSection = document.createElement('div');
+    contentSection.className = 'chart-content';
+
+    // Canvas for the chart - fixed position in content flow
     const canvas = document.createElement('canvas');
     canvas.id = canvasId;
-    container.appendChild(canvas);
+    canvas.style.width = '100%';
+
+    // Set a default height - will be properly sized later in createChart
+    canvas.style.height = '400px';
+    canvas.style.marginBottom = '10px';
+    contentSection.appendChild(canvas);
+
+    container.appendChild(contentSection);
+
+    // Create footer section for details
+    const footerSection = document.createElement('div');
+    footerSection.className = 'chart-footer';
 
     // Create details section for extra info
     const details = document.createElement('details');
@@ -387,7 +447,8 @@ function createChartContainer(data, canvasId, type) {
     extraInfo.innerHTML = generateExtraInfo(data, 'benchmark');
     details.appendChild(extraInfo);
 
-    container.appendChild(details);
+    footerSection.appendChild(details);
+    container.appendChild(footerSection);
 
     return container;
 }

devops/scripts/benchmarks/html/styles.css

@@ -18,8 +18,24 @@ h1, h2 {
     background: white;
     border-radius: 8px;
     padding: 24px;
-    margin-bottom: 24px;
-    box-shadow: 0 1px 3px rgba(0,0,0,0.1);
+    box-shadow: 0 1px 3px rgba(0, 0, 0, 0.1);
+    position: relative;
+    display: flex;
+    flex-direction: column;
+    box-sizing: border-box;
+    min-height: fit-content;
+    margin: 10px 0;
+    overflow: visible;
+}
+.chart-header {
+    padding: 10px;
+    border-radius: 6px 6px 0 0;
+}
+.chart-content {
+    padding: 10px;
+    display: flex;
+    flex-direction: column;
+    min-height: fit-content;
 }
 @media (max-width: 768px) {
     body {