Add ability to prime URL metrics #1850

b1ink0 · 2025-02-05T17:55:23Z

Summary

Fixes #1311

Relevant technical choices

This PR introduces a mechanism for priming URL metrics across the site:

The newly added submenu page in the Tools menu can be used to prime URL metrics for URLs that don't have any metrics yet or have an incomplete group.
Automatically primes URL metrics when a post is saved in the block editor or classic editor.
Introduces a CLI tool that uses Puppeteer to prime URL metrics, useful for sites with a large number of posts.

TODOS:

Add tests

Demos

Settings page:

UI.mp4

Saving post in Block Editor:

Block.Editor.mp4

Saving post in Classic Editor:

Classic.Editor.mp4

CLI:

cli-demo.mp4

…ing between parent and iframe

… on frontend

…ints logic to dedicated functions

… generating a new UUID

…cessing

…token

…ack config

… visibility hidden

…or not visible

codecov · 2025-02-05T18:04:15Z

Codecov Report

❌ Patch coverage is 6.59898% with 368 lines in your changes missing coverage. Please review.
✅ Project coverage is 65.86%. Comparing base (6d42df0) to head (4943768).
⚠️ Report is 21 commits behind head on trunk.

Files with missing lines	Patch %	Lines
plugins/optimization-detective/helper.php	0.00%	249 Missing ⚠️
plugins/optimization-detective/settings.php	0.00%	39 Missing ⚠️
...lass-od-rest-url-metrics-priming-mode-endpoint.php	0.00%	33 Missing ⚠️
...detective/storage/class-od-priming-mode-wp-cli.php	0.00%	18 Missing ⚠️
plugins/optimization-detective/detection.php	0.00%	16 Missing ⚠️
...orage/class-od-rest-url-metrics-store-endpoint.php	65.00%	7 Missing ⚠️
plugins/optimization-detective/load.php	0.00%	4 Missing ⚠️
...ins/optimization-detective/class-od-url-metric.php	84.61%	2 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##            trunk    #1850      +/-   ##
==========================================
- Coverage   68.81%   65.86%   -2.95%     
==========================================
  Files          90       93       +3     
  Lines        8006     8394     +388     
==========================================
+ Hits         5509     5529      +20     
- Misses       2497     2865     +368

Flag	Coverage Δ
multisite	`65.86% <6.59%> (-2.95%)`	⬇️
single	`33.97% <3.29%> (-1.50%)`	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

b1ink0 · 2025-02-06T08:22:30Z

plugins/optimization-detective/detect.js

+			'prime_url_metrics_verification_token',
+			odPrimeUrlMetricsVerificationToken
+		);
+	}


Authentication for REST API

WP Nonce Limitation: The default WordPress (WP) nonce does not function correctly when generated for the parent page and then passed to an iframe for REST API requests.

Custom Token Authentication: To address this, I have added a custom token-based authentication mechanism. This generates a time-limited token used to authenticate REST API requests made via the iframe.

In #1835 PR, WP nonces are introduced for REST API requests for logged-in users. This may allow us to eliminate the custom token authentication if URL metrics are collected exclusively from logged-in users.

b1ink0 · 2025-02-06T08:26:22Z

plugins/optimization-detective/prime-url-metrics.js

+			};
+
+			// Load the iframe
+			iframe.src = task.url;


Currently if the IFRAME shares the same origin as the parent, then it allows it to access the parent session. This ensures that the user session in the page loaded within the iframe (which is a frontend page) matches the logged-in user of the WordPress dashboard.

But if the WordPress admin dashboard and the frontend have different origins, WP nonces won’t work for REST API authentication because the iframe will not recognize the logged-in session. As the different origin does not allow iframe to access parents session. For context I am talking about the REST nonce introduced in #1835.

b1ink0 · 2025-02-06T08:30:29Z

plugins/optimization-detective/prime-url-metrics-block-editor.js

+	iframe.style.transform = 'scale(0.05)';
+	iframe.style.transformOrigin = '0 0';
+	iframe.style.pointerEvents = 'none';
+	iframe.style.opacity = '0.000001';


As the detect.js requires the iframe to be visible in the viewport to resolve the onLCP promise. Traditional methods like moving the iframe off-screen using translate, setting visibility: hidden, or opacity: 0 cause the promise to hang.

performance/plugins/optimization-detective/detect.js

Lines 496 to 503 in 6ca5c4b

// Obtain at least one LCP candidate. More may be reported before the page finishes loading.

await new Promise( ( resolve ) => {

onLCP(

( /** @type LCPMetric */ metric ) => {

lcpMetricCandidates.push( metric );

resolve();

},

{

I am using a workaround using the following CSS to keep the iframe minimally visible and functional:

position: fixed; top: 0px; left: 0px; transform: scale(0.05); transform-origin: 0px 0px; pointer-events: none; opacity: 1e-6; z-index: -9999;

I was wondering how you were going to resolve solve for this!

b1ink0 · 2025-02-06T08:33:27Z

plugins/optimization-detective/detect.js

+				'OD_PRIME_URL_METRICS_REQUEST_SUCCESS',
+				'*'
+			);
+			resolve();


Parent and IFRAME communication is handled via postMessage. A message is sent to the parent, and the promise resolves immediately.

If the promise isn't resolved immediately, navigating to a new URL causes the code following the promise to never execute. This is because changing the iframe.src does not trigger events like pagehide, pageswap, or visibilitychange.

performance/plugins/optimization-detective/detect.js

Lines 568 to 569 in 6ca5c4b

// Wait for the page to be hidden.

await new Promise( ( resolve ) => {

I wonder. Do we even need to post a message here? As soon as the iframe is destroyed won't it automatically cause the URL Metric to be sent, right?

The problem is we need to signal the parent that we can move to next URL or breakpoint using postMessage as the load event can't be used. Check this comment for detailed explanation #1850 (comment) .

Will it makes sense to send the postMessage after the navigator.sendBeacon then?

Yes, I think it makes sense to send the message after the beacon is sent, definitely.

Co-authored-by: Weston Ruter <[email protected]>

westonruter · 2025-04-22T19:25:40Z

plugins/optimization-detective/class-od-url-metric.php

+	 *
+	 * @since n.e.x.t
+	 *
+	 * @return non-empty-string|null Source.


The literal values could be used instead:

Suggested change

* @return non-empty-string|null Source.

* @return 'visitor'|'user'|'synthetic'|null Source.

westonruter · 2025-04-22T19:27:26Z

plugins/optimization-detective/detect.js

+		source: restApiNonce ? 'user' : 'visitor',
 	};

+	if ( odPrimeUrlMetricsVerificationToken ) {
+		urlMetric.source = 'synthetic';
+	}


Populating the source should probably be done in the endpoint itself. It should be a readonly param.

westonruter · 2025-04-22T19:27:54Z

plugins/optimization-detective/class-od-url-metric.php

+				'source'    => array(
+					'description' => __( 'The source of the URL Metric.', 'optimization-detective' ),
+					'type'        => 'string',
+					'required'    => false,
+					'enum'        => array(
+						'visitor',
+						'user',
+						'synthetic',
+					),
+				),


Should also be readonly, populated in the endpoint callback here:

performance/plugins/optimization-detective/storage/class-od-rest-url-metrics-store-endpoint.php

Lines 214 to 217 in ff8b62b

// Now supply the readonly args which were omitted from the REST API params due to being `readonly`.

'timestamp' => microtime( true ),

'uuid' => wp_generate_uuid4(),

'etag' => $request->get_param( 'current_etag' ),

The value can be synthetic if the prime_url_metrics_verification_token param is present, or else user if is_user_logged_in(). Otherwise, it can be visitor.

Done in 5e64b39

Co-authored-by: Weston Ruter <[email protected]>

…ripts

…oved code quality - Rename endpoint classes and functions to consistently use "priming mode" terminology - Add TypeScript definitions for URL metrics priming features - Extract verification token logic into a dedicated function - Fix navigation issue in classic editor when using Update button - Improve JavaScript documentation with comprehensive JSDoc comments - Replace nested loops with flatMap for more concise code in JavaScript files - Use more descriptive variable names throughout the codebase - Add validation for cursor parameters to prevent potential issues - Reorganize code structure for better readability and maintenance

…and clarity

westonruter · 2025-09-05T00:17:41Z

Re: #1850 (comment)

@b1ink0:

What about allowing fetchpriority=high when any two group have the IMG as a LCP element?

I don't think that is safe because mobile and phablet may have the same LCP element, but then desktop has something else. Or desktop and tablet could have the same LCP element, but mobile and tablet have something else. This would mean the LCP image would be incorrectly prioritized more often. The current logic is to only add fetchpriority=high when the smallest viewport group and largest group both have the same LCP element, unless there are any intermediate groups have a different LCP element. (Often the intermediate groups will have no URL Metrics reported, given the current breakpoints, since mobile and desktop users are by far the most common.)

But adding fetchpriority=high to the actual element isn't so important because we are adding the preload links anyway. So adding fetchpriority to the IMG is just a little something extra.

there is no viewport for the group between 480 and 600. Additionally, due to the current max widths (480, 600, 782) that determine the group, the URL metric for the 360 width viewport gets overridden by the 414 viewport.

I think this highlights how uncommon the "phablet" viewport is, or rather that the "small" breakpoint in Gutenberg is very uncommon:

$break-medium: 782px;	// adminbar goes big
$break-small: 600px;
$break-mobile: 480px;

When asking Gemini about devices with a viewport width between 480px and 600px, it says:

However, this range typically encompasses some older, larger smartphones when held in landscape mode, as well as some smaller tablets and the "portrait" view of some foldable phones.

And:

In the broader spectrum of device popularity, the 480px to 600px viewport range represents a smaller but still relevant segment of the market. Based on recent data, the most popular viewport widths for mobile devices tend to be narrower, clustering around the 360px to 430px range. For instance, resolutions like 360x800 and 390x844 hold a significant share of the mobile market.

Conversely, tablet and desktop viewports are considerably wider. Common tablet viewports often start at 768px, with popular resolutions being 768x1024 and 810x1080. Desktop resolutions are even larger, with 1920x1080 and 1536x864 being among the most prevalent.

Therefore, devices with viewport widths between 480px and 600px are less common than the dominant smaller mobile screen sizes and the much larger tablet and desktop screens. However, this intermediate range remains an important consideration for ensuring a truly comprehensive responsive design that caters to all users, regardless of their device. As a web developer, accounting for this "in-between" screen size in your media queries can help prevent layout issues and provide a seamless experience for users with these specific devices.

Now regards to this:

We could change the breakpoint max widths to ( 360, 414, 782 ), which would solve this issue. However, I am not sure about this. What do you think?

These are supposed to be the max widths for the breakpoints, not the most common viewports. We should align them with the breakpoints defined in the Gutenberg's _breakpoints.scss as much as possible. The problem is that there is no breakpoint between 480px and 280px, at least as used in Gutenberg. The stylesheet does reference three others which are commonly used in WordPress elsewhere, however:

// max-width: 400px *
// max-width: 380px
// max-width: 320px *

Given this, we may need to introduce a new breakpoint lower than 480px so that we can capture URL Metrics for more accurate optimizations since many flavors of mobile fit in the 0-480px viewport group. I asked Gemini to give me a bar chart showing the popular viewport widths (which I do not know whether it is accurate!):

Importantly: there are no popular devices between the 412px and 782px. So the current 600px max breakpoint width is actually going to be largely unused on real sites. It would seem more useful to add a breakpoint that subdivides the large mobile set of mobile viewports, for large smartphone and small smartphone:

According to the common WordPress breakpoints referenced in that Gutenberg stylesheet, it seems like 380px would be the logical choice to divide those two groups. The devices that fall into those two groups, again per Gemini:

Less Than or Equal to 380px

These devices have a viewport width of 380px or less. This category is dominated by many popular iPhones and smaller Android devices.

iPhone SE (1st gen): 320px

iPhone 6, 7, 8, SE (2nd/3rd gen): 375px

iPhone X, XS, 11 Pro, 12/13/14/15 mini: 375px

Samsung Galaxy S8, S9: 360px

Samsung Galaxy S10e, S20, S21, S22: 360px

Google Pixel 3, 4, 5, 6a: 360px

Nokia 8 Sirocco: 360px

Greater Than 380px (and less than 481px)

These devices have a viewport width greater than 380px but less than 481px. This group includes most standard and larger-sized modern smartphones.

iPhone 6/7/8 Plus: 414px

iPhone XR, 11: 414px

iPhone XS Max, 11 Pro Max: 414px

iPhone 12/13/14/15 Pro: 390px

iPhone 12/13/14 Pro Max, 14/15 Plus: 428-430px

Google Pixel 4 XL, 6, 7: 412px

Samsung Galaxy S10, S21 Ultra: 412px

Samsung Galaxy Note 9, 10, 20: 412-414px

OnePlus 7T, 8, 9: 412px

Samsung Galaxy Z Fold (unfolded): 480px

All this to say, we could add 380 to the existing breakpoint max widths: 480, 600, 782. This would segment the major mobile clusters as identified by Gemini:

For instance, resolutions like 360x800 and 390x844 hold a significant share of the mobile market.

This would allow you to use the following device viewports as you had compiled above:

$device_viewports = array(
  array( 'width' => 360, 'height' => 780 ), // Small Smartphone.
  array( 'width' => 414, 'height' => 896 ), // Large Smartphone.
  array( 'width' => 768, 'height' => 1024 ), // Tablet.
  array( 'width' => 1920, 'height' => 1080 ), // Desktop.
);

Each would get assigned to a different viewport group. And given the relative unpopularity of devices between 480px and 600px, it probably doesn't make sense to add a device specifically to test it. We even might want to consider removing 600 from the max viewport groups.

Co-authored-by: Weston Ruter <[email protected]>

…ax width

b1ink0 added 23 commits January 23, 2025 23:58

Add settings page for Optimization Detective

57f9958

Add initial admin script and UI elements for URL priming

71cd706

Implement batch URL priming functionality with REST API endpoint

da1d275

Include breakpoints in URL processing

bfed250

Add debug mode and better dynamic height generation for breakpoints

6ec82e3

Add verification token auth for protecting REST API, Add message pass…

af0172e

…ing between parent and iframe

Refactor to improve control flow and add support for pause and resume…

bff9973

… on frontend

Move batch URL fetching logic to helper function

c155fc1

Add support to filter URL whose metrics are already populated

7640ef7

Add support to filter breakpoints whose metrics are already populated

f05f6c0

Refactor by moving filtering batch URls and computing standard brekpo…

0631daf

…ints logic to dedicated functions

Fix sending empty batch URLs caused by filtering the batch URLs

5d93fdc

Save last batch cursor in options table to improve performance

3118fb3

Improve verification token handling by using a transient check before…

d8cfa02

… generating a new UUID

Add beforeunload event listener to prevent page navigation during pro…

8879f60

…cessing

Add new REST API routes for URL metrics breakpoints and verification …

72b18fd

…token

Add block editor support for priming URL metrics

f5bba48

Merge branch 'trunk' into add/url-metrics-priming

b0422eb

Add missing since and access annotations, Add new JavaScripts to webp…

db1fd81

…ack config

Remove usage of fetch and Fix URL metrics not priming due to usage of…

63564c4

… visibility hidden

Merge branch 'trunk' into add/url-metrics-priming

4fd36b6

Fix detect.js getting stuck on 'onLCP' due to IFRAME being offscreen …

82a16b5

…or not visible

Improve responsiveness of IFRAME in debug mode

a50f23e

b1ink0 mentioned this pull request Feb 5, 2025

Add ability to prime URL metrics across a site to avoid needing to collect from visitors #1311

Open

Merge branch 'WordPress:trunk' into add/url-metrics-priming

9201c86

b1ink0 commented Feb 6, 2025

View reviewed changes

b1ink0 requested review from mukeshpanchal27 and thelovekesh as code owners April 11, 2025 17:00

b1ink0 and others added 5 commits April 12, 2025 01:07

Improve communication of response of URL Metric sending request

3a795e9

Co-authored-by: Weston Ruter <[email protected]>

Refactor to use URL fragment for passing verification token

4f93e14

Co-authored-by: Weston Ruter <[email protected]>

Add source property to URL Metric schema

ff6ad0f

Add auto scrolling for URL Metric priming mode

ec5dcd7

Add function to force URL Metrics compression when in priming mode

2550f96

westonruter reviewed Apr 23, 2025

View reviewed changes

b1ink0 and others added 17 commits April 23, 2025 23:16

Refactor to make source property readonly

5e64b39

Co-authored-by: Weston Ruter <[email protected]>

Fix failing test because of new source property

7d42cac

Co-authored-by: Weston Ruter <[email protected]>

Improve logging for failed tasks, Improve DOC comment of variables

d21ac27

Standardize functions and improve event handling in priming helper sc…

3b41a12

…ripts

Improve URL metrics priming UI and CLI parameter handling

b201dbe

Merge branch 'trunk' into add/url-metrics-priming

5c7dca6

Merge branch 'trunk' into add/url-metrics-priming

d38e5ca

Add missing parameter type

ab9ce29

Add TypeScript types for Priming CLI

26dda1d

Add priming mode source tracking

aa7391b

Merge branch 'trunk' into add/url-metrics-priming

619dd8d

Fix missing sanitation of nonce for classic editor post update

5335d2a

Merge branch 'trunk' into add/url-metrics-priming

661dcf1

Refactor URL metrics priming functions and variables for consistency …

c934b86

…and clarity

Improve priming mode verification token handling with auto-refresh

78c838d

Cache sitemap URL count for priming mode

d73842a

b1ink0 and others added 3 commits September 9, 2025 19:34

Replace computed breakpoints with real device viewports

5b7a26c

Co-authored-by: Weston Ruter <[email protected]>

Fix failing tests caused by the addition of a new breakpoint max width

fd94d9d

Fix missed failing tests caused by the addition of a new breakpoint m…

4943768

…ax width

b1ink0 requested a review from adamsilverstein as a code owner September 12, 2025 08:33

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Add ability to prime URL metrics #1850

Add ability to prime URL metrics #1850

b1ink0 commented Feb 5, 2025 •

edited

Loading

Uh oh!

codecov bot commented Feb 5, 2025 •

edited

Loading

Uh oh!

b1ink0 Feb 6, 2025

Uh oh!

b1ink0 Feb 6, 2025

Uh oh!

b1ink0 Feb 6, 2025 •

edited

Loading

Uh oh!

westonruter Apr 10, 2025

Uh oh!

b1ink0 Feb 6, 2025

Uh oh!

westonruter Feb 7, 2025

Uh oh!

b1ink0 Feb 7, 2025 •

edited

Loading

Uh oh!

westonruter Feb 7, 2025

Uh oh!

westonruter Apr 22, 2025

Uh oh!

westonruter Apr 22, 2025

Uh oh!

westonruter Apr 22, 2025

Uh oh!

b1ink0 Apr 28, 2025

Uh oh!

westonruter commented Sep 5, 2025 •

edited

Loading

Less Than or Equal to 380px

Greater Than 380px (and less than 481px)

Uh oh!

Uh oh!

	// Obtain at least one LCP candidate. More may be reported before the page finishes loading.
	await new Promise( ( resolve ) => {
	onLCP(
	( /** @type LCPMetric */ metric ) => {
	lcpMetricCandidates.push( metric );
	resolve();
	},
	{

	// Wait for the page to be hidden.
	await new Promise( ( resolve ) => {

	* @return non-empty-string\|null Source.
	* @return 'visitor'\|'user'\|'synthetic'\|null Source.

	// Now supply the readonly args which were omitted from the REST API params due to being `readonly`.
	'timestamp' => microtime( true ),
	'uuid' => wp_generate_uuid4(),
	'etag' => $request->get_param( 'current_etag' ),

Add ability to prime URL metrics #1850

Are you sure you want to change the base?

Add ability to prime URL metrics #1850

Conversation

b1ink0 commented Feb 5, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Relevant technical choices

TODOS:

Demos

Uh oh!

codecov bot commented Feb 5, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

b1ink0 Feb 6, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

b1ink0 Feb 7, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

westonruter commented Sep 5, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Less Than or Equal to 380px

Greater Than 380px (and less than 481px)

Uh oh!

Uh oh!

b1ink0 commented Feb 5, 2025 •

edited

Loading

codecov bot commented Feb 5, 2025 •

edited

Loading

b1ink0 Feb 6, 2025 •

edited

Loading

b1ink0 Feb 7, 2025 •

edited

Loading

westonruter commented Sep 5, 2025 •

edited

Loading