Skip to content

Custom metric for content-visibility #171

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
burakguneli wants to merge 9 commits into from
Closed

Custom metric for content-visibility #171

Show file tree
Hide file tree
Changes from 2 commits
Commits
Show all changes
9 commits
Select commit Hold shift + click to select a range
5aa1f76
Custom metric for content-visibility
burakguneli Jun 30, 2025
3a78637
Move content_visibility to css.js
burakguneli Jul 1, 2025
1a8d5fe
Replace expensive DOM traversal with efficient regex-based CSS parsing
burakguneli Jul 4, 2025
6d19ce8
Fix linter issues
burakguneli Jul 4, 2025
cad76c8
Add debugging
burakguneli Jul 4, 2025
f7522aa
Add try catch
burakguneli Jul 4, 2025
a4f7467
Return data JSON-stringified
burakguneli Jul 4, 2025
7faf743
Use parsed_css
burakguneli Jul 5, 2025
9f7e264
Rename file to content_visibility
burakguneli Jul 5, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
34 changes: 34 additions & 0 deletions dist/css.js
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -75,4 +75,38 @@ return JSON.stringify({
externalCssInBody: countExternalCssInBody(),
inlineCssInHead: countInlineCssInHead(),
inlineCssInBody: countInlineCssInBody(),

content_visibility: (() => {
// Detects elements using the content-visibility CSS property
const elements = document.querySelectorAll('*');
const contentVisibilityElements = [];
const contentVisibilityValues = {};

for (const element of elements) {
try {
const computedStyle = getComputedStyle(element);
Copy link
Member

@tunetheweb tunetheweb Jul 1, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@pmeenan any concerns with running getComputedStyle on every element in the DOM?

It sounds like a terrible idea for performance but custom metrics are executed off page load so I guess it doesn't really matter for those does it?

Copy link
Member

@pmeenan pmeenan Jul 2, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm on the fence. I assume it will only force a layout once if one is pending and then can return the element position for everything else without having to recalculate anything expensive but doing anything on ALL page elements freaks me out a bit.

My query selector foo isn't great, but is there a way to query based on content-visibility and only retrieve the elements with the "interesting" content visibility types directly?

I'd recommend trying it on some extreme edge-case pages (like CNN) to see how bad it is.

Copy link
Member

@tunetheweb tunetheweb Jul 2, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

My query selector foo isn't great, but is there a way to query based on content-visibility and only retrieve the elements with the "interesting" content visibility types directly

@burakguneli did suggest doing this for inline styles. But that's very limited. We really need the calculated style, which won't be directly queryable.

The other option is to look at the parsed_css data. I'm not totally familiar with that to be honest and whether that's just "we have a selector that uses content-visibility but no idea if it's used?" or if it actually gives the usage. @burakguneli was going to look further in this.

Copy link
Author

@burakguneli burakguneli Jul 4, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@tunetheweb @pmeenan I worked on an alternative way instead of doing expensive calls like getComputedStyle. I will send my commit and try to see if that works as expected

Copy link
Member

@tunetheweb tunetheweb Jul 4, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm n ot sure what manually processing the CSS gains over using parsed_css (which basically does this already)?

Copy link
Author

@burakguneli burakguneli Jul 5, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@tunetheweb thanks for checking and your feedback. I think you are right, the best way would be to use already parsed css. I'm kind of trying to learn how these custom metrics work, sorry for the many commits and notifications you are getting because of it. I'm not clear on one thing. Do you know how I can use the already parsed CSS in my custom metric file content_visibility.js?

Copy link
Member

@tunetheweb tunetheweb Jul 5, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The idea would be you wouldn’t need this custom metric and instead would get the data you need by selecting from that table.

Try it out for last month for a single URL (make sure to bill the httparchive project).

As I said earlier the downside is that I think it only says if a selector exists using that property. Not how much it’s actually used (or even if at all!). But not familiar with that table so not 100% sure about that and maybe it is better than that.

Copy link
Author

@burakguneli burakguneli Jul 5, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

oh thanks I got it now! For that, I tried running a couple of queries on the 30th of June like that: https://console.cloud.google.com/bigquery?inv=1&invt=Ab19zQ&project=httparchive&ws=!1m10!1m4!4m3!1shttparchive!2ssample_data!3sparsed_css_10k!1m4!1m3!1shttparchive!2sbquxjob_69d0dc73_197c1eb5200!3sUS

but after running this query I got the impression that we don't have it and we need to create a custom metric. Maybe it was my bad and I couldn't write a proper SQL query. I will try to improve it and retry!

Copy link
Member

@tunetheweb tunetheweb Jul 7, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This seems to work:

#standardSQL
CREATE TEMPORARY FUNCTION hasContentVisibility(css STRING)
RETURNS ARRAY<STRUCT<property STRING, freq INT64>>
LANGUAGE js
OPTIONS (library = "gs://httparchive/lib/css-utils.js")
AS '''
try {
  var ast = JSON.parse(css);

  let ret = {};

  walkDeclarations(ast, ({property}) => {
    // Strip hacks like *property, _property etc and normalize to lowercase
    property = property.replace(/[^a-z-]/g, "").toLowerCase();

    if (matches(property, 'content-visibility')) {
      incrementByKey(ret, property);
    }
  });

  return Object.entries(ret).map(([property, freq]) => {
    return {property, freq};
  });
} catch (e) {
  return [];
}
''';

WITH totals AS (
  SELECT
    client,
    COUNT(distinct root_page) AS total_pages
  FROM
    `httparchive.crawl.parsed_css`
  WHERE
    date = '2025-06-01' AND
    rank <= 1000 AND -- remove to run on whole dataset
    is_root_page
  GROUP BY
    client
),
content_visibility_pages AS (
  SELECT
    client,
    COUNT(distinct root_page) AS pages_with_content_visibility
  FROM
    `httparchive.crawl.parsed_css`,
  UNNEST (hasContentVisibility(css))
  WHERE
    date = '2025-06-01' AND
    rank <= 1000 AND -- remove to run on whole dataset
    is_root_page
  GROUP BY
    client
)
SELECT
  totals.client,
  IFNULL(content_visibility_pages.pages_with_content_visibility, 0) AS pages_with_content_visibility,
  totals.total_pages,
  ROUND(IFNULL(content_visibility_pages.pages_with_content_visibility, 0) * 100.0 / totals.total_pages, 2) AS pct_pages
FROM
  totals
LEFT JOIN
  content_visibility_pages
USING (client)
ORDER BY
  totals.client

Copy link
Author

@burakguneli burakguneli Jul 7, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

that works amazing thanks!!! @tunetheweb

const contentVisibility = computedStyle.getPropertyValue('content-visibility');

if (contentVisibility && contentVisibility !== 'visible') {
contentVisibilityElements.push({
tagName: element.tagName.toLowerCase(),
contentVisibility: contentVisibility.trim(),
className: element.className || '',
id: element.id || ''
});

const value = contentVisibility.trim();
contentVisibilityValues[value] = (contentVisibilityValues[value] || 0) + 1;
}
} catch (e) {
// continue regardless of error
}
}

return {
total: contentVisibilityElements.length,
elements: contentVisibilityElements,
values: contentVisibilityValues
};
})()
});
32 changes: 32 additions & 0 deletions metric-summary.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -723,6 +723,38 @@ Example response:
[5, 12, 11]
```

## [css.js](https://github.com/HTTPArchive/custom-metrics/blob/main/dist/css.js) metrics

### content_visibility

Detects elements using the `content-visibility` CSS property for performance optimization analysis. Returns information about elements that have content-visibility set to values other than 'visible' (such as 'auto', 'hidden', or 'skip').

Example response:

```json
{
"total": 15,
"elements": [
{
"tagName": "div",
"contentVisibility": "auto",
"className": "lazy-section",
"id": "section-1"
},
{
"tagName": "section",
"contentVisibility": "hidden",
"className": "hidden-content",
"id": ""
}
],
"values": {
"auto": 12,
"hidden": 3
}
}
```

## [Images.js](https://github.com/HTTPArchive/custom-metrics/blob/main/dist/Images.js) metrics

A JSON array of `<img>` elements on the page.
Expand Down