Skip to content

feat(DO-2118): terms of use status query and DAG tweaks #8738

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
kik-kik wants to merge 3 commits into
base: main
Choose a base branch
from
Open

feat(DO-2118): terms of use status query and DAG tweaks #8738

kik-kik wants to merge 3 commits into from

Conversation

@kik-kik
Copy link
Contributor

@kik-kik kik-kik commented Jan 22, 2026

feat(DO-2118): terms of use status query and DAG tweaks

Description

This PR simplifies the query a little bit and opts for group by instead of WINDOW for when reducing client attributes for easier readability.

Also, changed depends_on_past to True to ensure the terms of use queries get executed in the correct order.

@kik-kik kik-kik requested a review from soGaussian January 22, 2026 17:43
@kik-kik kik-kik self-assigned this Jan 22, 2026
@kik-kik kik-kik requested a review from a team as a code owner January 22, 2026 17:43
@kik-kik kik-kik added the enhancement New feature or request label Jan 22, 2026
@dataops-ci-bot

This comment has been minimized.

Sign in to view
@kik-kik kik-kik force-pushed the fix/DO-2118/terms-of-use-status-missing-clients branch from ee0d358 to 3997620 Compare January 23, 2026 16:30
@dataops-ci-bot
Copy link

dataops-ci-bot commented Jan 23, 2026

Integration report for "feat: update tests to match new template"

sql.diff

Click to expand! 
diff -bur --no-dereference --new-file /tmp/workspace/main-generated-sql/dags/bqetl_terms_of_use.py /tmp/workspace/generated-sql/dags/bqetl_terms_of_use.py
--- /tmp/workspace/main-generated-sql/dags/bqetl_terms_of_use.py	2026-01-23 16:36:42.000000000 +0000
+++ /tmp/workspace/generated-sql/dags/bqetl_terms_of_use.py	2026-01-23 16:42:20.000000000 +0000
@@ -33,7 +33,7 @@
     "start_date": datetime.datetime(2025, 11, 3, 0, 0),
     "end_date": None,
     "email": ["[email protected]", "[email protected]"],
-    "depends_on_past": False,
+    "depends_on_past": True,
     "retry_delay": datetime.timedelta(seconds=1800),
     "email_on_failure": True,
     "email_on_retry": False,
diff -bur --no-dereference --new-file /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/fenix/broken_site_report/metadata.yaml /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/fenix/broken_site_report/metadata.yaml
--- /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/fenix/broken_site_report/metadata.yaml	2026-01-23 16:33:17.000000000 +0000
+++ /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/fenix/broken_site_report/metadata.yaml	2026-01-23 16:39:19.000000000 +0000
@@ -1,6 +1,10 @@
-friendly_name: Broken Site Report
+friendly_name: App-specific view for Glean ping "broken-site-report"
 description: |-
-  Please provide a description for the query
+  This a view that UNIONs the stable ping tables
+  across all channels of the Glean application "Firefox for Android"
+  (org_mozilla_firefox.broken_site_report, org_mozilla_firefox_beta.broken_site_report, org_mozilla_fenix.broken_site_report, org_mozilla_fenix_nightly.broken_site_report, org_mozilla_fennec_aurora.broken_site_report).
+
+  It is used by Looker.
 owners: []
 workgroup_access:
 - role: roles/bigquery.dataViewer
diff -bur --no-dereference --new-file /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/fenix/crash/metadata.yaml /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/fenix/crash/metadata.yaml
--- /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/fenix/crash/metadata.yaml	2026-01-23 16:33:17.000000000 +0000
+++ /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/fenix/crash/metadata.yaml	2026-01-23 16:39:19.000000000 +0000
@@ -1,6 +1,10 @@
-friendly_name: Crash
+friendly_name: App-specific view for Glean ping "crash"
 description: |-
-  Please provide a description for the query
+  This a view that UNIONs the stable ping tables
+  across all channels of the Glean application "Firefox for Android"
+  (org_mozilla_firefox.crash, org_mozilla_firefox_beta.crash, org_mozilla_fenix.crash, org_mozilla_fenix_nightly.crash, org_mozilla_fennec_aurora.crash).
+
+  It is used by Looker.
 owners: []
 workgroup_access:
 - role: roles/bigquery.dataViewer
diff -bur --no-dereference --new-file /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/fenix_derived/terms_of_use_status_v1/query.sql /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/fenix_derived/terms_of_use_status_v1/query.sql
--- /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/fenix_derived/terms_of_use_status_v1/query.sql	2026-01-23 16:33:18.000000000 +0000
+++ /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/fenix_derived/terms_of_use_status_v1/query.sql	2026-01-23 16:38:08.000000000 +0000
@@ -25,34 +25,38 @@
     DATE(submission_timestamp) AS submission_date,
     client_info.client_id,
     sample_id,
-    `moz-fx-data-shared-prod`.udf.mode_last(
-      ARRAY_AGG(app_version_major) OVER _window
+    mozfun.stats.mode_last(
+      ARRAY_AGG(app_version_major ORDER BY submission_timestamp ASC)
     ) AS app_version_major,
-    `moz-fx-data-shared-prod`.udf.mode_last(
-      ARRAY_AGG(app_version_minor) OVER _window
+    mozfun.stats.mode_last(
+      ARRAY_AGG(app_version_minor ORDER BY submission_timestamp ASC)
     ) AS app_version_minor,
-    `moz-fx-data-shared-prod`.udf.mode_last(
-      ARRAY_AGG(app_version_patch) OVER _window
+    mozfun.stats.mode_last(
+      ARRAY_AGG(app_version_patch ORDER BY submission_timestamp ASC)
     ) AS app_version_patch,
-    `moz-fx-data-shared-prod`.udf.mode_last(
-      ARRAY_AGG(normalized_channel) OVER _window
+    mozfun.stats.mode_last(
+      ARRAY_AGG(normalized_channel ORDER BY submission_timestamp ASC)
     ) AS normalized_channel,
-    `moz-fx-data-shared-prod`.udf.mode_last(
-      ARRAY_AGG(normalized_country_code) OVER _window
+    mozfun.stats.mode_last(
+      ARRAY_AGG(normalized_country_code ORDER BY submission_timestamp ASC)
     ) AS normalized_country_code,
-    `moz-fx-data-shared-prod`.udf.mode_last(ARRAY_AGG(normalized_os) OVER _window) AS normalized_os,
-    `moz-fx-data-shared-prod`.udf.mode_last(
-      ARRAY_AGG(normalized_os_version) OVER _window
+    mozfun.stats.mode_last(
+      ARRAY_AGG(normalized_os ORDER BY submission_timestamp ASC)
+    ) AS normalized_os,
+    mozfun.stats.mode_last(
+      ARRAY_AGG(normalized_os_version ORDER BY submission_timestamp ASC)
     ) AS normalized_os_version,
-    `moz-fx-data-shared-prod`.udf.mode_last(
-      ARRAY_AGG(is_bot_generated) OVER _window
+    mozfun.stats.mode_last(
+      ARRAY_AGG(is_bot_generated ORDER BY submission_timestamp ASC)
     ) AS is_bot_generated,
-    `moz-fx-data-shared-prod`.udf.mode_last(ARRAY_AGG(metadata.isp.name) OVER _window) AS isp_name,
-    `moz-fx-data-shared-prod`.udf.mode_last(
-      ARRAY_AGG(metrics.quantity.terms_of_use_version) OVER _window
+    mozfun.stats.mode_last(
+      ARRAY_AGG(metadata.isp.`name` ORDER BY submission_timestamp ASC)
+    ) AS isp_name,
+    mozfun.stats.mode_last(
+      ARRAY_AGG(metrics.quantity.terms_of_use_version ORDER BY submission_timestamp ASC)
     ) AS terms_of_use_version_accepted,
-    `moz-fx-data-shared-prod`.udf.mode_last(
-      ARRAY_AGG(metrics.datetime.terms_of_use_date) OVER _window
+    mozfun.stats.mode_last(
+      ARRAY_AGG(metrics.datetime.terms_of_use_date ORDER BY submission_timestamp ASC)
     ) AS terms_of_use_date_accepted,
   FROM
     `moz-fx-data-shared-prod.fenix.metrics`
@@ -60,21 +64,11 @@
     DATE(submission_timestamp) = @submission_date
     -- Adding a hard filter from when we want to start recording this data:
     AND DATE(submission_timestamp) >= "2025-09-15"
+    AND client_info.client_id IS NOT NULL
     AND app_version_major >= 142
     AND metrics.datetime.terms_of_use_date IS NOT NULL
-  QUALIFY
-    ROW_NUMBER() OVER (PARTITION BY client_id ORDER BY submission_timestamp DESC) = 1
-  WINDOW
-    _window AS (
-      PARTITION BY
-        sample_id,
-        client_info.client_id
-      ORDER BY
-        submission_timestamp
-      ROWS BETWEEN
-        UNBOUNDED PRECEDING
-        AND UNBOUNDED FOLLOWING
-    )
+  GROUP BY
+    ALL
 )
 SELECT
   -- update entry if `terms_of_use_version_accepted` or `terms_of_use_date_accepted` value changes:
@@ -82,11 +76,11 @@
     _previous.client_id IS NULL
     OR (
       (_current.terms_of_use_version_accepted <> _previous.terms_of_use_version_accepted)
-      AND (_current.terms_of_use_date_accepted <> _previous.terms_of_use_date_accepted)
+      OR (_current.terms_of_use_date_accepted <> _previous.terms_of_use_date_accepted)
     ),
     _current,
     _previous
-  ).*
+  ).*,
 FROM
   _current
 FULL OUTER JOIN
diff -bur --no-dereference --new-file /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/firefox_crashreporter/crash/metadata.yaml /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/firefox_crashreporter/crash/metadata.yaml
--- /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/firefox_crashreporter/crash/metadata.yaml	2026-01-23 16:33:18.000000000 +0000
+++ /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/firefox_crashreporter/crash/metadata.yaml	2026-01-23 16:39:21.000000000 +0000
@@ -1,6 +1,14 @@
-friendly_name: Crash
+friendly_name: Historical Pings for `firefox-crashreporter/crash`
 description: |-
-  Please provide a description for the query
+  A historical view of pings sent for the
+  `firefox-crashreporter/crash`
+  document type.
+
+  This view is guaranteed to contain only complete days
+  (per `submission_timestamp`)
+  and to contain only one row per distinct `document_id` within a given date.
+
+  Clustering fields: `normalized_channel`, `sample_id`
 owners: []
 labels:
   authorized: true
diff -bur --no-dereference --new-file /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/firefox_desktop/broken_site_report/metadata.yaml /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/firefox_desktop/broken_site_report/metadata.yaml
--- /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/firefox_desktop/broken_site_report/metadata.yaml	2026-01-23 16:33:17.000000000 +0000
+++ /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/firefox_desktop/broken_site_report/metadata.yaml	2026-01-23 16:39:19.000000000 +0000
@@ -1,6 +1,14 @@
-friendly_name: Broken Site Report
+friendly_name: Historical Pings for `firefox-desktop/broken-site-report`
 description: |-
-  Please provide a description for the query
+  A historical view of pings sent for the
+  `firefox-desktop/broken-site-report`
+  document type.
+
+  This view is guaranteed to contain only complete days
+  (per `submission_timestamp`)
+  and to contain only one row per distinct `document_id` within a given date.
+
+  Clustering fields: `normalized_channel`, `sample_id`
 owners: []
 labels:
   authorized: true
diff -bur --no-dereference --new-file /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/firefox_desktop/crash/metadata.yaml /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/firefox_desktop/crash/metadata.yaml
--- /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/firefox_desktop/crash/metadata.yaml	2026-01-23 16:33:17.000000000 +0000
+++ /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/firefox_desktop/crash/metadata.yaml	2026-01-23 16:39:19.000000000 +0000
@@ -1,6 +1,14 @@
-friendly_name: Crash
+friendly_name: Historical Pings for `firefox-desktop/crash`
 description: |-
-  Please provide a description for the query
+  A historical view of pings sent for the
+  `firefox-desktop/crash`
+  document type.
+
+  This view is guaranteed to contain only complete days
+  (per `submission_timestamp`)
+  and to contain only one row per distinct `document_id` within a given date.
+
+  Clustering fields: `normalized_channel`, `sample_id`
 owners: []
 labels:
   authorized: true
diff -bur --no-dereference --new-file /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/firefox_desktop/newtab/metadata.yaml /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/firefox_desktop/newtab/metadata.yaml
--- /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/firefox_desktop/newtab/metadata.yaml	2026-01-23 16:33:17.000000000 +0000
+++ /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/firefox_desktop/newtab/metadata.yaml	2026-01-23 16:39:19.000000000 +0000
@@ -1,6 +1,14 @@
-friendly_name: Newtab
+friendly_name: Historical Pings for `firefox-desktop/newtab`
 description: |-
-  Please provide a description for the query
+  A historical view of pings sent for the
+  `firefox-desktop/newtab`
+  document type.
+
+  This view is guaranteed to contain only complete days
+  (per `submission_timestamp`)
+  and to contain only one row per distinct `document_id` within a given date.
+
+  Clustering fields: `normalized_channel`, `sample_id`
 owners: []
 workgroup_access:
 - role: roles/bigquery.dataViewer
diff -bur --no-dereference --new-file /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/firefox_desktop_derived/terms_of_use_status_v1/query.sql /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/firefox_desktop_derived/terms_of_use_status_v1/query.sql
--- /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/firefox_desktop_derived/terms_of_use_status_v1/query.sql	2026-01-23 16:33:17.000000000 +0000
+++ /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/firefox_desktop_derived/terms_of_use_status_v1/query.sql	2026-01-23 16:38:08.000000000 +0000
@@ -26,37 +26,41 @@
     DATE(submission_timestamp) AS submission_date,
     client_info.client_id,
     sample_id,
-    `moz-fx-data-shared-prod`.udf.mode_last(
-      ARRAY_AGG(app_version_major) OVER _window
+    mozfun.stats.mode_last(
+      ARRAY_AGG(app_version_major ORDER BY submission_timestamp ASC)
     ) AS app_version_major,
-    `moz-fx-data-shared-prod`.udf.mode_last(
-      ARRAY_AGG(app_version_minor) OVER _window
+    mozfun.stats.mode_last(
+      ARRAY_AGG(app_version_minor ORDER BY submission_timestamp ASC)
     ) AS app_version_minor,
-    `moz-fx-data-shared-prod`.udf.mode_last(
-      ARRAY_AGG(app_version_patch) OVER _window
+    mozfun.stats.mode_last(
+      ARRAY_AGG(app_version_patch ORDER BY submission_timestamp ASC)
     ) AS app_version_patch,
-    `moz-fx-data-shared-prod`.udf.mode_last(
-      ARRAY_AGG(normalized_channel) OVER _window
+    mozfun.stats.mode_last(
+      ARRAY_AGG(normalized_channel ORDER BY submission_timestamp ASC)
     ) AS normalized_channel,
-    `moz-fx-data-shared-prod`.udf.mode_last(
-      ARRAY_AGG(normalized_country_code) OVER _window
+    mozfun.stats.mode_last(
+      ARRAY_AGG(normalized_country_code ORDER BY submission_timestamp ASC)
     ) AS normalized_country_code,
-    `moz-fx-data-shared-prod`.udf.mode_last(ARRAY_AGG(normalized_os) OVER _window) AS normalized_os,
-    `moz-fx-data-shared-prod`.udf.mode_last(
-      ARRAY_AGG(normalized_os_version) OVER _window
+    mozfun.stats.mode_last(
+      ARRAY_AGG(normalized_os ORDER BY submission_timestamp ASC)
+    ) AS normalized_os,
+    mozfun.stats.mode_last(
+      ARRAY_AGG(normalized_os_version ORDER BY submission_timestamp ASC)
     ) AS normalized_os_version,
-    `moz-fx-data-shared-prod`.udf.mode_last(
-      ARRAY_AGG(is_bot_generated) OVER _window
+    mozfun.stats.mode_last(
+      ARRAY_AGG(is_bot_generated ORDER BY submission_timestamp ASC)
     ) AS is_bot_generated,
-    `moz-fx-data-shared-prod`.udf.mode_last(ARRAY_AGG(metadata.isp.name) OVER _window) AS isp_name,
-    `moz-fx-data-shared-prod`.udf.mode_last(
-      ARRAY_AGG(metrics.quantity.termsofuse_version) OVER _window
+    mozfun.stats.mode_last(
+      ARRAY_AGG(metadata.isp.`name` ORDER BY submission_timestamp ASC)
+    ) AS isp_name,
+    mozfun.stats.mode_last(
+      ARRAY_AGG(metrics.quantity.termsofuse_version ORDER BY submission_timestamp ASC)
     ) AS terms_of_use_version_accepted,
-    `moz-fx-data-shared-prod`.udf.mode_last(
-      ARRAY_AGG(metrics.datetime.termsofuse_date) OVER _window
+    mozfun.stats.mode_last(
+      ARRAY_AGG(metrics.datetime.termsofuse_date ORDER BY submission_timestamp ASC)
     ) AS terms_of_use_date_accepted,
-    `moz-fx-data-shared-prod`.udf.mode_last(
-      ARRAY_AGG(metrics.uuid.legacy_telemetry_client_id) OVER _window
+    mozfun.stats.mode_last(
+      ARRAY_AGG(metrics.uuid.legacy_telemetry_client_id ORDER BY submission_timestamp ASC)
     ) AS legacy_telemetry_client_id,  -- firefox_desktop exclusive field
   FROM
     `moz-fx-data-shared-prod.firefox_desktop.metrics`
@@ -64,21 +68,11 @@
     DATE(submission_timestamp) = @submission_date
     -- Adding a hard filter from when we want to start recording this data:
     AND DATE(submission_timestamp) >= "2025-06-24"
+    AND client_info.client_id IS NOT NULL
     AND app_version_major >= 142
     AND metrics.datetime.termsofuse_date IS NOT NULL
-  QUALIFY
-    ROW_NUMBER() OVER (PARTITION BY client_id ORDER BY submission_timestamp DESC) = 1
-  WINDOW
-    _window AS (
-      PARTITION BY
-        sample_id,
-        client_info.client_id
-      ORDER BY
-        submission_timestamp
-      ROWS BETWEEN
-        UNBOUNDED PRECEDING
-        AND UNBOUNDED FOLLOWING
-    )
+  GROUP BY
+    ALL
 )
 SELECT
   -- update entry if `terms_of_use_version_accepted` or `terms_of_use_date_accepted` value changes:
@@ -86,11 +80,11 @@
     _previous.client_id IS NULL
     OR (
       (_current.terms_of_use_version_accepted <> _previous.terms_of_use_version_accepted)
-      AND (_current.terms_of_use_date_accepted <> _previous.terms_of_use_date_accepted)
+      OR (_current.terms_of_use_date_accepted <> _previous.terms_of_use_date_accepted)
     ),
     _current,
     _previous
-  ).*
+  ).*,
 FROM
   _current
 FULL OUTER JOIN
diff -bur --no-dereference --new-file /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/firefox_ios_derived/terms_of_use_status_v1/query.sql /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/firefox_ios_derived/terms_of_use_status_v1/query.sql
--- /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/firefox_ios_derived/terms_of_use_status_v1/query.sql	2026-01-23 16:33:18.000000000 +0000
+++ /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/firefox_ios_derived/terms_of_use_status_v1/query.sql	2026-01-23 16:38:08.000000000 +0000
@@ -25,34 +25,42 @@
     DATE(submission_timestamp) AS submission_date,
     client_info.client_id,
     sample_id,
-    `moz-fx-data-shared-prod`.udf.mode_last(
-      ARRAY_AGG(app_version_major) OVER _window
+    mozfun.stats.mode_last(
+      ARRAY_AGG(app_version_major ORDER BY submission_timestamp ASC)
     ) AS app_version_major,
-    `moz-fx-data-shared-prod`.udf.mode_last(
-      ARRAY_AGG(app_version_minor) OVER _window
+    mozfun.stats.mode_last(
+      ARRAY_AGG(app_version_minor ORDER BY submission_timestamp ASC)
     ) AS app_version_minor,
-    `moz-fx-data-shared-prod`.udf.mode_last(
-      ARRAY_AGG(app_version_patch) OVER _window
+    mozfun.stats.mode_last(
+      ARRAY_AGG(app_version_patch ORDER BY submission_timestamp ASC)
     ) AS app_version_patch,
-    `moz-fx-data-shared-prod`.udf.mode_last(
-      ARRAY_AGG(normalized_channel) OVER _window
+    mozfun.stats.mode_last(
+      ARRAY_AGG(normalized_channel ORDER BY submission_timestamp ASC)
     ) AS normalized_channel,
-    `moz-fx-data-shared-prod`.udf.mode_last(
-      ARRAY_AGG(normalized_country_code) OVER _window
+    mozfun.stats.mode_last(
+      ARRAY_AGG(normalized_country_code ORDER BY submission_timestamp ASC)
     ) AS normalized_country_code,
-    `moz-fx-data-shared-prod`.udf.mode_last(ARRAY_AGG(normalized_os) OVER _window) AS normalized_os,
-    `moz-fx-data-shared-prod`.udf.mode_last(
-      ARRAY_AGG(normalized_os_version) OVER _window
+    mozfun.stats.mode_last(
+      ARRAY_AGG(normalized_os ORDER BY submission_timestamp ASC)
+    ) AS normalized_os,
+    mozfun.stats.mode_last(
+      ARRAY_AGG(normalized_os_version ORDER BY submission_timestamp ASC)
     ) AS normalized_os_version,
-    `moz-fx-data-shared-prod`.udf.mode_last(
-      ARRAY_AGG(is_bot_generated) OVER _window
+    mozfun.stats.mode_last(
+      ARRAY_AGG(is_bot_generated ORDER BY submission_timestamp ASC)
     ) AS is_bot_generated,
-    `moz-fx-data-shared-prod`.udf.mode_last(ARRAY_AGG(metadata.isp.name) OVER _window) AS isp_name,
-    `moz-fx-data-shared-prod`.udf.mode_last(
-      ARRAY_AGG(metrics.quantity.user_terms_of_use_version_accepted) OVER _window
+    mozfun.stats.mode_last(
+      ARRAY_AGG(metadata.isp.`name` ORDER BY submission_timestamp ASC)
+    ) AS isp_name,
+    mozfun.stats.mode_last(
+      ARRAY_AGG(
+        metrics.quantity.user_terms_of_use_version_accepted
+        ORDER BY
+          submission_timestamp ASC
+      )
     ) AS terms_of_use_version_accepted,
-    `moz-fx-data-shared-prod`.udf.mode_last(
-      ARRAY_AGG(metrics.datetime.user_terms_of_use_date_accepted) OVER _window
+    mozfun.stats.mode_last(
+      ARRAY_AGG(metrics.datetime.user_terms_of_use_date_accepted ORDER BY submission_timestamp ASC)
     ) AS terms_of_use_date_accepted,
   FROM
     `moz-fx-data-shared-prod.firefox_ios.metrics`
@@ -60,21 +68,11 @@
     DATE(submission_timestamp) = @submission_date
     -- Adding a hard filter from when we want to start recording this data:
     AND DATE(submission_timestamp) >= "2025-09-15"
+    AND client_info.client_id IS NOT NULL
     AND app_version_major >= 142
     AND metrics.datetime.user_terms_of_use_date_accepted IS NOT NULL
-  QUALIFY
-    ROW_NUMBER() OVER (PARTITION BY client_id ORDER BY submission_timestamp DESC) = 1
-  WINDOW
-    _window AS (
-      PARTITION BY
-        sample_id,
-        client_info.client_id
-      ORDER BY
-        submission_timestamp
-      ROWS BETWEEN
-        UNBOUNDED PRECEDING
-        AND UNBOUNDED FOLLOWING
-    )
+  GROUP BY
+    ALL
 )
 SELECT
   -- update entry if `terms_of_use_version_accepted` or `terms_of_use_date_accepted` value changes:
@@ -82,11 +80,11 @@
     _previous.client_id IS NULL
     OR (
       (_current.terms_of_use_version_accepted <> _previous.terms_of_use_version_accepted)
-      AND (_current.terms_of_use_date_accepted <> _previous.terms_of_use_date_accepted)
+      OR (_current.terms_of_use_date_accepted <> _previous.terms_of_use_date_accepted)
     ),
     _current,
     _previous
-  ).*
+  ).*,
 FROM
   _current
 FULL OUTER JOIN
diff -bur --no-dereference --new-file /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/org_mozilla_fenix/broken_site_report/metadata.yaml /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/org_mozilla_fenix/broken_site_report/metadata.yaml
--- /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/org_mozilla_fenix/broken_site_report/metadata.yaml	2026-01-23 16:33:17.000000000 +0000
+++ /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/org_mozilla_fenix/broken_site_report/metadata.yaml	2026-01-23 16:39:23.000000000 +0000
@@ -1,6 +1,14 @@
-friendly_name: Broken Site Report
+friendly_name: Historical Pings for `org-mozilla-fenix/broken-site-report`
 description: |-
-  Please provide a description for the query
+  A historical view of pings sent for the
+  `org-mozilla-fenix/broken-site-report`
+  document type.
+
+  This view is guaranteed to contain only complete days
+  (per `submission_timestamp`)
+  and to contain only one row per distinct `document_id` within a given date.
+
+  Clustering fields: `normalized_channel`, `sample_id`
 owners: []
 labels:
   authorized: true
diff -bur --no-dereference --new-file /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/org_mozilla_fenix/crash/metadata.yaml /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/org_mozilla_fenix/crash/metadata.yaml
--- /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/org_mozilla_fenix/crash/metadata.yaml	2026-01-23 16:33:17.000000000 +0000
+++ /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/org_mozilla_fenix/crash/metadata.yaml	2026-01-23 16:39:23.000000000 +0000
@@ -1,6 +1,14 @@
-friendly_name: Crash
+friendly_name: Historical Pings for `org-mozilla-fenix/crash`
 description: |-
-  Please provide a description for the query
+  A historical view of pings sent for the
+  `org-mozilla-fenix/crash`
+  document type.
+
+  This view is guaranteed to contain only complete days
+  (per `submission_timestamp`)
+  and to contain only one row per distinct `document_id` within a given date.
+
+  Clustering fields: `normalized_channel`, `sample_id`
 owners: []
 labels:
   authorized: true
diff -bur --no-dereference --new-file /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/org_mozilla_fenix_nightly/broken_site_report/metadata.yaml /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/org_mozilla_fenix_nightly/broken_site_report/metadata.yaml
--- /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/org_mozilla_fenix_nightly/broken_site_report/metadata.yaml	2026-01-23 16:33:17.000000000 +0000
+++ /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/org_mozilla_fenix_nightly/broken_site_report/metadata.yaml	2026-01-23 16:39:19.000000000 +0000
@@ -1,6 +1,14 @@
-friendly_name: Broken Site Report
+friendly_name: Historical Pings for `org-mozilla-fenix-nightly/broken-site-report`
 description: |-
-  Please provide a description for the query
+  A historical view of pings sent for the
+  `org-mozilla-fenix-nightly/broken-site-report`
+  document type.
+
+  This view is guaranteed to contain only complete days
+  (per `submission_timestamp`)
+  and to contain only one row per distinct `document_id` within a given date.
+
+  Clustering fields: `normalized_channel`, `sample_id`
 owners: []
 labels:
   authorized: true
diff -bur --no-dereference --new-file /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/org_mozilla_fenix_nightly/crash/metadata.yaml /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/org_mozilla_fenix_nightly/crash/metadata.yaml
--- /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/org_mozilla_fenix_nightly/crash/metadata.yaml	2026-01-23 16:33:17.000000000 +0000
+++ /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/org_mozilla_fenix_nightly/crash/metadata.yaml	2026-01-23 16:39:19.000000000 +0000
@@ -1,6 +1,14 @@
-friendly_name: Crash
+friendly_name: Historical Pings for `org-mozilla-fenix-nightly/crash`
 description: |-
-  Please provide a description for the query
+  A historical view of pings sent for the
+  `org-mozilla-fenix-nightly/crash`
+  document type.
+
+  This view is guaranteed to contain only complete days
+  (per `submission_timestamp`)
+  and to contain only one row per distinct `document_id` within a given date.
+
+  Clustering fields: `normalized_channel`, `sample_id`
 owners: []
 labels:
   authorized: true
diff -bur --no-dereference --new-file /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/org_mozilla_fennec_aurora/broken_site_report/metadata.yaml /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/org_mozilla_fennec_aurora/broken_site_report/metadata.yaml
--- /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/org_mozilla_fennec_aurora/broken_site_report/metadata.yaml	2026-01-23 16:33:17.000000000 +0000
+++ /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/org_mozilla_fennec_aurora/broken_site_report/metadata.yaml	2026-01-23 16:39:19.000000000 +0000
@@ -1,6 +1,14 @@
-friendly_name: Broken Site Report
+friendly_name: Historical Pings for `org-mozilla-fennec-aurora/broken-site-report`
 description: |-
-  Please provide a description for the query
+  A historical view of pings sent for the
+  `org-mozilla-fennec-aurora/broken-site-report`
+  document type.
+
+  This view is guaranteed to contain only complete days
+  (per `submission_timestamp`)
+  and to contain only one row per distinct `document_id` within a given date.
+
+  Clustering fields: `normalized_channel`, `sample_id`
 owners: []
 labels:
   authorized: true
diff -bur --no-dereference --new-file /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/org_mozilla_fennec_aurora/crash/metadata.yaml /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/org_mozilla_fennec_aurora/crash/metadata.yaml
--- /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/org_mozilla_fennec_aurora/crash/metadata.yaml	2026-01-23 16:33:17.000000000 +0000
+++ /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/org_mozilla_fennec_aurora/crash/metadata.yaml	2026-01-23 16:39:19.000000000 +0000
@@ -1,6 +1,14 @@
-friendly_name: Crash
+friendly_name: Historical Pings for `org-mozilla-fennec-aurora/crash`
 description: |-
-  Please provide a description for the query
+  A historical view of pings sent for the
+  `org-mozilla-fennec-aurora/crash`
+  document type.
+
+  This view is guaranteed to contain only complete days
+  (per `submission_timestamp`)
+  and to contain only one row per distinct `document_id` within a given date.
+
+  Clustering fields: `normalized_channel`, `sample_id`
 owners: []
 labels:
   authorized: true
diff -bur --no-dereference --new-file /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/org_mozilla_firefox/broken_site_report/metadata.yaml /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/org_mozilla_firefox/broken_site_report/metadata.yaml
--- /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/org_mozilla_firefox/broken_site_report/metadata.yaml	2026-01-23 16:33:17.000000000 +0000
+++ /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/org_mozilla_firefox/broken_site_report/metadata.yaml	2026-01-23 16:39:23.000000000 +0000
@@ -1,6 +1,14 @@
-friendly_name: Broken Site Report
+friendly_name: Historical Pings for `org-mozilla-firefox/broken-site-report`
 description: |-
-  Please provide a description for the query
+  A historical view of pings sent for the
+  `org-mozilla-firefox/broken-site-report`
+  document type.
+
+  This view is guaranteed to contain only complete days
+  (per `submission_timestamp`)
+  and to contain only one row per distinct `document_id` within a given date.
+
+  Clustering fields: `normalized_channel`, `sample_id`
 owners: []
 labels:
   authorized: true
diff -bur --no-dereference --new-file /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/org_mozilla_firefox/crash/metadata.yaml /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/org_mozilla_firefox/crash/metadata.yaml
--- /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/org_mozilla_firefox/crash/metadata.yaml	2026-01-23 16:33:17.000000000 +0000
+++ /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/org_mozilla_firefox/crash/metadata.yaml	2026-01-23 16:39:23.000000000 +0000
@@ -1,6 +1,14 @@
-friendly_name: Crash
+friendly_name: Historical Pings for `org-mozilla-firefox/crash`
 description: |-
-  Please provide a description for the query
+  A historical view of pings sent for the
+  `org-mozilla-firefox/crash`
+  document type.
+
+  This view is guaranteed to contain only complete days
+  (per `submission_timestamp`)
+  and to contain only one row per distinct `document_id` within a given date.
+
+  Clustering fields: `normalized_channel`, `sample_id`
 owners: []
 labels:
   authorized: true
diff -bur --no-dereference --new-file /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/org_mozilla_firefox_beta/broken_site_report/metadata.yaml /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/org_mozilla_firefox_beta/broken_site_report/metadata.yaml
--- /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/org_mozilla_firefox_beta/broken_site_report/metadata.yaml	2026-01-23 16:33:17.000000000 +0000
+++ /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/org_mozilla_firefox_beta/broken_site_report/metadata.yaml	2026-01-23 16:39:21.000000000 +0000
@@ -1,6 +1,14 @@
-friendly_name: Broken Site Report
+friendly_name: Historical Pings for `org-mozilla-firefox-beta/broken-site-report`
 description: |-
-  Please provide a description for the query
+  A historical view of pings sent for the
+  `org-mozilla-firefox-beta/broken-site-report`
+  document type.
+
+  This view is guaranteed to contain only complete days
+  (per `submission_timestamp`)
+  and to contain only one row per distinct `document_id` within a given date.
+
+  Clustering fields: `normalized_channel`, `sample_id`
 owners: []
 labels:
   authorized: true
diff -bur --no-dereference --new-file /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/org_mozilla_firefox_beta/crash/metadata.yaml /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/org_mozilla_firefox_beta/crash/metadata.yaml
--- /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/org_mozilla_firefox_beta/crash/metadata.yaml	2026-01-23 16:33:17.000000000 +0000
+++ /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/org_mozilla_firefox_beta/crash/metadata.yaml	2026-01-23 16:39:21.000000000 +0000
@@ -1,6 +1,14 @@
-friendly_name: Crash
+friendly_name: Historical Pings for `org-mozilla-firefox-beta/crash`
 description: |-
-  Please provide a description for the query
+  A historical view of pings sent for the
+  `org-mozilla-firefox-beta/crash`
+  document type.
+
+  This view is guaranteed to contain only complete days
+  (per `submission_timestamp`)
+  and to contain only one row per distinct `document_id` within a given date.
+
+  Clustering fields: `normalized_channel`, `sample_id`
 owners: []
 labels:
   authorized: true

Link to full diff

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@scholtzan scholtzan scholtzan approved these changes

@soGaussian soGaussian soGaussian approved these changes

Assignees

@kik-kik kik-kik

Labels

enhancement New feature or request

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@kik-kik @dataops-ci-bot @scholtzan @soGaussian