Skip to content

Commit 74fb2e2

Browse files
committed
Clamp event timestamps to i64::MAX
BigQuery's integer type is a signed 64-bit value, while we collect unsigned ones. Timestamps are in milliseconds, so 9223372036854775807 (i64::MAX) ms is about 4.5 billion years, or roughly 1/15 of the age of earth. I think we should be fine with that for a little while longer. So any such value _should_ not happen. But it does in 0.002% of pings and thus data ends up in `additional_properties` instead of where it belongs. By clamping we might reduce that %age further. An event with that timestamp is most likely still not useful, but maybe the others are? One more problem is: Should we report this as an issue? If so how? Right now we don't have the APIs to tie that back to a specific event other then the one we own. But then our error types don't express the right things.
1 parent 62650f9 commit 74fb2e2

File tree

5 files changed

+103
-4
lines changed

5 files changed

+103
-4
lines changed

CHANGELOG.md

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -4,6 +4,8 @@
44

55
* General
66
* BUGFIX: Correct `glean.database.rkv_load_error`'s category and `glean.upload.discarded_exceeding_pings_size`'s name ([bug 2009475](https://bugzilla.mozilla.org/show_bug.cgi?id=2009475))
7+
* Event timestamps are now always clamped to the range of a signed 64-bit integer.
8+
An overflow is recorded in the new metric `glean.error.event_timestamp_clamped` in case this happens ([#3308](https://github.com/mozilla/glean/pull/3308)).
79

810
# v66.3.0 (2025-12-19)
911

docs/user/user/collected-metrics/metrics.md

Lines changed: 8 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -4,7 +4,7 @@
44

55
> If you are looking for the metrics collected by Glean.js,
66
> refer to the documentation over on the [`@mozilla/glean.js`](https://github.com/mozilla/glean.js/blob/main/docs/reference/metrics.md) repository.
7-
<!-- AUTOGENERATED BY glean_parser v18.1.1. DO NOT EDIT. -->
7+
<!-- AUTOGENERATED BY glean_parser v18.2.0. DO NOT EDIT. -->
88
99
# Metrics
1010

@@ -17,7 +17,7 @@ This means you might have to go searching through the dependency tree to get a f
1717
- [all-pings](#all-pings)
1818
- [baseline](#baseline)
1919
- [deletion-request](#deletion-request)
20-
- [glean_internal_info](#glean_internal_info)
20+
- [glean_client_info](#glean_client_info)
2121
- [health](#health)
2222
- [metrics](#metrics)
2323

@@ -121,7 +121,7 @@ All Glean pings contain built-in metrics in the [`ping_info`](https://mozilla.gi
121121

122122
This ping contains no metrics.
123123

124-
## glean_internal_info
124+
## glean_client_info
125125

126126
All Glean pings contain built-in metrics in the [`ping_info`](https://mozilla.github.io/glean/book/user/pings/index.html#the-ping_info-section) and [`client_info`](https://mozilla.github.io/glean/book/user/pings/index.html#the-client_info-section) sections.
127127

@@ -143,6 +143,9 @@ metric information. The `health` ping is automatically sent when the
143143
application calls Glean initialize before any operations are done on
144144
the data path with a reason of `pre_init`.
145145

146+
Previously a second ping with reason `post_init` was sent.
147+
This reason has been removed in Glean v66.2.0.
148+
146149

147150
This ping includes the [client id](https://mozilla.github.io/glean/book/user/pings/index.html#the-client_info-section).
148151

@@ -170,6 +173,7 @@ In addition to those built-in metrics, the following metrics are added to the pi
170173
| glean.database.rkv_load_error |[string](https://mozilla.github.io/glean/book/user/metrics/string.html) |If there was an error loading the RKV database, record it. |[Bug 1815253](https://bugzilla.mozilla.org/show_bug.cgi?id=1815253)||never |1 |
171174
| glean.database.size |[memory_distribution](https://mozilla.github.io/glean/book/user/metrics/memory_distribution.html) |The size of the database file at startup. |[Bug 1656589](https://bugzilla.mozilla.org/show_bug.cgi?id=1656589#c7)||never |1 |
172175
| glean.database.write_time |[timing_distribution](https://mozilla.github.io/glean/book/user/metrics/timing_distribution.html) |The time it takes for a write-commit for the Glean database. |[Bug 1896193](https://bugzilla.mozilla.org/show_bug.cgi?id=1896193#c4)||never |1 |
176+
| glean.error.event_timestamp_clamped |[counter](https://mozilla.github.io/glean/book/user/metrics/counter.html) |The number of times we had to clamp an event timestamp for exceeding the range of a signed 64-bit integer (9223372036854775807). |[Bug 1873482](https://bugzilla.mozilla.org/show_bug.cgi?id=1873482)||2026-06-30 |1 |
173177
| glean.error.io |[counter](https://mozilla.github.io/glean/book/user/metrics/counter.html) |The number of times we encountered an IO error when writing a pending ping to disk. |[Bug 1686233](https://bugzilla.mozilla.org/show_bug.cgi?id=1686233#c2)||never |1 |
174178
| glean.error.preinit_tasks_overflow |[counter](https://mozilla.github.io/glean/book/user/metrics/counter.html) |The number of tasks that overflowed the pre-initialization buffer. Only sent if the buffer ever overflows. In Version 0 this reported the total number of tasks enqueued. |[Bug 1609482](https://bugzilla.mozilla.org/show_bug.cgi?id=1609482#c3)||never |1 |
175179
| glean.health.data_directory_info |[object](https://mozilla.github.io/glean/book/user/metrics/object.html) |Information about the data directories and files used by FOG. Structure is an array of objects, each containing the following properties: - `dir_name`: The name of the directory. This is the subdirectory name relative to the FOG data directory and should only include "db", "events", and "pending_pings". - `dir_exists`: Whether the directory exists. This should only be false on the first run of FOG, or if the directory was deleted. - `dir_created`: The creation time of the directory, in seconds since the unix epoch. If the directory does not exist, this will be `null` and if the time cannot be determined, it will default to `0`. - `dir_modified`: The last modification time of the directory, in seconds since the unix epoch. If the directory does not exist, this will be `null` and if the time cannot be determined, it will default to `0`. - `file_count`: The number of files in the directory. If the directory does not exist, this will be `0`. - `error_message`: If there was an error accessing the directory, this will contain a brief description of the error. If there was no error, this will be null. - `files`: An array of objects, each containing: - `file_name`: The name of the file. Could be `data.safe.bin`, `events.safe.bin`, or A UUID representing the doc-id of a pending ping. - `file_created`: The creation time of the file, in seconds since the epoch. If the file does not exist, this will be `null` and if the time cannot be determined, it will default to `0`. - `file_modified`: The last modification time of the file, in seconds since the epoch. If the file does not exist, this will be `null` and if the time cannot be determined, it will default to `0`. - `file_size`: The size of the file in bytes. This can be just about any size but a `0` value indicates the file is empty. - `error_message`: If there was an error accessing the file, this will contain a brief description of the error. If there was no error, this will be null. |[Bug 1982711](https://bugzilla.mozilla.org/show_bug.cgi?id=1982711#c3)||never |1 |
@@ -255,5 +259,5 @@ In addition to those built-in metrics, the following metrics are added to the pi
255259

256260
Data categories are [defined here](https://wiki.mozilla.org/Firefox/Data_Collection).
257261

258-
<!-- AUTOGENERATED BY glean_parser v18.1.1. DO NOT EDIT. -->
262+
<!-- AUTOGENERATED BY glean_parser v18.2.0. DO NOT EDIT. -->
259263

glean-core/metrics.yaml

Lines changed: 18 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -649,6 +649,24 @@ glean.error:
649649
650650
expires: never
651651

652+
event_timestamp_clamped:
653+
type: counter
654+
description: |
655+
The number of times we had to clamp an event timestamp
656+
for exceeding the range of a signed 64-bit integer (9223372036854775807).
657+
send_in_pings:
658+
- health
659+
bugs:
660+
- https://bugzilla.mozilla.org/show_bug.cgi?id=1873482
661+
data_reviews:
662+
- https://bugzilla.mozilla.org/show_bug.cgi?id=1873482
663+
data_sensitivity:
664+
- technical
665+
notification_emails:
666+
667+
668+
expires: 2026-06-30
669+
652670
glean.upload:
653671
ping_upload_failure:
654672
type: labeled_counter

glean-core/src/event_database/mod.rs

Lines changed: 62 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -560,6 +560,22 @@ impl EventDatabase {
560560
// Let's fix cur_ec up and hope this isn't a sign something big is broken.
561561
cur_ec = execution_counter;
562562
}
563+
564+
// event timestamp is a `u64`, but BigQuery uses `i64` (signed!) everywhere. Let's clamp the value to make
565+
// sure we stay within bounds.
566+
if event.event.timestamp > i64::MAX as u64 {
567+
glean
568+
.additional_metrics
569+
.event_timestamp_clamped
570+
.add_sync(glean, 1);
571+
log::warn!(
572+
"Calculated event timestamp was too high. Got: {}, max: {}",
573+
event.event.timestamp,
574+
i64::MAX,
575+
);
576+
event.event.timestamp = event.event.timestamp.clamp(0, i64::MAX as u64);
577+
}
578+
563579
if highest_ts > event.event.timestamp {
564580
// Even though we sorted everything, something in the
565581
// execution_counter or glean.startup.date math went awry.
@@ -1357,4 +1373,50 @@ mod test {
13571373
)
13581374
.is_err());
13591375
}
1376+
1377+
#[test]
1378+
fn normalize_store_clamps_timestamp() {
1379+
let (glean, _dir) = new_glean(None);
1380+
1381+
let store_name = "store-name";
1382+
let event = RecordedEvent {
1383+
category: "category".into(),
1384+
name: "name".into(),
1385+
..Default::default()
1386+
};
1387+
1388+
let timestamps = [
1389+
0,
1390+
(i64::MAX / 2) as u64,
1391+
i64::MAX as _,
1392+
(i64::MAX as u64) + 1,
1393+
];
1394+
let mut store = timestamps
1395+
.into_iter()
1396+
.map(|timestamp| StoredEvent {
1397+
event: RecordedEvent {
1398+
timestamp,
1399+
..event.clone()
1400+
},
1401+
execution_counter: None,
1402+
})
1403+
.collect();
1404+
1405+
let glean_start_time = glean.start_time();
1406+
glean
1407+
.event_storage()
1408+
.normalize_store(&glean, store_name, &mut store, glean_start_time);
1409+
assert_eq!(4, store.len());
1410+
1411+
assert_eq!(0, store[0].event.timestamp);
1412+
assert_eq!((i64::MAX / 2) as u64, store[1].event.timestamp);
1413+
assert_eq!((i64::MAX as u64), store[2].event.timestamp);
1414+
assert_eq!((i64::MAX as u64), store[3].event.timestamp);
1415+
1416+
let error_count = glean
1417+
.additional_metrics
1418+
.event_timestamp_clamped
1419+
.get_value(&glean, "health");
1420+
assert_eq!(Some(1), error_count);
1421+
}
13601422
}

glean-core/src/internal_metrics.rs

Lines changed: 13 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -40,6 +40,10 @@ pub struct AdditionalMetrics {
4040
/// An experimentation identifier derived and provided by the application
4141
/// for the purpose of experimentation enrollment.
4242
pub experimentation_id: StringMetric,
43+
44+
/// The number of times we had to clamp an event timestamp
45+
/// for exceeding the range of a signed 64-bit integer (9223372036854775807).
46+
pub event_timestamp_clamped: CounterMetric,
4347
}
4448

4549
impl CoreMetrics {
@@ -198,6 +202,15 @@ impl AdditionalMetrics {
198202
disabled: false,
199203
dynamic_label: None,
200204
}),
205+
206+
event_timestamp_clamped: CounterMetric::new(CommonMetricData {
207+
name: "event_timestamp_clamped".into(),
208+
category: "glean.error".into(),
209+
send_in_pings: vec!["health".into()],
210+
lifetime: Lifetime::Ping,
211+
disabled: false,
212+
dynamic_label: None,
213+
}),
201214
}
202215
}
203216
}

0 commit comments

Comments
 (0)