refactor: align percentile semantics across metrics

On page 19 of the [IQB report 2025](
https://www.measurementlab.net/publications/IQB_report_2025.pdf)
we read:

> IQB uses the 95th percentile of a given
> dataset to evaluate a given metric. In this
> context, the 95th percentile is the value
> below which 95% of the observed
> measurements fall, which effectively
> captures the upper bound of a typical user
> experience while excluding extreme
> outliers. For example, to assess whether a
> region meets the network tier’s packet loss
> criteria for high-quality gaming, IQB
> calculates the 95th percentile of packet
> loss measurements collected from users in
> that region. The value is then compared to
> the predefined threshold. 

The formula example (page 33) shows taking the 95p of
metrics without considerations regarding polarity.

However, to produce consistent results, we need to take
the polarity into account when taking the 95p.

Let us illustrate the polarity issue with an example. We
assume that the following holds in a specific ISP:

(i) the 95p of latency being 10 ms means that 95% of
samples have 10 ms or less

(ii) the 95p of download speed being 22 Mbit/s means that
95% of samples have 22 Mbit/s or less

Let us also assume that we're evaluating online gaming and
that online gaming typically needs:

(a) download speed >= 20 Mbit/s

(b) latency <= 15 ms

The latency percentile allows us to say that "most
samples" (95%) in the given ISP show a latency (10 ms
or better) lower than the required latency (15 ms).

The speed percentile allows us to say that "few samples"
(5%) in the given ISP show a download speed (22 Mbit/s
or better) greater than the required one (20 Mbit/s).

So, is online gaming possible in the ISP? The answer
seems nonconclusive because of the imbalance in the
samples we are considering and the different polarity
between latency (higher is worse) and speed (higher
is better).

On paper, a better solution is to say "okay, for the
speed, instead, we consider the 5p". Now, let us assume:

(iii) the 5p of download speed being 21 Mbit/s means that
5% of samples have 21 Mbit/s or less

Based on this we can say that 95% of users have 21 Mbit/s
or more. Now, it's possible to write a statement regarding
the download speed concerning "most samples".

This allows us to conclude "most samples indicate that
users can play online with this ISP".

How to translate this into the actual code?

**Approach I**: modify `cache.py` so that, when we request
`percentile=X` we actually take the complementary percentile
for latency and loss (or for download and upload speed).

This change aligns the polarity and allows us to answer
questions using uniform sample sizes. It is basically
equivalent to what we manually did above.

**Approach II**: swap the labels for latency and loss (or for
download speed and upload speed) when querying BigQuery.

This means `cache.py` uniformly accesses `percentile=X` with
the understanding that 2/4 of the labels are swapped.

The second approach is more robust because it guarantees that,
if `percentile=X` is there then also the complementary
percentile is there. In both cases, people reading the code
will need to be aware of the polarity anyway.

Based on a discussion with @sermpezis, I am going to swap the
labels for latency and packet loss. The actual swapping operation
is anyway irrelevant (it's mostly a matter of convention) and
what matters is that we're aligning the polarity.

The meaning of the approach we are choosing is that 95p is the
cutoff where 95% of users have worse performance for the definition
of worse that is implied by the metric (e.g., lower speed or
higher latency). Obviously, it also holds the opposite, that is,
that 5% of users have better performance.

In conclusion, the implemented change aligns the sample size so
that the same percentile label picked up by `cache.py` allows
to make comparable statements with respect to better/worse.

Author: bassosimone

data/br_2024_10.json

@@ -10,48 +10,48 @@
   },
   "metrics": {
     "download_throughput_mbps": {
-      "p1": 0.15979623373499155,
-      "p5": 0.9501991252036766,
-      "p10": 3.101174869710966,
-      "p25": 15.0340700432778,
-      "p50": 51.9831305263177,
-      "p75": 158.38962702858973,
-      "p90": 330.3352983503099,
-      "p95": 456.0950392154999,
-      "p99": 696.5613392781584
+      "p1": 0.1607952665485899,
+      "p5": 0.9527639351896139,
+      "p10": 3.0995791948048126,
+      "p25": 15.027442481667213,
+      "p50": 52.05003588762424,
+      "p75": 157.9133050283035,
+      "p90": 330.56865817664186,
+      "p95": 456.85148065886335,
+      "p99": 699.4150527600466
     },
     "upload_throughput_mbps": {
-      "p1": 0.042563080079753776,
-      "p5": 0.07560071683921148,
-      "p10": 0.08980854096320207,
-      "p25": 5.545812099052701,
-      "p50": 30.78175191467136,
-      "p75": 88.37694460346944,
-      "p90": 181.64033113619195,
-      "p95": 255.97876412741525,
-      "p99": 394.3416893812533
+      "p1": 0.04271425174361129,
+      "p5": 0.07560308380999751,
+      "p10": 0.08981165041087474,
+      "p25": 5.53270523375538,
+      "p50": 30.774797230694343,
+      "p75": 88.4152374160525,
+      "p90": 181.77498611764298,
+      "p95": 255.68482970928804,
+      "p99": 393.05831520691316
     },
     "latency_ms": {
-      "p1": 1.394,
-      "p5": 3.637,
-      "p10": 4.958,
-      "p25": 9.079,
+      "p1": 274.874,
+      "p5": 234.07,
+      "p10": 184.463,
+      "p25": 52.107,
       "p50": 19.953,
-      "p75": 52.065,
-      "p90": 184.738,
-      "p95": 234.072,
-      "p99": 273.0
+      "p75": 9.072,
+      "p90": 4.958,
+      "p95": 3.64,
+      "p99": 1.394
     },
     "packet_loss": {
-      "p1": 0.0,
-      "p5": 0.0,
-      "p10": 0.0,
-      "p25": 1.1042755272820004e-05,
-      "p50": 0.004822712745559209,
-      "p75": 0.05811090765473097,
-      "p90": 0.13649207990035975,
-      "p95": 0.1987869577393624,
-      "p99": 0.3652163739953438
+      "p1": 0.3683998584485473,
+      "p5": 0.1989762612035137,
+      "p10": 0.13678163876238772,
+      "p25": 0.05818004802054541,
+      "p50": 0.004794188738463603,
+      "p75": 1.099611491732316e-05,
+      "p90": 0.0,
+      "p95": 0.0,
+      "p99": 0.0
     }
   }
 }

data/de_2024_10.json

@@ -10,48 +10,48 @@
   },
   "metrics": {
     "download_throughput_mbps": {
-      "p1": 0.22367850581560372,
-      "p5": 1.262769802856182,
-      "p10": 3.4166592054870026,
-      "p25": 13.817824595534129,
-      "p50": 45.24430302103892,
-      "p75": 100.56946051210859,
-      "p90": 248.78115747983244,
-      "p95": 377.8657642766346,
-      "p99": 741.7983223940372
+      "p1": 0.2225810589129801,
+      "p5": 1.2528819154630235,
+      "p10": 3.4209413897406646,
+      "p25": 13.810432910425352,
+      "p50": 45.27405796172478,
+      "p75": 100.56551588414163,
+      "p90": 248.79192117938703,
+      "p95": 377.46883271114353,
+      "p99": 741.7160739730066
     },
     "upload_throughput_mbps": {
-      "p1": 0.04798033204768874,
-      "p5": 0.07565187888251705,
-      "p10": 0.19852741925194242,
-      "p25": 3.5715003423978087,
-      "p50": 17.172955392453527,
-      "p75": 36.63458526768415,
-      "p90": 53.192909502396375,
-      "p95": 101.34444079000329,
-      "p99": 285.7324202068485
+      "p1": 0.04888864801257374,
+      "p5": 0.07565371155122488,
+      "p10": 0.20144847741476402,
+      "p25": 3.571516158290839,
+      "p50": 17.180642660658165,
+      "p75": 36.60604286113131,
+      "p90": 53.23036640523673,
+      "p95": 101.60189503128285,
+      "p99": 285.6942608280348
     },
     "latency_ms": {
-      "p1": 0.438,
-      "p5": 3.433,
-      "p10": 6.787,
-      "p25": 11.589,
-      "p50": 17.712,
-      "p75": 26.382,
-      "p90": 38.489,
-      "p95": 57.061,
-      "p99": 305.85
+      "p1": 304.0,
+      "p5": 57.103,
+      "p10": 38.461,
+      "p25": 26.383,
+      "p50": 17.711,
+      "p75": 11.597,
+      "p90": 6.791,
+      "p95": 3.482,
+      "p99": 0.438
     },
     "packet_loss": {
-      "p1": 0.0,
-      "p5": 0.0,
-      "p10": 0.0,
-      "p25": 0.0,
-      "p50": 0.00034573047467282084,
-      "p75": 0.016581558328885995,
-      "p90": 0.07073353719313655,
-      "p95": 0.11517449630011735,
-      "p99": 0.2521127443846117
+      "p1": 0.2521662550269682,
+      "p5": 0.11532512559748044,
+      "p10": 0.07065396520089343,
+      "p25": 0.016604288718428745,
+      "p50": 0.00034525252746869345,
+      "p75": 0.0,
+      "p90": 0.0,
+      "p95": 0.0,
+      "p99": 0.0
     }
   }
 }

data/query_downloads.sql

@@ -1,6 +1,37 @@
 SELECT
     client.Geo.CountryCode as country_code,
     COUNT(*) as sample_count,
+
+    -- ============================================================================
+    -- PERCENTILE LABELING CONVENTION FOR IQB QUALITY ASSESSMENT
+    -- ============================================================================
+    -- IQB Report (p19): "IQB uses the 95th percentile of a given dataset to
+    -- evaluate a given metric."
+    --
+    -- This implementation follows the report literally: use p95 for ALL metrics.
+    --
+    -- For "higher is better" metrics (throughput):
+    --   - Raw p95 = "95% of users have ≤ X Mbit/s"
+    --   - Label: OFFSET(95) → download_p95 (standard statistical definition)
+    --   - Interpretation: top ~5% of users have > p95 throughput
+    --
+    -- For "lower is better" metrics (latency, packet loss):
+    --   - Raw p95 = "95% of users have ≤ X ms latency" (worst-case typical)
+    --   - We want p95 to represent best-case typical (to match throughput semantics)
+    --   - Solution: Invert labels - use raw p5 labeled as p95
+    --   - Label: OFFSET(5) → latency_p95 (inverted!)
+    --   - Interpretation: top ~5% of users (best latency) have < p95
+    --
+    -- Result: Uniform comparison logic where p95 always means "typical best
+    -- performance" rather than "typical worst performance"
+    --
+    -- NOTE: This creates semantics where checking p95 thresholds asks
+    -- "Can the top ~5% of users perform this use case?" - empirical validation
+    -- against real data will determine if this interpretation is appropriate.
+    -- ============================================================================
+
+    -- Download throughput (higher is better - NO INVERSION)
+    -- Standard percentile labels matching statistical definition
     APPROX_QUANTILES(a.MeanThroughputMbps, 100)[OFFSET(1)] as download_p1,
     APPROX_QUANTILES(a.MeanThroughputMbps, 100)[OFFSET(5)] as download_p5,
     APPROX_QUANTILES(a.MeanThroughputMbps, 100)[OFFSET(10)] as download_p10,
@@ -10,24 +41,32 @@ SELECT
     APPROX_QUANTILES(a.MeanThroughputMbps, 100)[OFFSET(90)] as download_p90,
     APPROX_QUANTILES(a.MeanThroughputMbps, 100)[OFFSET(95)] as download_p95,
     APPROX_QUANTILES(a.MeanThroughputMbps, 100)[OFFSET(99)] as download_p99,
-    APPROX_QUANTILES(a.MinRTT, 100)[OFFSET(1)] as latency_p1,
-    APPROX_QUANTILES(a.MinRTT, 100)[OFFSET(5)] as latency_p5,
-    APPROX_QUANTILES(a.MinRTT, 100)[OFFSET(10)] as latency_p10,
-    APPROX_QUANTILES(a.MinRTT, 100)[OFFSET(25)] as latency_p25,
+
+    -- Latency/MinRTT (lower is better - INVERTED LABELS!)
+    -- ⚠️  OFFSET(99) = worst latency = top 1% worst users → labeled as p1
+    -- ⚠️  OFFSET(5) = 5th percentile = best ~5% of users → labeled as p95
+    APPROX_QUANTILES(a.MinRTT, 100)[OFFSET(99)] as latency_p1,
+    APPROX_QUANTILES(a.MinRTT, 100)[OFFSET(95)] as latency_p5,
+    APPROX_QUANTILES(a.MinRTT, 100)[OFFSET(90)] as latency_p10,
+    APPROX_QUANTILES(a.MinRTT, 100)[OFFSET(75)] as latency_p25,
     APPROX_QUANTILES(a.MinRTT, 100)[OFFSET(50)] as latency_p50,
-    APPROX_QUANTILES(a.MinRTT, 100)[OFFSET(75)] as latency_p75,
-    APPROX_QUANTILES(a.MinRTT, 100)[OFFSET(90)] as latency_p90,
-    APPROX_QUANTILES(a.MinRTT, 100)[OFFSET(95)] as latency_p95,
-    APPROX_QUANTILES(a.MinRTT, 100)[OFFSET(99)] as latency_p99,
-    APPROX_QUANTILES(a.LossRate, 100)[OFFSET(1)] as loss_p1,
-    APPROX_QUANTILES(a.LossRate, 100)[OFFSET(5)] as loss_p5,
-    APPROX_QUANTILES(a.LossRate, 100)[OFFSET(10)] as loss_p10,
-    APPROX_QUANTILES(a.LossRate, 100)[OFFSET(25)] as loss_p25,
+    APPROX_QUANTILES(a.MinRTT, 100)[OFFSET(25)] as latency_p75,
+    APPROX_QUANTILES(a.MinRTT, 100)[OFFSET(10)] as latency_p90,
+    APPROX_QUANTILES(a.MinRTT, 100)[OFFSET(5)] as latency_p95,
+    APPROX_QUANTILES(a.MinRTT, 100)[OFFSET(1)] as latency_p99,
+
+    -- Packet Loss Rate (lower is better - INVERTED LABELS!)
+    -- ⚠️  OFFSET(99) = worst loss = top 1% worst users → labeled as p1
+    -- ⚠️  OFFSET(5) = 5th percentile = best ~5% of users → labeled as p95
+    APPROX_QUANTILES(a.LossRate, 100)[OFFSET(99)] as loss_p1,
+    APPROX_QUANTILES(a.LossRate, 100)[OFFSET(95)] as loss_p5,
+    APPROX_QUANTILES(a.LossRate, 100)[OFFSET(90)] as loss_p10,
+    APPROX_QUANTILES(a.LossRate, 100)[OFFSET(75)] as loss_p25,
     APPROX_QUANTILES(a.LossRate, 100)[OFFSET(50)] as loss_p50,
-    APPROX_QUANTILES(a.LossRate, 100)[OFFSET(75)] as loss_p75,
-    APPROX_QUANTILES(a.LossRate, 100)[OFFSET(90)] as loss_p90,
-    APPROX_QUANTILES(a.LossRate, 100)[OFFSET(95)] as loss_p95,
-    APPROX_QUANTILES(a.LossRate, 100)[OFFSET(99)] as loss_p99
+    APPROX_QUANTILES(a.LossRate, 100)[OFFSET(25)] as loss_p75,
+    APPROX_QUANTILES(a.LossRate, 100)[OFFSET(10)] as loss_p90,
+    APPROX_QUANTILES(a.LossRate, 100)[OFFSET(5)] as loss_p95,
+    APPROX_QUANTILES(a.LossRate, 100)[OFFSET(1)] as loss_p99
 FROM
     `measurement-lab.ndt.unified_downloads`
 WHERE

data/query_uploads.sql

@@ -1,6 +1,23 @@
 SELECT
     client.Geo.CountryCode as country_code,
     COUNT(*) as sample_count,
+
+    -- ============================================================================
+    -- PERCENTILE LABELING CONVENTION FOR IQB QUALITY ASSESSMENT
+    -- ============================================================================
+    -- IQB Report (p19): "IQB uses the 95th percentile of a given dataset to
+    -- evaluate a given metric."
+    --
+    -- This implementation follows the report literally: use p95 for ALL metrics.
+    --
+    -- Upload throughput is "higher is better", so we use standard percentile
+    -- labels (no inversion).
+    --
+    -- See query_downloads.sql for detailed explanation and rationale.
+    -- ============================================================================
+
+    -- Upload throughput (higher is better - NO INVERSION)
+    -- Standard percentile labels matching statistical definition
     APPROX_QUANTILES(a.MeanThroughputMbps, 100)[OFFSET(1)] as upload_p1,
     APPROX_QUANTILES(a.MeanThroughputMbps, 100)[OFFSET(5)] as upload_p5,
     APPROX_QUANTILES(a.MeanThroughputMbps, 100)[OFFSET(10)] as upload_p10,

data/us_2024_10.json

@@ -10,48 +10,48 @@
   },
   "metrics": {
     "download_throughput_mbps": {
-      "p1": 0.37354810526833476,
-      "p5": 2.7494108827310177,
-      "p10": 7.6575433038007406,
-      "p25": 29.94873577502137,
-      "p50": 96.36533017831101,
-      "p75": 268.1810327939917,
-      "p90": 474.1768162996085,
-      "p95": 625.4494125653449,
-      "p99": 893.2782851912168
+      "p1": 0.3812327371493097,
+      "p5": 2.7476349376135296,
+      "p10": 7.638638876969803,
+      "p25": 29.89884264277766,
+      "p50": 96.2643838797045,
+      "p75": 268.4085597033553,
+      "p90": 474.14118932768235,
+      "p95": 626.3132682965055,
+      "p99": 893.0568504937463
     },
     "upload_throughput_mbps": {
-      "p1": 0.06279911698366483,
-      "p5": 0.15105079102447938,
-      "p10": 1.0130561597157441,
-      "p25": 8.030055616329323,
-      "p50": 20.95814566696693,
-      "p75": 65.73945359925672,
-      "p90": 223.9767416770114,
-      "p95": 370.4336035390081,
-      "p99": 813.7319533731953
+      "p1": 0.06257042412674083,
+      "p5": 0.15144845324010167,
+      "p10": 0.9992760254839029,
+      "p25": 8.03213984894271,
+      "p50": 20.98046809727222,
+      "p75": 65.669501568909,
+      "p90": 224.29692902729298,
+      "p95": 368.91185081459395,
+      "p99": 819.839280930373
     },
     "latency_ms": {
-      "p1": 0.16,
-      "p5": 0.808,
-      "p10": 2.886,
-      "p25": 7.778,
-      "p50": 16.124,
-      "p75": 30.0,
-      "p90": 51.303,
-      "p95": 80.55,
-      "p99": 251.545
+      "p1": 255.993,
+      "p5": 80.759,
+      "p10": 51.2,
+      "p25": 30.0,
+      "p50": 16.119,
+      "p75": 7.783,
+      "p90": 2.895,
+      "p95": 0.804,
+      "p99": 0.161
     },
     "packet_loss": {
-      "p1": 0.0,
-      "p5": 0.0,
-      "p10": 0.0,
-      "p25": 0.0,
-      "p50": 0.000516724336793541,
-      "p75": 0.019090240380880846,
-      "p90": 0.07332944466732425,
-      "p95": 0.12018590164702943,
-      "p99": 0.253111989432024
+      "p1": 0.2517569864889713,
+      "p5": 0.11998957375627733,
+      "p10": 0.07340665854872248,
+      "p25": 0.019064946948168876,
+      "p50": 0.0005185937475477769,
+      "p75": 0.0,
+      "p90": 0.0,
+      "p95": 0.0,
+      "p99": 0.0
     }
   }
 }