Skip to content
This repository was archived by the owner on Jan 6, 2026. It is now read-only.

Commit e724243

Browse files
pyropyjuliangruberbajtos
authored
Spark TTFB Stats (#14)
* Add TTFB chart to provider * Rename from ttfb to more generic name * Add ttfb chart to main dashboard * Remove console.log from src/provider/[provider].md Co-authored-by: Julian Gruber <julian@juliangruber.com> * Update routes for ttfb * Use ttfb_ms value from response * Update src/index.md Co-authored-by: Miroslav Bajtoš <oss@bajtos.net> * Update labels and titles for ttfb charts * Add TTFB value to miner table overview * Change TTFB chart ticks to day * Update src/data/spark-retrieval-timings.json.js Co-authored-by: Miroslav Bajtoš <oss@bajtos.net> * Update src/provider/[provider].md Co-authored-by: Miroslav Bajtoš <oss@bajtos.net> * Update src/data/[provider]-spark-retrieval-timings-summary.json.js Co-authored-by: Miroslav Bajtoš <oss@bajtos.net> * Update src/provider/[provider].md Co-authored-by: Miroslav Bajtoš <oss@bajtos.net> * Make charts start from 0 * Remove faked datapoints * Move comments * Turn miner ttfb into map for faster access --------- Co-authored-by: Julian Gruber <julian@juliangruber.com> Co-authored-by: Miroslav Bajtoš <oss@bajtos.net>
1 parent 0271d48 commit e724243

File tree

5 files changed

+109
-13
lines changed

5 files changed

+109
-13
lines changed

‎src/data/[provider]-spark-retrieval-timings-summary.json.js‎

Lines changed: 17 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,17 @@
1+
import pRetry from 'p-retry';
2+
import { jsonFetcher } from "./json-fetcher.js";
3+
import { getDateXDaysAgo } from "../utils/date-utils.js";
4+
import { parseArgs } from "node:util";
5+
6+
const {
7+
values: { provider }
8+
} = parseArgs({
9+
options: { provider: { type: "string" } }
10+
});
11+
12+
const start = '2025-01-14';
13+
const end = getDateXDaysAgo(1);
14+
15+
const summary = await pRetry(() => jsonFetcher(`https://stats.filspark.com/miner/${provider}/retrieval-timings/summary?from=${start}&to=${end}`), { retries: 3 });
16+
17+
process.stdout.write(JSON.stringify(summary));

‎src/data/spark-miners-retrieval-timings.json.js‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,8 @@
1+
import { jsonFetcher } from "./json-fetcher.js";
2+
import { getDateXDaysAgo } from "../utils/date-utils.js";
3+
4+
const from = getDateXDaysAgo(31);
5+
const to = getDateXDaysAgo(1);
6+
7+
const output = await jsonFetcher(`https://stats.filspark.com/miners/retrieval-timings/summary?from=${from}&to=${to}`);
8+
process.stdout.write(JSON.stringify(output));

‎src/data/spark-retrieval-timings.json.js‎

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,10 @@
1+
import pRetry from 'p-retry';
2+
import { jsonFetcher } from "./json-fetcher.js";
3+
import { getDateXDaysAgo } from "../utils/date-utils.js";
4+
5+
const start = '2025-01-14';
6+
const end = getDateXDaysAgo(1);
7+
8+
const summary = await pRetry(() => jsonFetcher(`https://stats.filspark.com/retrieval-timings/daily?from=${start}&to=${end}`), { retries: 3 });
9+
10+
process.stdout.write(JSON.stringify(summary));

‎src/index.md‎

Lines changed: 39 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -11,15 +11,25 @@ import { combine, move, clone } from "./utils/ratios-utils.js";
1111
const SparkRates = FileAttachment("./data/spark-rsr.json").json();
1212
const SparkNonZeroRates = FileAttachment("./data/spark-rsr-non-zero.json").json();
1313
const SparkMinerRates = FileAttachment("./data/spark-miners-rsr.json").json();
14+
const SparkMinerRetrievalTimings = FileAttachment("./data/spark-miners-retrieval-timings.json").json();
1415
const SparkRetrievalResultCodes = FileAttachment("./data/spark-retrieval-result-codes.json").json();
1516
const SparkMinerRsrSummaries = FileAttachment("./data/spark-miner-rsr-summaries.json").json();
17+
const SparkRetrievalTimes = FileAttachment("./data/spark-retrieval-timings.json").json();
1618
```
1719

1820
```js
21+
const sparkMinerRetrievalTimingsMap = SparkMinerRetrievalTimings.reduce((acc, record) => {
22+
acc[record.miner_id] = record;
23+
return acc;
24+
}, {});
25+
1926
const nonZeroSparkMinerRates = SparkMinerRates.filter((record) => record.success_rate != 0)
2027
const tidySparkMinerRates = SparkMinerRates
2128
.sort((recordA, recordB) => recordB.success_rate - recordA.success_rate)
22-
.map(record => ({ ...record, success_rate: `${(record.success_rate * 100).toFixed(2)}%`,success_rate_http: `${(record.success_rate_http * 100).toFixed(2)}%`}))
29+
.map(record => {
30+
const { ttfb_ms } = sparkMinerRetrievalTimingsMap[record.miner_id] ?? {};
31+
return { ...record, ttfb_ms, success_rate: `${(record.success_rate * 100).toFixed(2)}%`,success_rate_http: `${(record.success_rate_http * 100).toFixed(2)}%`}
32+
})
2333
```
2434
2535
<div class="hero">
@@ -146,8 +156,17 @@ const percentiles = Object.entries(SparkMinerRsrSummaries)
146156
147157
<div class="divider"></div>
148158
149-
<h4>Spark Retrieval Result Codes</h4>
150-
<body>This section shows the Spark Retrieval Result Codes breakdown.</body>
159+
160+
<div class="grid grid-cols-2">
161+
<div>
162+
<h4>Spark Retrieval Result Codes</h4>
163+
<body>This section shows the Spark Retrieval Result Codes breakdown.</body>
164+
</div>
165+
<div>
166+
<h4>Spark Time To First Byte (TTFB)</h4>
167+
<body>The section shows the median of all median TTFB values from all retrieval tasks.</body>
168+
</div>
169+
</div>
151170
152171
```js
153172
const mapping = {
@@ -233,6 +252,21 @@ const tidy = clone(SparkRetrievalResultCodes).flatMap(({ day, rates }) => {
233252
]
234253
})}
235254
</div>
255+
<div class="card">
256+
${Plot.plot({
257+
title: 'Time to First Byte (ms)',
258+
// TODO: Change tick to month once we have more data
259+
x: { type: 'utc', ticks: 'day' },
260+
y: { grid: true, zero: true},
261+
marks: [
262+
Plot.lineY(SparkRetrievalTimes, {
263+
x: 'day',
264+
y: 'ttfb_ms',
265+
stroke: "#FFBD3F",
266+
})
267+
]
268+
})}
269+
</div>
236270
</div>
237271
238272
<details>
@@ -253,8 +287,8 @@ ${JSON.stringify(
253287
254288
<div class="divider"></div>
255289
256-
<h4>Spark Miner RSR Table</h4>
257-
<body>The following table shows the Spark RSR values calculated in aggregate for each Filecoin Storage Provider over the past 30 days. Click on a miner id to view stats about this storage provider.</body>
290+
<h4>Spark Miner Stats Table</h4>
291+
<body>The following table shows the Spark RSR and TTFB values calculated in aggregate for each Filecoin Storage Provider over the past 30 days. Click on a miner id to view stats about this storage provider.</body>
258292
259293
```js
260294
const search = view(Inputs.search(tidySparkMinerRates, {placeholder: "Search Storage Providers…"}));

‎src/provider/[provider].md‎

Lines changed: 35 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -6,30 +6,57 @@ title: Storage Provider Summary
66
```js
77
import { LineGraph } from "../components/line-graph.js";
88
import { getDateXDaysAgo } from "../utils/date-utils.js";
9-
const data = FileAttachment(`../data/${observable.params.provider}-spark-rsr-summary.json`).json();
9+
10+
const rsrData = FileAttachment(`../data/${observable.params.provider}-spark-rsr-summary.json`).json();
11+
const ttfbData = FileAttachment(`../data/${observable.params.provider}-spark-retrieval-timings-summary.json`).json();
1012
```
1113

14+
1215
<div class="hero">
1316
<body><a href="/"><img src="../media/spark-logomark-blue-with-bbox.png" alt="Spark Logo" width="300" /></a><body>
1417
<h2>Dashboard Beta</h2>
1518
<body><a href="https://filspark.com/dashboard" target="_blank" rel="noopener noreferrer">(Click here for Legacy Spark Grafana Dashboard)</a><body>
1619
</div>
1720

18-
<h4>Storage Provider Spark RSR Summary</h4>
19-
<body>This section shows the storage provider Spark Retrieval Success Rate Score summary. You can adjust the date range. Records start on the 7th April 2024.</body>
20-
21-
2221
```js
2322
const start = view(Inputs.date({label: "Start", value: getDateXDaysAgo(180) }));
2423
const end = view(Inputs.date({label: "End", value: getDateXDaysAgo(1) }));
2524
```
2625

2726
<h3>Stats for ${observable.params.provider}</h3>
2827

29-
<div class="grid grid-cols" style="grid-auto-rows: 500px;">
28+
29+
<div class="grid grid-cols-2">
30+
<div>
31+
<h4>Storage Provider Spark RSR Summary</h4>
32+
<body>This section shows the storage provider Spark Retrieval Success Rate Score summary.</body>
33+
</div>
34+
<div>
35+
<h4>Storage Provider Spark Time To First Byte (TTFB)</h4>
36+
<body>The section shows the median of all TTFB values for successful retrieval checks of this storage provider.</body>
37+
</div>
38+
</div>
39+
40+
41+
<div class="grid grid-cols-2" style="grid-auto-rows: 500px;">
3042
<div class="card">${
31-
resize((width) => LineGraph(data, {width, title: "Retrieval Success Rate", start, end }))
43+
resize((width) => LineGraph(rsrData, {width, title: "Retrieval Success Rate", start, end }))
3244
}</div>
45+
<div class="card">
46+
${Plot.plot({
47+
title: 'Time to First Byte (ms)',
48+
// TODO: Change tick to month once we have more data
49+
x: { type: 'utc', ticks: 'day' },
50+
y: { grid: true, zero: true },
51+
marks: [
52+
Plot.lineY(ttfbData, {
53+
x: 'day',
54+
y: 'ttfb_ms',
55+
stroke: "#FFBD3F",
56+
})
57+
]
58+
})}
59+
</div>
3360
</div>
3461

3562
<style>
@@ -73,4 +100,4 @@ const end = view(Inputs.date({label: "End", value: getDateXDaysAgo(1) }));
73100
}
74101
}
75102

76-
</style>
103+
</style>

0 commit comments

Comments
 (0)