perf: add real e2e tracing and cold-path perf toolkit

Author: Lemonawa

docs/perf/2026-02-20-coldpath-baseline.md

@@ -0,0 +1,42 @@
+# Cold Path Real E2E Baseline (2026-02-20)
+
+## Context
+- Objective: reduce `/music` cold-path p95 latency with real-traffic evidence.
+- Decision metric: `e2e_total` p95 from structured `PERF` logs.
+- Synthetic benchmark is recorded as reference only.
+
+## Data Source
+- Log source: `data/manual-test/coldpath.log`
+- Parser: `scripts/perf_log_report.py`
+- Sample target: `>= 30` cold requests per topology
+
+## Topology Baseline
+
+| Topology | Requests | e2e p50 (ms) | e2e p95 (ms) | e2e max (ms) | Notes |
+|---|---:|---:|---:|---:|---|
+| official_api | TBD | TBD | TBD | TBD | |
+| selfhost_api_uri_upload / selfhost_api_multipart_upload | TBD | TBD | TBD | TBD | |
+
+## Stage Share Snapshot (p95)
+
+| Topology | Dominant Stage 1 | Dominant Stage 2 | Dominant Stage 3 |
+|---|---|---|---|
+| official_api | TBD | TBD | TBD |
+| selfhost_api_uri_upload / selfhost_api_multipart_upload | TBD | TBD | TBD |
+
+## Synthetic Reference (Non-decision)
+- Command: `python3 scripts/perf_compare.py`
+- Reference artifact: `docs/perf/2026-02-11-parallel-optimization-baseline.md`
+
+## Optimization Loop
+1. Collect baseline real logs.
+2. Apply minimal hot-path optimization.
+3. Re-sample with same workload shape.
+4. Compare `e2e_total` p95 and stage p95 shares.
+5. Iterate until soft target (~20%) or clear bottleneck plateau.
+
+## Result Summary (Fill After Run)
+- official_api p95 delta: `TBD`
+- selfhost_api_uri_upload/selfhost_api_multipart_upload p95 delta: `TBD`
+- achieved soft target (~20%): `TBD`
+- next priority if not reached: `TBD`

docs/runbooks/perf-coldpath-real-e2e.md

@@ -0,0 +1,75 @@
+# Cold Path Real E2E Performance Runbook
+
+## Goal
+Measure real `/music` cold-path latency using production-like traffic and parse structured `PERF|...` logs.
+
+## Scope
+- Primary path: first `/music` cold requests (cache miss)
+- Topologies:
+  - `official_api`
+  - `selfhost_api_uri_upload` (self-hosted API + URI upload mode)
+  - `selfhost_api_multipart_upload` (self-hosted API + multipart upload mode)
+- Decision metric: `e2e_total` p95
+
+## Prerequisites
+1. Build and run the bot with `loglevel=info` or `loglevel=debug`.
+2. Ensure requests are cold path samples (avoid repeated same `music_id` cache hits).
+3. Collect logs to file.
+
+Example:
+
+```bash
+cargo run --release -- --config config.ini 2>&1 | tee data/manual-test/coldpath.log
+```
+
+## Sampling Plan
+1. Official API: capture at least 30 cold-path requests.
+2. Self-hosted API mode (local or remote deployment): capture at least 30 cold-path requests.
+3. Separate self-hosted data by upload mode (`selfhost_api_uri_upload` vs `selfhost_api_multipart_upload`).
+4. Keep request set comparable (song size/category mix).
+
+## Parse and Report
+Generate overall report:
+
+```bash
+python3 scripts/perf_log_report.py \
+  --log-file data/manual-test/coldpath.log \
+  --cache-path miss_cold \
+  --markdown-output docs/perf/2026-02-20-coldpath-overall.md \
+  --json-output docs/perf/2026-02-20-coldpath-overall.json
+```
+
+Generate per-topology reports:
+
+```bash
+python3 scripts/perf_log_report.py \
+  --log-file data/manual-test/coldpath.log \
+  --topology official_api \
+  --cache-path miss_cold \
+  --markdown-output docs/perf/2026-02-20-coldpath-official.md \
+  --json-output docs/perf/2026-02-20-coldpath-official.json
+
+python3 scripts/perf_log_report.py \
+  --log-file data/manual-test/coldpath.log \
+  --topology selfhost_api_uri_upload \
+  --cache-path miss_cold \
+  --markdown-output docs/perf/2026-02-20-coldpath-local-uri.md \
+  --json-output docs/perf/2026-02-20-coldpath-local-uri.json
+```
+
+If URI upload is disabled in a run, use topology filter `selfhost_api_multipart_upload`.
+
+## Acceptance Rule
+- Soft target: cold-path `e2e_total` p95 improves by ~20%.
+- If target is not reached, still deliver:
+  - stage-level p95 shares (`upload_send`, `tag_process`, `select_url`, `db_save`, etc.)
+  - bottleneck evidence and next optimization priorities.
+
+## Reference (Synthetic)
+Synthetic benchmark remains useful for regression sanity checks only:
+
+```bash
+python3 scripts/perf_compare.py
+```
+
+Do not use synthetic output as final optimization decision data.