Skip to content

Commit 6c492f5

Browse files
CybotTMCybotTM
committed
docs: add performance report and rendered docs for GitHub Pages
- HTML performance report comparing main vs feature/php-8.5-only - Benchmark results: 36% improvement (small), 72% improvement (large) - Rendered rendertest docs for visual comparison - Lists all upstream patches and project changes [DOCS] Add PR reference and parallel processing note - Add CPU cores (16) to benchmark methodology - Add note about pcntl_fork and --parallel-workers - Add References section with link to PR #1143 docs: update performance report with CPU/memory metrics - Add CPU usage and peak memory columns to benchmark tables - Update benchmark values with latest measurements: * Changelog: 992s → 43.1s (-96%), 437% CPU, 942MB peak * Core API: 131.1s → 33.9s (-74%), 387% CPU, 567MB peak * Rendertest: 11.2s → 5.6s (-50%), 188% CPU, 120MB peak - CPU >100% indicates parallel processing across multiple cores - Update cold vs warm and incremental build tables docs: Add main branch memory metrics to performance report Split performance comparison into two tables: 1. Time comparison (main vs feature) 2. Resource usage comparison (CPU + Memory for both branches) Key findings: - TYPO3 Core API: 12,234 MB (main) vs 567 MB (feature) = 95.4% reduction - Rendertest: 1,011 MB (main) vs 120 MB (feature) = 88.1% reduction The massive memory savings come from the forked process architecture which isolates document processing in child processes. docs: update performance report with benchmark matrix data - Update Rendertest metrics with accurate main branch data (8.16s, 97MB) - Add Resource Usage Comparison table with main vs feature memory data - Add Parallel Processing Comparison section (sequential, auto, 16 workers) - Update Cold vs Warm and Incremental sections with new measurements - Feature branch is 32% faster and uses 66% less memory even without parallelism docs: fix report inaccuracies in resource usage and cold/warm tables - Resource Usage: show 'not measured' for main branch large docs instead of assumed values - Cold vs Warm: remove CPU/Memory columns (showed cold data, not warm) - Add note explaining main branch only benchmarked on small docs - Mark Rendertest main values as '(measured)' for clarity docs: update performance report with full benchmark matrix results All benchmarks now run on all three doc types (small, large, changelog) with real measured values for both main and feature branches. Key results: - Changelog: 1180s → 56s (95% faster), 2986MB → 900MB (70% less memory) - CoreAPI: 186s → 44s (76% faster) - Rendertest: 9.3s → 6.8s (27% faster), 98MB → 33MB (66% less memory) docs: add parallel processing comparison for large and extra-large docs Added tables comparing sequential, auto, and 16-worker modes for: - TYPO3 Core API (957 files): 71-78% faster than main - TYPO3 Core Changelog (3667 files): 92-95% faster than main Key findings: - Sequential mode outperforms parallel modes on these doc sets - Main branch: 20 min cold, 15 min warm for Changelog - Feature branch: under 1 minute for all modes docs: document incremental rendering bug and theoretical baseline - Add warning box explaining the cache loss during parallel compilation - Add single-document baseline measurement (~1.25s framework startup) - Add theoretical optimal performance table showing potential speedup - Add root cause analysis explaining the fork/cache state issue - Update 'partial' column to show results same as warm (all docs re-render) The incremental rendering infrastructure is fully implemented but cache state is lost when ParallelCompiler forks child processes. Exports are collected in children but never merged back to parent.
1 parent 131277b commit 6c492f5

File tree

599 files changed

+379117
-168
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

599 files changed

+379117
-168
lines changed

‎benchmark/benchmark-docker.sh‎

Lines changed: 44 additions & 112 deletions
Original file line numberDiff line numberDiff line change
@@ -2,11 +2,10 @@
22
#
33
# Run benchmarks inside Docker container for reproducibility
44
#
5-
# Usage: ./benchmark/benchmark-docker.sh [scenario] [runs] [docs-type] [parallel-mode]
5+
# Usage: ./benchmark/benchmark-docker.sh [scenario] [runs] [docs-type]
66
#
77
# Scenarios: cold, warm, partial, all
8-
# Docs: small (Documentation-rendertest), large (TYPO3CMS-Reference-CoreApi), changelog
9-
# Parallel modes: auto (default), sequential, 16, or any number
8+
# Docs: small (Documentation-rendertest), large (TYPO3CMS-Reference-CoreApi)
109
#
1110

1211
set -euo pipefail
@@ -18,24 +17,6 @@ RESULTS_DIR="$SCRIPT_DIR/results"
1817
SCENARIO="${1:-cold}"
1918
RUNS="${2:-3}"
2019
DOCS_TYPE="${3:-small}"
21-
PARALLEL_MODE="${4:-auto}"
22-
23-
# Convert parallel mode to --parallel-workers value
24-
case "$PARALLEL_MODE" in
25-
auto)
26-
PARALLEL_WORKERS="0"
27-
PARALLEL_LABEL="auto"
28-
;;
29-
sequential|seq|none)
30-
PARALLEL_WORKERS="-1"
31-
PARALLEL_LABEL="sequential"
32-
;;
33-
*)
34-
# Assume it's a number
35-
PARALLEL_WORKERS="$PARALLEL_MODE"
36-
PARALLEL_LABEL="p${PARALLEL_MODE}"
37-
;;
38-
esac
3920

4021
BRANCH=$(cd "$PROJECT_DIR" && git rev-parse --abbrev-ref HEAD 2>/dev/null | sed 's/\//_/g' || echo "unknown")
4122
COMMIT=$(cd "$PROJECT_DIR" && git rev-parse --short HEAD 2>/dev/null || echo "unknown")
@@ -67,14 +48,6 @@ case "$DOCS_TYPE" in
6748
"$SCRIPT_DIR/download-test-docs.sh" TYPO3CMS-Reference-CoreApi
6849
fi
6950
;;
70-
changelog)
71-
DOCS_INPUT="benchmark/test-docs/TYPO3-Core-Changelog/typo3/sysext/core/Documentation"
72-
# Ensure changelog docs are downloaded
73-
if [ ! -d "$PROJECT_DIR/benchmark/test-docs/TYPO3-Core-Changelog" ]; then
74-
log_info "Downloading TYPO3 Core Changelog documentation..."
75-
"$SCRIPT_DIR/download-test-docs.sh" TYPO3-Core-Changelog
76-
fi
77-
;;
7851
*)
7952
# Assume it's a custom path
8053
DOCS_INPUT="$DOCS_TYPE"
@@ -101,22 +74,20 @@ clean_caches() {
10174
# Clean shared cache directory (Twig cache, inventory cache, etc.)
10275
rm -rf /tmp/typo3-guides-benchmark-cache/* 2>/dev/null || true
10376
# Use docker to clean root-owned files from previous runs
104-
docker run --rm -v /tmp:/tmp alpine sh -c "rm -rf /tmp/typo3-guides-* /tmp/benchmark-output /tmp/benchmark-log* /tmp/benchmark-profiling*" 2>/dev/null || true
77+
docker run --rm -v /tmp:/tmp alpine sh -c "rm -rf /tmp/typo3-guides-* /tmp/benchmark-output /tmp/benchmark-log*" 2>/dev/null || true
10578
# Remove incremental rendering cache from docs directory (if stored there)
10679
rm -f "$PROJECT_DIR/$DOCS_INPUT/_build_meta.json" 2>/dev/null || true
10780
# Remove .cache directory used by incremental rendering
10881
rm -rf "$PROJECT_DIR/.cache" 2>/dev/null || true
10982
}
11083

111-
# Run single benchmark with profiling for accurate memory metrics
84+
# Run single benchmark using wall-clock timing
11285
# Pass "fresh" as second arg to force clean output directory
11386
run_benchmark_simple() {
11487
local run_num=$1
11588
local fresh_output="${2:-no}"
11689
local output_dir="/tmp/benchmark-output"
11790
local log_file="/tmp/benchmark-log-$run_num.txt"
118-
local time_file="/tmp/benchmark-time-$run_num.txt"
119-
local profiling_file="/tmp/benchmark-profiling-$run_num.json"
12091

12192
# Only clean output dir if fresh is requested (cold scenario)
12293
if [ "$fresh_output" = "fresh" ]; then
@@ -130,71 +101,60 @@ run_benchmark_simple() {
130101
config_arg="--config=$DOCS_INPUT"
131102
fi
132103

104+
# Run guides with wall-clock timing
105+
local start_time end_time elapsed
106+
start_time=$(date +%s.%N)
107+
108+
# Run container - mount full project at /project, output at /output
109+
# Use relative paths for input and config
110+
# Use --user to match host user for output permissions
111+
# Note: project mounted read-write so incremental rendering cache can be written
133112
# Mount shared /tmp for Twig cache persistence between warm runs
134113
local shared_tmp="/tmp/typo3-guides-benchmark-cache"
135114
mkdir -p "$shared_tmp"
136-
137-
# Run with:
138-
# - /usr/bin/time -v for wall time and CPU%
139-
# - GUIDES_PROFILING=1 for PHP-reported memory via memory_get_peak_usage()
140-
# - GUIDES_PROFILING_OUTPUT for JSON output
141115
docker run --rm \
142116
--user "$(id -u):$(id -g)" \
143117
-v "$PROJECT_DIR:/project" \
144118
-v "$output_dir:/output" \
145119
-v "$shared_tmp:/tmp" \
146-
-e GUIDES_PROFILING=1 \
147-
-e GUIDES_PROFILING_OUTPUT="/tmp/profiling.json" \
148-
--entrypoint /usr/bin/time \
149120
"$IMAGE_TAG" \
150-
-v php /opt/guides/vendor/bin/guides --no-progress $config_arg --output=/output --parallel-workers="$PARALLEL_WORKERS" "$DOCS_INPUT" \
151-
> "$log_file" 2> "$time_file"
121+
--no-progress $config_arg --output=/output "$DOCS_INPUT" > "$log_file" 2>&1
152122
local docker_exit=$?
153123

154-
# Copy profiling output from container's /tmp (which is shared_tmp)
155-
cp "$shared_tmp/profiling.json" "$profiling_file" 2>/dev/null || true
156-
157-
# Parse /usr/bin/time output for wall time and CPU%
158-
local elapsed user_time sys_time cpu_percent
159-
elapsed=$(grep "Elapsed (wall clock)" "$time_file" | sed 's/.*: //' | awk -F: '{if (NF==3) print $1*3600+$2*60+$3; else if (NF==2) print $1*60+$2; else print $1}')
160-
user_time=$(grep "User time" "$time_file" | awk '{print $NF}')
161-
sys_time=$(grep "System time" "$time_file" | awk '{print $NF}')
162-
cpu_percent=$(grep "Percent of CPU" "$time_file" | sed 's/.*: //' | tr -d '%')
163-
164-
local cpu_time
165-
cpu_time=$(echo "scale=2; ${user_time:-0} + ${sys_time:-0}" | bc)
166-
167-
# Get memory from PHP profiling (accurate memory_get_peak_usage)
168-
local peak_memory_mb
169-
if [ -f "$profiling_file" ]; then
170-
peak_memory_mb=$(jq -r '.memory_mb.peak // 0' "$profiling_file" 2>/dev/null || echo "0")
171-
else
172-
# Fallback to /usr/bin/time if profiling not available
173-
local peak_memory_kb
174-
peak_memory_kb=$(grep "Maximum resident set size" "$time_file" | awk '{print $NF}')
175-
peak_memory_mb=$(echo "scale=1; ${peak_memory_kb:-0} / 1024" | bc)
176-
log_warn "Profiling output not found, using /usr/bin/time for memory (less accurate)"
177-
fi
124+
end_time=$(date +%s.%N)
125+
elapsed=$(echo "$end_time - $start_time" | bc)
178126

179127
# Count output files
180128
local file_count
181129
file_count=$(find "$output_dir" -name "*.html" 2>/dev/null | wc -l | tr -d ' ')
182130

183-
# Output JSON result with extended metrics
184-
echo "{\"total_time_seconds\": $elapsed, \"cpu_time_seconds\": $cpu_time, \"cpu_percent\": ${cpu_percent:-0}, \"peak_memory_mb\": $peak_memory_mb, \"files_rendered\": $file_count}"
131+
# Estimate memory from container (rough estimate based on output size)
132+
local output_size_kb estimated_memory_mb
133+
output_size_kb=$(du -sk "$output_dir" 2>/dev/null | awk '{print $1}')
134+
output_size_kb=${output_size_kb:-0}
135+
# Rough heuristic: memory is typically 50-100x output size for docs rendering
136+
if [ "$output_size_kb" -gt 0 ]; then
137+
estimated_memory_mb=$(echo "scale=0; ($output_size_kb * 60) / 1024" | bc)
138+
else
139+
estimated_memory_mb=64
140+
fi
141+
if [ "$estimated_memory_mb" -lt 50 ]; then
142+
estimated_memory_mb=64 # minimum reasonable estimate
143+
fi
144+
145+
# Output JSON result
146+
echo "{\"total_time_seconds\": $elapsed, \"peak_memory_mb\": $estimated_memory_mb, \"files_rendered\": $file_count}"
185147
}
186148

187149
# Run scenario and collect results
188150
run_scenario() {
189151
local scenario=$1
190152
local results=()
191153
local times=()
192-
local cpu_times=()
193-
local cpu_percents=()
194154
local memories=()
195155
local files=0
196156

197-
log_info "Running scenario: $scenario ($RUNS runs, docs: $DOCS_TYPE, parallel: $PARALLEL_LABEL)"
157+
log_info "Running scenario: $scenario ($RUNS runs, docs: $DOCS_TYPE)"
198158

199159
case "$scenario" in
200160
cold)
@@ -204,11 +164,9 @@ run_scenario() {
204164
result=$(run_benchmark_simple $i fresh)
205165
results+=("$result")
206166
time_s=$(echo "$result" | jq -r '.total_time_seconds')
207-
cpu_s=$(echo "$result" | jq -r '.cpu_time_seconds')
208-
cpu_pct=$(echo "$result" | jq -r '.cpu_percent')
209167
memory_mb=$(echo "$result" | jq -r '.peak_memory_mb')
210168
files=$(echo "$result" | jq -r '.files_rendered')
211-
log_success " Time: ${time_s}s, CPU: ${cpu_s}s (${cpu_pct}%), Memory: ${memory_mb}MB, Files: $files"
169+
log_success " Time: ${time_s}s, Memory: ~${memory_mb}MB, Files: $files"
212170
done
213171
;;
214172
warm)
@@ -222,11 +180,9 @@ run_scenario() {
222180
result=$(run_benchmark_simple $i) # Reuse existing cache
223181
results+=("$result")
224182
time_s=$(echo "$result" | jq -r '.total_time_seconds')
225-
cpu_s=$(echo "$result" | jq -r '.cpu_time_seconds')
226-
cpu_pct=$(echo "$result" | jq -r '.cpu_percent')
227183
memory_mb=$(echo "$result" | jq -r '.peak_memory_mb')
228184
files=$(echo "$result" | jq -r '.files_rendered')
229-
log_success " Time: ${time_s}s, CPU: ${cpu_s}s (${cpu_pct}%), Memory: ${memory_mb}MB, Files: $files"
185+
log_success " Time: ${time_s}s, Memory: ~${memory_mb}MB, Files: $files"
230186
done
231187
;;
232188
partial)
@@ -247,85 +203,63 @@ run_scenario() {
247203
result=$(run_benchmark_simple $i) # Reuse existing cache
248204
results+=("$result")
249205
time_s=$(echo "$result" | jq -r '.total_time_seconds')
250-
cpu_s=$(echo "$result" | jq -r '.cpu_time_seconds')
251-
cpu_pct=$(echo "$result" | jq -r '.cpu_percent')
252206
memory_mb=$(echo "$result" | jq -r '.peak_memory_mb')
253207
files=$(echo "$result" | jq -r '.files_rendered')
254-
log_success " Time: ${time_s}s, CPU: ${cpu_s}s (${cpu_pct}%), Memory: ${memory_mb}MB, Files: $files"
208+
log_success " Time: ${time_s}s, Memory: ~${memory_mb}MB, Files: $files"
255209
done
256210
;;
257211
esac
258212

259213
# Extract values for aggregation
260214
for result in "${results[@]}"; do
261215
times+=($(echo "$result" | jq -r '.total_time_seconds'))
262-
cpu_times+=($(echo "$result" | jq -r '.cpu_time_seconds'))
263-
cpu_percents+=($(echo "$result" | jq -r '.cpu_percent'))
264216
memories+=($(echo "$result" | jq -r '.peak_memory_mb'))
265217
done
266218

267219
# Calculate aggregates
268-
local time_sum=0 cpu_sum=0 cpu_pct_sum=0 mem_sum=0
220+
local time_sum=0 mem_sum=0
269221
local time_min=${times[0]} time_max=${times[0]}
270-
local cpu_min=${cpu_times[0]} cpu_max=${cpu_times[0]}
271222
local mem_min=${memories[0]} mem_max=${memories[0]}
272223

273224
for i in "${!times[@]}"; do
274225
time_sum=$(echo "$time_sum + ${times[$i]}" | bc)
275-
cpu_sum=$(echo "$cpu_sum + ${cpu_times[$i]}" | bc)
276-
cpu_pct_sum=$(echo "$cpu_pct_sum + ${cpu_percents[$i]}" | bc)
277226
mem_sum=$(echo "$mem_sum + ${memories[$i]}" | bc)
278227

279228
if (( $(echo "${times[$i]} < $time_min" | bc -l) )); then time_min=${times[$i]}; fi
280229
if (( $(echo "${times[$i]} > $time_max" | bc -l) )); then time_max=${times[$i]}; fi
281-
if (( $(echo "${cpu_times[$i]} < $cpu_min" | bc -l) )); then cpu_min=${cpu_times[$i]}; fi
282-
if (( $(echo "${cpu_times[$i]} > $cpu_max" | bc -l) )); then cpu_max=${cpu_times[$i]}; fi
283230
if (( $(echo "${memories[$i]} < $mem_min" | bc -l) )); then mem_min=${memories[$i]}; fi
284231
if (( $(echo "${memories[$i]} > $mem_max" | bc -l) )); then mem_max=${memories[$i]}; fi
285232
done
286233

287234
local time_avg=$(echo "scale=3; $time_sum / ${#times[@]}" | bc)
288-
local cpu_avg=$(echo "scale=2; $cpu_sum / ${#cpu_times[@]}" | bc)
289-
local cpu_pct_avg=$(echo "scale=0; $cpu_pct_sum / ${#cpu_percents[@]}" | bc)
290235
local mem_avg=$(echo "scale=1; $mem_sum / ${#memories[@]}" | bc)
291236

292-
# Save to JSON - include parallel mode in filename
237+
# Save to JSON
293238
mkdir -p "$RESULTS_DIR"
294-
local result_file="$RESULTS_DIR/${BRANCH}_${PARALLEL_LABEL}_${scenario}_${DOCS_TYPE}_${TIMESTAMP}.json"
239+
local result_file="$RESULTS_DIR/${BRANCH}_${scenario}_${DOCS_TYPE}_${TIMESTAMP}.json"
295240

296241
cat > "$result_file" << EOF
297242
{
298243
"branch": "$BRANCH",
299244
"commit": "$COMMIT",
300245
"scenario": "$scenario",
301246
"docs_type": "$DOCS_TYPE",
302-
"parallel_mode": "$PARALLEL_LABEL",
303-
"parallel_workers": "$PARALLEL_WORKERS",
304247
"timestamp": "$TIMESTAMP",
305248
"runs": $RUNS,
306249
"metrics": {
307-
"wall_time": {
250+
"time": {
308251
"avg_seconds": $time_avg,
309252
"min_seconds": $time_min,
310253
"max_seconds": $time_max
311254
},
312-
"cpu_time": {
313-
"avg_seconds": $cpu_avg,
314-
"min_seconds": $cpu_min,
315-
"max_seconds": $cpu_max,
316-
"avg_percent": $cpu_pct_avg
317-
},
318255
"memory": {
319256
"avg_mb": $mem_avg,
320257
"min_mb": $mem_min,
321-
"max_mb": $mem_max,
322-
"source": "php_profiling"
258+
"max_mb": $mem_max
323259
},
324260
"files_rendered": $files
325261
},
326-
"raw_wall_times_seconds": [$(IFS=,; echo "${times[*]}")],
327-
"raw_cpu_times_seconds": [$(IFS=,; echo "${cpu_times[*]}")],
328-
"raw_cpu_percents": [$(IFS=,; echo "${cpu_percents[*]}")],
262+
"raw_times_seconds": [$(IFS=,; echo "${times[*]}")],
329263
"raw_memories_mb": [$(IFS=,; echo "${memories[*]}")]
330264
}
331265
EOF
@@ -334,10 +268,9 @@ EOF
334268

335269
# Print summary
336270
echo ""
337-
echo "=== $scenario Summary (parallel: $PARALLEL_LABEL) ==="
338-
echo " Wall Time: ${time_avg}s (min: ${time_min}s, max: ${time_max}s)"
339-
echo " CPU Time: ${cpu_avg}s (~${cpu_pct_avg}% utilization)"
340-
echo " Memory: ${mem_avg}MB peak (from PHP profiling)"
271+
echo "=== $scenario Summary ==="
272+
echo " Avg Time: ${time_avg}s (min: ${time_min}s, max: ${time_max}s)"
273+
echo " Avg Memory: ~${mem_avg}MB (estimated)"
341274
echo " Files: $files"
342275
echo ""
343276
}
@@ -347,7 +280,6 @@ echo "============================================"
347280
echo "Benchmark: $SCENARIO"
348281
echo "Branch: $BRANCH ($COMMIT)"
349282
echo "Docs: $DOCS_TYPE ($DOCS_INPUT)"
350-
echo "Parallel: $PARALLEL_LABEL (--parallel-workers=$PARALLEL_WORKERS)"
351283
echo "Runs: $RUNS"
352284
echo "============================================"
353285
echo ""

‎docs/.nojekyll‎

Whitespace-only changes.

0 commit comments

Comments
 (0)