Drop in performance at Tech Empower benchmark

[NBM1977]

Hi, what happened to fasthttp scores at Tech Empower benchmark, it used to be one of the best framework, but the performance is dropping like a rock at this benchmark since 2022. It was some kind of security bug?

This is the plaintext benchmark:

2020 fasthttp 5,962,266 81.2%
2021 fasthttp 6,671,354 95.1%
2022 fasthttp 929,880 13.2%
2023 fasthttp 367,504 5.2%
2025 fasthttp 569,793 2.0%

Thanks.

[erikdubbelboer]

I have no idea what could be causing this. Lots of changes have been made over the years, but none should affect performance this much. Has the benchmark changed?

If I have some time I'll try to run my own benchmark against older code and see if I can find anything.

[NBM1977]

I am not sure if they changed something, In 2021 I decided to use fasthttp in my project after I saw the great performance at their benchmark. From time to time I check that website for updates and saw this sharp drop in performance in 2022. I thought it was some error in their benchmark but later the results continued bad, now I am kind of worried about this.

[erikdubbelboer]

I have tested all versions and only found this specific commit that caused a ~15% slowdown in handling requests: 3b117f8
I'm trying to find a fix for this. It's using the Mutex too often.

Here is the result of my running a wrk benchmark on all versions (printing requests per second):

v1.10.0: 362579.16
v1.11.0: 359667.31
v1.12.0: 364035.49
v1.13.0: 363075.05
v1.14.0: 363688.84
v1.15.0: 364585.41
v1.16.0: 365185.00
v1.17.0: 358476.87
v1.18.0: 363085.11
v1.19.0: 364730.17
v1.20.0: 360243.44
v1.21.0: 364885.29
v1.22.0: 360594.45
v1.23.0: 360707.36
v1.24.0: 366293.64
v1.25.0: 364223.53
v1.26.0: 368327.07
v1.27.0: 360010.40
v1.28.0: 362552.11
v1.29.0: 361046.06
v1.30.0: 361533.39
v1.31.0: 363819.79
v1.32.0: 312502.24 <-- this is where the rps dropped
v1.33.0: 314383.42
v1.34.0: 310461.47
v1.35.0: 304573.11
v1.36.0: 310698.89
v1.37.0: 315389.47
v1.38.0: 312370.17
v1.39.0: 300507.58
v1.40.0: 307602.40
v1.41.0: 305047.70
v1.42.0: 299546.40
v1.43.0: 299316.18
v1.44.0: 297426.66
v1.45.0: 293567.19
v1.46.0: 298349.67
v1.47.0: 300466.07
v1.48.0: 295501.55
v1.49.0: 301550.28
v1.50.0: 300503.12
v1.51.0: 297285.80
v1.52.0: 305689.94
v1.53.0: 296296.28
v1.54.0: 300613.45
v1.55.0: 300131.14
v1.56.0: 300373.90
v1.57.0: 305786.78
v1.58.0: 303318.66
v1.59.0: 298994.68

[NBM1977]

Did you test the normal or the prefork version? In their test, the prefork version didn't lose too much performance. The normal version is the one that lost a lot of performance.

2020: fasthttp-prefork | 6,505,718 | 88.6%
2021: fasthttp-prefork | 6,832,182 | 97.4%
2022: fasthttp-prefork | 6,268,456 | 89.2%
2023: fasthttp-prefork | 5,321,131 | 75.9%
2025: fasthttp-prefork | 14,566,665 | 51.3%

2020: fasthttp | 5,962,266 | 81.2%
2021: fasthttp | 6,671,354 | 95.1%
2022: fasthttp | 929,880 | 13.2%
2023: fasthttp | 367,504 | 5.2%
2025: fasthttp | 569,793 | 2.0%

Comparing their 2025 test, the prefork version is 25 times faster than the normal version.

[erikdubbelboer]

It would make both the normal and prefork faster. Here is the pull: #1986
I'll release this and then wait for them to run using this version to see the improvements on their end.

[gaby]

What is the wrk benchmark value with v1.60 ?

[erikdubbelboer]

I already deleted the machine I did the tests on, but it was something like 350000, so almost back to it's original value.

[gaby]

Awesome 💪 great find.

Did you save the code/command for testing this? I wonder if Fiber has the same issue

[erikdubbelboer]

I created a sub folder in the fasthttp dir. In this subfolder I did go mod init bench and created a simple main.go which just started a server that responded with nothing.

Then in the go.mod in that subfolder you do:

replace github.com/valyala/fasthttp v0.0.0 => ../

require (
	github.com/valyala/fasthttp v0.0.0
)

This will make the server use the repo version of fasthttp.

Then save and run this script in that subfolder.

#!/bin/bash

for version in $(seq 10 60); do
    tag="v1.${version}.0"
    echo -n "$tag: "

    # Checkout tag
    git checkout $tag --quiet || { echo "checkout failed"; exit 1; }

    # Ensure dependencies
    go mod tidy >/dev/null 2>&1 || { echo "go mod failed"; exit 1; }

    # Start the Go server in a new process group
    setsid go run main.go >/dev/null 2>&1 &
    SERVER_PID=$!

    # Wait for server to be ready (max 10 seconds)
    for i in {1..10}; do
        if curl -s http://localhost:8080 >/dev/null; then
            break
        fi
        sleep 1
    done

    # Check if the server is still running
    if ! kill -0 $SERVER_PID 2>/dev/null; then
        echo "go run failed"
        exit 1
    fi

    # Run wrk benchmark
    result=$(wrk -t16 -c64 -d60s http://localhost:8080 2>/dev/null | grep "Requests/sec")

    # Kill the whole process group
    kill -- -$SERVER_PID 2>/dev/null
    wait $SERVER_PID 2>/dev/null

    # Print result
    if [[ $result =~ ([0-9]+\.[0-9]+) ]]; then
        echo "${BASH_REMATCH[1]}"
    else
        echo "N/A"
    fi
done

[gaby]

Thanks! Looks reasonable. I will give that a try 💪 Thanks!

[NBM1977]

Hey Erik, could you contact the guys from Tech Empower and tell them about this find? They run a continuous benchmarking and would be nice if they apply this fix.

Thanks.

[gaby]

@NBM1977 A pull request to be sent to https://github.com/TechEmpower/FrameworkBenchmarks

Once merged, it will be in the next run

[erikdubbelboer]

TechEmpower/FrameworkBenchmarks#9771

[gaby]

@erikdubbelboer I manage to make your script into a GitHub workflow for Fiber. Still work in progress since I haven't figure out to test each tag 😂

https://github.com/gaby/fiber-benchmarks/blob/main/.github/workflows/benchmark.yml

Maybe i can make this generic enough to be reusable as a GitHub Action