Add framework to measure code base performance ✨ 💎

[RobertoPrevato]

• Add framework to measure code base performance. This is to refactor more comfortably Cython code and core features, keeping an eye on performance.
• Download the benchmark-reports artifact from the GitHub Workflow to see it in action. https://github.com/Neoteroi/BlackSheep/actions/runs/14950087869

Tip

The benchmark-reports artifacts include Excel files with tables and charts.

[bymoye]

Blacksheep performs best in Python 3.11! Why not 3.13?

[RobertoPrevato]

@bymoye I was surprised, too. I have no idea. But I think it must be related to CPython itself, because I executed the tests on the same commit. I am pretty satisfied with what I did with these benchmarks, the worklow is pretty cool! 😄

[RobertoPrevato]

However, the best performance in terms of memory consumption is with:

3.12.10	Windows-2022Server-10.0.20348-SP0
3.11.9	Windows-10-10.0.20348-SP0

👍🏼

---

The worse in terms of memoy consumption:

3.13.3	Linux-6.11.0-1013-azure-x86_64-with-glibc2.39
3.13.3	macOS-14.7.5-arm64-arm-64bit-Mach-O

👎🏼

[bymoye]

However, the best performance in terms of memory consumption is with:

3.12.10 Windows-2022Server-10.0.20348-SP0
3.11.9 Windows-10-10.0.20348-SP0
👍🏼

The worse in terms of memoy consumption:

3.13.3 Linux-6.11.0-1013-azure-x86_64-with-glibc2.39
3.13.3 macOS-14.7.5-arm64-arm-64bit-Mach-O
👎🏼

This is very strange... I was under the impression that Linux systems should perform best... maybe it has something to do with the memory management mechanism of each system?

[RobertoPrevato]

I find it strange, too. The specs for GitHub runners are the same for Windows and Linux. macOS has less resources and I got better results at each run.

[bymoye]

Hi, @RobertoPrevato

I did some research on the possible reason, maybe the memory_profiler results between different platforms cannot be compared together. Because memory_profiler uses psutil, and psutil works differently in Linux and Windows and MacOS. psutil-windows-process-memory-usage

Related documentation: memory_full_info

Maybe you should use mem_usage = memory_usage(wrapper, interval=0.01, timeout=30, backend='psutil_uss') to test, it defaults to psutil(rss), using uss will measure the memory unique to the process instead of shared memory.

[RobertoPrevato]

Hi @bymoye
Thank you for the heads up. I fix it asap (I am documenting the new features of 2.3.0 right now).

[RobertoPrevato]

@bymoye backend='psutil_uss' fails on macOS. 👀 https://github.com/Neoteroi/BlackSheep/actions/runs/14958347197/job/42017019493

I'll check more another day.

[bymoye]

@RobertoPrevato

I used psutil to replace the original writing, I am not sure whether it is consistent with the original functional requirements. bymoye@2614f00

The following results were obtained:

https://github.com/bymoye/BlackSheep/actions/runs/14961845051

[RobertoPrevato]

@RobertoPrevato

I used psutil to replace the original writing, I am not sure whether it is consistent with the original functional requirements. bymoye@2614f00

The following results were obtained:

https://github.com/bymoye/BlackSheep/actions/runs/14961845051

Thanks, very much appreciated. You are correct that USS should be used here.

I would add garbage collection before each function execution:

while time.time() - start_time < 30:  # timeout=30 seconds
+   gc.collect()
    wrapper()

The results of the app_handle_text_response_peak_mb look a bit weird.

app_handle_text_response_peak_mb	app_handle_text_response_controller_peak_mb
356.65000000	28.05625000
353.15000000	27.10000000
350.40312500	26.65937500
331.13437500	23.27265625
328.94531250	28.15234375
319.26328125	27.78984375
311.47109375	23.67968750
306.60546875	22.79296875
285.61015625	29.01171875