Effects of order on results

[myaaghubi]

@joanhey Managed to run it without docker. 15.9% more rps for maravel vs lumen with this PR

UPDATE
The results vary by the order from config...

When done one at a time

vs

vs

vs

Originally posted by @macropay-solutions in #77 (comment)

[myaaghubi]

Thanks @macropay-solutions for the benchmarks. I ran many tests to ensure that the order, logs, etc., don't affect the results of other frameworks; everything was fine unless having opcatch ON for Laravel and a couple of other frameworks that I don't remember. Can you share some details about your benchmark environment?

[marius-ciclistu]

Ubuntu 24.04 with php 8.3 on a thinkpad p52 32 gb ram.
Opcache not enabled.

[marius-ciclistu]

@myaaghubi Could you please try in one run just maravel and lumen and then change their order from config and run again?

[myaaghubi]

I'll check it

[joanhey]

I made and know a lot of benchmarks, teorically it's necessary ~10 sec between frameworks, that's only using PHP-FPM.
When you benchmark async fws (persistents) always is good to make it separate for each fw. I have seen benchmarks where some persitents frameworks use +5% CPU usage without any request.

[joanhey]

Ideally all fw need to be benchmarked alone.
Examples on the old Kenjis:

• Separate php.ini for phalcon? kenjis/php-framework-benchmark#8
• 'ab' is not fast enough kenjis/php-framework-benchmark#62

Also with PHP-FPM, some frameworks made slower the rest of fws, because they consume CPU without any request (phalcon, yaf,...)

[myaaghubi]

I use this command to run benchmarks

$ bash benchmark.sh -f -rapache

PHP-Frameworks-Bench/README.md

Lines 159 to 160 in ff13aa3

	# use bash benchmark.sh --help
	$ bash benchmark.sh -f -rapache -t pure-php

-f for a fresh install, every time composer takes a few minutes at least

PHP-Frameworks-Bench/base/hello_world.sh

Lines 45 to 48 in ff13aa3

	if [ "$param_fresh" = true ]; then
	bash clean.sh
	bash setup.sh -t "$fw"
	fi

-rapache to reset Apache

PHP-Frameworks-Bench/base/hello_world.sh

Lines 63 to 66 in ff13aa3

	if [ "$param_restart_apache" = true ]; then
	echo 'systemctl restart apache2'
	sudo systemctl restart apache2
	fi

-For each benchmark, there is an opcache reset and a 2 sec space

PHP-Frameworks-Bench/base/hello_world.sh

Line 60 in ff13aa3

opcacherest=`curl -s "$base/libs/reset_opcache.php"`

PHP-Frameworks-Bench/base/hello_world.sh

Lines 73 to 74 in ff13aa3

	# a small gap between composer, reset_opcache, webserver restarted and benchmark
	sleep 2

These are all new on this repo rather than the older one

[myaaghubi]

Check out bash benchmark.sh --help

PHP-Frameworks-Bench/benchmark.sh

Lines 26 to 31 in ff13aa3

	Optional Arguments:
	-f, --fresh Clean all frameworks and install the target framework -fresh install- before benchmark.
	-h, --help Show this help message and exit
	-rapache, --restart-apache Restart apache "sudo systemctl restart apache2" before each benchmark.
	-rnginx, --restart-nginx Restart apache "sudo systemctl restart nginx" before each benchmark.
	-t, --target Specify your target framework/s for benchmarking

[myaaghubi]

Thanks @macropay-solutions for the benchmarks. I ran many tests to ensure that the order, logs, etc., don't affect the results of other frameworks; everything was fine unless having opcatch ON for Laravel and a couple of other frameworks that I don't remember. Can you share some details about your benchmark environment?

There should be no problem with bash benchmark.sh -f -rapache
Let me know if it fixes the issue

[marius-ciclistu]

@myaaghubi I had to install apache just to run the test. Then uninstall it because it was messing up with my docker setup. Would help if you confirm that with that command the order from config is not altering the result. (I am from phone).

[myaaghubi]

Based on my tests, everything seems to be fine. Yet for your environment, you'll need to make sure by running it yourself.

[marius-ciclistu]

My goal was to check it against lumen. It is faster and I am happy with that. You can update the documentation for this project with this situation for others to know about it.

[marius-ciclistu]

@myaaghubi I think you can close this issue.

[marius-ciclistu]

@myaaghubi
I tested with benchmark.sh -f -rapache and the config order does not seem to count anymore:

PHP 8.1

But the latest features added to Maravel (event:cache with observers included, autowiring:cache and prevent circular dependency) made it drop below lumen.

[marius-ciclistu]

@myaaghubi however these new cache commands almost double the RPS when used in Maravel vs Lumen and Maravelith vs Laravel has also an increase when both are optimized (without octane):