Merge pull request #3 from myaghobi/v1.4

V1.4

Author: myaaghubi

README.md

@@ -18,7 +18,6 @@ Components like Template engines or ORM/Database libraries are out of scope in t
   - [Commands](#commands)
   - [Benchmarks](#benchmarks)
     - [Latest](#latest)
-    - [Old](#old)
     - [OPCache](#results-with-opcache)
   - [References](#references)
   - [License](#license)
@@ -29,64 +28,73 @@ Components like Template engines or ORM/Database libraries are out of scope in t
 
 This is `master` branch.
 
-* Install a framework according to the official documentation.
+* Install a framework via `composer` according to the official documentation.
 * Use the default configuration.
   * Don't remove any components/configurations even if they are not used.
   * With minimum changes to run this benchmark.
 * Set environment production/Turn off debug mode.
 * Run the general optimization in your production environment, like `--optimize-autoloader` for the composer.
-* Use controller or action class.
+* Use controller class.
 
-Some frameworks are optimized, some are not, so some people may think using default configuration is not fair. The dept of optimizing a framework depends on the developer's knowledge & experienced, it's the rabbit hole and there is no point in that for benchmarking. I think the default configuration of frameworks is a good starting point to have the ranking points.
+Some frameworks are optimized, some are not, so some people may think using default configuration is not fair. The dept of optimizing a framework depends on the developer's experienced too, so it's the rabbit hole and there is no point in that for benchmarking. I think the default configuration of frameworks is a good starting point to have the ranking points.
 
-If you find something wrong with my code, please feel free to send Pull Requests. But please note optimizing for the "Hello World" is not acceptable! Building the fastest "Hello World" application is not the goal of this project.
+If you find something wrong with my code, feel free to send PR. But please note optimizing for the "Hello World" is not acceptable! Building the fastest "Hello World" application is not the goal of this project.
 
 
 ## How to Benchmark
 
 If you want to benchmark PHP extension frameworks like Phalcon, you need to install the extenstions.
 
-1- Install source code as <http://localhost/php-frameworks-bench/>:
+1- Download & install the library in your localhost:
 
 ```
-# you can use --branch vx.x
+# instead you can use --branch vx.x
 $ git clone https://github.com/myaghobi/php-frameworks-bench.git
 $ cd php-frameworks-bench
+# run the setup & follow the progress
 $ bash setup.sh
 ```
 
-2- Run benchmarks:
+2- To make sure everything is ok:
+
+```
+# this is optional
+$ bash check.sh
+```
+
+3- Run benchmarks:
 
 ```
 $ bash benchmark.sh
 ```
 
-3- Check the resuts:
+4- Check the resuts:
 
 <http://localhost/php-frameworks-bench/>
 
+
 ## Commands
 
 ```
-# install | create-project frameworks
-$ bash setup.sh
-
-# run the benchmark
-$ bash benchmark.sh
-
-# composer update
+# composer update for each framework
 $ bash update.sh
 
 # to clean them all
-# then you can use setup.sh 
-# trobble? use it with `sudo`
+# trouble? run it then
+# you can use setup.sh 
 $ bash clean.sh
+
+# clear the cache of frameworks
+$ bash clear-cache.sh
+
+# show the table of last benchmark
+$ bash show-table.sh
 ```
 
 To specify the frameworks, put them after each command:
 
 ```
-$ bash benchmark.sh laravel-9.1.3/ slim-4.10/ fatfree-3.7.3/
+$ bash benchmark.sh laravel-9.1/ slim-4.10/ fatfree-3.8.0/
 ...
 ```
 
@@ -97,13 +105,15 @@ $ bash benchmark.sh laravel-9.1.3/ slim-4.10/ fatfree-3.7.3/
 
 #### Environment
 
-* Ubuntu 20.04 LTS 64bit
-  * PHP 8.1.4 (LAMP)
-  * CPU Core i7 [email protected]
+* Ubuntu 22.04 LTS 64bit
+  * PHP 8.1.5
+  * OPCache disabled
+  * Apache 2.4.52
+  * CPU Core i7 2600K@4Ghz
   * Memory 16G 
 
 
-#### Results (2022/5/9)
+#### Results (2022/5/15)
 
 These are my benchmarks, not yours. **I encourage you to run on your (production equivalent) environments.**
 
@@ -112,59 +122,32 @@ These are my benchmarks, not yours. **I encourage you to run on your (production
 ![Benchmark Results Graph Execution Time](screenshots/php-frameworks-bench-exectime.png)
 ![Benchmark Results Graph Included Files](screenshots/php-frameworks-bench-includedfiles.png)
 
-PHP 8.1.4 (LAMP)
-|framework          |requests per second|relative|peak memory|relative|
-|-------------------|------------------:|-------:|----------:|-------:|
-|pure-php           |          25,363.33|25,363.3|       0.34|     0.3|
-|phroute-2.1        |          20,871.63|20,871.6|       0.34|     0.3|
-|fastroute-1.3      |          20,692.36|20,692.4|       0.34|     0.3|
-|siler-1.7.9        |          15,140.00|15,140.0|       0.36|     0.4|
-|fatfree-3.8.0      |          13,172.39|13,172.4|       0.39|     0.4|
-|slim-4.10          |          10,863.08|10,863.1|       0.38|     0.4|
-|slim-3.12          |          10,630.10|10,630.1|       0.39|     0.4|
-|yii-2.0-basic      |           8,382.51| 8,382.5|       0.69|     0.7|
-|silex-2.3          |           8,017.40| 8,017.4|       0.39|     0.4|
-|symfony-5.4        |           7,309.46| 7,309.5|       0.46|     0.5|
-|symfony-6.0        |           7,272.85| 7,272.9|       0.45|     0.5|
-|lumen-9.0.0        |           4,990.60| 4,990.6|       0.39|     0.4|
-|cakephp-4.3        |           4,471.16| 4,471.2|       0.43|     0.4|
-|fuelphp-1.9        |           3,728.35| 3,728.4|       0.45|     0.5|
-|codeigniter-4.1.9  |           1,671.15| 1,671.2|       1.46|     1.5|
-|laravel-9.1.3      |             826.64|   826.6|       0.77|     0.8|
-
-
-### Old
-
-#### Environment
-
-* Windows 10 64bit (bash executed on WSL 2)
-  * PHP 8.0 (XAMPP x64)
-  * CPU Core i7 [email protected]
-  * Memory 16G 
-
-
-#### Results (2021/12/01)
-
 |framework          |requests per second|relative|peak memory|relative|
 |-------------------|------------------:|-------:|----------:|-------:|
-|pure-php           |           2,476.19| 2,476.2|       0.42|     0.4|
-|fastroute-1.3      |           1,208.93| 1,208.9|       0.57|     0.6|
-|phroute-2.1        |           1,201.83| 1,201.8|       0.58|     0.6|
-|fatfree-3.7.3      |             695.23|   695.2|       1.67|     1.7|
-|siler-1.7.9        |             520.31|   520.3|       1.19|     1.2|
-|yii-2.0.14-basic   |             269.47|   269.5|       2.76|     2.8|
-|slim-4.9           |             239.09|   239.1|       1.60|     1.6|
-|silex-2.3          |             224.29|   224.3|       2.28|     2.3|
-|slim-4.3           |             223.43|   223.4|       1.55|     1.6|
-|fuelphp-1.8        |             180.84|   180.8|       2.72|     2.7|
-|lumen-8.1.2        |             162.19|   162.2|       3.45|     3.5|
-|cakephp-4.3        |             108.52|   108.5|       4.49|     4.5|
-|symfony-5.3        |              96.57|    96.6|       5.49|     5.5|
-|codeigniter-4.1.5  |              75.94|    75.9|       3.37|     3.4|
-|laravel-8.6.6      |              39.96|    40.0|      11.90|    11.9|
-
-
-### Results with OPCache
+|pure-php           |          22,427.34|22,427.3|       0.42|     0.4|
+|fastroute-1.3      |           4,224.76| 4,224.8|       0.56|     0.6|
+|phroute-2.2        |           3,901.59| 3,901.6|       0.58|     0.6|
+|leaf-3.0           |           1,539.38| 1,539.4|       1.05|     1.1|
+|fatfree-3.8.0      |           1,414.71| 1,414.7|       1.67|     1.7|
+|siler-1.7.9        |           1,314.15| 1,314.2|       1.17|     1.2|
+|frameworkx-dev     |             972.97|   973.0|       1.41|     1.4|
+|slim-3.12          |             969.54|   969.5|       1.40|     1.4|
+|slim-4.10          |             752.96|   753.0|       1.60|     1.6|
+|yii-2.0-basic      |             510.90|   510.9|       2.72|     2.7|
+|silex-2.3          |             510.51|   510.5|       2.18|     2.2|
+|fuelphp-1.9        |             410.60|   410.6|       2.50|     2.5|
+|ubiquity-2.4.x.dev |             379.51|   379.5|       2.33|     2.3|
+|lumen-9.0          |             329.70|   329.7|       3.39|     3.4|
+|symfony-6.0        |             293.11|   293.1|       3.85|     3.9|
+|symfony-5.4        |             291.41|   291.4|       3.77|     3.8|
+|codeigniter-4.1    |             290.94|   290.9|       3.39|     3.4|
+|laminas-2.0.x.dev  |             284.43|   284.4|       3.52|     3.5|
+|cakephp-4.3        |             232.68|   232.7|       4.45|     4.5|
+|laravel-9.1        |              93.47|    93.5|      11.04|    11.0|
+
+
+
+### Results with OPCache (old version)
 [![IMAGE ALT TEXT](http://img.youtube.com/vi/BUcxalvD92U/0.jpg)](http://www.youtube.com/watch?v=BUcxalvD92U "PHP OPCache Benchmarks, a Comparison for Popular Frameworks")
 
 
@@ -176,19 +159,23 @@ Note: This project is based on
 * [CodeIgniter](https://github.com/codeigniter4/CodeIgniter4)
 * [FastRoute](https://github.com/nikic/FastRoute)
 * [FatFree](https://github.com/bcosca/fatfree)
+* [FrameworkX](https://github.com/clue/framework-x)
 * [FuelPHP](https://github.com/fuelphp/fuelphp)
+* [Laminas](https://github.com/laminas)
 * [Laravel](https://github.com/laravel/laravel)
+* [Leaf](https://github.com/leafsphp/leaf)
 * [Lumen](https://github.com/laravel/lumen)
 * [PhRoute](https://github.com/mrjgreen/phroute)
 * [Siler](https://github.com/leocavalcante/siler)
 * [Silex](https://github.com/silexphp/Silex)
 * [Slim](https://github.com/slimphp/Slim)
 * [Symfony](https://github.com/symfony/symfony)
+* [Ubiquity](https://github.com/phpMv/ubiquity)
 * [Yii](https://github.com/yiisoft/yii2)
 
 
 ## License
 
-You are allowed to use this plugin under the terms of the GNU General Public License version 2.
+You are allowed to use this plugin/project under the terms of the GNU General Public License version 2.
 
 Copyright (C) 2022 Mohammad Yaghobi

benchmarks/_functions.sh

@@ -4,23 +4,48 @@ benchmark () {
     output_wrk="output/$fw.wrk.log"
     output="output/$fw.output"
 
-    # get rps
-    # for this version of wrk -R (--rate) is necessary to use for wsl
+    url_output=$(curl -s "$url")
+    reqsubstr="Hello World!"
+
+    # to make sure the appropriate response is reachable
+    # otherwise, there is no point running the benchmark
+    if ! [ -z "${url_output##*$reqsubstr*}" ] ;then
+        tmp=`cat "$output"`
+        error="$error$tmp"
+        echo "error!"
+        echo -e "$fw\n$error" >> "$error_file"
+        echo "---" >> "$error_file"
+
+        echo "$fw: 0: 0: 0: 0" >> "$results_file"
+        return 1
+    fi
+
+    config_wrk="wrk -t50 -c1000 -d10s"
+
+    # is it wsl!?
+    # if you're using wsl, it's necessary to put -R (--rate)
+    # in this version of wrk
     # I used a large number to make sure there be no limitation of rate
     # for a high end server, you can increase first two parameters
-    # is it wsl!?
     if grep -q Microsoft /proc/version; then
-        echo "wrk -t50 -c1000 -d10s -R1000g $url"
-        wrk -t50 -c1000 -d10s -R1000g "$url" > "$output_wrk"
+        config_wrk="$config_wrk -R1000g $url"
     else
-        echo "wrk -t50 -c1000 -d10s $url"
-        wrk -t50 -c1000 -d10s "$url" > "$output_wrk"
+        config_wrk="$config_wrk $url"
     fi
 
+    echo $config_wrk
+
+    config_wrk="$config_wrk '$url' > '$output_wrk'"
+    eval $config_wrk
+
     rps=`grep "Requests/sec:" "$output_wrk" | tr -dc '0-9.'`
 
-    count=5
+    numfmt --g "$rps rps"
 
+    # to make a small gap between the WRK and CURL
+    sleep 1
+
+    count=5
     total=0
     i=0
     # The for (( expr ; expr ; expr )) syntax is not available in sh, so:

benchmarks/hello_world.sh

@@ -18,11 +18,16 @@ mv "$check_file" "$check_file.old"
 mv "$error_file" "$error_file.old"
 mv "$url_file" "$url_file.old"
 
+
+phpv=`php -r 'echo phpversion();'`
+echo "/------- PHP $phpv -------/"
+
 for fw in `echo $targets`
 do
     if [ -d "$fw" ]; then
         echo "/------- $fw: benchmarking -------/"
         echo "$fw/_benchmark/hello_world.sh"
+        php ./libs/reset_opcache.php
         . "$fw/_benchmark/hello_world.sh"
         benchmark "$fw" "$url"
     fi

bin/show_results_table.php

@@ -4,6 +4,5 @@
 require __DIR__ . '/../libs/build_table.php';
 
 $results = parse_results(__DIR__ . '/../output/results.hello_world.log');
-//var_dump($results);
 
 echo build_table($results);

cakephp-4.3/_benchmark/clear-cache.sh

@@ -0,0 +1,4 @@
+#!/bin/sh
+# clear cache
+bin/cake cache clear_all
+echo -e "done"

cakephp-4.3/_benchmark/hello_world.sh

@@ -1,2 +1,2 @@
 #!/bin/sh
-url="$base/$fw/index.php/hello/index"
+url="$base/$fw/webroot/index.php/hello/index"

cakephp-4.3/_benchmark/setup.sh

@@ -7,7 +7,7 @@ yes|mv ./_benchmark/temp/{.,}* ./
 # have the route & controller
 yes|cp -r _benchmark/cakephp/* ./
 
-# some enhancement
+# some enhancements
 composer dump-autoload -o
-sudo rm -rf tmp/*
-composer install --no-interaction --no-dev -o
+composer install --no-interaction --no-dev -o
+rm ./webroot/.htaccess

cakephp-4.3/_benchmark/update.sh

@@ -4,7 +4,7 @@ composer update
 # have the route & controller
 yes|cp -r _benchmark/cakephp/* ./
 
-# some enhancement
+# some enhancements
 composer dump-autoload -o
-sudo rm -rf tmp/*
-composer install --no-interaction --no-dev -o
+composer install --no-interaction --no-dev -o
+rm ./webroot/.htaccess

check.sh

@@ -0,0 +1,30 @@
+#!/bin/sh
+
+base="http://127.0.0.1/php-frameworks-bench"
+
+# cd `dirname $0`#
+
+if [ $# -eq 0 ]; then
+    # include framework list
+    . ./list.sh
+    export targets="$list"
+else
+    export targets="${@%/}"
+fi
+
+for fw in `echo $targets`
+do
+    if [ -d "$fw" ]; then
+        echo -n "/------- $fw: checking... "
+        . "$fw/_benchmark/hello_world.sh"
+
+        url_output=$(curl -s "$url")
+
+        if ! [[ "$url_output" =~ ^('Hello World!')(.*)(([0-9]*):(([0-9]+([.][0-9]*)?|[.][0-9]+)):([0-9]*))$ ]]; then
+            echo -e "error: \n$url"
+            echo "$url_output"
+        else
+            echo "done."
+        fi
+    fi
+done