Multiple stacks

.gitignore

@@ -1,3 +1,2 @@
 /nbproject/
-/output/
 vendor/

OTHER_RESULTS.md

@@ -0,0 +1,115 @@
+# PHP ORM Benchmark
+
+## Other Results
+
+### [motin](https://github.com/motin)
+
+(2015-12-10)
+
+Using [the supplied Docker Stack]((docker/README.md)):
+
+* Ubuntu 15.04 64bit (Docker)
+  * Nginx 1.7.12
+  * PHP-FPM 5.6.4
+    * Zend OPcache 7.0.4-dev
+    * PhalconPHP 2.0.9
+  * PHP-FPM 7.0.0
+    * Zend OPcache 7.0.6-dev
+    * PhalconPHP 2.0.9
+  * HHVM 3.10.1
+
+Running on a MacBook Pro (Retina, 15-inch, Mid 2014).
+
+By sharing underlying software stacks, the benchmark results vary only according to the host machine's hardware specs and differing code implementations.
+
+Note: Frameworks that were currently not reporting complete benchmark results are zeroed out in the graphs
+
+#### PHP-FPM 5.6.4 with opcode cache
+
+<img width="1002" alt="php_framework_benchmark" src="https://cloud.githubusercontent.com/assets/793037/11716938/cf4d6c5e-9f55-11e5-90f3-c177a0f6f35c.png">
+
+|framework          |requests per second|relative|peak memory|relative|
+|-------------------|------------------:|-------:|----------:|-------:|
+|phalcon-2.0        |             965.16|    53.7|       0.27|     1.0|
+|ice-1.0            |             887.56|    49.4|       0.26|     1.0|
+|tipsy-0.10         |           1,186.60|    66.0|       0.32|     1.2|
+|fatfree-3.5        |             549.20|    30.5|       0.43|     1.6|
+|slim-2.6           |             839.07|    46.7|       0.48|     1.8|
+|ci-3.0             |             114.04|     6.3|       0.43|     1.6|
+|nofuss-1.2         |             282.49|    15.7|       0.59|     2.2|
+|slim-3.0           |             578.63|    32.2|       0.62|     2.3|
+|bear-1.0           |              50.19|     2.8|       0.77|     2.9|
+|lumen-5.1          |               0.00|     0.0|       0.00|     0.0|
+|ze-1.0             |             270.54|    15.0|       0.80|     3.0|
+|radar-1.0-dev      |             271.33|    15.1|       0.71|     2.7|
+|yii-2.0            |             316.65|    17.6|       1.36|     5.1|
+|silex-1.3          |               0.00|     0.0|       0.00|     0.0|
+|cygnite-1.3        |             129.65|     7.2|       0.76|     2.9|
+|fuel-1.8-dev       |              56.63|     3.1|       0.71|     2.7|
+|phpixie-3.2        |              82.51|     4.6|       1.30|     4.9|
+|aura-2.0           |             139.26|     7.7|       0.90|     3.4|
+|cake-3.1           |               0.00|     0.0|       0.00|     0.0|
+|symfony-2.7        |               0.00|     0.0|       0.00|     0.0|
+|laravel-5.1        |               0.00|     0.0|       0.00|     0.0|
+|zf-2.5             |              17.98|     1.0|       3.22|    12.2|
+|typo3f-3.0         |               0.00|     0.0|       0.00|     0.0|
+
+#### PHP-FPM 7.0.0 with opcode cache
+
+<img width="1002" alt="php_framework_benchmark" src="https://cloud.githubusercontent.com/assets/793037/11716920/c005b09e-9f55-11e5-8cb3-b932a435e725.png">
+
+|framework          |requests per second|relative|peak memory|relative|
+|-------------------|------------------:|-------:|----------:|-------:|
+|phalcon-2.0        |               0.00|     0.0|       0.00|     0.0|
+|ice-1.0            |               0.00|     0.0|       0.00|     0.0|
+|tipsy-0.10         |           1,497.81|    35.7|       0.37|     1.0|
+|fatfree-3.5        |             664.94|    15.9|       0.44|     1.2|
+|slim-2.6           |           1,174.32|    28.0|       0.49|     1.3|
+|ci-3.0             |             120.21|     2.9|       0.38|     1.0|
+|nofuss-1.2         |             332.31|     7.9|       0.66|     1.8|
+|slim-3.0           |             907.03|    21.6|       0.58|     1.6|
+|bear-1.0           |               0.00|     0.0|       0.00|     0.0|
+|lumen-5.1          |               0.00|     0.0|       0.00|     0.0|
+|ze-1.0             |             407.75|     9.7|       0.68|     1.8|
+|radar-1.0-dev      |             395.55|     9.4|       0.62|     1.7|
+|yii-2.0            |             497.33|    11.9|       1.02|     2.8|
+|silex-1.3          |               0.00|     0.0|       0.00|     0.0|
+|cygnite-1.3        |             169.69|     4.0|       0.65|     1.8|
+|fuel-1.8-dev       |              69.26|     1.7|       0.63|     1.7|
+|phpixie-3.2        |             158.76|     3.8|       1.01|     2.7|
+|aura-2.0           |             202.28|     4.8|       0.74|     2.0|
+|cake-3.1           |               0.00|     0.0|       0.00|     0.0|
+|symfony-2.7        |               0.00|     0.0|       0.00|     0.0|
+|laravel-5.1        |               0.00|     0.0|       0.00|     0.0|
+|zf-2.5             |              41.94|     1.0|       1.88|     5.1|
+|typo3f-3.0         |               0.00|     0.0|       0.00|     0.0|
+
+#### HHVM 3.10.1 (Corresponding roughly to an up-to-date PHP 5.6)
+
+<img width="1002" alt="php_framework_benchmark" src="https://cloud.githubusercontent.com/assets/793037/11716924/c83b4724-9f55-11e5-9a3f-a5cf7abf23e4.png">
+
+|framework          |requests per second|relative|peak memory|relative|
+|-------------------|------------------:|-------:|----------:|-------:|
+|phalcon-2.0        |               0.00|     0.0|       0.00|     0.0|
+|ice-1.0            |               0.00|     0.0|       0.00|     0.0|
+|tipsy-0.10         |             180.92| 1,292.3|       0.04|     1.0|
+|fatfree-3.5        |             200.91| 1,435.1|       0.07|     1.7|
+|slim-2.6           |             124.64|   890.3|       0.05|     1.2|
+|ci-3.0             |              41.37|   295.5|       0.05|     1.2|
+|nofuss-1.2         |             156.60| 1,118.6|       0.05|     1.2|
+|slim-3.0           |              38.04|   271.7|       0.07|     1.7|
+|bear-1.0           |               0.22|     1.6|       0.13|     3.2|
+|lumen-5.1          |              47.54|   339.6|       0.24|     5.9|
+|ze-1.0             |              20.85|   148.9|       0.16|     4.0|
+|radar-1.0-dev      |              22.95|   163.9|       0.09|     2.2|
+|yii-2.0            |              31.43|   224.5|       0.51|    12.6|
+|silex-1.3          |              26.85|   191.8|       0.11|     2.7|
+|cygnite-1.3        |              10.27|    73.4|       0.18|     4.4|
+|fuel-1.8-dev       |              14.93|   106.6|       0.13|     3.2|
+|phpixie-3.2        |               0.21|     1.5|       0.31|     7.7|
+|aura-2.0           |              21.55|   153.9|       0.17|     4.2|
+|cake-3.1           |               4.95|    35.4|       0.18|     4.4|
+|symfony-2.7        |               0.16|     1.1|       0.72|    17.8|
+|laravel-5.1        |               0.00|     0.0|       0.00|     0.0|
+|zf-2.5             |               0.14|     1.0|       0.66|    16.3|
+|typo3f-3.0         |               0.00|     0.0|       0.00|     0.0|

README.md

@@ -99,6 +99,8 @@ These are my benchmarks, not yours. **I encourage you to run on your (production
 
 Note(1): All the results are run on php with `phalcon.so` and `ice.so`. If you don't load phalcon.so or ice.so, the rps except for Phalcon or Ice probably increase a bit.
 
+If you are interested in other results, see [OTHER_RESULTS.md](OTHER_RESULTS.md).
+
 ## How to Benchmark
 
 Install source code as <http://localhost/php-framework-benchmark/>:

bear-1.0/var/log/.gitignore

@@ -0,0 +1,2 @@
+*
+!.gitignore

benchmark.sh

@@ -1,6 +1,12 @@
 #!/bin/sh
 
-base="http://127.0.0.1/php-framework-benchmark"
+# exit on failure
+set -e
+
+if [ "$stack" = "" ]; then
+  stack="local"
+fi
+export stack;
 
 cd `dirname $0`
 
@@ -14,6 +20,6 @@ fi
 
 cd benchmarks
 
-sh hello_world.sh "$base"
+sh hello_world.sh "$stack"
 
 php ../bin/show_results_table.php

benchmarks/_functions.sh

@@ -2,38 +2,52 @@ benchmark ()
 {
     fw="$1"
     url="$2"
-    ab_log="output/$fw.ab.log"
-    output="output/$fw.output"
+    ab_log="$output_dir/$fw.ab.log"
+    output="$output_dir/$fw.output"
+    benchmark_data="$output_dir/$fw.benchmark_data"
 
     echo "ab -c 10 -t 3 $url"
     ab -c 10 -t 3 "$url" > "$ab_log"
-    curl "$url" > "$output"
+    curl -H 'X-Include-Benchmark-Output-Data: 1' --dump-header "$benchmark_data" "$url" > "$output"
 
     rps=`grep "Requests per second:" "$ab_log" | cut -f 7 -d " "`
-    memory=`tail -1 "$output" | cut -f 1 -d ':'`
-    time=`tail -1 "$output" | cut -f 2 -d ':'`
-    file=`tail -1 "$output" | cut -f 3 -d ':'`
+    memory=`grep "X-Benchmark-Output-Data:" "$benchmark_data" | cut -f 2 -d ':' | cut -f 2 -d ' '`
+    time=`grep "X-Benchmark-Output-Data:" "$benchmark_data" | cut -f 3 -d ':'`
+    file=`grep "X-Benchmark-Output-Data:" "$benchmark_data" | cut -f 4 -d ':'`
     echo "$fw: $rps: $memory: $time: $file" >> "$results_file"
 
     echo "$fw" >> "$check_file"
     grep "Document Length:" "$ab_log" >> "$check_file"
     grep "Failed requests:" "$ab_log" >> "$check_file"
     grep 'Hello World!' "$output" >> "$check_file"
+    grep ':' "$output" >> "$check_file"
     echo "---" >> "$check_file"
 
     # check errors
     touch "$error_file"
     error=''
-    x=`grep 'Failed requests:        0' "$ab_log"`
+    x=`grep 'Failed requests:        0' "$ab_log" || true`
     if [ "$x" = "" ]; then
         tmp=`grep "Failed requests:" "$ab_log"`
         error="$error$tmp"
     fi
-    x=`grep 'Hello World!' "$output"`
+    x=`grep 'Hello World!' "$output" || true`
     if [ "$x" = "" ]; then
         tmp=`cat "$output"`
         error="$error$tmp"
     fi
+    if [ "$memory" = "" ]; then
+        tmp=`cat "$benchmark_data"`
+        error="$error$tmp"
+    fi
+    if [ "$time" = "" ]; then
+        tmp=`cat "$benchmark_data"`
+        error="$error$tmp"
+    fi
+    if [ "$file" = "" ]; then
+        tmp=`cat "$benchmark_data"`
+        error="$error$tmp"
+    fi
     if [ "$error" != "" ]; then
         echo -e "$fw\n$error" >> "$error_file"
         echo "---" >> "$error_file"

benchmarks/hello_world.sh

@@ -1,22 +1,42 @@
 #!/bin/sh
 
+# exit on failure - use during development
+#set -e
+
 cd `dirname $0`
 . ./_functions.sh
 
-base="$1"
+stack="$1"
 bm_name=`basename $0 .sh`
 
-results_file="output/results.$bm_name.log"
-check_file="output/check.$bm_name.log"
-error_file="output/error.$bm_name.log"
-url_file="output/urls.log"
+if [ "$stack" = "local" ]; then
+  base="http://127.0.0.1/php-framework-benchmark"
+fi
+if [ "$stack" = "docker_nginx_php_5_6_4" ]; then
+  base="http://nginx_php_5_6_4/php-framework-benchmark"
+fi
+if [ "$stack" = "docker_nginx_hhvm_3_10_1" ]; then
+  base="http://nginx_hhvm_3_10_1/php-framework-benchmark"
+fi
+if [ "$stack" = "docker_nginx_php_7_0_0" ]; then
+  base="http://nginx_php_7_0_0/php-framework-benchmark"
+fi
+
+output_dir="output/$stack"
+
+results_file="$output_dir/results.$bm_name.log"
+check_file="$output_dir/check.$bm_name.log"
+error_file="$output_dir/error.$bm_name.log"
+url_file="$output_dir/urls.log"
 
 cd ..
 
-mv "$results_file" "$results_file.old"
-mv "$check_file" "$check_file.old"
-mv "$error_file" "$error_file.old"
-mv "$url_file" "$url_file.old"
+mkdir -p "$output_dir"
+
+mv "$results_file" "$results_file.old" || true
+mv "$check_file" "$check_file.old" || true
+mv "$error_file" "$error_file.old" || true
+mv "$url_file" "$url_file.old" || true
 
 for fw in `echo $targets`
 do
@@ -27,4 +47,4 @@ do
     fi
 done
 
-cat "$error_file"
+cat "$error_file" || true

bin/show_comparison_table.php

@@ -4,16 +4,19 @@
 require __DIR__ . '/../libs/build_table.php';
 require __DIR__ . '/../libs/recalc_relative.php';
 
+$stack = getenv('stack') ? getenv('stack') : 'local';
+$output_dir = __DIR__ . '/../output/' . $stack;
+
 $list = [
     'cake-3.0',
     'symfony-2.6',
     'zf-2.4',
 ];
 
 system('git checkout master');
-$results_master = parse_results(__DIR__ . '/../output/results.hello_world.log');
+$results_master = parse_results($output_dir . '/results.hello_world.log');
 system('git checkout optimize');
-$results_optimize = parse_results(__DIR__ . '/../output/results.hello_world.log');
+$results_optimize = parse_results($output_dir . '/results.hello_world.log');
 //var_dump($results_master, $results_optimize);
 
 $is_fisrt = true;

bin/show_fw_array.sh

@@ -3,17 +3,4 @@
 cd `dirname $0`
 cd ..
 
-# include framework list
-. ./list.sh
-targets="$list"
-
-echo '['
-
-for fw in $targets
-do
-    if [ -d "$fw" ]; then
-        echo "\t'$fw',"
-    fi
-done
-
-echo ']'
+php -r 'require("list.php"); var_export(frameworks());'

bin/show_results_table.php

@@ -3,7 +3,10 @@
 require __DIR__ . '/../libs/parse_results.php';
 require __DIR__ . '/../libs/build_table.php';
 
-$results = parse_results(__DIR__ . '/../output/results.hello_world.log');
+$stack = getenv('stack') ? getenv('stack') : 'local';
+$output_dir = __DIR__ . '/../output/' . $stack;
+
+$results = parse_results($output_dir . '/results.hello_world.log');
 //var_dump($results);
 
 echo build_table($results);

docker/README.md

@@ -0,0 +1,69 @@
+## Benchmarking using the supplied Docker Stack
+
+Use the supplied Docker Stack in order to automatically set up the following benchmarking environments:
+
+* Ubuntu 15.04 64bit (Docker)
+  * Nginx 1.7.12
+  * PHP-FPM 5.6.4
+    * Zend OPcache 7.0.4-dev
+    * PhalconPHP 2.0.9
+  * PHP-FPM 7.0.0
+    * Zend OPcache 7.0.6-dev
+    * PhalconPHP 2.0.9
+  * HHVM 3.10.1
+
+By sharing underlying software stacks, the benchmark results vary only according to the host machine's hardware specs and differing code implementations.
+
+### Getting Started
+
+If running locally, install [Docker Toolbox](https://www.docker.com/docker-toolbox).
+
+Clone the source code:
+~~~
+$ git clone https://github.com/kenjis/php-framework-benchmark.git
+$ cd php-framework-benchmark
+~~~
+
+Cd into the docker directory of this repo and make sure that docker toolbox is available:
+~~~
+$ cd docker
+$ eval "$(docker-machine env default)"
+~~~
+
+Start the Nginx/PHP server stacks:
+~~~
+$ docker-compose up -d
+~~~
+
+Start the supplied docker shell from within this repository's `docker` folder:
+~~~
+$ docker-compose build shell
+$ docker-compose run shell
+~~~
+
+Run the set-up script:
+~~~
+# sh setup.sh
+~~~
+
+Run benchmarks against each docker stack:
+~~~
+$ docker/bin/benchmark.sh
+~~~
+
+### Check the results
+
+To see the results tables in markdown, run the following from within the docker shell:
+
+~~~
+$ docker/bin/results.sh
+~~~
+
+To see the results graph, run the following script from **outside** the docker shell, from the repository root:
+
+~~~
+$ docker/bin/urls.sh
+~~~
+
+It echoes URLs, which you should open up in your browser.
+