Running | Benchmark Numbers
A first thing to notice about the benchmarks reported in the OOPSLA paper is that they rely on transformations done by hand based on the miniboxing transformation rules. The reasons for not using the miniboxing plugin are twofold:
- using files coded by hand gives a much faster turnaround time for exploring new translations
- the plugin has a limited set of language features it is capable of translating, mainly due to technical reasons. Limiting to these features would have significantly hindered exploration. Of course, as the project matures, more and more language features will be supported, but for now, the transformation is pretty basic, as you can see from the page on known limitations.
Fortunately, the Scala language is very versatile, allowing users to express both high and low-level features at the same time. This topic is further explored in the transformation prototyping page.
This said, the benchmarks can be ran using the Terminal (with sbt) link in the Miniboxing directory on the desktop. To run the benchmarks, you can use the miniboxing-benchmarks/run command. The following is an example of running this command:
[info] Loading project definition from /home/acc/Workspace/miniboxing-plugin/project
[info] Set current project to miniboxing (in build file:/home/acc/Workspace/miniboxing-plugin/)
> miniboxing-benchmarks/run
[warn] Potentially incompatible versions of dependencies of {file:/home/acc/Workspace/miniboxing-plugin/}miniboxing-benchmarks:
[warn] org.scala-lang: 2.10.3-SNAPSHOT, 2.10.1, 2.10.0
[info] Compiling 1 Scala source to /home/acc/Workspace/miniboxing-plugin/tests/benchmarks/target/scala-2.10/classes...
[info] Running miniboxing.benchmarks.launch.BenchmarkingTest
ideal.array.insert : Parameters(size -> 1000): 0.01426
ideal.array.reverse: Parameters(size -> 1000): 0.00617
... [ after some good minutes or hours, depending on test size] ...
array.insert:
ideal : 0.01426 +/- 0.00012
miniboxed dispatch mono : ....... +/- .......
miniboxed dispatch mega : ....... +/- .......
miniboxed dispatch w/cl mono : ....... +/- .......
miniboxed dispatch w/cl mega : ....... +/- .......
miniboxed standard FS mono : ....... +/- .......
miniboxed standard FS mega : ....... +/- .......
miniboxed standard SS mono : ....... +/- .......
miniboxed standard SS mega : ....... +/- .......
miniboxed standard DT mono : ....... +/- .......
miniboxed standard DT mega : ....... +/- .......
miniboxed standard LI mono : ....... +/- .......
miniboxed standard LI mega : ....... +/- .......
miniboxed standard NI mono : ....... +/- .......
miniboxed standard NI mega : ....... +/- .......
miniboxed standard w/cl mono : ....... +/- .......
miniboxed standard w/cl mega : ....... +/- .......
specialized mono : ....... +/- .......
specialized mega : ....... +/- .......
generic mono : ....... +/- .......
generic mega : ....... +/- .......
array.reverse:
ideal : 0.00617 +/- 0.00001
.................................
array.find:
.................................
list.insert:
.................................
list.hashCode:
.................................
list.find:
.................................
The size of the tests can be controlled by starting Eclipse with Miniboxing Project and opening the BenchmarkingTest.scala file in the miniboxing-benchmarks project, in the miniboxing.benchmarks.launch package:
lazy val testSizes = {
List(1000)
}The tests can also be enabled and disabled from this file, in order to obtain the exact equivalents in the paper. Please notice that the paper runs examples on 3 million (3000000) elements to eliminate as much of the noise as possible. Also, please keep in mind that the paper benchmarks were executed directly on the bare metal instead of a virtual machine. Finally, the Java distribution in the virtual machine is different form the one used for the paper benchmarks: Due to licensing issues, the virtual machine comes with the OpenJDK Java Runtime Environment 1.7.0_09 instead of the Oracle/Sun Java SE Runtime Environment (build 1.6.0_26-b03) used for benchmarking.
The first time you execute the benchmarks, the labels probably don't look very familiar. Below is the correspondence between labels in the paper and labels in the benchmarks:
Label from benchmarks => Label from paper
------------------------------------------------------
array.insert => ArrayBuffer.append
array.reverse => ArrayBuffer.reverse
array.find => ArrayBuffer.contains
list.insert => List creation
list.hashCode => List.hashCode
list.find => List.contains
mono => Single Context
multi => Multi Context
ideal => monomorphic
miniboxed dispatch => miniboxing with dispatcher
miniboxed dispatch w/cl => miniboxing with dispatcher and runtime support
miniboxed standard FS => miniboxing full-switching support
miniboxed standard SS => miniboxing semi-switching support
miniboxed standard DT => miniboxing decision tree support
miniboxed standard LI => miniboxing linear if support
miniboxed standard NI => miniboxing non-inlining full-switch support
miniboxed standard w/cl => miniboxing full-switching and runtime support
specialized => specialization
generic => generic
MBResizableArray => ArrayBuffer
MBList => List
The authors apologize for using mismatching labels. This mismatch was caused by attempting to use the most relevant labels in the paper to better transmit the message. On the other hand, the labels in the project reflect the realities in the code.
In order to run the miniboxing benchmarks in interpreter-only mode one needs to modify the ScalameterBenchTest.scala file in the miniboxing-benchmarks project, in the miniboxing.benchmarks.launch.scalameter package:
def test[T](transformation: String, tag: String, setup: Int => Unit, benchmark: => Unit, teardown: => Unit) = {
performance of transformation in {
measure method tag in {
using(sizes) config (exec.independentSamples -> 20) setUp {
size => testSize = size; System.gc(); setup(size); System.gc();
} tearDown {
teardown; size => testSize = 0
} in {
size => benchmark
}
}
}
}The config line needs to contain exec.jvmflags -> "-Xint":
using(sizes) config (exec.independentSamples -> 20, exec.jvmflags -> "-Xint") setUp {And then you can leave the computer crunching numbers for the next day. :)
The Graal benchmarks are only available for local installations. See the Graal benchmark instructions page for more details.
In the Miniboxing OOPSLA13 paper we included bytecode size benchmarks, so you can check the miniboxed class size. This will be done with the miniboxed class:
$ cd ~/Workspace/miniboxing-plugin/sandbox
$ mb-scalac -optimize ../tests/benchmarks/src/miniboxing/benchmarks/minibox/MBResizableArray.scala
$ find ./miniboxing/benchmarks/minibox/MBResizableArray* -name "*.class" -ls | awk '{total += $7} END {print total}'
12044
$ scalac -optimize ../tests/benchmarks/src/miniboxing/benchmarks/generic/ResizableArray.scala
$ find ./miniboxing/benchmarks/generic/ResizableArray* -name "*.class" -ls | awk '{total += $7} END {print total}'
4505
$ mb-scalac -optimize ../tests/benchmarks/src/miniboxing/benchmarks/minibox/MBList.scala
$ find ./miniboxing/benchmarks/minibox/MBList* -name "*.class" -ls | awk '{total += $7} END {print total}'
9083
$ scalac -optimize ../tests/benchmarks/src/miniboxing/benchmarks/generic/List.scala
$ find ./miniboxing/benchmarks/generic/List* -name "*.class" -ls | awk '{total += $7} END {print total}'
3146
In our case, these reported 12044 bytes for MBResizableArray and 9083 for MBList, which are better than the sizes reported in the paper, 24.5 Kbytes for MBResizableArray and 11.5 Kbytes for MBList.
You can continue with the following resources:
