CONTRIBUTION.md

Contribution Guide

Code organization and internals

The code is organized into three main parts:

• renaissance-core: these are the core APIs that a benchmark needs to work with, such as the runtime configuration, the benchmark base class or the policy.
• renaissance-harness: this is the overall suite project, which is responsible for parsing the arguments, loading the classes, and running the benchmark.
• a set of projects in the benchmarks directory: these are the individual groups of benchmarks that share a common set of dependencies. A group is typically thematic, relating to a particular framework or a JVM language, and benchmarks in different groups depend on a different set of libraries, or even the same libraries with different versions. The bundling mechanism of the suite takes care that the dependencies of the different groups never clash with each other.

Adding a new benchmark

To add a new benchmark to an existing group, identify the respective project in the benchmarks directory, and add a new top-level Scala (Java) class that extends (implements) the Benchmark interface.

Here is an example:

import org.renaissance._
import org.renaissance.Benchmark._

@Summary("Runs some performance-critical Java code.")
final class MyJavaBenchmark extends Benchmark {
  override def runIteration(config: Config): BenchmarkResult = {
    // This is the benchmark body, which in this case calls some Java code.
    JavaCode.runSomeJavaCode()
    // Return object for later validation of the iteration.
    return new MyJavaBenchmarkResult()
  }
}

Above, the name of the benchmark will be automatically generated from the class name. In this case, the name will be my-java-benchmark.

To create a new group of benchmarks (for example, benchmarks that depend on a new framework), create an additional sbt project in the benchmarks directory, using an existing project, such as scala-stdlib, as an example. The project will be automatically picked up by the build system and included into the Renaissance distribution.

Once the benchmark has been added, one needs to make sure to be compliant with the code formatting of the project (rules defined in .scalafmt.conf). A convenient sbt task can do that check:

$ tools/sbt/bin/sbt renaissanceFormatCheck

Another one can directly update the source files to match the desired format:

$ tools/sbt/bin/sbt renaissanceFormat

Moreover, the contents of the README and CONTRIBUTION files are automatically generated from the codebase. Updating those files can be done with the --readme command-line flag. Using sbt, one would do:

$ tools/sbt/bin/sbt runMain org.renaissance.core.Launcher --readme

IDE development

IntelliJ

In order to work on the project with IntelliJ, one needs to install the following plugins :

• Scala : part of the codebase uses Scala and Renaissance is organized in sbt projects.
• scalafmt (optional) : adds an action that can be triggered with Code -> Reformat with scalafmt which will reformat the code according to the formatting rule defined in .scalafmt.conf (same as the renaissanceFormat sbt task).

Then, the root of this repository can be opened as an IntelliJ project.

Benchmark evaluation and release process

In the open-source spirit, anyone can propose an additional benchmark by opening a pull request. The code is then inspected by the community -- typically, the suite maintainers are involved in the review, but anybody else is free join the discussion. During the discussion, the reviewers suggest the ways in which the benchmark could be improved.

Before submitting a pull request, it is recommendable to open an issue first, and discuss the benchmark proposal with the maintainers.

Benchmark criteria

Here is some of the quality criteria that a new benchmark should satisfy:

• Stylistically conformant: the benchmark code must conform to existing formatting and stylistic standards in the suite.
• Meaningful: it must represent a meaningful workload that is either frequently executed, or it consists of code patterns and coding styles that are desired or arguably preferable, or it encodes some important algorithm, a data structure or a framework, or is significant in some other important way. If the benchmark is automatically generated by some tool, then it must be accompanied with a detailed explanation of why the workload is useful.
• Observable: it must constitute a run that is observable and measurable. For example, the run should last sufficiently long so that it is possible to accurately measure its running time, gather profiling information or observe a performance effect. Typically, this means that the benchmark duration should be between 0.5 and 10 seconds.
• Deterministic: the performance effects of the benchmark should be reproducible. Even if the benchmark consists of, for example, concurrent code that is inherently non-deterministic, the benchmark should behave relatively deterministically in terms of the number of threads that it creates, the objects it allocates, and the total amount of computation that it needs to do. For example, each of the benchmark repetitions should have a relatively stable running time on major JVMs.
• New: it must contain some new functionality that is not yet reflected in any of the existing benchmarks. If there is significant overlap with an existing benchmark, then it should be explained why the new benchmark is necessary.
• Self-contained: it must be runnable within a single JVM instance, without any additional software installation, operating system functionality, operating system services, other running processes, online services, networked deployments or similar. The JVM installation should be sufficient to run all the code of the benchmark. For example, if the benchmark is exercising Apache Spark, then the workers should run in the local mode, and if the benchmark targets a database, then it should be embedded. The benefit is that the performance effects of the benchmark are easy to measure, and the code is reproducible everywhere.
• Open-source: the benchmark must consist of open-source code, with well-defined licenses.

Code of Conduct

We would also like to point you to the Renaissance Code of Conduct. As a member of the Renaissance community, make sure that you follow it to guarantee an enjoyable experience for every member of the community.

Release process

While the open-source process is designed to accept contributions on an ongoing basis, we expect that this benchmark suite will grow considerably over the course of time. We will therefore regularly release snapshots of this suite, which will be readily available. These will be known as minor releases.

Although we will strive to have high-quality, meaningful benchmarks, it will be necessary to proliferate the most important ones, and publish them as major releases. This way, researchers and developers will be able to test their software against those benchmarks that were deemed most relevant. A major release will still include all the benchmarks in the suite, but the list of highlighted benchmarks might evolve between major releases.

Once a year, a committee convenes and discusses which important benchmarks were contributed since the last release, and should become highlighted in the next major release. The committee consists of members from various universities and companies, who are involved in research and development in virtual machine, compiler, memory management, static and dynamic analysis, and performance engineering.

The committee goes over the benchmark list, and inspect the new ones since the last release. The members can recommend specific benchmarks for inclusion in the highlighted list, and can present their arguments about why those benchmarks should be included. In addition, the members can recommend the exclusion of some benchmarks. The committee members then vote, and the majority is the basis for a decision.

The new major release is then bundled and the binaries are made available publicly.

The current members of the committee are:

• Walter Binder, Universita della Svizzera italiana
• Steve Blackburn, Australian National University
• Lubomir Bulej, Charles University
• Gilles Duboscq, Oracle Labs
• Francois Farquet, Oracle Labs
• Vojtech Horky, Charles University
• David Leopoldseder, Johannes Kepler University Linz
• Guillaume Martres, Ecole Polytechnique Federale de Lausanne
• Aleksandar Prokopec, Oracle Labs
• Andrea Rosa, Universita della Svizzera italiana
• Denys Shabalin, Ecole Polytechnique Federale de Lausanne
• Petr Tuma, Charles University
• Alex Villazon, Universidad Privada Boliviana