Understanding the Performance of
Highly Configurable Systems: An Empirical Study

Supplementary Material

Sergiy Kolesnikov Norbert Siegmund Christian Kästner Alexander Grebhahn Sven Apel
University of Passau University of Passau Carnegie Mellon University University of Passau University of Passau
Germany Germany USA Germany Germany

Performance-Influence Models for the Subject Systems

Performance-
System Influence Model Variability Model
AJstats ajstats.txt ajstats_vm.xml ajstats_vm.png
Apache apache.txt apache_vm.xml apache_vm.png
BDB-C bdb-c.txt bdb-c_vm.xml bdb-c_vm.png
DBD-J bdb-j.txt bdb-j_vm.xml bdb-j_vm.png
Clasp clasp.txt clasp_vm.xml clasp_vm.png
DUNE dune.txt dune_vm.xml dune_vm.png
HSMGP hsmgp.txt hsmgp_vm.xml hsmgp_vm.png
LLVM llvm_nh.txt llvm_vm.xml llvm_vm.png
Lrzip lrzip.txt lrzip_vm.xml lrzip_vm.png
x264 x264.txt x264_vm.xml x264_vm.png

Note: XML files with variability models are in the SPL Conqueror format.

Download all models: models.zip

Influence of Measurement Errors on the Performance-Influence Model Accuracy


Download plots: plt_noise_effect.pdf

For each algorithm variant there is one plot. A plot shows the influence of noise of different strength (x-axis) on the accuracy of the models (y-axis) of different systems. For each subject system there is a series of data points connected by a line. Each data point represents a performance-influence model of the system that was learned from the measurement data with corresponding noise (x-axis) and has the corresponding accuracy (y-axis).


Interviews with Domain Experts and Programmers

We performed semi-structured interviews with three experts from the domain of high performance computing, who develop and work with performance critical systems. During the interview we focused on the following issues:

After a short introduction about our performance-influence models, we presented three different performance-influence models of a system to the interviewee. To lighten understanding of the models, we select models for systems the interviewee work with and also presented them a variability model describing the variability we considered in our case study. The models differ in complexity, prediction accuracy, and computation time required to build the model. To improve readability of the models, we performed a preprocessing of the models, where we rounded the coefficients of the terms to three significant digits. During the interview, we asked them which of the models they prefer and why. When they mentioned the complexity we asked which attributes they used to asses the complexity of the model. To get an impression about the complexity growth of a single term, we presented four terms describing the influence of an individual option and interactions up to a degree of four (four configuration options interacting with each other).

Material presented to the interviewees

Download the material presented to the interviewees: Models.pdf