Tradeoffs in Modeling Performance
of Highly-Configurable Software Systems

Supplementary Material

Experimental Data

To ensure the replicability of our study, we provide the complete set of our experimental data and scripts. The data set includes:

• performance measurement data
• variability models
• SPLConqueror (the learner) automation scripts
• SPLConqueror learning logs containing the learned performance-influence models
• analysis and visualisation Python 3 script in form of a portable jupyter notebook.

For reproducing the results of the study it is easier to download the complete data set with all data in one direcotry structure. See the README file in the archive for a detailed description of the structure.

Steps for reproducing the results:

1. Run SPLConqueror using the automation scripts. The automation scripts use the performance measurement data and variability models as input and produce learning logs as output.
2. Run the jupyter notebook that analyzes the learning logs and generates plots.

Download the complete data set: dataset.zip

Influence of Measurement Errors on the Performance-Influence Model Accuracy

Download plots: plt_noise_effect.pdf

For each algorithm variant there is one plot. A plot shows the influence of noise of different strength (x-axis) on the accuracy of the models (y-axis) of different systems. For each subject system there is a series of data points connected by a line. Each data point represents a performance-influence model of the system that was learned from the measurement data with corresponding noise (x-axis) and has the corresponding accuracy (y-axis).

Interviews with Domain Experts and Programmers

We performed semi-structured interviews with three experts from the domain of high performance computing, who develop and work with performance-critical systems. During the interview we focused on the following issues:

• What use cases for the performance-influence models the experts can think of?
• What are acceptable tradeoffs among accuracy, complexity, and learning time of a model with respect to the use cases?
• How do the experts perceive the complexity of models?

After a short introduction about our performance-influence models, we presented three different performance-influence models of a system to the interviewee. To lighten understanding of the models, we select models for systems the interviewee work with and also presented them a variability model describing the variability we considered in our case study. The models differ in complexity, prediction accuracy, and computation time required to build the model. To improve readability of the models, we performed a preprocessing of the models, where we rounded the coefficients of the terms to three significant digits. During the interview, we asked them which of the models they prefer and why. When they mentioned the complexity, we asked which attributes they used to asses the complexity of the model. To get an impression about the complexity growth of a single term, we presented four terms describing the influence of an individual option and interactions up to a degree of four (four configuration options interacting with each other).

Material presented to the interviewees

Download the material presented to the interviewees: Models.pdf