IJPP Supplementary Webpage
Andreas Simbürger
06 March, 2018, 14:05
1 Versions
For this empirical study we used the following tools with their given versions. The project PolyJIT contains a meta project that downloads and builds all necessary libraries in the given configuration automatically.
1.1 Tools
| Tool | URL | Version |
|---|---|---|
| PolyJIT | https://github.com/PolyJIT/PolyJIT.git | 9c69b69 |
| LLVM | https://llvm.org/git/llvm.git | e8afbbe |
| clang | https://llvm.org/git/clang.git | 11fbce0 |
| Polly | https://llvm.org/git/polly.git | 7f80afe |
| Benchbuild | https://github.com/PolyJIT/benchbuild.git | 7cf4556 |
The PolyJIT project serves as a meta project that generates a source distribution with all necessary. Using this project you can specifiy all dependent versions at configure time. You need a standard unix build environment with a few additional libraries. Ubuntu 16.04 is known to work with the meta project, given that necessary libraries like libpqxx and likwid are available on the system. The cmake configuration will guide you through all missing dependencies.
1.2 Raw Data
All experiment results and plots are derived from the following raw data.
| Data | File |
|---|---|
| Whole program (per config) | complete |
| Whole program (per config #2) | runtime |
| Regions (With Speedups) | region compare |
| Regions (Timings) | regions |
1.3 Benchbuild
In the following section you can find the projects involved in this empirical study. All projects are listed with their version and source url. Benchbuild will download these automatically in the given version and cache the projects in a temporary folder for future reuse.
Many projects found in the later sections do not have a corresponding name in the project list. This is caused by dynamic projects that are handled by BenchBuild, for example lnt based projects are listed as their top category SingleSourceBenchmark but not each project from the category is listed individually. However, individual projects are visible in the later stages of this document. In some cases we cannot provide a fixed version number, as it is missing from the project description.
2 Input data used for this analysis.
We use the following three datasets for this analysis. The .csv files can be generated from benchbuild with a successfull run of the ijpp experiment.
$> benchbuild run -E ijppThis runs a fresh benchbuild experiment with a new experiment ID exp_id. The input data can be regenerated using the following command.
$> benchbuild report -E ijpp -e <exp_id>If your experiment did not filter any projects, this will take a while. For example, for our analysis here we use exp_id=4ae29f2f-32d3-4e7c-977e-972c39d01b58.
ijpp_all <- read_csv("data/ijpp_report_runtime_4ae29f2f-32d3-4e7c-977e-972c39d01b58.csv")
ijpp_report_complete <- read_csv("data/ijpp_report_complete_4ae29f2f-32d3-4e7c-977e-972c39d01b58.csv")
irc <- read_csv("data/ijpp_report_region_compare_4ae29f2f-32d3-4e7c-977e-972c39d01b58.csv")
ijpp_regions <- read_csv("data/ijpp_report_regions_4ae29f2f-32d3-4e7c-977e-972c39d01b58.csv")3 Complete overview
The following table provides a complete overview over all programs that managed to fully complete the experiment ijpp.
t_per_config <- gen_t_per_config(ijpp_all)
ijpp_report_runtime <- ijpp_all[ijpp_all$group != "polybench-mod",]
ijpp_report_runtime <- ijpp_report_runtime[ijpp_report_runtime$project %in% t_per_config$Project,]
ijpp_with_baselines <- add_baseline_comparison(ijpp_report_runtime)
DT::datatable(t_per_config)4 Regionwise Results
Histogram of all regionwise measurements, excluding the band between speedup values of –1.1 and 1.1. We used a binwidth of 0.5.
plot_region_histogram(irc_gb, store=FALSE, w=7, h=6, "figures/region_histogram")
Pointcloud and violin plots of the regionwise results.
plot_region_wise_points(ijpp_regions_f, store=FALSE, w=7, h=6, filename="figures/region-per-config-speedups")
plot_region_wise_violin(ijpp_regions_f, store=FALSE, w=7, h=6, filename="figures/region-per-config-vio-speedup-polly")
5 Programwise results
5.1 Overall
plot_pointcloud2D(ijpp_with_baselines, filename="figures/per-config-speedups")
5.2 Single baseline: Static (-O3)
5.2.1 Pointcloud 2D (O3, Good)
plot_pointcloud2D(ijpp_good_o3, filename="figures/per-config-speedups-better-than-o3")
5.2.2 VioPlot (O3)
plot_vioplot(ijpp_with_baselines, use_polly=FALSE, filename="figures/per-config-vio-speedup-o3")## Warning: Removed 56 rows containing non-finite values (stat_ydensity).
5.2.3 BarPlot (O3)
plot_bar_speedup(ijpp_with_baselines, use_polly=FALSE, filename="figures/per-config-bar-all-o3")
5.2.4 BoxPlot (BL O3, Good)
plot_speedup_boxplot(ijpp_good_o3, filename="figures/per-config-bp-speedup-o3")
5.3 VioBar (O3)
Violin plot with baseline configuration Static (-O3). The dataset contains values that include at least one configuration with a speedup greater than 1.1.
plot_viobar(ijpp_good_o3, use_polly=FALSE, filename="figures/vio-good-bp-per-config_o3")
5.4 Barplot (O3, Good)
Bar plot with baseline configuration Static (-O3). The dataset contains values that include at least one configuration with a speedup greater than 1.1.
plot_bar_speedup(ijpp_good_o3, use_polly=FALSE, filename="figures/per-config-bar-speedup-o3")
5.5 Boxplot (O3, Bad)
Box plot with baseline configuration Static (-O3). The dataset contains values that include at least one configuration with a slowdown greater (speedup less than) than -1.1.
plot_speedup_boxplot(ijpp_bad_o3, filename="figures/per-config-bp-slowdown-o3")
5.6 VioBar (O3, Bad)
Violin plot with baseline configuration Static (-O3). The dataset contains values that include at least one configuration with a slowdown greater (speedup less than) than -1.1.
plot_viobar(ijpp_bad_o3, use_polly=FALSE, filename="figures/vio-bad-bp-per-config_o3")## Warning: Removed 56 rows containing non-finite values (stat_ydensity).## Warning: Removed 56 rows containing non-finite values (stat_boxplot).
5.7 Single Baseline: Static (-O3 -polly -parallel)
The following plots consider a baseline configuration of Static (-O3 -polly -parallel).
5.7.1 Pointcloud 2D (Polly, Good)
plot_pointcloud2D(ijpp_good_polly, filename="figures/per-config-speedups-better-than-polly")
5.7.2 VioPlot (Polly)
plot_vioplot(ijpp_with_baselines, filename="figures/per-config-vio-speedup-polly")## Warning: Removed 50 rows containing non-finite values (stat_ydensity).
5.7.3 VioBar (Polly)
Violin plot with baseline configuration Static (-O3 -polly -parallel). The dataset contains values that include at least one configuration with a speedup greater than 1.1.
plot_viobar(ijpp_good_polly, use_polly=TRUE, filename="figures/vio-good-bp-per-config_polly")
5.8 Barplot (Polly, Good)
Bar plot with baseline configuration Static (-O3 -polly -parallel). The dataset contains values that include at least one configuration with a speedup greater than 1.1.
plot_bar_speedup(ijpp_good_polly, use_polly=TRUE, filename="figures/per-config-bar-speedup-polly")
5.9 VioBar (Polly, Bad)
Violin plot with baseline configuration Static (-O3 -polly -parallel). The dataset contains values that include at least one configuration with a slowdown greater (speedup less than) than -1.1.
plot_viobar(ijpp_bad_polly, use_polly=TRUE, filename="figures/vio-bad-bp-per-config_polly")## Warning: Removed 50 rows containing non-finite values (stat_ydensity).## Warning: Removed 50 rows containing non-finite values (stat_boxplot).