1*4d6fc14bSjoerg# Benchmark Tools 2*4d6fc14bSjoerg 3*4d6fc14bSjoerg## compare.py 4*4d6fc14bSjoerg 5*4d6fc14bSjoergThe `compare.py` can be used to compare the result of benchmarks. 6*4d6fc14bSjoerg 7*4d6fc14bSjoerg**NOTE**: the utility relies on the scipy package which can be installed using [these instructions](https://www.scipy.org/install.html). 8*4d6fc14bSjoerg 9*4d6fc14bSjoerg### Displaying aggregates only 10*4d6fc14bSjoerg 11*4d6fc14bSjoergThe switch `-a` / `--display_aggregates_only` can be used to control the 12*4d6fc14bSjoergdisplayment of the normal iterations vs the aggregates. When passed, it will 13*4d6fc14bSjoergbe passthrough to the benchmark binaries to be run, and will be accounted for 14*4d6fc14bSjoergin the tool itself; only the aggregates will be displayed, but not normal runs. 15*4d6fc14bSjoergIt only affects the display, the separate runs will still be used to calculate 16*4d6fc14bSjoergthe U test. 17*4d6fc14bSjoerg 18*4d6fc14bSjoerg### Modes of operation 19*4d6fc14bSjoerg 20*4d6fc14bSjoergThere are three modes of operation: 21*4d6fc14bSjoerg 22*4d6fc14bSjoerg1. Just compare two benchmarks 23*4d6fc14bSjoergThe program is invoked like: 24*4d6fc14bSjoerg 25*4d6fc14bSjoerg``` bash 26*4d6fc14bSjoerg$ compare.py benchmarks <benchmark_baseline> <benchmark_contender> [benchmark options]... 27*4d6fc14bSjoerg``` 28*4d6fc14bSjoergWhere `<benchmark_baseline>` and `<benchmark_contender>` either specify a benchmark executable file, or a JSON output file. The type of the input file is automatically detected. If a benchmark executable is specified then the benchmark is run to obtain the results. Otherwise the results are simply loaded from the output file. 29*4d6fc14bSjoerg 30*4d6fc14bSjoerg`[benchmark options]` will be passed to the benchmarks invocations. They can be anything that binary accepts, be it either normal `--benchmark_*` parameters, or some custom parameters your binary takes. 31*4d6fc14bSjoerg 32*4d6fc14bSjoergExample output: 33*4d6fc14bSjoerg``` 34*4d6fc14bSjoerg$ ./compare.py benchmarks ./a.out ./a.out 35*4d6fc14bSjoergRUNNING: ./a.out --benchmark_out=/tmp/tmprBT5nW 36*4d6fc14bSjoergRun on (8 X 4000 MHz CPU s) 37*4d6fc14bSjoerg2017-11-07 21:16:44 38*4d6fc14bSjoerg------------------------------------------------------ 39*4d6fc14bSjoergBenchmark Time CPU Iterations 40*4d6fc14bSjoerg------------------------------------------------------ 41*4d6fc14bSjoergBM_memcpy/8 36 ns 36 ns 19101577 211.669MB/s 42*4d6fc14bSjoergBM_memcpy/64 76 ns 76 ns 9412571 800.199MB/s 43*4d6fc14bSjoergBM_memcpy/512 84 ns 84 ns 8249070 5.64771GB/s 44*4d6fc14bSjoergBM_memcpy/1024 116 ns 116 ns 6181763 8.19505GB/s 45*4d6fc14bSjoergBM_memcpy/8192 643 ns 643 ns 1062855 11.8636GB/s 46*4d6fc14bSjoergBM_copy/8 222 ns 222 ns 3137987 34.3772MB/s 47*4d6fc14bSjoergBM_copy/64 1608 ns 1608 ns 432758 37.9501MB/s 48*4d6fc14bSjoergBM_copy/512 12589 ns 12589 ns 54806 38.7867MB/s 49*4d6fc14bSjoergBM_copy/1024 25169 ns 25169 ns 27713 38.8003MB/s 50*4d6fc14bSjoergBM_copy/8192 201165 ns 201112 ns 3486 38.8466MB/s 51*4d6fc14bSjoergRUNNING: ./a.out --benchmark_out=/tmp/tmpt1wwG_ 52*4d6fc14bSjoergRun on (8 X 4000 MHz CPU s) 53*4d6fc14bSjoerg2017-11-07 21:16:53 54*4d6fc14bSjoerg------------------------------------------------------ 55*4d6fc14bSjoergBenchmark Time CPU Iterations 56*4d6fc14bSjoerg------------------------------------------------------ 57*4d6fc14bSjoergBM_memcpy/8 36 ns 36 ns 19397903 211.255MB/s 58*4d6fc14bSjoergBM_memcpy/64 73 ns 73 ns 9691174 839.635MB/s 59*4d6fc14bSjoergBM_memcpy/512 85 ns 85 ns 8312329 5.60101GB/s 60*4d6fc14bSjoergBM_memcpy/1024 118 ns 118 ns 6438774 8.11608GB/s 61*4d6fc14bSjoergBM_memcpy/8192 656 ns 656 ns 1068644 11.6277GB/s 62*4d6fc14bSjoergBM_copy/8 223 ns 223 ns 3146977 34.2338MB/s 63*4d6fc14bSjoergBM_copy/64 1611 ns 1611 ns 435340 37.8751MB/s 64*4d6fc14bSjoergBM_copy/512 12622 ns 12622 ns 54818 38.6844MB/s 65*4d6fc14bSjoergBM_copy/1024 25257 ns 25239 ns 27779 38.6927MB/s 66*4d6fc14bSjoergBM_copy/8192 205013 ns 205010 ns 3479 38.108MB/s 67*4d6fc14bSjoergComparing ./a.out to ./a.out 68*4d6fc14bSjoergBenchmark Time CPU Time Old Time New CPU Old CPU New 69*4d6fc14bSjoerg------------------------------------------------------------------------------------------------------ 70*4d6fc14bSjoergBM_memcpy/8 +0.0020 +0.0020 36 36 36 36 71*4d6fc14bSjoergBM_memcpy/64 -0.0468 -0.0470 76 73 76 73 72*4d6fc14bSjoergBM_memcpy/512 +0.0081 +0.0083 84 85 84 85 73*4d6fc14bSjoergBM_memcpy/1024 +0.0098 +0.0097 116 118 116 118 74*4d6fc14bSjoergBM_memcpy/8192 +0.0200 +0.0203 643 656 643 656 75*4d6fc14bSjoergBM_copy/8 +0.0046 +0.0042 222 223 222 223 76*4d6fc14bSjoergBM_copy/64 +0.0020 +0.0020 1608 1611 1608 1611 77*4d6fc14bSjoergBM_copy/512 +0.0027 +0.0026 12589 12622 12589 12622 78*4d6fc14bSjoergBM_copy/1024 +0.0035 +0.0028 25169 25257 25169 25239 79*4d6fc14bSjoergBM_copy/8192 +0.0191 +0.0194 201165 205013 201112 205010 80*4d6fc14bSjoerg``` 81*4d6fc14bSjoerg 82*4d6fc14bSjoergWhat it does is for the every benchmark from the first run it looks for the benchmark with exactly the same name in the second run, and then compares the results. If the names differ, the benchmark is omitted from the diff. 83*4d6fc14bSjoergAs you can note, the values in `Time` and `CPU` columns are calculated as `(new - old) / |old|`. 84*4d6fc14bSjoerg 85*4d6fc14bSjoerg2. Compare two different filters of one benchmark 86*4d6fc14bSjoergThe program is invoked like: 87*4d6fc14bSjoerg 88*4d6fc14bSjoerg``` bash 89*4d6fc14bSjoerg$ compare.py filters <benchmark> <filter_baseline> <filter_contender> [benchmark options]... 90*4d6fc14bSjoerg``` 91*4d6fc14bSjoergWhere `<benchmark>` either specify a benchmark executable file, or a JSON output file. The type of the input file is automatically detected. If a benchmark executable is specified then the benchmark is run to obtain the results. Otherwise the results are simply loaded from the output file. 92*4d6fc14bSjoerg 93*4d6fc14bSjoergWhere `<filter_baseline>` and `<filter_contender>` are the same regex filters that you would pass to the `[--benchmark_filter=<regex>]` parameter of the benchmark binary. 94*4d6fc14bSjoerg 95*4d6fc14bSjoerg`[benchmark options]` will be passed to the benchmarks invocations. They can be anything that binary accepts, be it either normal `--benchmark_*` parameters, or some custom parameters your binary takes. 96*4d6fc14bSjoerg 97*4d6fc14bSjoergExample output: 98*4d6fc14bSjoerg``` 99*4d6fc14bSjoerg$ ./compare.py filters ./a.out BM_memcpy BM_copy 100*4d6fc14bSjoergRUNNING: ./a.out --benchmark_filter=BM_memcpy --benchmark_out=/tmp/tmpBWKk0k 101*4d6fc14bSjoergRun on (8 X 4000 MHz CPU s) 102*4d6fc14bSjoerg2017-11-07 21:37:28 103*4d6fc14bSjoerg------------------------------------------------------ 104*4d6fc14bSjoergBenchmark Time CPU Iterations 105*4d6fc14bSjoerg------------------------------------------------------ 106*4d6fc14bSjoergBM_memcpy/8 36 ns 36 ns 17891491 211.215MB/s 107*4d6fc14bSjoergBM_memcpy/64 74 ns 74 ns 9400999 825.646MB/s 108*4d6fc14bSjoergBM_memcpy/512 87 ns 87 ns 8027453 5.46126GB/s 109*4d6fc14bSjoergBM_memcpy/1024 111 ns 111 ns 6116853 8.5648GB/s 110*4d6fc14bSjoergBM_memcpy/8192 657 ns 656 ns 1064679 11.6247GB/s 111*4d6fc14bSjoergRUNNING: ./a.out --benchmark_filter=BM_copy --benchmark_out=/tmp/tmpAvWcOM 112*4d6fc14bSjoergRun on (8 X 4000 MHz CPU s) 113*4d6fc14bSjoerg2017-11-07 21:37:33 114*4d6fc14bSjoerg---------------------------------------------------- 115*4d6fc14bSjoergBenchmark Time CPU Iterations 116*4d6fc14bSjoerg---------------------------------------------------- 117*4d6fc14bSjoergBM_copy/8 227 ns 227 ns 3038700 33.6264MB/s 118*4d6fc14bSjoergBM_copy/64 1640 ns 1640 ns 426893 37.2154MB/s 119*4d6fc14bSjoergBM_copy/512 12804 ns 12801 ns 55417 38.1444MB/s 120*4d6fc14bSjoergBM_copy/1024 25409 ns 25407 ns 27516 38.4365MB/s 121*4d6fc14bSjoergBM_copy/8192 202986 ns 202990 ns 3454 38.4871MB/s 122*4d6fc14bSjoergComparing BM_memcpy to BM_copy (from ./a.out) 123*4d6fc14bSjoergBenchmark Time CPU Time Old Time New CPU Old CPU New 124*4d6fc14bSjoerg-------------------------------------------------------------------------------------------------------------------- 125*4d6fc14bSjoerg[BM_memcpy vs. BM_copy]/8 +5.2829 +5.2812 36 227 36 227 126*4d6fc14bSjoerg[BM_memcpy vs. BM_copy]/64 +21.1719 +21.1856 74 1640 74 1640 127*4d6fc14bSjoerg[BM_memcpy vs. BM_copy]/512 +145.6487 +145.6097 87 12804 87 12801 128*4d6fc14bSjoerg[BM_memcpy vs. BM_copy]/1024 +227.1860 +227.1776 111 25409 111 25407 129*4d6fc14bSjoerg[BM_memcpy vs. BM_copy]/8192 +308.1664 +308.2898 657 202986 656 202990 130*4d6fc14bSjoerg``` 131*4d6fc14bSjoerg 132*4d6fc14bSjoergAs you can see, it applies filter to the benchmarks, both when running the benchmark, and before doing the diff. And to make the diff work, the matches are replaced with some common string. Thus, you can compare two different benchmark families within one benchmark binary. 133*4d6fc14bSjoergAs you can note, the values in `Time` and `CPU` columns are calculated as `(new - old) / |old|`. 134*4d6fc14bSjoerg 135*4d6fc14bSjoerg3. Compare filter one from benchmark one to filter two from benchmark two: 136*4d6fc14bSjoergThe program is invoked like: 137*4d6fc14bSjoerg 138*4d6fc14bSjoerg``` bash 139*4d6fc14bSjoerg$ compare.py filters <benchmark_baseline> <filter_baseline> <benchmark_contender> <filter_contender> [benchmark options]... 140*4d6fc14bSjoerg``` 141*4d6fc14bSjoerg 142*4d6fc14bSjoergWhere `<benchmark_baseline>` and `<benchmark_contender>` either specify a benchmark executable file, or a JSON output file. The type of the input file is automatically detected. If a benchmark executable is specified then the benchmark is run to obtain the results. Otherwise the results are simply loaded from the output file. 143*4d6fc14bSjoerg 144*4d6fc14bSjoergWhere `<filter_baseline>` and `<filter_contender>` are the same regex filters that you would pass to the `[--benchmark_filter=<regex>]` parameter of the benchmark binary. 145*4d6fc14bSjoerg 146*4d6fc14bSjoerg`[benchmark options]` will be passed to the benchmarks invocations. They can be anything that binary accepts, be it either normal `--benchmark_*` parameters, or some custom parameters your binary takes. 147*4d6fc14bSjoerg 148*4d6fc14bSjoergExample output: 149*4d6fc14bSjoerg``` 150*4d6fc14bSjoerg$ ./compare.py benchmarksfiltered ./a.out BM_memcpy ./a.out BM_copy 151*4d6fc14bSjoergRUNNING: ./a.out --benchmark_filter=BM_memcpy --benchmark_out=/tmp/tmp_FvbYg 152*4d6fc14bSjoergRun on (8 X 4000 MHz CPU s) 153*4d6fc14bSjoerg2017-11-07 21:38:27 154*4d6fc14bSjoerg------------------------------------------------------ 155*4d6fc14bSjoergBenchmark Time CPU Iterations 156*4d6fc14bSjoerg------------------------------------------------------ 157*4d6fc14bSjoergBM_memcpy/8 37 ns 37 ns 18953482 204.118MB/s 158*4d6fc14bSjoergBM_memcpy/64 74 ns 74 ns 9206578 828.245MB/s 159*4d6fc14bSjoergBM_memcpy/512 91 ns 91 ns 8086195 5.25476GB/s 160*4d6fc14bSjoergBM_memcpy/1024 120 ns 120 ns 5804513 7.95662GB/s 161*4d6fc14bSjoergBM_memcpy/8192 664 ns 664 ns 1028363 11.4948GB/s 162*4d6fc14bSjoergRUNNING: ./a.out --benchmark_filter=BM_copy --benchmark_out=/tmp/tmpDfL5iE 163*4d6fc14bSjoergRun on (8 X 4000 MHz CPU s) 164*4d6fc14bSjoerg2017-11-07 21:38:32 165*4d6fc14bSjoerg---------------------------------------------------- 166*4d6fc14bSjoergBenchmark Time CPU Iterations 167*4d6fc14bSjoerg---------------------------------------------------- 168*4d6fc14bSjoergBM_copy/8 230 ns 230 ns 2985909 33.1161MB/s 169*4d6fc14bSjoergBM_copy/64 1654 ns 1653 ns 419408 36.9137MB/s 170*4d6fc14bSjoergBM_copy/512 13122 ns 13120 ns 53403 37.2156MB/s 171*4d6fc14bSjoergBM_copy/1024 26679 ns 26666 ns 26575 36.6218MB/s 172*4d6fc14bSjoergBM_copy/8192 215068 ns 215053 ns 3221 36.3283MB/s 173*4d6fc14bSjoergComparing BM_memcpy (from ./a.out) to BM_copy (from ./a.out) 174*4d6fc14bSjoergBenchmark Time CPU Time Old Time New CPU Old CPU New 175*4d6fc14bSjoerg-------------------------------------------------------------------------------------------------------------------- 176*4d6fc14bSjoerg[BM_memcpy vs. BM_copy]/8 +5.1649 +5.1637 37 230 37 230 177*4d6fc14bSjoerg[BM_memcpy vs. BM_copy]/64 +21.4352 +21.4374 74 1654 74 1653 178*4d6fc14bSjoerg[BM_memcpy vs. BM_copy]/512 +143.6022 +143.5865 91 13122 91 13120 179*4d6fc14bSjoerg[BM_memcpy vs. BM_copy]/1024 +221.5903 +221.4790 120 26679 120 26666 180*4d6fc14bSjoerg[BM_memcpy vs. BM_copy]/8192 +322.9059 +323.0096 664 215068 664 215053 181*4d6fc14bSjoerg``` 182*4d6fc14bSjoergThis is a mix of the previous two modes, two (potentially different) benchmark binaries are run, and a different filter is applied to each one. 183*4d6fc14bSjoergAs you can note, the values in `Time` and `CPU` columns are calculated as `(new - old) / |old|`. 184*4d6fc14bSjoerg 185*4d6fc14bSjoerg### U test 186*4d6fc14bSjoerg 187*4d6fc14bSjoergIf there is a sufficient repetition count of the benchmarks, the tool can do 188*4d6fc14bSjoerga [U Test](https://en.wikipedia.org/wiki/Mann%E2%80%93Whitney_U_test), of the 189*4d6fc14bSjoergnull hypothesis that it is equally likely that a randomly selected value from 190*4d6fc14bSjoergone sample will be less than or greater than a randomly selected value from a 191*4d6fc14bSjoergsecond sample. 192*4d6fc14bSjoerg 193*4d6fc14bSjoergIf the calculated p-value is below this value is lower than the significance 194*4d6fc14bSjoerglevel alpha, then the result is said to be statistically significant and the 195*4d6fc14bSjoergnull hypothesis is rejected. Which in other words means that the two benchmarks 196*4d6fc14bSjoergaren't identical. 197*4d6fc14bSjoerg 198*4d6fc14bSjoerg**WARNING**: requires **LARGE** (no less than 9) number of repetitions to be 199*4d6fc14bSjoergmeaningful! 200