1# Automated script for NVMe performance test 2 3## Compile SPDK with LTO 4 5The link time optimization (lto) gcc flag allows the linker to run a post-link optimization pass on the code. During 6that pass the linker inlines thin wrappers such as those around DPDK calls which results in a shallow call stack and 7significantly improves performance. Therefore, we recommend compiling SPDK with the lto flag prior to running this 8benchmark script to achieve optimal performance. 9Link time optimization can be enabled in SPDK by doing the following: 10 11~{.sh} 12./configure --enable-lto 13~ 14 15## Configuration 16 17Test is configured by using command-line options. 18 19### Available options 20 21#### -h, --help 22 23Prints available commands and help. 24 25#### --run-time 26 27Tell fio to terminate processing after the specified period of time. Value in seconds. 28 29#### --ramp-time 30 31Fio will run the specified workload for this amount of time before logging any performance numbers. 32Value in seconds. 33 34#### --fio-bin 35 36Path to fio binary. 37 38#### --driver 39 40Select between SPDK driver and kernel driver. The Linux Kernel driver has three configurations: 41Default mode, Hybrid Polling and Classic Polling. The SPDK driver supports 2 fio_plugin modes: bdev and NVMe PMD. 42Before running test with spdk, you will need to bind NVMe devics to the Linux uio_pci_generic or vfio-pci driver. 43When running test with the Kernel driver, NVMe devices use the Kernel driver. The 5 valid values for this option are: 44'bdev', 'nvme', 'kernel-libaio', 'kernel-classic-polling' and 'kernel-hybrid-polling'. 45 46#### --max-disk 47 48This option will run multiple fio jobs with varying number of NVMe devices. First it will start with 49max-disk number of devices then decrease number of disk by two until there are no more devices. 50If set to 'all' then max-disk number will be set to all available devices. 51Only one of the max-disk or disk-no option can be used. 52 53#### --disk-no 54 55This option will run fio job on specified number of NVMe devices. If set to 'all' then max-disk number 56will be set to all available devices. Only one of the max-disk or disk-no option can be used. 57 58#### --cpu-allowed 59 60Specifies the CPU cores that will be used by fio to execute the performance test cases. When spdk driver is chosen, 61the script attempts to assign NVMe devices to CPU cores on the same NUMA node. The script will try to align each 62core with devices matching core's NUMA first but if the is no devices left within the CPU core NUMA then it will use 63devices from the other NUMA node. It is important to choose cores that will ensure best NUMA node alignment. For example: 64On System with 8 devices on NUMA node 0 and 8 devices on NUMA node 1, cores 0-27 on numa node 0 and 28-55 65on numa node 1, if test is set to use 16 disk and four cores then "--cpu-allowed=1,2,28,29" can be used 66resulting with 4 devices with node0 per core 1 and 2 and 4 devices with node1 per core 28 and 29. If 10 cores 67are required then best option would be "--cpu-allowed=1,2,3,4,28,29,30,31,32,33" because cores 1-4 will be 68aligned with 2 devices on numa0 per core and cores 28-33 will be aligned with 1 device on numa1 per core. 69If kernel driver is chosen then for each job with NVME device, all cpu cores with corresponding NUMA node are picked. 70 71#### --rw 72 73Type of I/O pattern. Accepted values are: randrw, rw 74 75#### --rwmixread 76 77Percentage of a mixed workload that should be reads. 78 79#### --iodepth 80 81Number of I/O units to keep in flight against each file. 82 83#### --block-size 84 85The block size in bytes used for I/O units. 86 87#### --numjobs 88 89Create the specified number of clones of a job. 90 91#### --repeat-no 92 93Specifies how many times run each workload. End results are averages of these workloads 94 95#### "--no-io-scaling" 96 97For SPDK fio plugin iodepth is multiplied by number of devices. When this option is set this multiplication will be disabled. 98 99## Results 100 101Results are stored in "results" folder. After each workload, to this folder are copied files with: 102fio configuration file, json files with fio results and logs with latencies with sampling interval 250 ms. 103Number of copied files depends from number of repeats of each workload. Additionally csv file is created with averaged 104results of all workloads. 105