xref: /spdk/app/fio/bdev/README.md (revision 83ba9086796471697a4975a58f60e2392bccd08c) 
1# Introduction
2
3This directory contains a plug-in module for fio to enable use with SPDK. Fio is free software
4published under version 2 of the GPL license.
5
6## Compiling fio
7
8Clone the [fio source repository](https://github.com/axboe/fio)
9
10```bash
11    git clone https://github.com/axboe/fio
12    cd fio
13```
14
15Compile the fio code and install:
16
17```bash
18    make
19    make install
20```
21
22## Compiling SPDK
23
24Clone the [SPDK source repository](https://github.com/spdk/spdk)
25
26```bash
27    git clone https://github.com/spdk/spdk
28    cd spdk
29    git submodule update --init
30```
31
32Then, run the SPDK configure script to enable fio (point it to the root of the fio repository):
33
34```bash
35    cd spdk
36    ./configure --with-fio=/path/to/fio/repo <other configuration options>
37```
38
39Finally, build SPDK:
40
41```bash
42    make
43```
44
45**Note to advanced users**: These steps assume you're using the DPDK submodule. If you are using your
46own version of DPDK, the fio plugin requires that DPDK be compiled with -fPIC. You can compile DPDK
47with -fPIC by modifying your DPDK configuration file and adding the line:
48
49```bash
50EXTRA_CFLAGS=-fPIC
51```
52
53## Usage
54
55To use the SPDK fio plugin with fio, specify the plugin binary using LD_PRELOAD when running
56fio and set ioengine=spdk_bdev in the fio configuration file (see example_config.fio in the same
57directory as this README). Following example command assumes `fio` is in your system `$PATH` environment variable.
58
59```bash
60LD_PRELOAD=<path to spdk repo>/build/fio/spdk_bdev fio
61```
62
63The fio configuration file must contain parameter pointing to a JSON configuration file containing SPDK bdev configuration:
64
65```bash
66spdk_json_conf=./app/fio/bdev/bdev.json
67```
68
69You can specify which block device to run against by setting the filename parameter
70to the block device name:
71
72```bash
73filename=Malloc0
74```
75
76Or for NVMe devices:
77
78```bash
79filename=Nvme0n1
80```
81
82fio by default forks a separate process for every job. It also supports just spawning a separate
83thread in the same process for every job. The SPDK fio plugin is limited to this latter thread
84usage model, so fio jobs must also specify thread=1 when using the SPDK fio plugin. The SPDK fio
85plugin supports multiple threads - in this case, the "1" just means "use thread mode".
86
87fio also currently has a race condition on shutdown if dynamically loading the ioengine by specifying the
88engine's full path via the ioengine parameter - LD_PRELOAD is recommended to avoid this race condition.
89
90When testing random workloads, it is recommended to set norandommap=1.  fio's random map
91processing consumes extra CPU cycles which will degrade performance over time with
92the fio_plugin since all I/O are submitted and completed on a single CPU core.
93
94### Step-by-step usage examples
95
96These examples assume you have built fio and SPDK with `--with-fio` option enabled.
97
98#### Using fio bdev plugin with local NVMe storage
99
100- Bind local NVMe drives to userspace driver
101
102- Run gen_nvme.sh script to create a JSON file with bdev subsystem configuration
103
104    ```bash
105    scripts/gen_nvme.sh --json-with-subsystems > /tmp/bdev.json
106
107    cat /tmp/bdev.json  | jq
108    {
109        "subsystems": [
110        {
111            "subsystem": "bdev",
112            "config": [
113            {
114                "method": "bdev_nvme_attach_controller",
115                "params": {
116                "trtype": "PCIe",
117                "name": "Nvme0",
118                "traddr": "0000:0a:00.0"
119                }
120            },
121            {
122                "method": "bdev_nvme_attach_controller",
123                "params": {
124                "trtype": "PCIe",
125                "name": "Nvme1",
126                "traddr": "0000:85:00.0"
127                }
128            }
129            ]
130        }
131        ]
132    }
133    ```
134
135- Prepare fio configuration file
136
137    ```bash
138    cat /tmp/fio.conf
139
140    [global]
141    ioengine=/spdk/build/fio/spdk_bdev
142    spdk_json_conf=/tmp/bdev.json
143
144    thread=1
145    direct=1
146    group_reporting=1
147
148    bs=4k
149    rw=randread
150    rwmixread=70
151    time_based=1
152    runtime=10
153    norandommap=1
154
155    [filename0]
156    filename=Nvme0n1
157    filename=Nvme1n1
158    iodepth=8
159    ```
160
161- Run fio with spdk bdev plugin
162
163    ```bash
164    /usr/src/fio/fio /tmp/fio.conf
165    ```
166
167#### Using fio bdev plugin as NVMe-oF initiator with remote storage
168
169- Start SPDK NVMe-oF Target process and configure it with block devices and NVMe-oF subsystems
170
171    ```bash
172    build/bin/nvmf_tgt &
173    sleep 3
174    scripts/rpc.py bdev_malloc_create 10 512 -b Malloc0
175    scripts/rpc.py nvmf_create_transport -t TCP
176    scripts/rpc.py nvmf_create_subsystem nqn.2018-09.io.spdk:cnode1 -a -s S000001
177    scripts/rpc.py nvmf_subsystem_add_listener nqn.2018-09.io.spdk:cnode1 -t tcp -f ipv4 -s 4420 -a 10.0.0.1
178    scripts/rpc.py nvmf_subsystem_add_ns nqn.2018-09.io.spdk:cnode1 Malloc0
179    ```
180
181- Run gen_nvme.sh script to prepare a JSON file containing bdev subsystem configuration
182  for initiator which will allow it to connect to target
183
184    ```bash
185    scripts/gen_nvme.sh --json-with-subsystems --mode=remote \
186    --trid=tcp:10.0.0.1:4420:nqn.2018-09.io.spdk:cnode1 > /tmp/bdev.json
187
188    cat /tmp/bdev.json | jq
189    {
190        "subsystems": [
191        {
192            "subsystem": "bdev",
193            "config": [
194            {
195                "method": "bdev_nvme_attach_controller",
196                "params": {
197                "trtype": "tcp",
198                "adrfam": "IPv4",
199                "name": "Nvme0",
200                "subnqn": "nqn.2018-09.io.spdk:cnode1",
201                "traddr": "10.0.0.1",
202                "trsvcid": "4420"
203                }
204            }
205            ]
206        }
207        ]
208    }
209    ```
210
211- Prepare fio configuration file
212
213    ```bash
214    cat /tmp/fio.conf
215
216    [global]
217    ioengine=/spdk/build/fio/spdk_bdev
218    spdk_json_conf=/tmp/bdev.json
219
220    thread=1
221    direct=1
222    group_reporting=1
223
224    bs=4k
225    rw=randread
226    rwmixread=70
227    time_based=1
228    runtime=10
229    norandommap=1
230
231    [filename0]
232    filename=Nvme0n1
233    iodepth=8
234    ```
235
236- Run fio bdev plugin as initiator
237
238    ```bash
239    /usr/src/fio/fio /tmp/fio.conf
240    ```
241
242## Zoned Block Devices
243
244SPDK has a zoned block device API (bdev_zone.h) which currently supports Open-channel SSDs,
245NVMe Zoned Namespaces (ZNS), and the virtual zoned block device SPDK module.
246
247If you wish to run fio against a SPDK zoned block device, you can use the fio option:
248
249```bash
250zonemode=zbd
251```
252
253It is recommended to use a fio version newer than version 3.26, if using --numjobs > 1.
254If using --numjobs=1, fio version >= 3.23 should suffice.
255
256See zbd_example.fio in this directory for a zoned block device example config.
257
258### Maximum Open Zones
259
260Most zoned block devices have a resource constraint on the amount of zones which can be in an opened
261state at any point in time. It is very important to not exceed this limit.
262
263You can control how many zones fio will keep in an open state by using the
264`--max_open_zones` option.
265
266If you use a fio version newer than 3.26, fio will automatically detect and set the proper value.
267If you use an old version of fio, make sure to provide the proper --max_open_zones value yourself.
268
269### Maximum Active Zones
270
271Zoned block devices may also have a resource constraint on the number of zones that can be active at
272any point in time. Unlike `max_open_zones`, fio currently does not manage this constraint, and
273there is thus no option to limit it either.
274
275Since the max active zones limit (by definition) has to be greater than or equal to the max open
276zones limit, the easiest way to work around that fio does not manage this constraint, is to start
277with a clean state each run (except for read-only workloads), by resetting all zones before fio
278starts running its jobs by using the engine option:
279
280```bash
281--initial_zone_reset=1
282```
283
284### Zone Append
285
286When running fio against a zoned block device you need to specify --iodepth=1 to avoid
287"Zone Invalid Write: The write to a zone was not at the write pointer." I/O errors.
288However, if your zoned block device supports Zone Append, you can use the engine option:
289
290```bash
291--zone_append=1
292```
293
294To send zone append commands instead of write commands to the zoned block device.
295When using zone append, you will be able to specify a --iodepth greater than 1.
296