1# NVMe-oF Multipath HOWTO {#nvmf_multipath_howto} 2 3This HOWTO provides step-by-step instructions for setting-up a simple SPDK deployment and testing multipath. 4It demonstrates configuring path preferences with Asymmetric Namespace Access (ANA), as well as round-robin 5path load balancing. 6 7## Build SPDK on both the initiator and target servers 8 9Clone the repo: 10~~~{.sh} 11git clone https://github.com/spdk/spdk --recursive 12~~~ 13 14Configure and build SPDK: 15~~~{.sh} 16cd spdk/ 17./configure 18make -j16 19~~~ 20 21## Setup hugepages 22 23This should be run once on each server (and after reboots): 24~~~{.sh} 25cd spdk/ 26./scripts/setup.sh 27~~~ 28 29## On target: start and configure SPDK 30 31Start the target in the background and configure it: 32~~~{.sh} 33cd spdk/ 34./build/bin/nvmf_tgt -m 0x3 & 35./scripts/rpc.py nvmf_create_transport -t tcp -o -u 8192 36~~~ 37 38Create a subsystem, with `-r` to enable ANA reporting feature: 39~~~{.sh} 40./scripts/rpc.py nvmf_create_subsystem nqn.2022-02.io.spdk:cnode0 -a -s SPDK00000000000001 -r 41~~~ 42 43Create and add a malloc block device: 44~~~{.sh} 45./scripts/rpc.py bdev_malloc_create 64 512 -b Malloc0 46./scripts/rpc.py nvmf_subsystem_add_ns nqn.2022-02.io.spdk:cnode0 Malloc0 47~~~ 48 49Add two listeners, each with a different `IP:port` pair: 50~~~{.sh} 51./scripts/rpc.py nvmf_subsystem_add_listener -t tcp -a 172.17.1.13 -s 4420 nqn.2022-02.io.spdk:cnode0 52./scripts/rpc.py nvmf_subsystem_add_listener -t tcp -a 172.18.1.13 -s 5520 nqn.2022-02.io.spdk:cnode0 53~~~ 54 55## On initiator: start and configure bdevperf 56 57Launch the bdevperf process in the background: 58~~~{.sh} 59cd spdk/ 60./build/examples/bdevperf -m 0x4 -z -r /tmp/bdevperf.sock -q 128 -o 4096 -w verify -t 90 &> bdevperf.log & 61~~~ 62 63Configure bdevperf and add two paths: 64~~~{.sh} 65./scripts/rpc.py -s /tmp/bdevperf.sock bdev_nvme_set_options -r -1 66./scripts/rpc.py -s /tmp/bdevperf.sock bdev_nvme_attach_controller -b Nvme0 -t tcp -a 172.17.1.13 -s 4420 -f ipv4 -n nqn.2022-02.io.spdk:cnode0 -l -1 -o 10 67./scripts/rpc.py -s /tmp/bdevperf.sock bdev_nvme_attach_controller -b Nvme0 -t tcp -a 172.18.1.13 -s 5520 -f ipv4 -n nqn.2022-02.io.spdk:cnode0 -x multipath -l -1 -o 10 68~~~ 69 70## Launch a bdevperf test 71 72Connect to the RPC socket of the bdevperf process and start the test: 73~~~{.sh} 74PYTHONPATH=$PYTHONPATH:/root/src/spdk/python ./examples/bdev/bdevperf/bdevperf.py -t 1 -s /tmp/bdevperf.sock perform_tests 75~~~ 76 77The RPC command will return, leaving the test to run for 90 seconds in the background. On the target server, 78observe that only the first path (port) is receiving packets by checking the queues with `ss -t`. 79 80You can view the paths available to the initiator with: 81~~~{.sh} 82./scripts/rpc.py -s /tmp/bdevperf.sock bdev_nvme_get_io_paths -n Nvme0n1 83~~~ 84 85## Switching paths 86 87This can be done on the target server by setting the first path's ANA to `non_optimized`: 88~~~{.sh} 89./scripts/rpc.py nvmf_subsystem_listener_set_ana_state nqn.2022-02.io.spdk:cnode0 -t tcp -a 172.17.1.13 -s 4420 -n non_optimized 90~~~ 91 92Use `ss -t` to verify that the traffic has switched to the second path. 93 94## Use round-robin (active_active) path load balancing 95 96First, ensure the ANA for both paths is configured as `optimized` on the target. Then, change the 97multipath policy on the initiator to `active_active` (multipath policy is per bdev, so 98`bdev_nvme_set_multipath_policy` must be called after `bdev_nvme_attach_controller`): 99~~~{.sh} 100./scripts/rpc.py -s /tmp/bdevperf.sock bdev_nvme_set_multipath_policy -b Nvme0n1 -p active_active 101~~~ 102 103Observe with `ss -t` that both connections are receiving traffic (queues build up). 104