1# SPDK Docker suite 2 3This suite is meant to serve as an example of how SPDK can be encapsulated 4into docker container images. The example containers consist of SPDK NVMe-oF 5target sharing devices to another SPDK NVMe-oF application. Which serves 6as both initiator and target. Finally a traffic generator based on FIO 7issues I/O to the connected devices. 8Please note that some simplifications have been made to the configuration files 9for the purpose of the example, please do not use the files directly in 10the production environment. 11 12## Prerequisites 13 14docker: We recommend version 20.10 and above because it supports cgroups v2 for 15customization of host resources like CPUs, memory, and block I/O. 16 17docker-compose: We recommend using 1.29.2 version or newer. 18 19kernel: Hugepages must be allocated prior running the containers and hugetlbfs 20mount must be available under /dev/hugepages. Also, tmpfs should be mounted 21under /dev/shm. Depending on the use-case, some kernel modules should be also 22loaded into the kernel prior running the containers. 23 24proxy: If you are working behind firewall make sure dockerd is aware of the 25proxy. Please refer to: 26[docker-proxy](https://docs.docker.com/config/daemon/systemd/#httphttps-proxy) 27 28To pass `$http_proxy` to docker-compose build use: 29~~~{.sh} 30docker-compose build --build-arg PROXY=$http_proxy 31~~~ 32 33## How-To 34 35`docker-compose.yaml` shows an example deployment of the storage containers based on SPDK. 36Running `docker-compose build` creates 5 docker images: 37 38- build_base 39- storage-target 40- proxy-container 41- traffic-generator-nvme 42- traffic-generator-virtio 43 44The `build_base` image provides the core components required to containerize SPDK 45applications. The fedora:35 image from the Fedora Container Registry is used and then SPDK is installed. SPDK is installed out of `build_base/spdk.tar.gz` provided. 46See `build_base` folder for details on what's included in the final image. 47 48Running `docker-compose up` creates 3 docker containers: 49 50-- storage-target: Contains SPDK NVMe-oF target exposing single subsystem to 51`proxy-container` based on malloc bdev. 52-- proxy-container: Contains SPDK NVMe-oF target connecting to `storage-target` 53and then exposing the same devices to `traffic-generator-nvme` using NVMe-oF and 54to `traffic-generator-virtio` using Virtio. 55-- traffic-generator-nvme: Contains FIO using SPDK plugin to connect to `proxy-container` 56and runs a sample workload. 57-- traffic-generator-virtio: Contains FIO using SPDK plugin to connect to `proxy-container` 58and runs a sample workload. 59 60Each container is connected to a separate "spdk" network which is created before 61deploying the containers. See `docker-compose.yaml` for the network's detailed setup and ip assignment. 62 63All the above boils down to: 64 65~~~{.sh} 66cd docker 67tar -czf build_base/spdk.tar.gz --exclude='docker/*' -C .. . 68docker-compose build 69docker-compose up 70~~~ 71 72The `storage-target` and `proxy-container` can be started as services. 73Allowing for multiple traffic generator containers to connect. 74 75~~~{.sh} 76docker-compose up -d proxy-container 77docker-compose run traffic-generator-nvme 78docker-compose run traffic-generator-virtio 79~~~ 80 81Environment variables to containers can be passed as shown in 82[docs](https://docs.docker.com/compose/environment-variables/). 83For example extra arguments to fio can be passed as so: 84 85~~~{.sh} 86docker-compose run -e FIO_ARGS="--minimal" traffic-generator-nvme 87~~~ 88 89As each container includes SPDK installation it is possible to use rpc.py to 90examine the final setup. E.g.: 91 92~~~{.sh} 93docker-compose exec storage-target rpc.py bdev_get_bdevs 94docker-compose exec proxy-container rpc.py nvmf_get_subsystems 95~~~ 96 97## Caveats 98 99- If you run docker < 20.10 under distro which switched fully to cgroups2 100 (e.g. f33) make sure that /sys/fs/cgroup/systemd exists otherwise docker/build 101 will simply fail. 102- Each SPDK app inside the containers is limited to single, separate CPU. 103