1# SPDK Docker suite 2 3This suite is meant to serve as an example of how SPDK can be encapsulated 4into docker container images. The example containers consist of SPDK NVMe-oF 5target sharing devices to another SPDK NVMe-oF application. Which serves 6as both initiator and target. Finally a traffic generator based on FIO 7issues I/O to the connected devices. 8 9## Prerequisites 10 11docker: We recommend version 20.10 and above because it supports cgroups v2 for 12customization of host resources like CPUs, memory, and block I/O. 13 14docker-compose: We recommend using 1.29.2 version or newer. 15 16kernel: Hugepages must be allocated prior running the containers and hugetlbfs 17mount must be available under /dev/hugepages. Also, tmpfs should be mounted 18under /dev/shm. Depending on the use-case, some kernel modules should be also 19loaded into the kernel prior running the containers. 20 21proxy: If you are working behind firewall make sure dockerd is aware of the 22proxy. Please refer to: 23[docker-proxy](https://docs.docker.com/config/daemon/systemd/#httphttps-proxy) 24 25To pass `$http_proxy` to docker-compose build use: 26~~~{.sh} 27docker-compose build --build-arg PROXY=$http_proxy 28~~~ 29 30## How-To 31 32`docker-compose.yaml` shows an example deployment of the storage containers based on SPDK. 33Running `docker-compose build` creates 5 docker images: 34 35- build_base 36- storage-target 37- proxy-container 38- traffic-generator-nvme 39- traffic-generator-virtio 40 41The `build_base` image provides the core components required to containerize SPDK 42applications. The fedora:35 image from the Fedora Container Registry is used and then SPDK is installed. SPDK is installed out of `build_base/spdk.tar.gz` provided. 43See `build_base` folder for details on what's included in the final image. 44 45Running `docker-compose up` creates 3 docker containers: 46 47-- storage-target: Contains SPDK NVMe-oF target exposing single subsystem to 48`proxy-container` based on malloc bdev. 49-- proxy-container: Contains SPDK NVMe-oF target connecting to `storage-target` 50and then exposing the same devices to `traffic-generator-nvme` using NVMe-oF and 51to `traffic-generator-virtio` using Virtio. 52-- traffic-generator-nvme: Contains FIO using SPDK plugin to connect to `proxy-container` 53and runs a sample workload. 54-- traffic-generator-virtio: Contains FIO using SPDK plugin to connect to `proxy-container` 55and runs a sample workload. 56 57Each container is connected to a separate "spdk" network which is created before 58deploying the containers. See `docker-compose.yaml` for the network's detailed setup and ip assignment. 59 60All the above boils down to: 61 62~~~{.sh} 63cd docker 64tar -czf build_base/spdk.tar.gz --exclude='docker/*' -C .. . 65docker-compose build 66docker-compose up 67~~~ 68 69The `storage-target` and `proxy-container` can be started as services. 70Allowing for multiple traffic generator containers to connect. 71 72~~~{.sh} 73docker-compose up -d proxy-container 74docker-compose run traffic-generator-nvme 75docker-compose run traffic-generator-virtio 76~~~ 77 78Environment variables to containers can be passed as shown in 79[docs](https://docs.docker.com/compose/environment-variables/). 80For example extra arguments to fio can be passed as so: 81 82~~~{.sh} 83docker-compose run -e FIO_ARGS="--minimal" traffic-generator-nvme 84~~~ 85 86As each container includes SPDK installation it is possible to use rpc.py to 87examine the final setup. E.g.: 88 89~~~{.sh} 90docker-compose exec storage-target rpc.py bdev_get_bdevs 91docker-compose exec proxy-container rpc.py nvmf_get_subsystems 92~~~ 93 94## Caveats 95 96- If you run docker < 20.10 under distro which switched fully to cgroups2 97 (e.g. f33) make sure that /sys/fs/cgroup/systemd exists otherwise docker/build 98 will simply fail. 99- Each SPDK app inside the containers is limited to single, separate CPU. 100