# SPDK Docker suite

This suite is meant to serve as an example of how SPDK can be encapsulated
into docker container images. The example containers consist of SPDK NVMe-oF
target sharing devices to another SPDK NVMe-oF application. Which serves
as both initiator and target. Finally a traffic generator based on FIO
issues I/O to the connected devices.
Please note that some simplifications have been made to the configuration files
for the purpose of the example, please do not use the files directly in
the production environment.

## Prerequisites

docker: We recommend version 20.10 and above because it supports cgroups v2 for
customization of host resources like CPUs, memory, and block I/O.

docker-compose: We recommend using 1.29.2 version or newer.

kernel: Hugepages must be allocated prior running the containers and hugetlbfs
mount must be available under /dev/hugepages. Also, tmpfs should be mounted
under /dev/shm. Depending on the use-case, some kernel modules should be also
loaded into the kernel prior running the containers.

proxy: If you are working behind firewall make sure dockerd is aware of the
proxy. Please refer to:
[docker-proxy](https://docs.docker.com/config/daemon/systemd/#httphttps-proxy)

To pass `$http_proxy` to docker-compose build use:
~~~{.sh}
docker-compose build --build-arg PROXY=$http_proxy
~~~

## How-To

`docker-compose.yaml` shows an example deployment of the storage containers based on SPDK.
Running `docker-compose build` creates 5 docker images:

- build_base
- storage-target
- proxy-container
- traffic-generator-nvme
- traffic-generator-virtio

The `build_base` image provides the core components required to containerize SPDK
applications. The fedora:35 image from the Fedora Container Registry is used and then SPDK is installed. SPDK is installed out of `build_base/spdk.tar.gz` provided.
See `build_base` folder for details on what's included in the final image.

Running `docker-compose up` creates 3 docker containers:

- storage-target: Contains SPDK NVMe-oF target exposing single subsystem to `proxy-container` based on malloc bdev.
- proxy-container: Connecting to `storage-target` and then exposing the same devices to `traffic-generator-nvme` using NVMe-oF and to `traffic-generator-virtio` using Virtio.
- traffic-generator-nvme: Contains FIO using SPDK plugin to connect to `proxy-container` and runs a sample workload.
- traffic-generator-virtio: Contains FIO using SPDK plugin to connect to `proxy-container` and runs a sample workload.

Each container is connected to a separate "spdk" network which is created before
deploying the containers. See `docker-compose.yaml` for the network's detailed setup and ip assignment.

All the above boils down to:

~~~{.sh}
cd docker
tar -czf build_base/spdk.tar.gz --exclude='docker/*' -C .. .
docker-compose build
docker-compose up
~~~

The `storage-target` and `proxy-container` can be started as services.
Allowing for multiple traffic generator containers to connect.

~~~{.sh}
docker-compose up -d proxy-container
docker-compose run traffic-generator-nvme
docker-compose run traffic-generator-virtio
~~~

Environment variables to containers can be passed as shown in
[docs](https://docs.docker.com/compose/environment-variables/).
For example extra arguments to fio can be passed as so:

~~~{.sh}
docker-compose run -e FIO_ARGS="--minimal" traffic-generator-nvme
~~~

As each container includes SPDK installation it is possible to use rpc.py to
examine the final setup. E.g.:

~~~{.sh}
docker-compose exec storage-target rpc.py bdev_get_bdevs
docker-compose exec proxy-container rpc.py nvmf_get_subsystems
~~~

## Monitoring

`docker-compose.monitoring.yaml` shows an example deployment of the storage containers based on SPDK.

Running `docker-compose -f docker-compose.monitoring.yaml up` creates 3 docker containers:

- storage-target: Contains SPDK NVMe-oF target exposing single subsystem based on malloc bdev.
- [telegraf](https://www.influxdata.com/time-series-platform/telegraf/) is a very minimal memory footprint agent for collecting and sending metrics and events.
- [prometheus](https://prometheus.io/) is leading open-source monitoring solution.

`telegraf` connects to `spdk` via `rpc_http_proxy.py` and uses `bdev_get_iostat` commands to fetch bdev statistics.

In order to see data change, once all of the 3 containers are brought up, use `docker-compose run traffic-generator-nvme` to generate some traffic.

Open Prometheus UI or query via cmdline. E.g.:

~~~{.sh}
curl --fail http://127.0.0.1:9090/api/v1/query?query=spdk_bytes_read
curl --fail http://127.0.0.1:9090/api/v1/query?query=spdk_bytes_written
~~~

## Caveats

- If you run docker < 20.10 under distro which switched fully to cgroups2
  (e.g. f33) make sure that /sys/fs/cgroup/systemd exists otherwise docker/build
  will simply fail.
- Each SPDK app inside the containers is limited to single, separate CPU.