1# Storage Management Agent {#sma} 2 3Storage Management Agent (SMA) is a service providing a gRPC interface for 4orchestrating SPDK applications. It's a standalone application that allows 5users to create and manage various types of devices (e.g. NVMe, virtio-blk, 6etc.). The major difference between SMA's API and the existing SPDK-RPC 7interface is that it's designed to abstract the low level details exposed by 8SPDK-RPCs, which enables it to be more easily consumed by orchestration 9frameworks, such as k8s or OpenStack. This is especially important for 10deployments on IPUs (Infrastructure Processing Unit), which usually require a 11lot of hardware-specific options. 12 13## Interface 14 15The interface is defined in a protobuf files located in `proto` 16directory. The generic interface common to all types of devices is defined in 17`sma.proto` file, while device-specific options are defined in their separate 18files (e.g. `nvme.proto` for NVMe). 19 20Currently, the interface consists of four methods. Additionally, it defines two 21main types of objects: volumes and devices. A volume is a representation of 22some storage media. It is equivalent to a SPDK bdev and/or an NVMe namespace 23and can exist even if it's not presented to the host system. A device is 24usually a virtual/physical PCIe function that is exposed to a host. It is 25capable of presenting one or more volumes (depending on the type of the device) 26to a host. 27 28The following sections provide a high-level description of each method. For 29more details, consult the protobuf definitions. 30 31### CreateDevice 32 33This method creates a device. If a device with given parameters already exists, 34it becomes a no-op and returns a handle to that device. 35 36Input: 37 38- `volume`: Volume parameters describing a volume to immediately attach to the 39 created device. This field may be optional for some device types (e.g. NVMe), 40 while it may be required for others (e.g. virtio-blk). Extending parameters with 41 crypto attributes like encryption type, keys, tweak mode allows the user to configure 42 crypto when attaching a volume to a device. Users must specify the crypto engine 43 to use under `crypto` section in config. It is also possible to register out-of-tree 44 crypto engines by inheriting from the `CryptoEngine` class. 45- `params`: Device-specific parameters. The type of this structure determines 46 the type of device to create. 47 48Output: 49 50- `handle`: Opaque handle identifying the device. 51 52### DeleteDevice 53 54This method deletes a device. Volumes that are still attached to a device being 55deleted will be automatically detached. 56 57Input: 58 59- `handle`: Device handle obtained from `CreateDevice`. 60 61### AttachVolume 62 63This method creates a volume and attaches it to a device exposing it to the 64host. It might lead to establishing a connection to remote storage target. 65However, this is not always the case, even if the volume is remote. For 66instance, if a volume describes an NVMe namespace, it might already be connected 67if another volume on the same subsystem was created previously. It may be 68unsupported by some types of devices (e.g. virtio-blk). 69 70Input: 71 72- `volume`: Parameters describing the volume to attach. The type of this 73 structure determines the method to create it (e.g. direct NVMe-oF connection, 74 NVMe-oF through discovery service, iSCSI, etc.). Extending parameters with 75 crypto attributes like type, keys, tweak mode allowing the user to configure crypto 76 when attaching a volume to a device. Users must specify the crypto engine to use under 77 `crypto` section in config. It is also possible to register out-of-tree crypto engines 78 by inheriting from the `CryptoEngine` class. 79- `device_handle`: Device handle obtained from `CreateDevice`. 80 81### DetachVolume 82 83This method detaches a volume from a device making it unavailable to the host. 84It may be unsupported by some types of devices (e.g. virtio-blk). 85 86Input: 87 88- `volume_id`: Volume UUID/GUID. 89- `device_handle`: Device handle obtained from `CreateDevice`. 90 91### SetQos 92 93This method configures QoS on a per-device or per-volume level. 94Not all QoS settings have to be supported by each device, so users 95can use `GetQosCapabilities` to get capabilities. 96 97Input: 98 99- `handle`: Device handle obtained from `CreateDevice`. 100- `volume_id`: Volume UUID/GUID. If this parameter is omitted, the QoS will be set up 101 on the whole device (all volumes attached to that device will share QoS settings). 102 Some device types might only support configuring QoS on per-device 103 (volume_id must be empty) or per-volume level (volume_id cannot be empty). 104 This information can be obtained by sending a GetQosCapabilities request. 105- `maximum`: Maximum allowed IOPS/bandwidth values. 106 107### GetQosCapabilities 108 109This method queries supported QoS settings. 110 111Input: 112 113- `device_type`: Type of a device to query for QoS capabilities. 114 115Output: 116 117- `capabilities`: QoS capabilities response including device/volume 118 QoS limits like read IOPS, write IOPS, read/write IOPS, read bandwidth, 119 write bandwidth, read/write bandwidth. 120 121## Running and Configuration 122 123In order to run SMA, SPDK needs to be configured with the `--with-sma` flag. 124Then, SMA can be started using a script located in `scripts/sma.py`. It 125requires a YAML configuration file that specifies which types of devices to 126service, as well as several other options (e.g. listen address, SPDK-RPC socket, 127etc.). Device types not listed in the configuration will be disabled and it 128won't be possible to manage them. Below is an example configuration enabling 129two device types (NVMe/vfiouser and vhost-blk): 130 131```yaml 132address: 'localhost' 133socket: '/var/tmp/spdk.sock' 134port: 8080 135devices: 136 - name: 'vfiouser' 137 params: 138 root_path: '/var/tmp/vfiouser' 139 bus: 'bus0' 140 address: '127.0.0.1' 141 port: 4444 142 - name: 'vhost-blk' 143``` 144 145## Plugins 146 147SMA provides a way to load external plugins implementing support for specific 148device types. A plugin will be loaded if it's specified in the `SMA_PLUGINS` 149environment variable (multiple plugins are separated with a colon) or if it's 150specified in the `plugins` section of the config file. For example, the 151following two methods are equivalent: 152 153```sh 154$ SMA_PLUGINS=plugin1:plugin2 scripts/sma.py 155 156$ cat sma.yaml 157address: 'localhost' 158port: 8080 159plugins: 160 - 'plugin1' 161 - 'plugin2' 162devices: 163 - name: 'device-from-plugin1' 164 - name: 'device-from-plugin2' 165$ scripts/sma.py -c sma.yaml 166``` 167 168Each plugin needs to be in the python search path (either in one of the default 169directories or added to `PYTHONPATH`). 170 171A plugin is required to define a global variable called `devices` storing a list 172of classes deriving from `spdk.sma.DeviceManager`. This base class define the 173interface each device needs to implement. Additionally, each DeviceManager 174needs to define a unique name that will be used to identify it in config file as 175well as the name of the protocol it supports. There can be many DeviceManagers 176supporting the same protocol, but only one can be active at a time. The name of 177the protocol shall match the type specified in `CreateDeviceRequest.params` 178(e.g. "nvme", "virtio_blk", etc.), as it'll be used to select the DeviceManager 179to handle a gRPC request. Finally, a DeviceManager needs to implement the 180`own_device()` method returning a boolean value indicating whether a given 181device handle is owned by that DeviceManager. 182