add (c) and SPDX header to python files as needed

per Intel policy to include file commit date using git cmd
below. The policy does not apply to non-Intel (C) notices.

git log --follow -C90% --for

add (c) and SPDX header to python files as needed

per Intel policy to include file commit date using git cmd
below. The policy does not apply to non-Intel (C) notices.

git log --follow -C90% --format=%ad --date default <file> | tail -1

and then pull just the year from the result.

Intel copyrights were not added to files where Intel either had
no contribution ot the contribution lacked substance (ie license
header updates, formatting changes, etc)

Note that several files in this patch didn't end the license/(c)
block with a blank comment line so these were added as the vast
majority of files do have this last blank line. Simply there for
consistency.

Signed-off-by: paul luse <paul.e.luse@intel.com>
Change-Id: I6cd3f18d1b469d5ef249d26ddb2923ca6b970bd4
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/15208
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>

show more ...

sma: initial crypto definitions

This patch defines the interface for crypto engines, which provide
support for configuring crypto on a given volume. Only a single crypto
engine can be active at a t

sma: initial crypto definitions

This patch defines the interface for crypto engines, which provide
support for configuring crypto on a given volume. Only a single crypto
engine can be active at a time and it's selected in the "crypto" section
of the config file. Similarly to device managers, external crypto
engines can be loaded from plugins.

Signed-off-by: Konrad Sztyber <konrad.sztyber@intel.com>
Change-Id: Id942ef876e070816827d7ad1937eb510a85c8f8d
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/13869
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: <sebastian.brzezinka@intel.com>

show more ...

sma: nvmf/vfiouser device manager implementation

This patch utilize generic sma implementation by adding vfiouser
devices manager. It's allow to expose virtualized block devices to
QEMU instances or

sma: nvmf/vfiouser device manager implementation

This patch utilize generic sma implementation by adding vfiouser
devices manager. It's allow to expose virtualized block devices to
QEMU instances or other arbitrary processes.

Max device capacity depend on available `pci-bridge`
```yaml
devices:
- name: 'vfiouser'
params:
buses:
- name: 'pci.spdk.0'
count: 32
- name: 'pci.spdk.1'
count: 32
qmp_addr: 127.0.0.1
qmp_port: 9090
```

Signed-off-by: Milosz Linkiewicz <milosz.linkiewicz@intel.com>
Signed-off-by: Sebastian Brzezinka <sebastian.brzezinka@intel.com>
Change-Id: I5ab43f4b877c371fa16a4daf4212ac2686991bd4
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/13004
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Konrad Sztyber <konrad.sztyber@intel.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>

show more ...

sma: vhost-user sma implementation

This patch utilize generic sma implementation by adding vhost-user
devices manager. It's allow to expose virtualized block devices to
QEMU instances or other arbit

sma: vhost-user sma implementation

This patch utilize generic sma implementation by adding vhost-user
devices manager. It's allow to expose virtualized block devices to
QEMU instances or other arbitrary processes.

Max device capacity depend on available `pci-bridge`
```
devices:
- name: 'vhost_blk'
params:
buses:
- name: 'pci.spdk.0'
count: 32
- name: 'pci.spdk.1'
count: 32
qmp_addr: 127.0.0.1
qmp_port: 9090
```

To attach `pci-bridge` at boot time we need to run qemu with this option
```
device pci-bridge,chassis_nr=1,id=pci.spdk.0
device pci-bridge,chassis_nr=2,id=pci.spdk.1
``

Change-Id: Idbe841b12cf59975ff0e6717f8dc905d38379357
Signed-off-by: Sebastian Brzezinka <sebastian.brzezinka@intel.com>
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/12207
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Community-CI: Broadcom CI <spdk-ci.pdl@broadcom.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Konrad Sztyber <konrad.sztyber@intel.com>
Reviewed-by: Artek Koltun <artsiom.koltun@intel.com>

show more ...

sma: add volume cleanup thread

Since SMA keeps track of attached volumes and a volume might get
disconnected at any time (in which case they're also removed from the
host side), a mechanism is neede

sma: add volume cleanup thread

Since SMA keeps track of attached volumes and a volume might get
disconnected at any time (in which case they're also removed from the
host side), a mechanism is needed to monitor if any of the volumes are
no longer accessible.

This patch implements that mechanism by adding a new thread running in
the background that will periodically (60s by default) send a
bdev_get_bdevs RPC and check that all previously attached volumes are
available. If any of them are not, it'll remove it and stop the
associated discovery services (if their refcount goes down to zero).
The period can be changed through the `volume_cleanup_period` variable
in the config file.

One important thing to note is that we assume that any intermittent
connection issues are handled internally by the SPDK application and a
bdev is only removed after all reconnection attempts are performed.

Signed-off-by: Konrad Sztyber <konrad.sztyber@intel.com>
Change-Id: I5b9e63698879527d9f79156a0eda1c8bc5e66def
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/12699
Community-CI: Broadcom CI <spdk-ci.pdl@broadcom.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>

show more ...

sma: initial support for volume connection via discovery

This patch adds support for connecting volumes via discovery service.
The user specifies a volume UUID/GUID and a list of discovery endpoints

sma: initial support for volume connection via discovery

This patch adds support for connecting volumes via discovery service.
The user specifies a volume UUID/GUID and a list of discovery endpoints,
which are then used to start the discovery service on and attach the
volume to a device.

SMA will keep track of the attached volumes and will also refcount the
connections to discovery services. So if two volumes are attached using
the same discovery endpoint, it'll be disconnected only after both of
them are detached.

Signed-off-by: Konrad Sztyber <konrad.sztyber@intel.com>
Change-Id: Ie8ea50a2a784cf0db8a5953234c6bb2b68685d7c
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/12413
Community-CI: Broadcom CI <spdk-ci.pdl@broadcom.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>

show more ...

sma: pass config to StorageManagementAgent's __init__

This will make it easier to control various options through config file.

Signed-off-by: Konrad Sztyber <konrad.sztyber@intel.com>
Change-Id: I1

sma: pass config to StorageManagementAgent's __init__

This will make it easier to control various options through config file.

Signed-off-by: Konrad Sztyber <konrad.sztyber@intel.com>
Change-Id: I19ab9c020d536521f5985227bc43965bd04d2707
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/12724
Community-CI: Broadcom CI <spdk-ci.pdl@broadcom.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>

show more ...

sma: wait until SPDK process is responding

Most of the devices need to send RPCs during initialization (e.g. create
a transport), so we need to take care that they're initialized only
after we are c

sma: wait until SPDK process is responding

Most of the devices need to send RPCs during initialization (e.g. create
a transport), so we need to take care that they're initialized only
after we are certain that the SPDK process is listening.

The mechanisim is similar to the `waitforlisten` function used in our
test scripts - it sends a series of `rpc_get_methods` calls and times
out after a period of time if no response is received. For now, the
timeout is hardcoded to 60s.

Signed-off-by: Konrad Sztyber <konrad.sztyber@intel.com>
Change-Id: Iddadc04ad4c486d2894bc40e1a899a9d204400fc
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/11802
Community-CI: Broadcom CI <spdk-ci.pdl@broadcom.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>

show more ...

sma: handle SIGINT and SIGTERM signals

If the app is killed with either one of these signals, it'll shutdown
gracefully and exit with zero.

Signed-off-by: Konrad Sztyber <konrad.sztyber@intel.com>

sma: handle SIGINT and SIGTERM signals

If the app is killed with either one of these signals, it'll shutdown
gracefully and exit with zero.

Signed-off-by: Konrad Sztyber <konrad.sztyber@intel.com>
Change-Id: I630f9e34c3ccb382c8e1b53d0f589f3ec4d1483b
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/11727
Community-CI: Broadcom CI <spdk-ci.pdl@broadcom.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>

show more ...

sma: register devices based on config file

This allows a user to specify the types of devices to support. Those
not specified in the config, won't get initialized and will not service
user's reques

sma: register devices based on config file

This allows a user to specify the types of devices to support. Those
not specified in the config, won't get initialized and will not service
user's requests.

Additionally, each device manager can now receive its own configuration
in its init() method.

The device configuration is structured as a list of objects with two
properties: "name" and "params". The former identifies a device to
enable, while the latter contains a set of options (if any) specific to
that device manager. For instance:

```
devices:
- name: 'nvmf-tcp'
params:
max_queue_depth: 256
io_unit_size: 8192
- name: 'nvmf-vfiouser'
```

Signed-off-by: Konrad Sztyber <konrad.sztyber@intel.com>
Change-Id: I967016502ad93c243b3a7af58992bde14c44953c
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/11713
Community-CI: Broadcom CI <spdk-ci.pdl@broadcom.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>

show more ...

sma: load configuration from file

This patch adds support for loading configuration from a yaml-formatted
file. The command-line options takes precedence over those from the
config file.

Signed-of

sma: load configuration from file

This patch adds support for loading configuration from a yaml-formatted
file. The command-line options takes precedence over those from the
config file.

Signed-off-by: Konrad Sztyber <konrad.sztyber@intel.com>
Change-Id: Iafb5b4aec0a64afe358fce2d6c55361fc4093223
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/11712
Community-CI: Broadcom CI <spdk-ci.pdl@broadcom.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Karol Latecki <karol.latecki@intel.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Monica Kenguva <monica.kenguva@intel.com>

show more ...

sma: loading out-of-tree plugins

It's now possible to register SMA device managers defined in a module
outside of the regular directory. To do that, a global variable called
`devices` containing a

sma: loading out-of-tree plugins

It's now possible to register SMA device managers defined in a module
outside of the regular directory. To do that, a global variable called
`devices` containing a list of device manager classes needs to be defined
in a module that's added to SMA_PLUGINS environment variable.

For example:
```
$ cat /path/to/plugins/external_devices/__init__.py
import spdk.sma as sma

class MyDeviceManager1(sma.DeviceManager):
pass

class MyDeviceManager2(sma.DeviceManager):
pass

devices = [MyDeviceManager1, MyDeviceManager2]

$ SMA_PLUGINS=external_devices scripts/sma.py
```

Multiple plugins can be specified by separating them with a colon, e.g.:
```
SMA_PLUGINS=plugin1:plugin2
```
Of course, the location at which these modules reside must be in
PYTHONPATH.

Signed-off-by: Konrad Sztyber <konrad.sztyber@intel.com>
Change-Id: I2bb4cfea5191710f97d66abe3a21e4deacc6338a
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/11412
Community-CI: Broadcom CI <spdk-ci.pdl@broadcom.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>

show more ...

sma: make SMA parameters configurable

It's now possible to specify the gRPC interface listen address and the
SPDK-RPC socket path.

Change-Id: I957cf4ca4e03eac6f4f6f7ffab5ee95c5b6f7521
Signed-off-by

sma: make SMA parameters configurable

It's now possible to specify the gRPC interface listen address and the
SPDK-RPC socket path.

Change-Id: I957cf4ca4e03eac6f4f6f7ffab5ee95c5b6f7521
Signed-off-by: Sebastian Brzezinka <sebastian.brzezinka@intel.com>
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/10979
Community-CI: Broadcom CI <spdk-ci.pdl@broadcom.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>

show more ...

sma: initial Storage Management Agent structures

Storage Management Agent is a gRPC server that provides an abstraction
layer above the SPDK RPC interface. The interface aims to provide a set
of me

sma: initial Storage Management Agent structures

Storage Management Agent is a gRPC server that provides an abstraction
layer above the SPDK RPC interface. The interface aims to provide a set
of methods for managing various protocols (e.g. NVMe, virtio-blk) while
hiding the details of a particular transport.

The external API is defined by `lib/python/spdk/sma/proto/sma.proto`
protobuf file. It defines the generic gRPC service methods and their
requests/responses. Device-specific messages are defined in their own
files. This patch also defines messages for creating NVMe and NVMe/TCP
devices.

This patch implements a gRPC service that delegates the work to a
specific device type. A DeviceManager is a class that implements some
of the methods defined by the service for a given type of devices (e.g.
NVMe, virtio-blk, NVMe/TCP, etc.). For now, only the RPC for creating a
device is implemented, others are added in subsequent patches.

The series implements the generic calls as well as their NVMe/TCP
implementation. Support for other devce types could be easily added by
creating a new device manager and defining its protobuf parameter
definition.

Signed-off-by: Konrad Sztyber <konrad.sztyber@intel.com>
Change-Id: I17cde3b31d3514878f1027cfcd112b48848f6123
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/10273
Community-CI: Broadcom CI <spdk-ci.pdl@broadcom.com>
Community-CI: Mellanox Build Bot
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>

show more ...