bdev/raid: add qos for raid process

Change-Id: I5383d8658c59760ccaaec163e25d5e6d0cd8e1d0
Signed-off-by: xupeng-mingtu <jingmamour@gmail.com>
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/

bdev/raid: add qos for raid process

Change-Id: I5383d8658c59760ccaaec163e25d5e6d0cd8e1d0
Signed-off-by: xupeng-mingtu <jingmamour@gmail.com>
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/21732
Reviewed-by: Artur Paszkiewicz <artur.paszkiewicz@intel.com>
Reviewed-by: Aleksey Marchuk <alexeymar@nvidia.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <jim.harris@samsung.com>
Reviewed-by: Konrad Sztyber <konrad.sztyber@intel.com>
Community-CI: Mellanox Build Bot

show more ...

ut/raid: move out raid0-specific tests to separate file

This was a leftover from the time when raid0 was the only supported
level and was implemented directly in bdev_raid.c.

Signed-off-by: Artur P

ut/raid: move out raid0-specific tests to separate file

This was a leftover from the time when raid0 was the only supported
level and was implemented directly in bdev_raid.c.

Signed-off-by: Artur Paszkiewicz <artur.paszkiewicz@intel.com>
Change-Id: I416bd49843b04bf5c0c5fdc43e4aaf8566885630
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/22820
Reviewed-by: Jim Harris <jim.harris@samsung.com>
Community-CI: Mellanox Build Bot
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Konrad Sztyber <konrad.sztyber@intel.com>

show more ...

ut/raid: remove unused globals and functions

Signed-off-by: Artur Paszkiewicz <artur.paszkiewicz@intel.com>
Change-Id: Iafe94381a0b72c7b5c9c056d6c2798877bacdf19
Reviewed-on: https://review.spdk.io/g

ut/raid: remove unused globals and functions

Signed-off-by: Artur Paszkiewicz <artur.paszkiewicz@intel.com>
Change-Id: Iafe94381a0b72c7b5c9c056d6c2798877bacdf19
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/22774
Reviewed-by: Jim Harris <jim.harris@samsung.com>
Community-CI: Mellanox Build Bot
Reviewed-by: Konrad Sztyber <konrad.sztyber@intel.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Mateusz Kozlowski <mateusz.kozlowski@solidigm.com>
Reviewed-by: Krzysztof Karas <krzysztof.karas@intel.com>

show more ...

raid: fix race between process starting and removing a base bdev

Abort starting the process if a base bdev was removed in the meantime.

Signed-off-by: Artur Paszkiewicz <artur.paszkiewicz@intel.com

raid: fix race between process starting and removing a base bdev

Abort starting the process if a base bdev was removed in the meantime.

Signed-off-by: Artur Paszkiewicz <artur.paszkiewicz@intel.com>
Change-Id: I45345322c6e21837d257cbe0598c5eb19f3f41be
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/22773
Reviewed-by: Krzysztof Karas <krzysztof.karas@intel.com>
Reviewed-by: Konrad Sztyber <konrad.sztyber@intel.com>
Reviewed-by: Mateusz Kozlowski <mateusz.kozlowski@solidigm.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Community-CI: Mellanox Build Bot
Reviewed-by: Jim Harris <jim.harris@samsung.com>

show more ...

ut/raid: fix getting process thread

Use the new_thread_fn callback of spdk_thread_lib_init() to reliably get
the process thread.

The tests_single_run workaround can be removed now.

Signed-off-by:

ut/raid: fix getting process thread

Use the new_thread_fn callback of spdk_thread_lib_init() to reliably get
the process thread.

The tests_single_run workaround can be removed now.

Signed-off-by: Artur Paszkiewicz <artur.paszkiewicz@intel.com>
Change-Id: Id03671d411d89db746411bbffecb836a195f6b5d
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/21729
Reviewed-by: Konrad Sztyber <konrad.sztyber@intel.com>
Reviewed-by: Shuhei Matsumoto <smatsumoto@nvidia.com>
Community-CI: Mellanox Build Bot
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Mateusz Kozlowski <mateusz.kozlowski@solidigm.com>

show more ...

ut/raid: don't use ut_multithread.c

test_raid_process() abuses this framework because it needs to test a
thread created during the test. Initialize the thread lib and set up the
app thread manually

ut/raid: don't use ut_multithread.c

test_raid_process() abuses this framework because it needs to test a
thread created during the test. Initialize the thread lib and set up the
app thread manually instead.

Signed-off-by: Artur Paszkiewicz <artur.paszkiewicz@intel.com>
Change-Id: I5631fcbbc84a8dfc8a033f4ea8bb780f004d0957
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/21728
Community-CI: Mellanox Build Bot
Reviewed-by: Jim Harris <jim.harris@samsung.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Konrad Sztyber <konrad.sztyber@intel.com>
Reviewed-by: Shuhei Matsumoto <smatsumoto@nvidia.com>

show more ...

raid: move blocklen_shift to r5f_info

This is not generally useful now but there is one case in raid5f where
it is used to avoid a division, so move it there.

Signed-off-by: Artur Paszkiewicz <artu

raid: move blocklen_shift to r5f_info

This is not generally useful now but there is one case in raid5f where
it is used to avoid a division, so move it there.

Signed-off-by: Artur Paszkiewicz <artur.paszkiewicz@intel.com>
Change-Id: I2662160e72a71b2bae499655fe4fb7d236fb766f
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/21684
Reviewed-by: Konrad Sztyber <konrad.sztyber@intel.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Shuhei Matsumoto <smatsumoto@nvidia.com>
Reviewed-by: Mateusz Kozlowski <mateusz.kozlowski@solidigm.com>
Community-CI: Mellanox Build Bot

show more ...

raid: refactor to use write_named_uuid function

Use spdk_json_write_named_uuid() in
raid_bdev_write_*_json() functions

Change-Id: I589815c4328a39d7d3da99051671a8e0a497397a
Signed-off-by: Eugene Kob

raid: refactor to use write_named_uuid function

Use spdk_json_write_named_uuid() in
raid_bdev_write_*_json() functions

Change-Id: I589815c4328a39d7d3da99051671a8e0a497397a
Signed-off-by: Eugene Kobyak <eugene.kobyak@intel.com>
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/22880
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Konrad Sztyber <konrad.sztyber@intel.com>
Reviewed-by: Jim Harris <jim.harris@samsung.com>
Community-CI: Mellanox Build Bot
Reviewed-by: Shuhei Matsumoto <smatsumoto@nvidia.com>
Reviewed-by: Artur Paszkiewicz <artur.paszkiewicz@intel.com>

show more ...

module/raid: DIX support for RAID 0

Added an ability for the application to use DIX protection
(separate metadata) with the RAID bdev, with RAID level 0.

In order to handle the protection informati

module/raid: DIX support for RAID 0

Added an ability for the application to use DIX protection
(separate metadata) with the RAID bdev, with RAID level 0.

In order to handle the protection information correctly for
RAID bdevs, the patch introduces Reference Tag remapping between
the RAID bdev and base bdevs on read and write paths.

Change-Id: I4056a3e4410a4df3282b993b655f6e6e2715e5a9
Signed-off-by: Slawomir Ptak <slawomir.ptak@intel.com>
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/21618
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Shuhei Matsumoto <smatsumoto@nvidia.com>
Community-CI: Mellanox Build Bot
Reviewed-by: Jim Harris <jim.harris@samsung.com>

show more ...

raid: alloc sb buffer when configuring the raid bdev

Move the allocation to a later point, when the block size is known. This
will be needed for upcoming patches for interleaved metadata.

Signed-of

raid: alloc sb buffer when configuring the raid bdev

Move the allocation to a later point, when the block size is known. This
will be needed for upcoming patches for interleaved metadata.

Signed-off-by: Artur Paszkiewicz <artur.paszkiewicz@intel.com>
Change-Id: Ief8d59c2fec06dd0c4e3edc43153795177829e08
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/21679
Community-CI: Mellanox Build Bot
Reviewed-by: Konrad Sztyber <konrad.sztyber@intel.com>
Reviewed-by: Jim Harris <jim.harris@samsung.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>

show more ...

raid: add raid_bdev->superblock_enabled flag

The next patch will change when raid_bdev->sb is allocated and this flag
will be necessary.

Signed-off-by: Artur Paszkiewicz <artur.paszkiewicz@intel.co

raid: add raid_bdev->superblock_enabled flag

The next patch will change when raid_bdev->sb is allocated and this flag
will be necessary.

Signed-off-by: Artur Paszkiewicz <artur.paszkiewicz@intel.com>
Change-Id: If8043c24e9689104a80b18595e83269b79894aee
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/21726
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Community-CI: Mellanox Build Bot
Reviewed-by: Konrad Sztyber <konrad.sztyber@intel.com>

show more ...

raid: adjust block size calculations to support interleaved md

Signed-off-by: Artur Paszkiewicz <artur.paszkiewicz@intel.com>
Change-Id: Id26b3814d9f4195beb54f2c0ae42137b0685e77b
Reviewed-on: https:

raid: adjust block size calculations to support interleaved md

Signed-off-by: Artur Paszkiewicz <artur.paszkiewicz@intel.com>
Change-Id: Id26b3814d9f4195beb54f2c0ae42137b0685e77b
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/21678
Reviewed-by: Konrad Sztyber <konrad.sztyber@intel.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <jim.harris@samsung.com>
Community-CI: Mellanox Build Bot

show more ...

module/raid: stop rebuild before the raid bdev is unregistered

Open the raid bdev internally to receive SPDK_BDEV_EVENT_REMOVE and
stop the rebuild process from the event callback. Unregistering wil

module/raid: stop rebuild before the raid bdev is unregistered

Open the raid bdev internally to receive SPDK_BDEV_EVENT_REMOVE and
stop the rebuild process from the event callback. Unregistering will
be performed after the descriptor is closed. The process needs to be
stopped before the destruct callback, because stopping the rebuild
process may require unquiescing a range, which requires the bdev
spin lock, which is destroyed before the bdev module's destruct()
callback is called.

We will look later at trying to fix this problem at bdev layer, to
avoid bdev modules having to open themselves to avoid these kinds of
problems.

Change-Id: Ic9c5bf14f785aaf7629bf1c9d2e1889e254cec5f
Signed-off-by: Artur Paszkiewicz <artur.paszkiewicz@intel.com>
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/20830
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Community-CI: Mellanox Build Bot
Reviewed-by: Jim Harris <jim.harris@samsung.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Tested-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>

show more ...

module/raid: raid_io handling during rebuild

Handle I/Os for the processed (rebuilt) range differently than for the
unprocessed range. Use a copy of raid_bdev_io_channel, corresponding to
the proces

module/raid: raid_io handling during rebuild

Handle I/Os for the processed (rebuilt) range differently than for the
unprocessed range. Use a copy of raid_bdev_io_channel, corresponding to
the processed range, and assign it to the I/Os for this range. I/Os that
span both ranges will be queued by the bdev layer if the process window
range is quiesced. Otherwise, split them and handle separately.

Change-Id: I802fd5071d37fec07aa43d5dc8ce353aba41a2fe
Signed-off-by: Artur Paszkiewicz <artur.paszkiewicz@intel.com>
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/18741
Reviewed-by: Jim Harris <jim.harris@samsung.com>
Reviewed-by: Konrad Sztyber <konrad.sztyber@intel.com>
Community-CI: Mellanox Build Bot
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>

show more ...

module/raid: rebuild support

Add support for running a background "process" for performing the
rebuild I/O. In the future, different types of processes can be added,
like resync or migration.

The p

module/raid: rebuild support

Add support for running a background "process" for performing the
rebuild I/O. In the future, different types of processes can be added,
like resync or migration.

The process performs its work inside a "window" - a range of LBAs of
predefined maximum size. The basic flow of the process is:
- lock (quiesce) the range
- submit process requests, handled by the raid module implementation
- completion of process requests
- update io channels - move the processed range offset
- unlock (unquiesce) the range
- repeat until all blocks are processed or the process is stopped

Change-Id: I6a89bf06791683cac2e178211efe7b66a34394eb
Signed-off-by: Artur Paszkiewicz <artur.paszkiewicz@intel.com>
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/18739
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Community-CI: Mellanox Build Bot
Reviewed-by: Jim Harris <jim.harris@samsung.com>
Reviewed-by: Konrad Sztyber <konrad.sztyber@intel.com>
Reviewed-by: LiYankun <845245370@qq.com>

show more ...

ut/raid: clean up creating/destroying io channels in tests

Use spdk_get_io_channel() and spdk_put_io_channel() instead of
allocating and freeing the channels directly.

Change-Id: Ifaa1f1fc2025421ab

ut/raid: clean up creating/destroying io channels in tests

Use spdk_get_io_channel() and spdk_put_io_channel() instead of
allocating and freeing the channels directly.

Change-Id: Ifaa1f1fc2025421abe6c2756002d394cde5a8572
Signed-off-by: Artur Paszkiewicz <artur.paszkiewicz@intel.com>
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/18713
Reviewed-by: Konrad Sztyber <konrad.sztyber@intel.com>
Community-CI: Mellanox Build Bot
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <jim.harris@samsung.com>

show more ...

module/raid: show base bdev details in json

Change-Id: I0da3e91e7736bc651e284f68238ace864def87b2
Signed-off-by: Artur Paszkiewicz <artur.paszkiewicz@intel.com>
Reviewed-on: https://review.spdk.io/ge

module/raid: show base bdev details in json

Change-Id: I0da3e91e7736bc651e284f68238ace864def87b2
Signed-off-by: Artur Paszkiewicz <artur.paszkiewicz@intel.com>
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/16165
Reviewed-by: Konrad Sztyber <konrad.sztyber@intel.com>
Reviewed-by: Jim Harris <jim.harris@samsung.com>
Community-CI: Mellanox Build Bot
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>

show more ...

module/raid: assemble raid bdev from superblock

Change the bdev_raid examine procedure to read the superblock from the
examined base bdev. If a valid superblock is found, re-create the
raid_bdev fro

module/raid: assemble raid bdev from superblock

Change the bdev_raid examine procedure to read the superblock from the
examined base bdev. If a valid superblock is found, re-create the
raid_bdev from it.

Change-Id: I4bd589647a207a216ecf0dec9baf11c5d691f5d5
Signed-off-by: Artur Paszkiewicz <artur.paszkiewicz@intel.com>
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/16163
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Community-CI: Mellanox Build Bot
Reviewed-by: Konrad Sztyber <konrad.sztyber@intel.com>
Reviewed-by: Jim Harris <jim.harris@samsung.com>

show more ...

module/raid: write initial superblock

When creating the raid_bdev with enabled superblock option, write the
superblock to the base bdevs before bringing the array online.

Change-Id: I24659202ef3bbe

module/raid: write initial superblock

When creating the raid_bdev with enabled superblock option, write the
superblock to the base bdevs before bringing the array online.

Change-Id: I24659202ef3bbe6c87ca8603d514bd81660c9b41
Signed-off-by: Artur Paszkiewicz <artur.paszkiewicz@intel.com>
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/16162
Reviewed-by: Ben Walker <ben@nvidia.com>
Community-CI: Mellanox Build Bot
Reviewed-by: Konrad Sztyber <konrad.sztyber@intel.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>

show more ...

module/raid: data offset and data size implementation

When raid bdev is created with superblock parameter then all data on
this bdev should be shifted by some offset. Such space at the beginning
of

module/raid: data offset and data size implementation

When raid bdev is created with superblock parameter then all data on
this bdev should be shifted by some offset. Such space at the beginning
of bdev will be used to store on-disk raid metadata.

Change-Id: I2545a2b00a651ef5332ca1757da0110a63914a43
Signed-off-by: Krzysztof Smolinski <krzysztof.smolinski@intel.com>
Signed-off-by: Artur Paszkiewicz <artur.paszkiewicz@intel.com>
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/15246
Reviewed-by: Jim Harris <jim.harris@samsung.com>
Reviewed-by: Konrad Sztyber <konrad.sztyber@intel.com>
Community-CI: Mellanox Build Bot
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>

show more ...

rpc/bdev_raid_create: added superblock parameter

Introduction of superblock parameter for bdev_raid_create rpc. This
parameter determines whether raid bdev should be created with support
for on-disk

rpc/bdev_raid_create: added superblock parameter

Introduction of superblock parameter for bdev_raid_create rpc. This
parameter determines whether raid bdev should be created with support
for on-disk metadata (support for raid on-disk metadata is going to be
implemented in the future).

Change-Id: Ie8c64f837dd7eb3ba788b7c5d7bc98e8f1368ba7
Signed-off-by: Krzysztof Smolinski <krzysztof.smolinski@intel.com>
Signed-off-by: Artur Paszkiewicz <artur.paszkiewicz@intel.com>
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/15245
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Community-CI: Mellanox Build Bot
Reviewed-by: Jim Harris <jim.harris@samsung.com>
Reviewed-by: Konrad Sztyber <konrad.sztyber@intel.com>

show more ...

rpc: use spdk_json_decode_uuid() to decode UUIDs

Signed-off-by: Konrad Sztyber <konrad.sztyber@intel.com>
Change-Id: I99e5d44d2cea4d50219b44e4595379a3a17bd11f
Reviewed-on: https://review.spdk.io/ger

rpc: use spdk_json_decode_uuid() to decode UUIDs

Signed-off-by: Konrad Sztyber <konrad.sztyber@intel.com>
Change-Id: I99e5d44d2cea4d50219b44e4595379a3a17bd11f
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/20475
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Community-CI: Mellanox Build Bot
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Reviewed-by: Jim Harris <jim.harris@samsung.com>

show more ...

test/unit: use spdk_ut_run_tests()

Replaced direct calls to the CUnit's functions to run the tests with
spdk_ut_run_tests(). That way, each test will have the ability to run
a specific test case.

test/unit: use spdk_ut_run_tests()

Replaced direct calls to the CUnit's functions to run the tests with
spdk_ut_run_tests(). That way, each test will have the ability to run
a specific test case.

The blob.c unit test wasn't changed, because it runs all tests multiple
times with different parameter combinations, so it cannot be easily
converted. In the future, each such combination could be split into a
separate test suite, which would make it compatible with
spdk_ut_run_tests().

Signed-off-by: Konrad Sztyber <konrad.sztyber@intel.com>
Change-Id: I4463f808f89844e9bf32b5b31eda197c5d729d1d
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/19288
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Community-CI: Mellanox Build Bot
Reviewed-by: Aleksey Marchuk <alexeymar@nvidia.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>

show more ...

test/unit: move spdk_cunit.h to include/spdk_internal

It'll make it easier to include this file outside of unit tests.

Signed-off-by: Konrad Sztyber <konrad.sztyber@intel.com>
Change-Id: I171ddb864

test/unit: move spdk_cunit.h to include/spdk_internal

It'll make it easier to include this file outside of unit tests.

Signed-off-by: Konrad Sztyber <konrad.sztyber@intel.com>
Change-Id: I171ddb8649f67b5786f08647560e2907603d0574
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/19284
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Community-CI: Mellanox Build Bot
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Aleksey Marchuk <alexeymar@nvidia.com>

show more ...

module/raid: remove bdev pointer from base_bdev_info

There is a spdk_bdev_desc pointer in base_bdev_info, which is always set
together with the bdev pointer, so use that instead. It makes checking
w

module/raid: remove bdev pointer from base_bdev_info

There is a spdk_bdev_desc pointer in base_bdev_info, which is always set
together with the bdev pointer, so use that instead. It makes checking
whether a base bdev is present more consistent. Before, in some places
the bdev pointer was checked and in others the desc pointer.

Change-Id: I5ac305b7d4b07c609b759a59f7f0fa540532ea1e
Signed-off-by: Artur Paszkiewicz <artur.paszkiewicz@intel.com>
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/18286
Reviewed-by: Shuhei Matsumoto <smatsumoto@nvidia.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Community-CI: Mellanox Build Bot
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>

show more ...