blob: Add support for variable metadata page size

Currently SPDK blobstore depends on 4KB atomicity of metadata.
So, Metadata page size will be made variable based on the physical
block size reporte

blob: Add support for variable metadata page size

Currently SPDK blobstore depends on 4KB atomicity of metadata.
So, Metadata page size will be made variable based on the physical
block size reported by the underlying device. This will enable
blobstore to store metadata in IU-sized chunks, not always 4KB.

blobstore.c uses SPDK_BS_PAGE_SIZE (4KB) hardcoded at many places.
To remove this make the metadata page size variable,
phys_blocklen is added to spdk_bs_dev struct and
md_page_size is added to spdk_bs_super_block struct.

Change-Id: I29d073eb4f4341a94a0675e70492b9186382f97f
Signed-off-by: Atul Malakar <a.malakar@samsung.com>
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/25130
Community-CI: Broadcom CI <spdk-ci.pdl@broadcom.com>
Community-CI: Mellanox Build Bot
Reviewed-by: Ben Walker <ben@nvidia.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Community-CI: Community CI Samsung <spdk.community.ci.samsung@gmail.com>
Reviewed-by: Konrad Sztyber <konrad.sztyber@intel.com>
Reviewed-by: Jim Harris <jim.harris@nvidia.com>

show more ...

so_ver: increase all major versions after SPDK 24.01

To allow SO_MINOR updates on LTS for the whole year it is supported,
the major version for all components needs to be increased.
This is to preve

so_ver: increase all major versions after SPDK 24.01

To allow SO_MINOR updates on LTS for the whole year it is supported,
the major version for all components needs to be increased.
This is to prevent scenario where two versions exists with matching
versions, but conflicting ABI.
Ex. Next SPDK release adds an API call increasing the minor version,
then LTS needs just a subset of those additions.

Increasing major so version after LTS, allows the future releases
to update versions as needed. Yet allowing LTS to increase minor
version separately.

This patch:
- increases SO_VER by 1 for all components
- resets SO_MINOR to 0 for all components

Due to patch below being introduced after v24.01 code freeze,
bdev lib version should not be increased:
61623c5ca9 bdev: add spdk_bdev_io padding in place of io_submit_ch

Short reference to how the versions were changed:
MAX=$(git grep "SO_VER := " | cut -d" " -f 3 | sort -ubnr | head -1)
for((i=$MAX;i>0;i-=1)); do find . -name "Makefile" -exec \
sed -i -e "s/SO_VER := $i\$/SO_VER := $(($i+1))/g" {} +; done
find . -name "Makefile" -exec \
sed -i -e "s/SO_MINOR := .*/SO_MINOR := 0/g" {} +

Signed-off-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Change-Id: I0166d6be6c17bbbf985d48b5dfcb36f1d4af1b48
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/21582
Reviewed-by: Konrad Sztyber <konrad.sztyber@intel.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>

show more ...

bdev/lvol: use spdk_bs_grow_live() in grow lvs RPC

This patch adds spdk_lvs_grow_live() pipes through to
spdk_bs_grow_live() and use it when
bdev_lvol_grow_lvstore RPC is called.

Growing the size o

bdev/lvol: use spdk_bs_grow_live() in grow lvs RPC

This patch adds spdk_lvs_grow_live() pipes through to
spdk_bs_grow_live() and use it when
bdev_lvol_grow_lvstore RPC is called.

Growing the size of lvs still needs to be explicit
RPC call by the user.

Moved include for blob_bdev.h to header of the whole
bdev module. This way spdk_bdev_update_bs_blockcnt()
can be called when RPC is called.

Please note that there already exists a test with this
RPC - blobstore_grow.sh. Next patch in series
adds test calling grow while I/O is being performed.

Signed-off-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Change-Id: I9e896c65a8b017d0d20b77d3758901e588e9e70c
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/20322
Community-CI: Mellanox Build Bot
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <jim.harris@samsung.com>
Reviewed-by: Shuhei Matsumoto <smatsumoto@nvidia.com>

show more ...

so_ver: increase all major versions

To allow SO_MINOR updates on LTS for the whole year it is supported,
the major version for all components needs to be increased.
This is to prevent scenario where

so_ver: increase all major versions

To allow SO_MINOR updates on LTS for the whole year it is supported,
the major version for all components needs to be increased.
This is to prevent scenario where two versions exists with matching
versions, but conflicting ABI.
Ex. Next SPDK release adds an API call increasing the minor version,
then LTS needs just a subset of those additions.

Increasing major so version after LTS, allows the future releases
to update versions as needed. Yet allowing LTS to increase minor
version separately.

Disabled test for increasing SO version without ABI change, as
that is goal of this patch. This check shall be removed with SPDK 23.05
release.
Looks like this was left over from prior LTS, to avoid that
make sure it is only skipped when running against v23.01.x as latest
release.

This patch:
- increases SO_VER by 1 for all components
- resets SO_MINOR to 0 for all components
- removes suppressions for ABI tests


Short reference to how the versions were changed:
MAX=$(git grep "SO_VER := " | cut -d" " -f 3 | sort -ubnr | head -1)
for((i=$MAX;i>0;i-=1)); do find . -name "Makefile" -exec \
sed -i -e "s/SO_VER := $i\$/SO_VER := $(($i+1))/g" {} +; done
find . -name "Makefile" -exec \
sed -i -e "s/SO_MINOR := .*/SO_MINOR := 0/g" {} +

Change-Id: I3e5681802c0a5ac6d7d652a18896997cd07cc8bf
Signed-off-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/16419
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Konrad Sztyber <konrad.sztyber@intel.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>

show more ...

blob: Add translate_lba operation

New `translate_lba` operation allows to translate blob lba to lba on
the underlying bdev. It recurses down the whole chain of bs_dev's. The
operation may fail to do

blob: Add translate_lba operation

New `translate_lba` operation allows to translate blob lba to lba on
the underlying bdev. It recurses down the whole chain of bs_dev's. The
operation may fail to do the translation when blob lba is not backed
by the real bdev. For example, when we eventually hit zeroes device in
the chain.

This operation is used in the next commit to get source LBA for copy
operation.

Signed-off-by: Evgeniy Kochetov <evgeniik@nvidia.com>
Change-Id: I89c2d03d1982d66b9137a3a3653a98c361984fab
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/14528
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Shuhei Matsumoto <smatsumoto@nvidia.com>
Reviewed-by: Aleksey Marchuk <alexeymar@nvidia.com>

show more ...

update Intel copyright notices

per Intel policy to include file commit date using git cmd
below. The policy does not apply to non-Intel (C) notices.

git log --follow -C90% --format=%ad --date defa

update Intel copyright notices

per Intel policy to include file commit date using git cmd
below. The policy does not apply to non-Intel (C) notices.

git log --follow -C90% --format=%ad --date default <file> | tail -1

and then pull just the 4 digit year from the result.

Intel copyrights were not added to files where Intel either had
no contribution ot the contribution lacked substance (ie license
header updates, formatting changes, etc). Contribution date used
"--follow -C95%" to get the most accurate date.

Note that several files in this patch didn't end the license/(c)
block with a blank comment line so these were added as the vast
majority of files do have this last blank line. Simply there for
consistency.

Signed-off-by: paul luse <paul.e.luse@intel.com>
Change-Id: Id5b7ce4f658fe87132f14139ead58d6e285c04d4
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/15192
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Community-CI: Mellanox Build Bot

show more ...

blobstore: reserve space for growing blobstore

Reserve space for used_cluster bitmap. The reserved space is calculated
according to the num_md_pages. The reserved space would be used when
the blobst

blobstore: reserve space for growing blobstore

Reserve space for used_cluster bitmap. The reserved space is calculated
according to the num_md_pages. The reserved space would be used when
the blobstore is extended in the future.
Add the num_md_pages_per_cluster_ratio parameter to the
bdev_lvol_create_lvstore API. Then calculate the num_md_pages
according to the num_md_pages_per_cluster_ratio and bdev total size, then
pass the num_md_pages to the blobstore.

Signed-off-by: Peng Yu <yupeng0921@gmail.com>
Change-Id: I61a28a3c931227e0fd3e1ef6b145fc18a3657751
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/9517
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Community-CI: Broadcom CI <spdk-ci.pdl@broadcom.com>
Community-CI: Mellanox Build Bot
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>

show more ...

Replace most BSD 3-clause license text with SPDX identifier.

Many open source projects have moved to using SPDX identifiers
to specify license information, reducing the amount of
boilerplate code in

Replace most BSD 3-clause license text with SPDX identifier.

Many open source projects have moved to using SPDX identifiers
to specify license information, reducing the amount of
boilerplate code in every source file. This patch replaces
the bulk of SPDK .c, .cpp and Makefiles with the BSD-3-Clause
identifier.

Almost all of these files share the exact same license text,
and this patch only modifies the files that contain the
most common license text. There can be slight variations
because the third clause contains company names - most say
"Intel Corporation", but there are instances for Nvidia,
Samsung, Eideticom and even "the copyright holder".

Used a bash script to automate replacement of the license text
with SPDX identifier which is checked into scripts/spdx.sh.

Signed-off-by: Jim Harris <james.r.harris@intel.com>
Change-Id: Iaa88ab5e92ea471691dc298cfe41ebfb5d169780
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/12904
Community-CI: Broadcom CI <spdk-ci.pdl@broadcom.com>
Community-CI: Mellanox Build Bot
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Aleksey Marchuk <alexeymar@nvidia.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: Dong Yi <dongx.yi@intel.com>
Reviewed-by: Konrad Sztyber <konrad.sztyber@intel.com>
Reviewed-by: Paul Luse <paul.e.luse@intel.com>
Reviewed-by: <qun.wan@intel.com>

show more ...

so_ver: increase all major versions

To allow SO_MINOR updates on LTS for the whole year it is supported,
the major version for all components needs to be increased.
This is to prevent scenario where

so_ver: increase all major versions

To allow SO_MINOR updates on LTS for the whole year it is supported,
the major version for all components needs to be increased.
This is to prevent scenario where two versions exists with matching
versions, but conflicting ABI.
Ex. Next SPDK release adds an API call increasing the minor version,
then LTS needs just a subset of those additions.

Increasing major so version after LTS, allows the future releases
to update versions as needed. Yet allowing LTS to increase minor
version separately.

Disabled test for increasing SO version without ABI change, as
that is goal of this patch. This check shall be removed with SPDK 22.05
release.

This patch:
- increases SO_VER by 1 for all components
- resets SO_MINOR to 0 for all components
- removes suppressions for ABI tests

Signed-off-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Change-Id: Id1a5358882dc496faa5b0b5c9a63b326c378c551
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/11361
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Community-CI: Broadcom CI <spdk-ci.pdl@broadcom.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>

show more ...

blob: use uint64_t for unmap and write_zeroes lba count

Previous patches (5363eb3c) tried to work around the
32-bit unmap and write_zeroes LBA counts by breaking
up larger operations into smaller ch

blob: use uint64_t for unmap and write_zeroes lba count

Previous patches (5363eb3c) tried to work around the
32-bit unmap and write_zeroes LBA counts by breaking
up larger operations into smaller chunks of max size
UINT32_MAX lba chunks.

But some SSDs may just ignore unmap operations that
are not aligned to full physical block boundaries -
and a UINT32_MAX lba unmap on a 512B logical /
4KiB physical SSD would not be aligned. If the SSD
decided to ignore the unmap/deallocate (which it is
allowed to do according to NVMe spec), we could end
up with not unmapping *any* blocks. Probably SSDs
should always be trying hard to unmap as many
blocks as possible, but let's not try to depend on
that in blobstore.

So one option would be to break them into chunks
close to UINT32_MAX which are still aligned to
4KiB boundaries. But the better fix is to just
change the unmap and write_zeroes APIs to take
64-bit arguments, and then we can avoid the
chunking altogether.

Fixes issue #2190.

Signed-off-by: Jim Harris <james.r.harris@intel.com>
Change-Id: I23998e493a764d466927c3520c7a8c7f943000a6
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/9737
Community-CI: Broadcom CI <spdk-ci.pdl@broadcom.com>
Community-CI: Mellanox Build Bot
Reviewed-by: Xiaodong Liu <xiaodong.liu@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: Aleksey Marchuk <alexeymar@mellanox.com>
Reviewed-by: Dong Yi <dongx.yi@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>

show more ...

so_ver: increase all major versions

To allow SO_MINOR updates on LTS for the whole year it is supported,
the major version for all components needs to be increased.
This is to prevent scenario where

so_ver: increase all major versions

To allow SO_MINOR updates on LTS for the whole year it is supported,
the major version for all components needs to be increased.
This is to prevent scenario where two versions exists with matching
versions, but conflicting ABI.
Ex. Next SPDK release adds an API call increasing the minor version,
then LTS needs just a subset of those additions.

Increasing major so version after LTS, allows the quarterly releases
to update versions as needed. Yet allowing LTS to increase minor
version separately.

Disabled test for increasing SO version without ABI change, as
that is goal of this patch. This check shall be removed with SPDK 21.04
release.

This patch:
- increases SO_VER by 1 for all components
- resets SO_MINOR to 0 for all components
- removes suppressions for ABI tests

Signed-off-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Change-Id: I44d01154430a074103bd21c7084f44932e81fe72
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/6167
Community-CI: Broadcom CI
Community-CI: Mellanox Build Bot
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>

show more ...

bdev: extend interface with get module ctx

Currently only nvme bdev module implements this interface. Bdev module
context (in this case spdk_nvme_ctrlr opaque handle) allows for nvme
interface usage

bdev: extend interface with get module ctx

Currently only nvme bdev module implements this interface. Bdev module
context (in this case spdk_nvme_ctrlr opaque handle) allows for nvme
interface usage for additional management.

Signed-off-by: Jacek Kalwas <jacek.kalwas@intel.com>
Change-Id: I6302c9229d5f7f294a3c1472d9e8dc1519637ffb
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/4924
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Reviewed-by: Aleksey Marchuk <alexeymar@mellanox.com>

show more ...

mk/lib: add a check that major and minor version is set for libs.

Also, while we are here, consolidate setting SO_SUFFIX to one spot.

Previously, it was possible for a library to slip through
witho

mk/lib: add a check that major and minor version is set for libs.

Also, while we are here, consolidate setting SO_SUFFIX to one spot.

Previously, it was possible for a library to slip through
without an SO version.

Signed-off-by: Seth Howell <seth.howell@intel.com>
Change-Id: I4db5fa5839502d266c6259892e5719b05134518c
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/2361
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Community-CI: Mellanox Build Bot
Community-CI: Broadcom CI
Reviewed-by: Paul Luse <paul.e.luse@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Aleksey Marchuk <alexeymar@mellanox.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>

show more ...

lib/lvol: add a map file.

Signed-off-by: Seth Howell <seth.howell@intel.com>
Change-Id: Iec2a50ca4af8fe24f011d4cc6e04606115ba45a0
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/2237
Commun

lib/lvol: add a map file.

Signed-off-by: Seth Howell <seth.howell@intel.com>
Change-Id: Iec2a50ca4af8fe24f011d4cc6e04606115ba45a0
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/2237
Community-CI: Mellanox Build Bot
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>

show more ...

make: rev SO versions individually for libraries.

This will allow us to keep track of compatibility issues on a
per-library basis.

Change-Id: Ib0c796adb1efe1570212a503ed660bef6f142b6e
Signed-off-by

make: rev SO versions individually for libraries.

This will allow us to keep track of compatibility issues on a
per-library basis.

Change-Id: Ib0c796adb1efe1570212a503ed660bef6f142b6e
Signed-off-by: Seth Howell <seth.howell@intel.com>
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/1067
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>

show more ...

build: remove $(ENV_CFLAGS) where not necessary

Only Makefiles for libraries that directly depend on DPDK (rather than
the SPDK env abstraction) should add $(ENV_CFLAGS).

Change-Id: Ifdf44d3ef8c42b

build: remove $(ENV_CFLAGS) where not necessary

Only Makefiles for libraries that directly depend on DPDK (rather than
the SPDK env abstraction) should add $(ENV_CFLAGS).

Change-Id: Ifdf44d3ef8c42bbf7f20edd524b330d00658235b
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
Reviewed-on: https://review.gerrithub.io/392818
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>

show more ...

lvol: Logical volume implementation

Change-Id: Ia96ae78ff9530d953181ac5f7255a38f3c8ec430
Signed-off-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Signed-off-by: Maciej Szwed <maciej.szwed@intel.co

lvol: Logical volume implementation

Change-Id: Ia96ae78ff9530d953181ac5f7255a38f3c8ec430
Signed-off-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Signed-off-by: Maciej Szwed <maciej.szwed@intel.com>
Signed-off-by: Piotr Pelplinski <piotr.pelplinski@intel.com>
Reviewed-on: https://review.gerrithub.io/375392
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Daniel Verkamp <daniel.verkamp@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>

show more ...