blob: remove all references to pages as io_units

When blobstore was first created, everything was in terms of 4KB
pages - this included metadata page size and the unit for I/O
operations.

A bit lat

blob: remove all references to pages as io_units

When blobstore was first created, everything was in terms of 4KB
pages - this included metadata page size and the unit for I/O
operations.

A bit later, we introduced concept of "io_unit". If a blobstore
was put on a bdev with 512 blocksize, then the io_unit could be
512 bytes.

But when this happened, we should have changed all of the blobstore
code such that remaining "page" references only referred to metadata
pages. Instead, we left a bunch of places where we would convert
various values to/from a number of 4KB pages, and then to the number
of io_units. This made the code quite confusing, since direct
conversion to/from io_units would have been much clearer.

This existing problem was exacerbated with the upcoming patch to support
variable metadata page sizes. We need things like spdk_bs_get_page_size()
to return the size of the metadata pages, which may not be 4KB.

So make all of the changes necessary such that all references to
"page" now means "metadata page". This includes removing the
spdk_blob_get_num_pages() function, which no longer makes sense.

Signed-off-by: Jim Harris <jim.harris@samsung.com>
Change-Id: I66e93e7a4325a3b032bb16edaf657ef12044e1fd
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/25368
Community-CI: Mellanox Build Bot
Community-CI: Broadcom CI <spdk-ci.pdl@broadcom.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Ben Walker <ben@nvidia.com>
Community-CI: Community CI Samsung <spdk.community.ci.samsung@gmail.com>
Reviewed-by: Konrad Sztyber <konrad.sztyber@intel.com>

show more ...

blob: Handle range check for all snapshot back_bs_dev up in the chain

When there are multiple snapshots in the backing for a blob,
we traverse all of them. During traversal, the is_range_valid
check

blob: Handle range check for all snapshot back_bs_dev up in the chain

When there are multiple snapshots in the backing for a blob,
we traverse all of them. During traversal, the is_range_valid
check need to be done for every one of them.

Change-Id: Ib8318a5d3cdc558e7f295e0543587a20a2c32917
Signed-off-by: Diwakar Sharma <diwakar.sharma@datacore.com>
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/23020
Community-CI: Mellanox Build Bot
Reviewed-by: Jim Harris <jim.harris@samsung.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>

show more ...

lib/blob: fixed potential expression overflow

Fixed issue found by Coverity.

'payload_bytes' and 'zero_bytes' are 64-bit (uint64_t)
integers and are calculated as a product of two 32-bit
integers.

lib/blob: fixed potential expression overflow

Fixed issue found by Coverity.

'payload_bytes' and 'zero_bytes' are 64-bit (uint64_t)
integers and are calculated as a product of two 32-bit
integers. To avoid using of 32-bit arithmetic potentially
overflowing the result (truncating to 32-bit), we need
to cast at least one of the integers to 64-bit.

Change-Id: I323a3d1caa39f8358830f3fb06a5cc3fa0ca0555
Signed-off-by: Marcin Spiewak <marcin.spiewak@intel.com>
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/23024
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Community-CI: Mellanox Build Bot
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Reviewed-by: Jim Harris <jim.harris@samsung.com>

show more ...

blob: handle IO to a resized blob

When a blob is resized and is backed by a another blob like
a snapshot, then the resized blob has more number of clusters
compared to backing device. Since the blob

blob: handle IO to a resized blob

When a blob is resized and is backed by a another blob like
a snapshot, then the resized blob has more number of clusters
compared to backing device. Since the blob is thin, either
originally or as a result of creating snapshot, the IOs to
the new cluster ranges fail as there is no cluster to be looked
at in the backing. This patch introduces mechanism to handle reads
by zeroeing out the payload which is trailing beyond cluster range,
and handle writes by validating backing dev cluster range during
copy-on-write.

Change-Id: Ib204b922819775396e53ffb718e8230ebf8fa46b
Signed-off-by: Diwakar Sharma <diwakar.sharma@datacore.com>
Co-authored-by: Diwakar Sharma <diwakar.sharma@datacore.com>
Co-authored-by: Jim Harris <james.r.harris@intel.com>
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/22377
Community-CI: Mellanox Build Bot
Reviewed-by: Shuhei Matsumoto <smatsumoto@nvidia.com>
Reviewed-by: Jim Harris <jim.harris@samsung.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>

show more ...

blob: add is_degraded() to spdk_blob_bs_dev

The health of clones of esnap clones depends on the health of the esnap
clone. This allows recursion through a chain of clones so that degraded
state prop

blob: add is_degraded() to spdk_blob_bs_dev

The health of clones of esnap clones depends on the health of the esnap
clone. This allows recursion through a chain of clones so that degraded
state propagates up from any back_bs_dev that is degraded.

Signed-off-by: Mike Gerdts <mgerdts@nvidia.com>
Change-Id: Iadd879d589f6ce4d0b654945db065d304b0c8357
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/17517
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>

show more ...

blob: Add translate_lba operation

New `translate_lba` operation allows to translate blob lba to lba on
the underlying bdev. It recurses down the whole chain of bs_dev's. The
operation may fail to do

blob: Add translate_lba operation

New `translate_lba` operation allows to translate blob lba to lba on
the underlying bdev. It recurses down the whole chain of bs_dev's. The
operation may fail to do the translation when blob lba is not backed
by the real bdev. For example, when we eventually hit zeroes device in
the chain.

This operation is used in the next commit to get source LBA for copy
operation.

Signed-off-by: Evgeniy Kochetov <evgeniik@nvidia.com>
Change-Id: I89c2d03d1982d66b9137a3a3653a98c361984fab
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/14528
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Shuhei Matsumoto <smatsumoto@nvidia.com>
Reviewed-by: Aleksey Marchuk <alexeymar@nvidia.com>

show more ...

update Intel copyright notices

per Intel policy to include file commit date using git cmd
below. The policy does not apply to non-Intel (C) notices.

git log --follow -C90% --format=%ad --date defa

update Intel copyright notices

per Intel policy to include file commit date using git cmd
below. The policy does not apply to non-Intel (C) notices.

git log --follow -C90% --format=%ad --date default <file> | tail -1

and then pull just the 4 digit year from the result.

Intel copyrights were not added to files where Intel either had
no contribution ot the contribution lacked substance (ie license
header updates, formatting changes, etc). Contribution date used
"--follow -C95%" to get the most accurate date.

Note that several files in this patch didn't end the license/(c)
block with a blank comment line so these were added as the vast
majority of files do have this last blank line. Simply there for
consistency.

Signed-off-by: paul luse <paul.e.luse@intel.com>
Change-Id: Id5b7ce4f658fe87132f14139ead58d6e285c04d4
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/15192
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Community-CI: Mellanox Build Bot

show more ...

blob: Optimize copy-on-write flow for clusters backed by zeroes device

Writing to unallocated cluster triggers copy-on-write sequence. If
this cluster is backed by zeroes device we can skip the copy

blob: Optimize copy-on-write flow for clusters backed by zeroes device

Writing to unallocated cluster triggers copy-on-write sequence. If
this cluster is backed by zeroes device we can skip the copy part. For
a simple thin provisioned volume copy this shortcut is already
implemented because `blob->parent_id == SPDK_BLOBID_INVALID`. But this
will not work for thin provisioned volumes created from snapshot. In
this case we need to traverse the whole stack of underlying
`spdk_bs_dev` devices for specific cluster to check if it is zeroes
backed.

This patch adds `is_zeroes` operation to `spdk_bs_dev`. For zeroes
device it always returns 'true', for real bdev (`blob_bs_dev`) always
returns false, for another layer of `blob_bs_dev` does lba conversion
and forwards to backing device.

In blobstore's cluster copy flow we check if cluster is backed by
zeroes device and skip copy part if it is.

Signed-off-by: Evgeniy Kochetov <evgeniik@nvidia.com>
Change-Id: I640773ac78f8f466b96e96a34c3a6c3c91f87dab
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/13446
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Community-CI: Mellanox Build Bot
Community-CI: Broadcom CI <spdk-ci.pdl@broadcom.com>
Reviewed-by: Shuhei Matsumoto <smatsumoto@nvidia.com>
Reviewed-by: Aleksey Marchuk <alexeymar@nvidia.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>

show more ...

Replace most BSD 3-clause license text with SPDX identifier.

Many open source projects have moved to using SPDX identifiers
to specify license information, reducing the amount of
boilerplate code in

Replace most BSD 3-clause license text with SPDX identifier.

Many open source projects have moved to using SPDX identifiers
to specify license information, reducing the amount of
boilerplate code in every source file. This patch replaces
the bulk of SPDK .c, .cpp and Makefiles with the BSD-3-Clause
identifier.

Almost all of these files share the exact same license text,
and this patch only modifies the files that contain the
most common license text. There can be slight variations
because the third clause contains company names - most say
"Intel Corporation", but there are instances for Nvidia,
Samsung, Eideticom and even "the copyright holder".

Used a bash script to automate replacement of the license text
with SPDX identifier which is checked into scripts/spdx.sh.

Signed-off-by: Jim Harris <james.r.harris@intel.com>
Change-Id: Iaa88ab5e92ea471691dc298cfe41ebfb5d169780
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/12904
Community-CI: Broadcom CI <spdk-ci.pdl@broadcom.com>
Community-CI: Mellanox Build Bot
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Aleksey Marchuk <alexeymar@nvidia.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: Dong Yi <dongx.yi@intel.com>
Reviewed-by: Konrad Sztyber <konrad.sztyber@intel.com>
Reviewed-by: Paul Luse <paul.e.luse@intel.com>
Reviewed-by: <qun.wan@intel.com>

show more ...

blob: Add readv/writev_ext functions

These function accept optional spdk_blob_ext_io_opts
structure. If this structure is provided by the user
then readv/writev_ext ops of base dev will be used
in d

blob: Add readv/writev_ext functions

These function accept optional spdk_blob_ext_io_opts
structure. If this structure is provided by the user
then readv/writev_ext ops of base dev will be used
in data path

Signed-off-by: Alexey Marchuk <alexeymar@mellanox.com>
Change-Id: I370dd43f8c56f5752f7a52d0780bcfe3e3ae2d9e
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/11371
Community-CI: Broadcom CI <spdk-ci.pdl@broadcom.com>
Community-CI: Mellanox Build Bot
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Shuhei Matsumoto <smatsumoto@nvidia.com>

show more ...

blob: use uint64_t for unmap and write_zeroes lba count

Previous patches (5363eb3c) tried to work around the
32-bit unmap and write_zeroes LBA counts by breaking
up larger operations into smaller ch

blob: use uint64_t for unmap and write_zeroes lba count

Previous patches (5363eb3c) tried to work around the
32-bit unmap and write_zeroes LBA counts by breaking
up larger operations into smaller chunks of max size
UINT32_MAX lba chunks.

But some SSDs may just ignore unmap operations that
are not aligned to full physical block boundaries -
and a UINT32_MAX lba unmap on a 512B logical /
4KiB physical SSD would not be aligned. If the SSD
decided to ignore the unmap/deallocate (which it is
allowed to do according to NVMe spec), we could end
up with not unmapping *any* blocks. Probably SSDs
should always be trying hard to unmap as many
blocks as possible, but let's not try to depend on
that in blobstore.

So one option would be to break them into chunks
close to UINT32_MAX which are still aligned to
4KiB boundaries. But the better fix is to just
change the unmap and write_zeroes APIs to take
64-bit arguments, and then we can avoid the
chunking altogether.

Fixes issue #2190.

Signed-off-by: Jim Harris <james.r.harris@intel.com>
Change-Id: I23998e493a764d466927c3520c7a8c7f943000a6
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/9737
Community-CI: Broadcom CI <spdk-ci.pdl@broadcom.com>
Community-CI: Mellanox Build Bot
Reviewed-by: Xiaodong Liu <xiaodong.liu@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: Aleksey Marchuk <alexeymar@mellanox.com>
Reviewed-by: Dong Yi <dongx.yi@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>

show more ...

lib/blob: remove _spdk prefix from all functions.

Signed-off-by: Seth Howell <seth.howell@intel.com>
Change-Id: Idb33816e5b66266987845172c27c87667ac0a596
Reviewed-on: https://review.spdk.io/gerrit/c

lib/blob: remove _spdk prefix from all functions.

Signed-off-by: Seth Howell <seth.howell@intel.com>
Change-Id: Idb33816e5b66266987845172c27c87667ac0a596
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/2437
Community-CI: Mellanox Build Bot
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Aleksey Marchuk <alexeymar@mellanox.com>

show more ...

lib/blob: remove spdk_ from non-public APIs

We have an unofficial naming convention that the
spdk_ namespace is reserved for public API functions only.
This patch is attempting to bring the blob lib

lib/blob: remove spdk_ from non-public APIs

We have an unofficial naming convention that the
spdk_ namespace is reserved for public API functions only.
This patch is attempting to bring the blob library into compliance
with that naming convention.

Signed-off-by: Seth Howell <seth.howell@intel.com>
Change-Id: Ie298e41d1b741dae01744826c208378ee60f9d0a
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/1700
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Community-CI: Broadcom CI

show more ...

blobstore: allow I/O operations to use io unit size smaller than page size.

Signed-off-by: Piotr Pelplinski <piotr.pelplinski@intel.com>
Change-Id: I994b5d46faffd34430cb39e66225929c4cba90ba
Reviewed

blobstore: allow I/O operations to use io unit size smaller than page size.

Signed-off-by: Piotr Pelplinski <piotr.pelplinski@intel.com>
Change-Id: I994b5d46faffd34430cb39e66225929c4cba90ba
Reviewed-on: https://review.gerrithub.io/414935
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Chandler-Test-Pool: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>

show more ...

blobstore: Add a blob_bs_dev that provides back_bs_dev for clones

Unit tests implemented in following patches.

This is rebased patch from https://review.gerrithub.io/#/c/396648
merged as commit c11

blobstore: Add a blob_bs_dev that provides back_bs_dev for clones

Unit tests implemented in following patches.

This is rebased patch from https://review.gerrithub.io/#/c/396648
merged as commit c1174e6895ca44242d12b2e5f0742f84ccbf0b8f
and reverted in 0847f27b54c82c29a67f779f6a464454015d448b.

Change-Id: I3d152bf7847c83bf75149edd61564c1f393927d8
Signed-off-by: Piotr Pelplinski <piotr.pelplinski@intel.com>
Reviewed-on: https://review.gerrithub.io/402529
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Daniel Verkamp <daniel.verkamp@intel.com>

show more ...

blobstore: Add a blob_bs_dev that provides back_bs_dev for clones

Unit tests implemented in following patches.

Signed-off-by: Piotr Pelplinski <piotr.pelplinski@intel.com>
Change-Id: Ib18c9060f527b

blobstore: Add a blob_bs_dev that provides back_bs_dev for clones

Unit tests implemented in following patches.

Signed-off-by: Piotr Pelplinski <piotr.pelplinski@intel.com>
Change-Id: Ib18c9060f527bd22bfdbed74e96871a6e0551ead
Reviewed-on: https://review.gerrithub.io/396648
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Maciej Szwed <maciej.szwed@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Daniel Verkamp <daniel.verkamp@intel.com>

show more ...