lib/blob: fix data inconsistency when unmap a thin-provisioned blob that is backed

The issue may happen in this case:
1.create lvol A from snapshot B
2.write data to A_cluster1
3.unmap A_cluster1

lib/blob: fix data inconsistency when unmap a thin-provisioned blob that is backed

The issue may happen in this case:
1.create lvol A from snapshot B
2.write data to A_cluster1
3.unmap A_cluster1
4.read data from lba coverd by A_cluster1
the result of step 4 should be zero, but it’s actually the data in B_cluster1

Change-Id: If08bdf392d8d3dee737fcacca03bd9f04faa3933
Signed-off-by: xupeng-mingtu <jingmamour@gmail.com>
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/22325
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <jim.harris@samsung.com>
Reviewed-by: Tomasz Zawadzki <tomasz@tzawadzki.com>
Community-CI: Mellanox Build Bot
Reviewed-by: Mateusz Kozlowski <mateusz.kozlowski@solidigm.com>

show more ...

blob: handle IO to a resized blob

When a blob is resized and is backed by a another blob like
a snapshot, then the resized blob has more number of clusters
compared to backing device. Since the blob

blob: handle IO to a resized blob

When a blob is resized and is backed by a another blob like
a snapshot, then the resized blob has more number of clusters
compared to backing device. Since the blob is thin, either
originally or as a result of creating snapshot, the IOs to
the new cluster ranges fail as there is no cluster to be looked
at in the backing. This patch introduces mechanism to handle reads
by zeroeing out the payload which is trailing beyond cluster range,
and handle writes by validating backing dev cluster range during
copy-on-write.

Change-Id: Ib204b922819775396e53ffb718e8230ebf8fa46b
Signed-off-by: Diwakar Sharma <diwakar.sharma@datacore.com>
Co-authored-by: Diwakar Sharma <diwakar.sharma@datacore.com>
Co-authored-by: Jim Harris <james.r.harris@intel.com>
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/22377
Community-CI: Mellanox Build Bot
Reviewed-by: Shuhei Matsumoto <smatsumoto@nvidia.com>
Reviewed-by: Jim Harris <jim.harris@samsung.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>

show more ...

blob: Add translate_lba operation

New `translate_lba` operation allows to translate blob lba to lba on
the underlying bdev. It recurses down the whole chain of bs_dev's. The
operation may fail to do

blob: Add translate_lba operation

New `translate_lba` operation allows to translate blob lba to lba on
the underlying bdev. It recurses down the whole chain of bs_dev's. The
operation may fail to do the translation when blob lba is not backed
by the real bdev. For example, when we eventually hit zeroes device in
the chain.

This operation is used in the next commit to get source LBA for copy
operation.

Signed-off-by: Evgeniy Kochetov <evgeniik@nvidia.com>
Change-Id: I89c2d03d1982d66b9137a3a3653a98c361984fab
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/14528
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Shuhei Matsumoto <smatsumoto@nvidia.com>
Reviewed-by: Aleksey Marchuk <alexeymar@nvidia.com>

show more ...

update Intel copyright notices

per Intel policy to include file commit date using git cmd
below. The policy does not apply to non-Intel (C) notices.

git log --follow -C90% --format=%ad --date defa

update Intel copyright notices

per Intel policy to include file commit date using git cmd
below. The policy does not apply to non-Intel (C) notices.

git log --follow -C90% --format=%ad --date default <file> | tail -1

and then pull just the 4 digit year from the result.

Intel copyrights were not added to files where Intel either had
no contribution ot the contribution lacked substance (ie license
header updates, formatting changes, etc). Contribution date used
"--follow -C95%" to get the most accurate date.

Note that several files in this patch didn't end the license/(c)
block with a blank comment line so these were added as the vast
majority of files do have this last blank line. Simply there for
consistency.

Signed-off-by: paul luse <paul.e.luse@intel.com>
Change-Id: Id5b7ce4f658fe87132f14139ead58d6e285c04d4
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/15192
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Community-CI: Mellanox Build Bot

show more ...

blob: Optimize copy-on-write flow for clusters backed by zeroes device

Writing to unallocated cluster triggers copy-on-write sequence. If
this cluster is backed by zeroes device we can skip the copy

blob: Optimize copy-on-write flow for clusters backed by zeroes device

Writing to unallocated cluster triggers copy-on-write sequence. If
this cluster is backed by zeroes device we can skip the copy part. For
a simple thin provisioned volume copy this shortcut is already
implemented because `blob->parent_id == SPDK_BLOBID_INVALID`. But this
will not work for thin provisioned volumes created from snapshot. In
this case we need to traverse the whole stack of underlying
`spdk_bs_dev` devices for specific cluster to check if it is zeroes
backed.

This patch adds `is_zeroes` operation to `spdk_bs_dev`. For zeroes
device it always returns 'true', for real bdev (`blob_bs_dev`) always
returns false, for another layer of `blob_bs_dev` does lba conversion
and forwards to backing device.

In blobstore's cluster copy flow we check if cluster is backed by
zeroes device and skip copy part if it is.

Signed-off-by: Evgeniy Kochetov <evgeniik@nvidia.com>
Change-Id: I640773ac78f8f466b96e96a34c3a6c3c91f87dab
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/13446
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Community-CI: Mellanox Build Bot
Community-CI: Broadcom CI <spdk-ci.pdl@broadcom.com>
Reviewed-by: Shuhei Matsumoto <smatsumoto@nvidia.com>
Reviewed-by: Aleksey Marchuk <alexeymar@nvidia.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>

show more ...

Replace most BSD 3-clause license text with SPDX identifier.

Many open source projects have moved to using SPDX identifiers
to specify license information, reducing the amount of
boilerplate code in

Replace most BSD 3-clause license text with SPDX identifier.

Many open source projects have moved to using SPDX identifiers
to specify license information, reducing the amount of
boilerplate code in every source file. This patch replaces
the bulk of SPDK .c, .cpp and Makefiles with the BSD-3-Clause
identifier.

Almost all of these files share the exact same license text,
and this patch only modifies the files that contain the
most common license text. There can be slight variations
because the third clause contains company names - most say
"Intel Corporation", but there are instances for Nvidia,
Samsung, Eideticom and even "the copyright holder".

Used a bash script to automate replacement of the license text
with SPDX identifier which is checked into scripts/spdx.sh.

Signed-off-by: Jim Harris <james.r.harris@intel.com>
Change-Id: Iaa88ab5e92ea471691dc298cfe41ebfb5d169780
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/12904
Community-CI: Broadcom CI <spdk-ci.pdl@broadcom.com>
Community-CI: Mellanox Build Bot
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Aleksey Marchuk <alexeymar@nvidia.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: Dong Yi <dongx.yi@intel.com>
Reviewed-by: Konrad Sztyber <konrad.sztyber@intel.com>
Reviewed-by: Paul Luse <paul.e.luse@intel.com>
Reviewed-by: <qun.wan@intel.com>

show more ...

blob: Add readv/writev ext ops to spdk_bs_dev

Introduce spdk_blob_ext_io_opts structure which
is used in the new *_ext functions.
Zeroes dev is updated with implementation of
readv_ext which uses m

blob: Add readv/writev ext ops to spdk_bs_dev

Introduce spdk_blob_ext_io_opts structure which
is used in the new *_ext functions.
Zeroes dev is updated with implementation of
readv_ext which uses memory domains memzero
or regular memset().

Signed-off-by: Alexey Marchuk <alexeymar@mellanox.com>
Change-Id: Id94542196eff999827bf00591fd43804256fccb4
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/11369
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Community-CI: Mellanox Build Bot
Community-CI: Broadcom CI <spdk-ci.pdl@broadcom.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Shuhei Matsumoto <smatsumoto@nvidia.com>

show more ...

blob: use uint64_t for unmap and write_zeroes lba count

Previous patches (5363eb3c) tried to work around the
32-bit unmap and write_zeroes LBA counts by breaking
up larger operations into smaller ch

blob: use uint64_t for unmap and write_zeroes lba count

Previous patches (5363eb3c) tried to work around the
32-bit unmap and write_zeroes LBA counts by breaking
up larger operations into smaller chunks of max size
UINT32_MAX lba chunks.

But some SSDs may just ignore unmap operations that
are not aligned to full physical block boundaries -
and a UINT32_MAX lba unmap on a 512B logical /
4KiB physical SSD would not be aligned. If the SSD
decided to ignore the unmap/deallocate (which it is
allowed to do according to NVMe spec), we could end
up with not unmapping *any* blocks. Probably SSDs
should always be trying hard to unmap as many
blocks as possible, but let's not try to depend on
that in blobstore.

So one option would be to break them into chunks
close to UINT32_MAX which are still aligned to
4KiB boundaries. But the better fix is to just
change the unmap and write_zeroes APIs to take
64-bit arguments, and then we can avoid the
chunking altogether.

Fixes issue #2190.

Signed-off-by: Jim Harris <james.r.harris@intel.com>
Change-Id: I23998e493a764d466927c3520c7a8c7f943000a6
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/9737
Community-CI: Broadcom CI <spdk-ci.pdl@broadcom.com>
Community-CI: Mellanox Build Bot
Reviewed-by: Xiaodong Liu <xiaodong.liu@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: Aleksey Marchuk <alexeymar@mellanox.com>
Reviewed-by: Dong Yi <dongx.yi@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>

show more ...

lib/blob: remove spdk_ from non-public APIs

We have an unofficial naming convention that the
spdk_ namespace is reserved for public API functions only.
This patch is attempting to bring the blob lib

lib/blob: remove spdk_ from non-public APIs

We have an unofficial naming convention that the
spdk_ namespace is reserved for public API functions only.
This patch is attempting to bring the blob library into compliance
with that naming convention.

Signed-off-by: Seth Howell <seth.howell@intel.com>
Change-Id: Ie298e41d1b741dae01744826c208378ee60f9d0a
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/1700
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Community-CI: Broadcom CI

show more ...

blob: return error for write_zeroes and unmap requests

Actually write/writev/write_zeroes/unmap are never be called, and we add
the error code here to keep it same style with snapshot bs_bdev.

Chan

blob: return error for write_zeroes and unmap requests

Actually write/writev/write_zeroes/unmap are never be called, and we add
the error code here to keep it same style with snapshot bs_bdev.

Change-Id: I32ad051c1902bd7080b894e36f7c89f1c8d27434
Signed-off-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/456924
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>

show more ...

blobstore: allow I/O operations to use io unit size smaller than page size.

Signed-off-by: Piotr Pelplinski <piotr.pelplinski@intel.com>
Change-Id: I994b5d46faffd34430cb39e66225929c4cba90ba
Reviewed

blobstore: allow I/O operations to use io unit size smaller than page size.

Signed-off-by: Piotr Pelplinski <piotr.pelplinski@intel.com>
Change-Id: I994b5d46faffd34430cb39e66225929c4cba90ba
Reviewed-on: https://review.gerrithub.io/414935
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Chandler-Test-Pool: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>

show more ...

blob: Add a bs_dev that always returns 0

This will be useful for backing thin provisioned
blobs in the future.

Change-Id: I78cf8cda39e8dff42da69b79ed460797d7494af1
Signed-off-by: Ben Walker <benjam

blob: Add a bs_dev that always returns 0

This will be useful for backing thin provisioned
blobs in the future.

Change-Id: I78cf8cda39e8dff42da69b79ed460797d7494af1
Signed-off-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-on: https://review.gerrithub.io/397043
Reviewed-by: Daniel Verkamp <daniel.verkamp@intel.com>
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>

show more ...