blobstore: Hold the original buffer to free it when realloc() is failed

Change-Id: I1f2c62b2607fb6efb82c9de59244ba3e32fa67ca
Signed-off-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Reviewe

blobstore: Hold the original buffer to free it when realloc() is failed

Change-Id: I1f2c62b2607fb6efb82c9de59244ba3e32fa67ca
Signed-off-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Reviewed-on: https://review.gerrithub.io/412753
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Daniel Verkamp <daniel.verkamp@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>

show more ...

blobstore: Use concrete error codes and add callback when failed.

Change-Id: I5670ed4d5f03c3d2e5719829f1991d2d2ed38750
Signed-off-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Reviewed-on:

blobstore: Use concrete error codes and add callback when failed.

Change-Id: I5670ed4d5f03c3d2e5719829f1991d2d2ed38750
Signed-off-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Reviewed-on: https://review.gerrithub.io/412726
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: Daniel Verkamp <daniel.verkamp@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>

show more ...

blob: remove deprecated bs_io_*_blob functions

These were marked as deprecated in v18.04 and scheduled for removal in
v18.07.

Change-Id: I2587bcaf89cdcc757ad902ac42ccd9adce9c8f92
Signed-off-by: Dan

blob: remove deprecated bs_io_*_blob functions

These were marked as deprecated in v18.04 and scheduled for removal in
v18.07.

Change-Id: I2587bcaf89cdcc757ad902ac42ccd9adce9c8f92
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
Reviewed-on: https://review.gerrithub.io/410727
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>

show more ...

blobstore: add inflate call

Inflate call can be used on thin provisioned blob or clones.

Function allocates all unallocated clusters on specified blob and:
- For clones, copies data from backing b

blobstore: add inflate call

Inflate call can be used on thin provisioned blob or clones.

Function allocates all unallocated clusters on specified blob and:
- For clones, copies data from backing blob.
- For thin provisioned blobs, clusters are zeroed.

After this call all dependency from specified blob is removed
what allows deletion i.e. snapshots.

Signed-off-by: Piotr Pelplinski <piotr.pelplinski@intel.com>
Signed-off-by: Tomasz Kulasek <tomaszx.kulasek@intel.com>
Change-Id: Ibff569e45b12068b2fb46557156be348b36c252b
Reviewed-on: https://review.gerrithub.io/399367
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Daniel Verkamp <daniel.verkamp@intel.com>

show more ...

blob: don't try to claim cluster 0 in recovery code

Thin provisioned blobs mark unallocated clusters with
cluster ID 0. During recovery from a dirty shutdown,
we must not try to claim cluster 0 - w

blob: don't try to claim cluster 0 in recovery code

Thin provisioned blobs mark unallocated clusters with
cluster ID 0. During recovery from a dirty shutdown,
we must not try to claim cluster 0 - we should ignore
them instead.

Fixes issue #291.

Signed-off-by: Jim Harris <james.r.harris@intel.com>
Change-Id: If0dd42416f5de8d9972073bf6ed44eb8bc655415
Reviewed-on: https://review.gerrithub.io/410065
Reviewed-by: Daniel Verkamp <daniel.verkamp@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>

show more ...

blobstore: fix payload offset incrementation on operation split

On operation split, payload pointer should be incremented by the number
of bytes, not by op_length which indicates the number of pages

blobstore: fix payload offset incrementation on operation split

On operation split, payload pointer should be incremented by the number
of bytes, not by op_length which indicates the number of pages.

Change-Id: I5d40b6ff7f39b599fe8c8072ee7879848a6af848
Signed-off-by: Tomasz Kulasek <tomaszx.kulasek@intel.com>
Reviewed-on: https://review.gerrithub.io/409201
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Daniel Verkamp <daniel.verkamp@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>

show more ...

blobstore: do not produce error when requesting number of clones

One of intended uses of spdk_blob_get_clones() is to retrive
number of clones from a snapshot. This is done by passing
NULL pointer f

blobstore: do not produce error when requesting number of clones

One of intended uses of spdk_blob_get_clones() is to retrive
number of clones from a snapshot. This is done by passing
NULL pointer for destination array.

In this case SPDK_ERRLOG is superfluous, as ENOMEM should
be handled appropriately by called.
Example of correct usage producing this error log is in vbdev_lvol.c
vbdev_lvol_dump_info_json().

Change-Id: I032ca12af01caddf6f540e39d49c2adba40a6ff1
Signed-off-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Reviewed-on: https://review.gerrithub.io/409164
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Daniel Verkamp <daniel.verkamp@intel.com>

show more ...

blobstore: clone-snapshot blobstore relations

This commit provides an API to obtain an information about
snapshot and clone relations.

The main objective is:

1) Determinate if we can delete snaps

blobstore: clone-snapshot blobstore relations

This commit provides an API to obtain an information about
snapshot and clone relations.

The main objective is:

1) Determinate if we can delete snapshot (if have some created
clones),

2) Provide an information about parent/children nodes to the upper
layer (e.g. lvol)

Realization:

1) Structure parent-children is stored in the blob store object
and updated on:

a) blob store load,

b) blob create/delete,

2) Full information about parent-children is provided via new API:
spdk_blob_get_parent() and spdk_blob_get_children(),

Note:

While we don't store an information about these relations in the
blob store, we need to open all blobs on blob store load to create
it. It should be considered that it have an impact on the blobstore
loading performance.

Change-Id: Ie0237fa5b93af01aa73d1f68ac1694e653fb75e5
Signed-off-by: Tomasz Kulasek <tomaszx.kulasek@intel.com>
Reviewed-on: https://review.gerrithub.io/405025
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Maciej Szwed <maciej.szwed@intel.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Daniel Verkamp <daniel.verkamp@intel.com>

show more ...

blobstore: clone-snapshot classification

This patch introduces API to get some blobs capabilites:

bool spdk_blob_is_read_only(struct spdk_blob *blob);
bool spdk_blob_is_thin_provisioned(struct spdk

blobstore: clone-snapshot classification

This patch introduces API to get some blobs capabilites:

bool spdk_blob_is_read_only(struct spdk_blob *blob);
bool spdk_blob_is_thin_provisioned(struct spdk_blob *blob);

to be used in upper level in the unified way.

Change-Id: I4411bb3f4dd0c64826ae16a66141b2911cbaab79
Signed-off-by: Tomasz Kulasek <tomaszx.kulasek@intel.com>
Reviewed-on: https://review.gerrithub.io/405022
Reviewed-by: Daniel Verkamp <daniel.verkamp@intel.com>
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>

show more ...

blob: copy xattr name with memcpy()

We know exactly how long the name is, so there is no need to use a C
string function to copy it.

Change-Id: I21b5f1e318555b46729582ab6a1e6bd163c85205
Signed-off-

blob: copy xattr name with memcpy()

We know exactly how long the name is, so there is no need to use a C
string function to copy it.

Change-Id: I21b5f1e318555b46729582ab6a1e6bd163c85205
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
Reviewed-on: https://review.gerrithub.io/406984
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>

show more ...

blobstore: allow creating clones out of read-only snapshots

Signed-off-by: Piotr Pelplinski <piotr.pelplinski@intel.com>
Signed-off-by: Tomasz Kulasek <tomaszx.kulasek@intel.com>
Change-Id: Iad67be7

blobstore: allow creating clones out of read-only snapshots

Signed-off-by: Piotr Pelplinski <piotr.pelplinski@intel.com>
Signed-off-by: Tomasz Kulasek <tomaszx.kulasek@intel.com>
Change-Id: Iad67be79d0ddd8c498950c4f7b1b3203e47a7a41
Reviewed-on: https://review.gerrithub.io/393936
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Maciej Szwed <maciej.szwed@intel.com>
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>

show more ...

blobstore: add snapshot functionality

This patch adds new feature of blobstore.
New call creates a read-only snapshot of specified blob with provided options.

NOTE:

This patch doesn't cover recove

blobstore: add snapshot functionality

This patch adds new feature of blobstore.
New call creates a read-only snapshot of specified blob with provided options.

NOTE:

This patch doesn't cover recovery operation if snapshotting fails. This operation
will be implemented and added later.

Signed-off-by: Piotr Pelplinski <piotr.pelplinski@intel.com>
Signed-off-by: Tomasz Kulasek <tomaszx.kulasek@intel.com>
Change-Id: I470ca13525638fa6df485d508b3adf71b6b69c0b
Reviewed-on: https://review.gerrithub.io/393935
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Maciej Szwed <maciej.szwed@intel.com>
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>

show more ...

blob: make spdk_blob_resize an async operation

To support resize operations during I/O, we will need
to send messages to each thread to quiesce I/O while
the resize operation is in progress to guard

blob: make spdk_blob_resize an async operation

To support resize operations during I/O, we will need
to send messages to each thread to quiesce I/O while
the resize operation is in progress to guard against
the cluster map memory changing while another thread
is accessing the cluster map.

Therefore, spdk_blob_resize needs to be asynchronous.

Signed-off-by: Jim Harris <james.r.harris@intel.com>
Change-Id: Ida037334739b4b80a1dbc76e8f1c70bca8b73582

Reviewed-on: https://review.gerrithub.io/404616
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Daniel Verkamp <daniel.verkamp@intel.com>

show more ...

blobstore: destroy bs_dev on spdk_bs_load fail

Some error paths before _spdk_bs_alloc did not
destroy bs_dev.

After succesfull _spdk_bs_alloc, destroying is done
in _spdk_bs_free.

Change-Id: Ib69a

blobstore: destroy bs_dev on spdk_bs_load fail

Some error paths before _spdk_bs_alloc did not
destroy bs_dev.

After succesfull _spdk_bs_alloc, destroying is done
in _spdk_bs_free.

Change-Id: Ib69ae9707e12a646af80f7892af49cc4f79c199e
Signed-off-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Reviewed-on: https://review.gerrithub.io/405223
Reviewed-by: Daniel Verkamp <daniel.verkamp@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>

show more ...

blob: disallow bs load on device with unsupported block length

Blobstore supports only block lengths that are less than
SPDK_BS_PAGE_SIZE or when multiplication of block len
results in SPDK_BS_PAGE_

blob: disallow bs load on device with unsupported block length

Blobstore supports only block lengths that are less than
SPDK_BS_PAGE_SIZE or when multiplication of block len
results in SPDK_BS_PAGE_SIZE.

This was checked only on spdk_bs_init(), but not spdk_bs_load().

When not checked, it caused issues with lvol store tasting.
During tasting, there is an attempt to perform spdk_bs_load()
on a given device.
It was possible to hit asserts in blobstore by creating
malloc with block size 8192.

Change-Id: I30b62bebad405b581eb2158925884adc616d9b92
Signed-off-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Reviewed-on: https://review.gerrithub.io/404537
Reviewed-by: Daniel Verkamp <daniel.verkamp@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>

show more ...

blob: use internal _spdk_blob_resize when creating blob

Upcoming patches will make spdk_blob_resize() asynchronous
to allow for resizing while I/O is in progress. During
blob creation, it is not po

blob: use internal _spdk_blob_resize when creating blob

Upcoming patches will make spdk_blob_resize() asynchronous
to allow for resizing while I/O is in progress. During
blob creation, it is not possible for I/O to be in progress,
so just use the internal _spdk_blob_resize which will
remain synchronous and called only after I/O is frozen.

Signed-off-by: Jim Harris <james.r.harris@intel.com>
Change-Id: I02128612a5f1bcd8bb236da113b05e88aa52c310

Reviewed-on: https://review.gerrithub.io/404613
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Reviewed-by: Daniel Verkamp <daniel.verkamp@intel.com>
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>

show more ...

blob: rename _spdk_resize_blob to _spdk_blob_resize

This internal function then will match noun/verb order
of the public spdk_blob_resize function.

Signed-off-by: Jim Harris <james.r.harris@intel.c

blob: rename _spdk_resize_blob to _spdk_blob_resize

This internal function then will match noun/verb order
of the public spdk_blob_resize function.

Signed-off-by: Jim Harris <james.r.harris@intel.com>
Change-Id: I0d49af6502438b3571eebe5c87abe96a8fb77658

Reviewed-on: https://review.gerrithub.io/404612
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Reviewed-by: Daniel Verkamp <daniel.verkamp@intel.com>

show more ...

blobstore: process split operation sequentially

Previously when operation for blobstore was spanning multiple
clusters, it was split into multiple operation processed in a batch.

This made it possi

blobstore: process split operation sequentially

Previously when operation for blobstore was spanning multiple
clusters, it was split into multiple operation processed in a batch.

This made it possible to max out channel ops when using large enough
operation. It was encountered when issuing unmap for whole device.

Now single large operation is processed sequentially, making
it take at most one ops from channel.

Fixes #251
Fixes #250

Change-Id: I132309877ba3b2d7217332daf1025fb5f9ee74d0
Signed-off-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Reviewed-on: https://review.gerrithub.io/403306
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Daniel Verkamp <daniel.verkamp@intel.com>

show more ...

Revert "blob: queue sync requests if one already in progress"

BlobFS shutdown path needs to be investigated more with these
changes.

This reverts commit a137b9afd0bc4e81df38a3a0847612269d038289.

S

Revert "blob: queue sync requests if one already in progress"

BlobFS shutdown path needs to be investigated more with these
changes.

This reverts commit a137b9afd0bc4e81df38a3a0847612269d038289.

Signed-off-by: Jim Harris <james.r.harris@intel.com>
Change-Id: I8b04b24e178945d62db20668b9e500f278ae955b
Reviewed-on: https://review.gerrithub.io/403600
Reviewed-by: Daniel Verkamp <daniel.verkamp@intel.com>
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>

show more ...

blob: queue sync requests if one already in progress

For any given blob, if an spdk_blob_sync_md() operation
is already in progress, queue additional spdk_blob_sync_md()
operations until the previou

blob: queue sync requests if one already in progress

For any given blob, if an spdk_blob_sync_md() operation
is already in progress, queue additional spdk_blob_sync_md()
operations until the previous one completes.

This ensures proper ordering of writing metadata to
disk.

Signed-off-by: Jim Harris <james.r.harris@intel.com>
Change-Id: I2051e8cb5b8d1a033ec1238cb4811232110aa0f4

Reviewed-on: https://review.gerrithub.io/401257
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Daniel Verkamp <daniel.verkamp@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>

show more ...

blobstore: add internal xattrs for creating blob

This patch adds possibility to set internal xattrs on blob

Signed-off-by: Piotr Pelplinski <piotr.pelplinski@intel.com>
Change-Id: I2d0f14558e4a1af7

blobstore: add internal xattrs for creating blob

This patch adds possibility to set internal xattrs on blob

Signed-off-by: Piotr Pelplinski <piotr.pelplinski@intel.com>
Change-Id: I2d0f14558e4a1af7071ee1a4f59aaf9f14bea2d9
Reviewed-on: https://review.gerrithub.io/396418
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Daniel Verkamp <daniel.verkamp@intel.com>

show more ...

blobstore: Add a blob_bs_dev that provides back_bs_dev for clones

Unit tests implemented in following patches.

This is rebased patch from https://review.gerrithub.io/#/c/396648
merged as commit c11

blobstore: Add a blob_bs_dev that provides back_bs_dev for clones

Unit tests implemented in following patches.

This is rebased patch from https://review.gerrithub.io/#/c/396648
merged as commit c1174e6895ca44242d12b2e5f0742f84ccbf0b8f
and reverted in 0847f27b54c82c29a67f779f6a464454015d448b.

Change-Id: I3d152bf7847c83bf75149edd61564c1f393927d8
Signed-off-by: Piotr Pelplinski <piotr.pelplinski@intel.com>
Reviewed-on: https://review.gerrithub.io/402529
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Daniel Verkamp <daniel.verkamp@intel.com>

show more ...

blobstore: allow internaly get xattr value even in loading state

For some xattrs we need to know its value even if blob is in loading
state, e.g. BLOB_SNAPSHOT xattr value when blob is loading.

Cha

blobstore: allow internaly get xattr value even in loading state

For some xattrs we need to know its value even if blob is in loading
state, e.g. BLOB_SNAPSHOT xattr value when blob is loading.

Change-Id: I1cd7805cf33be64cf59792f85a270e9b536e23bd
Signed-off-by: Tomasz Kulasek <tomaszx.kulasek@intel.com>
Reviewed-on: https://review.gerrithub.io/403062
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Daniel Verkamp <daniel.verkamp@intel.com>

show more ...

blob: persist super_blob id on blobstore immediately

Before this patch super_blob id for blobstore was persisted
only during spdk_bs_unload. If power fail occurred after creating and
syncing blob, s

blob: persist super_blob id on blobstore immediately

Before this patch super_blob id for blobstore was persisted
only during spdk_bs_unload. If power fail occurred after creating and
syncing blob, super_blob id was lost within blobstore.

Lvol store metadata would be lost, if proper shutdown
didn't occur in first SPDK instance run since creation of lvs.

This fix changes setting super blob to be instantly persisted
on disk in super block. Without affecting clean bit in super block.

Change-Id: I578f1fc8717e2d7968ad506fa4dead7507a5e0b4
Signed-off-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Reviewed-on: https://review.gerrithub.io/398804
Reviewed-by: Maciej Szwed <maciej.szwed@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Daniel Verkamp <daniel.verkamp@intel.com>
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>

show more ...

scripts/check_format: check for spaces before tabs

Automatically detect more whitespace errors.

All existing cases are fixed; only whitespace change (verify with
diff -w) except for one comment sty

scripts/check_format: check for spaces before tabs

Automatically detect more whitespace errors.

All existing cases are fixed; only whitespace change (verify with
diff -w) except for one comment style fixup in include/spdk/nvme.h.

Change-Id: If750e54b9c8e3421ea6feda5f20184a31431631e
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
Reviewed-on: https://review.gerrithub.io/402360
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>

show more ...