blob: handle FLAGS descriptor during dirty shutdown recovery

Found during unit testing for blobid_mask coming in a future
patch - the unit test will be added as part of that future
patch.

Signed-of

blob: handle FLAGS descriptor during dirty shutdown recovery

Found during unit testing for blobid_mask coming in a future
patch - the unit test will be added as part of that future
patch.

Signed-off-by: Jim Harris <james.r.harris@intel.com>
Change-Id: Iecdde6ba16c5af9caf59214f328ddc22aae71e94

Reviewed-on: https://review.gerrithub.io/391692
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Daniel Verkamp <daniel.verkamp@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>

show more ...

blob: change spdk_bs_iter_next parameter to spdk_blob *

Similar to previous change, the ** paradigm is a bit
problematic for asynchronous routines that could fail.

Signed-off-by: Jim Harris <james.

blob: change spdk_bs_iter_next parameter to spdk_blob *

Similar to previous change, the ** paradigm is a bit
problematic for asynchronous routines that could fail.

Signed-off-by: Jim Harris <james.r.harris@intel.com>
Change-Id: Ife7748280482356c4c51a796817b71cd7bc7e479

Reviewed-on: https://review.gerrithub.io/391483
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Daniel Verkamp <daniel.verkamp@intel.com>

show more ...

blob: change spdk_blob_close parameter to spdk_blob *

Using the ** paradigm is a bit problematic for asynchronous
routines that could fail. Currently we were inconsistent in
that some error paths w

blob: change spdk_blob_close parameter to spdk_blob *

Using the ** paradigm is a bit problematic for asynchronous
routines that could fail. Currently we were inconsistent in
that some error paths would zero the pointer while others
did not. So make this just a plain pointer, which simplifies
the API and its implementation.

Signed-off-by: Jim Harris <james.r.harris@intel.com>
Change-Id: I67147931c6e8350896a4505022a6a314655de3d3

Reviewed-on: https://review.gerrithub.io/391482
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Daniel Verkamp <daniel.verkamp@intel.com>

show more ...

blob: always rely on spdk_bs_open_blob in _spdk_bs_iter_cpl

We do not need to separately call _spdk_blob_lookup() in
_spdk_bs_iter_cpl() - spdk_bs_open_blob() does this
already. This minimizes the

blob: always rely on spdk_bs_open_blob in _spdk_bs_iter_cpl

We do not need to separately call _spdk_blob_lookup() in
_spdk_bs_iter_cpl() - spdk_bs_open_blob() does this
already. This minimizes the number of entry points to
_spdk_blob_lookup() which will be important with some
upcoming changes around multiple threads performing
metadata operations.

Signed-off-by: Jim Harris <james.r.harris@intel.com>
Change-Id: I032cd55c862267298cbe1674dd13d7a83eef7c0f

Reviewed-on: https://review.gerrithub.io/391474
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Daniel Verkamp <daniel.verkamp@intel.com>

show more ...

blob: remove md_channel

Remove the metadata channel, and instead use the same
channel for metadata and data operations on the metadata
thread.

This prepares for future changes which will allow
for

blob: remove md_channel

Remove the metadata channel, and instead use the same
channel for metadata and data operations on the metadata
thread.

This prepares for future changes which will allow
for metadata operations on any thread - not just the
thread where spdk_bs_load() or spdk_bs_init() was
called.

Signed-off-by: Jim Harris <james.r.harris@intel.com>
Change-Id: I6b11a58fcb237a9a7603841d118b3729d83c6c98

Reviewed-on: https://review.gerrithub.io/391311
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Daniel Verkamp <daniel.verkamp@intel.com>
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>

show more ...

blob: rename bs_md functions

As part of clarifying the API and preparing for some
future changes, rename the following functions:

spdk_bs_md_create_blob => spdk_bs_create_blob
spdk_bs_md_open_blob

blob: rename bs_md functions

As part of clarifying the API and preparing for some
future changes, rename the following functions:

spdk_bs_md_create_blob => spdk_bs_create_blob
spdk_bs_md_open_blob => spdk_bs_open_blob
spdk_bs_md_delete_blob => spdk_bs_delete_blob
spdk_bs_md_iter_first => spdk_bs_iter_first
spdk_bs_md_iter_next => spdk_bs_iter_next

Signed-off-by: Jim Harris <james.r.harris@intel.com>
Change-Id: I86bf792717b68379484a6108396bb891fe1c221e

Reviewed-on: https://review.gerrithub.io/391031
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Daniel Verkamp <daniel.verkamp@intel.com>
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>

show more ...

blob: remove spdk_bs_io_flush_channel

This function was a nop and is not needed.

lvol was calling this function when an lvol bdev
gets a FLUSH I/O, but that is not needed either. So
lvol will now

blob: remove spdk_bs_io_flush_channel

This function was a nop and is not needed.

lvol was calling this function when an lvol bdev
gets a FLUSH I/O, but that is not needed either. So
lvol will now report it does not support flush.

Signed-off-by: Jim Harris <james.r.harris@intel.com>
Change-Id: I92df83243f7ebce81c69040a8874891dc2ffc961

Reviewed-on: https://review.gerrithub.io/391023
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Daniel Verkamp <daniel.verkamp@intel.com>
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>

show more ...

blob: remove spdk_bs_[un]register_md_thread from public API

The implementations in blobstore.c still remain for now, but
those will be removed after some upcoming changes which will
eliminate a glob

blob: remove spdk_bs_[un]register_md_thread from public API

The implementations in blobstore.c still remain for now, but
those will be removed after some upcoming changes which will
eliminate a global md thread and instead allow caller to
specify an I/O channel for each blobstore level API call.

Signed-off-by: Jim Harris <james.r.harris@intel.com>
Change-Id: Idf800a7f061ffc9c42488951262e28e660871356

Reviewed-on: https://review.gerrithub.io/391020
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Daniel Verkamp <daniel.verkamp@intel.com>

show more ...

blob: rename spdk blob metadata functions

spdk_bs_md_resize_blob => spdk_blob_resize
spdk_bs_md_sync_blob => spdk_blob_sync_md
spdk_bs_md_get_xattr_value => spdk_blob_get_xattr_value
spdK_bs_md_get_

blob: rename spdk blob metadata functions

spdk_bs_md_resize_blob => spdk_blob_resize
spdk_bs_md_sync_blob => spdk_blob_sync_md
spdk_bs_md_get_xattr_value => spdk_blob_get_xattr_value
spdK_bs_md_get_xattr_names => spdk_blob_get_xattr_names
spdk_blob_md_set_xattr => spdk_blob_set_xattr
spdk_blob_md_remove_xattr => spdk_blob_remove_xattr
spdk_bs_md_close_blob => spdk_blob_close

Signed-off-by: Jim Harris <james.r.harris@intel.com>
Change-Id: I99b593c03bc61354ba6aa5a446b0fc82df1e9ce2

Reviewed-on: https://review.gerrithub.io/391006
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Daniel Verkamp <daniel.verkamp@intel.com>

show more ...

blob: decouple spdk_blob from its actual data

Moving forward, the spdk_blob returned to users will
actually be an I/O channel - not the blob structure
itself. So rename the existing spdk_blob to sp

blob: decouple spdk_blob from its actual data

Moving forward, the spdk_blob returned to users will
actually be an I/O channel - not the blob structure
itself. So rename the existing spdk_blob to spdk_blob_data.
spdk_blob_data will continue to contain global state for
the blob. In the future spdk_blob will point to an
I/O channel for the blob - for now it effectively still
points to the spdk_blob_data, but by changing the
structure names here it will reduce the code churn in
future patches.

Signed-off-by: Jim Harris <james.r.harris@intel.com>
Change-Id: I7d0cbc0553f68f96c24173c833091a80d058eb89

Reviewed-on: https://review.gerrithub.io/390900
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Daniel Verkamp <daniel.verkamp@intel.com>

show more ...

blob: move data_ro check into rw_iov function

This keeps the top level functions as simple pass-throughs
which will simplify some future commits.

Signed-off-by: Jim Harris <james.r.harris@intel.com

blob: move data_ro check into rw_iov function

This keeps the top level functions as simple pass-throughs
which will simplify some future commits.

Signed-off-by: Jim Harris <james.r.harris@intel.com>
Change-Id: I6578a1440d0404f600c055a5e37f28468b633d6f

Reviewed-on: https://review.gerrithub.io/390899
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Daniel Verkamp <daniel.verkamp@intel.com>

show more ...

log: rename SPDK_TRACE_* to SPDK_LOG_*

Disambiguate the log components from the trace functionality
(include/spdk/trace.h).

The internal spdk_trace_flag structure and related functions will be
rena

log: rename SPDK_TRACE_* to SPDK_LOG_*

Disambiguate the log components from the trace functionality
(include/spdk/trace.h).

The internal spdk_trace_flag structure and related functions will be
renamed in a later commit - this is just a find and replace on
SPDK_TRACE_* and SPDK_LOG_REGISTER_TRACE_FLAG().

Change-Id: I617bd5a9fbe35ffb44ae6020b292658c094a0ad6
Signed-off-by: Daniel Verkamp <daniel.verkamp@intel.com>
Reviewed-on: https://review.gerrithub.io/376421
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>

show more ...

blob: spdk_bs_destroy now only zeroes the super block

The function just needs to zero out metadata so that the
blobstore is effectively destroyed. If the user wants
to unmap the rest of the disk aft

blob: spdk_bs_destroy now only zeroes the super block

The function just needs to zero out metadata so that the
blobstore is effectively destroyed. If the user wants
to unmap the rest of the disk after the blobstore is
destroyed, they are free to do so. On future initializations
of blobstores the code will do the unmapping, so performance
is not impacted.

While here, implement the zeroing using the new
write_zeroes functionality instead of allocating a buffer
full of zeroes.

Change-Id: I7f18be0fd5e13a48b171ab3f4d5f5e12876023bc
Signed-off-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-on: https://review.gerrithub.io/390307
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Reviewed-by: Daniel Verkamp <daniel.verkamp@intel.com>

show more ...

blob: add per-blob feature flags

Similar flags will be added at the blobstore level in a future
patch.

This allows backwards compatibility - i.e. allow older blobstore
applications to open blobstor

blob: add per-blob feature flags

Similar flags will be added at the blobstore level in a future
patch.

This allows backwards compatibility - i.e. allow older blobstore
applications to open blobstores created by newer blobstore
applications with new features. Any blob's using a new feature
should have an associated flag set in one of three new flag masks:

- invalid: if a bit is set in this mask that the application is not
aware of, do not allow the blob to be opened
- data_ro: if a bit is set in this mask that the application is not
aware of, allow the blob to be opened, but do not allow
write I/O nor any operation that changes metadata
- md_ro: if a bit is set in this mask that the application is not
aware of, allow the blob to be opened for performing any
kind of I/O, but do not allow any operation that changes
metadata

While here, bump SPDK_BS_VERSION to 3. We intend this to be the
last change made to SPDK_BS_VERSION - future versioning will be
done via blobstore or per-blob feature flags instead.

Signed-off-by: Jim Harris <james.r.harris@intel.com>
Change-Id: If059e38bfffbeec25c849a7629a81193b12302c4

Reviewed-on: https://review.gerrithub.io/388703
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Daniel Verkamp <daniel.verkamp@intel.com>

show more ...

blob: fail _spdk_blob_parse_page on bad md descriptors

Currently there are a bunch of asserts() on metadata
descriptors - change these to fail the blob parsing
instead.

While here also return -ENOM

blob: fail _spdk_blob_parse_page on bad md descriptors

Currently there are a bunch of asserts() on metadata
descriptors - change these to fail the blob parsing
instead.

While here also return -ENOMEM if any of the memory
allocations fail.

Signed-off-by: Jim Harris <james.r.harris@intel.com>
Change-Id: Ie37b73c57b304d05a45d10a8d33bcc1d47e7a1be

Reviewed-on: https://review.gerrithub.io/388702
Reviewed-by: Maciej Szwed <maciej.szwed@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Daniel Verkamp <daniel.verkamp@intel.com>
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>

show more ...

blob: add data_ro and md_ro flags to spdk_blob

data_ro means that write, write_zeroes and unmap operations are not
allowed.

md_ro means that resize, set_xattr and remove_xattr are not
allowed.

The

blob: add data_ro and md_ro flags to spdk_blob

data_ro means that write, write_zeroes and unmap operations are not
allowed.

md_ro means that resize, set_xattr and remove_xattr are not
allowed.

There is no code yet that can activate this - it is coming in a future
patch. Two usages are planned though:

1) a user explicitly marks a blob as read-only - this is persisted so that
future loads of the blob will ensure the blob cannot be modified - neither
metadata nor data
2) a future feature flag framework (how's that for alliteration) may allow
a blob to be opened, but not allow metadata modifications, if there are
feature flags set in the blob's or blobstore's metadata that the
application does not understand

Signed-off-by: Jim Harris <james.r.harris@intel.com>
Change-Id: I247fd900430c56f7176edfb80dddd5a1a6c8dc87

Reviewed-on: https://review.gerrithub.io/388663
Reviewed-by: Daniel Verkamp <daniel.verkamp@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>

show more ...

blobstore: on init write zeroes to metadata and unmap rest

Currently on blobstore creation we use write zeros everytime.
Some drives does not support write zeros, but support unmap.
We should do wri

blobstore: on init write zeroes to metadata and unmap rest

Currently on blobstore creation we use write zeros everytime.
Some drives does not support write zeros, but support unmap.
We should do write zeroes only on metadata and try to unmap
data clusters

Signed-off-by: Maciej Szwed <maciej.szwed@intel.com>
Change-Id: Iae36c1ccacc08340e79ad40c4c9a2c53dda920ba
Reviewed-on: https://review.gerrithub.io/387152
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Reviewed-by: Daniel Verkamp <daniel.verkamp@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>

show more ...

blobstore: add unmap and write zeros support

Signed-off-by: Maciej Szwed <maciej.szwed@intel.com>
Change-Id: If89275acfb1560982e332148a99ed3c83f8cb34f

Reviewed-on: https://review.gerrithub.io/38760

blobstore: add unmap and write zeros support

Signed-off-by: Maciej Szwed <maciej.szwed@intel.com>
Change-Id: If89275acfb1560982e332148a99ed3c83f8cb34f

Reviewed-on: https://review.gerrithub.io/387609
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Daniel Verkamp <daniel.verkamp@intel.com>

show more ...

blob: handle the case that channel is not created

There is case like no memory to create the channel.
Needs to handle this properly.

Change-Id: I5d13d18037e6aa8f057769b1ef345f45597b22af
Signed-off-

blob: handle the case that channel is not created

There is case like no memory to create the channel.
Needs to handle this properly.

Change-Id: I5d13d18037e6aa8f057769b1ef345f45597b22af
Signed-off-by: GangCao <gang.cao@intel.com>
Reviewed-on: https://review.gerrithub.io/386016
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Daniel Verkamp <daniel.verkamp@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>

show more ...

blobstore: store usable number of clusters in blobstore structure

At the moment there was no way to a user of blobstore api to know,
how many clusters are availible to him. Total_clusters describes

blobstore: store usable number of clusters in blobstore structure

At the moment there was no way to a user of blobstore api to know,
how many clusters are availible to him. Total_clusters describes
number of clusters for metadata and user data.

New field added total_data_clusters, keeping number of clusters
that can be used to create blobs - meaning just user data.

Signed-off-by: Maciej Szwed <maciej.szwed@intel.com>
Signed-off-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Change-Id: I60555217644557410844f74628375a6b46fd2ac7
Reviewed-on: https://review.gerrithub.io/385633
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>

show more ...

blob: fix bug when specifing cluster size < 4096

Signed-off-by: Piotr Pelplinski <piotr.pelplinski@intel.com>
Change-Id: I59cbef4ce1bfe8af113c66c2c9cb9f208440c0aa

Reviewed-on: https://review.gerrit

blob: fix bug when specifing cluster size < 4096

Signed-off-by: Piotr Pelplinski <piotr.pelplinski@intel.com>
Change-Id: I59cbef4ce1bfe8af113c66c2c9cb9f208440c0aa

Reviewed-on: https://review.gerrithub.io/383887
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Daniel Verkamp <daniel.verkamp@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>

show more ...

blobstore: do not allow for spdk_bs_unload if blobs are still open

It is possible that a user will call spdk_bs_unload() with blobs
list not-empty. Instead of just asserting that, now the call fails

blobstore: do not allow for spdk_bs_unload if blobs are still open

It is possible that a user will call spdk_bs_unload() with blobs
list not-empty. Instead of just asserting that, now the call fails
with appropriate error.

Signed-off-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Change-Id: I83818453d6c90ff9b5bf657c90e12b2f9d5ca013
Reviewed-on: https://review.gerrithub.io/383220
Reviewed-by: Paul Luse <paul.e.luse@intel.com>
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Daniel Verkamp <daniel.verkamp@intel.com>

show more ...

blob: add write_zeroes support

Unmap does not guarantee that erased blocks will return all zeroes.
using write_zeroes when unmapping metadata gives the
desired behavior for a blob.

Only metadata pa

blob: add write_zeroes support

Unmap does not guarantee that erased blocks will return all zeroes.
using write_zeroes when unmapping metadata gives the
desired behavior for a blob.

Only metadata pages will be cleared with write_zeroes in this patch;
blob data clusters will still call unmap. This behavior may be made
configurable in a later patch (to allow the user to request zeroing of
clusters rather than just unmapping).

Change-Id: I1b210abac110867ce703bcfdeb634eb45aa9d5c9
Signed-off-by: Seth Howell <seth.howell@intel.com>
Reviewed-on: https://review.gerrithub.io/372004
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Daniel Verkamp <daniel.verkamp@intel.com>

show more ...

blobstore: New API call to destroy blobstore [1/3]

Currently exposed API allows to load/unload and to
initialize blobstore on a device.

A spdk_bs_destroy() call is added in order to reach
functiona

blobstore: New API call to destroy blobstore [1/3]

Currently exposed API allows to load/unload and to
initialize blobstore on a device.

A spdk_bs_destroy() call is added in order to reach
functional parity with spdk_bs_init(). It was not
possible to remove blobstore from device from within
SPDK previously.

spdk_bs_destroy() takes blobstore pointer as argument
(instead of bs_dev), because blobstore has to be already
loaded to destroy it.

Signed-off-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Change-Id: I2c493a4407868fcf08fd1766a19fc8463f634ef5
Reviewed-on: https://review.gerrithub.io/382019
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Daniel Verkamp <daniel.verkamp@intel.com>

show more ...

blob: Make the files persisted cross power failure.

We only sync the metadata and data in the runtime of blobstore, which
means we only update the used md bitmap and used clusters bitmap in memory.

blob: Make the files persisted cross power failure.

We only sync the metadata and data in the runtime of blobstore, which
means we only update the used md bitmap and used clusters bitmap in memory.
if the system crushed, we have no chance to sync the used md bitmap and
used clusters bitmap into disk, then next time when we try to load the
blobstore, all the data will lost, this patch add the logic to recover the
valid data from last dirty shutdown. We will go through all the metadata pages
to find all valid data and rebuild them.

Change-Id: Ieb7c5f932206b1b68fdde0cee35f2d2cb3a4f309
Signed-off-by: Cunyin Chang <cunyin.chang@intel.com>
Reviewed-on: https://review.gerrithub.io/376470
Reviewed-by: Daniel Verkamp <daniel.verkamp@intel.com>
Tested-by: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>

show more ...