blob: Add support for variable metadata page size

Currently SPDK blobstore depends on 4KB atomicity of metadata.
So, Metadata page size will be made variable based on the physical
block size reporte

blob: Add support for variable metadata page size

Currently SPDK blobstore depends on 4KB atomicity of metadata.
So, Metadata page size will be made variable based on the physical
block size reported by the underlying device. This will enable
blobstore to store metadata in IU-sized chunks, not always 4KB.

blobstore.c uses SPDK_BS_PAGE_SIZE (4KB) hardcoded at many places.
To remove this make the metadata page size variable,
phys_blocklen is added to spdk_bs_dev struct and
md_page_size is added to spdk_bs_super_block struct.

Change-Id: I29d073eb4f4341a94a0675e70492b9186382f97f
Signed-off-by: Atul Malakar <a.malakar@samsung.com>
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/25130
Community-CI: Broadcom CI <spdk-ci.pdl@broadcom.com>
Community-CI: Mellanox Build Bot
Reviewed-by: Ben Walker <ben@nvidia.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Community-CI: Community CI Samsung <spdk.community.ci.samsung@gmail.com>
Reviewed-by: Konrad Sztyber <konrad.sztyber@intel.com>
Reviewed-by: Jim Harris <jim.harris@nvidia.com>

show more ...

blob: remove all references to pages as io_units

When blobstore was first created, everything was in terms of 4KB
pages - this included metadata page size and the unit for I/O
operations.

A bit lat

blob: remove all references to pages as io_units

When blobstore was first created, everything was in terms of 4KB
pages - this included metadata page size and the unit for I/O
operations.

A bit later, we introduced concept of "io_unit". If a blobstore
was put on a bdev with 512 blocksize, then the io_unit could be
512 bytes.

But when this happened, we should have changed all of the blobstore
code such that remaining "page" references only referred to metadata
pages. Instead, we left a bunch of places where we would convert
various values to/from a number of 4KB pages, and then to the number
of io_units. This made the code quite confusing, since direct
conversion to/from io_units would have been much clearer.

This existing problem was exacerbated with the upcoming patch to support
variable metadata page sizes. We need things like spdk_bs_get_page_size()
to return the size of the metadata pages, which may not be 4KB.

So make all of the changes necessary such that all references to
"page" now means "metadata page". This includes removing the
spdk_blob_get_num_pages() function, which no longer makes sense.

Signed-off-by: Jim Harris <jim.harris@samsung.com>
Change-Id: I66e93e7a4325a3b032bb16edaf657ef12044e1fd
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/25368
Community-CI: Mellanox Build Bot
Community-CI: Broadcom CI <spdk-ci.pdl@broadcom.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Ben Walker <ben@nvidia.com>
Community-CI: Community CI Samsung <spdk.community.ci.samsung@gmail.com>
Reviewed-by: Konrad Sztyber <konrad.sztyber@intel.com>

show more ...

blob: use uint64_t for cluster start page index

When creating blobstore on devices with more than 16 TiB space, the
start page index of clusters can execeed UINT32_MAX and result in
overflow in uint

blob: use uint64_t for cluster start page index

When creating blobstore on devices with more than 16 TiB space, the
start page index of clusters can execeed UINT32_MAX and result in
overflow in uint32_t representation.

Use uint64_t instead of uint32_t to avoid the overflow.

Change-Id: I8e2586251b1b31ef7520b870af537c3f5792508e
Signed-off-by: Jinlong Chen <chenjinlong.cjl@alibaba-inc.com>
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/24619
Reviewed-by: Konrad Sztyber <konrad.sztyber@intel.com>
Community-CI: Broadcom CI <spdk-ci.pdl@broadcom.com>
Reviewed-by: Jim Harris <jim.harris@samsung.com>
Community-CI: Mellanox Build Bot
Reviewed-by: Changpeng Liu <changpeliu@tencent.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: GangCao <gang.cao@intel.com>

show more ...

lib/blob: fix data inconsistency when unmap a thin-provisioned blob that is backed

The issue may happen in this case:
1.create lvol A from snapshot B
2.write data to A_cluster1
3.unmap A_cluster1

lib/blob: fix data inconsistency when unmap a thin-provisioned blob that is backed

The issue may happen in this case:
1.create lvol A from snapshot B
2.write data to A_cluster1
3.unmap A_cluster1
4.read data from lba coverd by A_cluster1
the result of step 4 should be zero, but it’s actually the data in B_cluster1

Change-Id: If08bdf392d8d3dee737fcacca03bd9f04faa3933
Signed-off-by: xupeng-mingtu <jingmamour@gmail.com>
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/22325
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <jim.harris@samsung.com>
Reviewed-by: Tomasz Zawadzki <tomasz@tzawadzki.com>
Community-CI: Mellanox Build Bot
Reviewed-by: Mateusz Kozlowski <mateusz.kozlowski@solidigm.com>

show more ...

blob: keep count of allocated clusters number

A new variable has been added in spdk_blob_mut_data to store the
number of allocated clusters in clusters array. Also a new method
to get this value has

blob: keep count of allocated clusters number

A new variable has been added in spdk_blob_mut_data to store the
number of allocated clusters in clusters array. Also a new method
to get this value has been added.

Change-Id: Ibac9344bcf7e4c5e6e12cf78a6eae0a4d6755acb
Signed-off-by: Damiano Cipriani <damiano.cipriani@suse.com>
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/19712
Reviewed-by: Jim Harris <jim.harris@samsung.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Community-CI: Mellanox Build Bot

show more ...

blob: defer unload until channel destroy done

As the blobstore is being unlaoded, async esnap channel destructions may
be in flight. In such a case, spdk_bs_unload() needs to defer the unload
of the

blob: defer unload until channel destroy done

As the blobstore is being unlaoded, async esnap channel destructions may
be in flight. In such a case, spdk_bs_unload() needs to defer the unload
of the blobstore until channel destructions are complete.

The following commands lead to the illustrated states.

bdev_malloc_create -b malloc0
bdev_lvol_clone_bdev lvs1 malloc0 eclone

.---------. .--------.
| malloc0 |<--| eclone |
`---------' `--------'

bdev_lvol_snapshot lvs1/eclone snap

.---------. .------. .--------.
| malloc0 |<--| snap |<--| eclone |
`---------' `------' `--------'

bdev_lvol_clone lvs1/snap eclone

.--------.
,-| eclone |
.---------. .------.<-' `--------'
| malloc0 |<--| snap |
`---------' `------'<-. .-------.
`-| clone |
`-------'

As the blobstore is preparing to be unloaded spdk_blob_unload(snap) is
called once for eclone, once for clone, and once for snap. The last of
these calls happens just before spdk_bs_unload() is called.
spdk_blob_unload() needs to destroy channels on each thread. During this
thread iteration, spdk_bs_unload() starts. The work performed in the
iteration maintains a reference to the blob, and as such it
spdk_bs_unload() cannot do its work until the iteration is complete.

Change-Id: Id9b92ad73341fb3437441146110055c84ee6dc52
Signed-off-by: Mike Gerdts <mgerdts@nvidia.com>
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/14975
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Community-CI: Mellanox Build Bot
Reviewed-by: Ben Walker <benjamin.walker@intel.com>

show more ...

blob: add IO channels for esnap clones

The channel passed to blob IO operations is useful for tracking
operations within the blobstore and the bs_dev that the blobstore
resides on. Esnap clone blobs

blob: add IO channels for esnap clones

The channel passed to blob IO operations is useful for tracking
operations within the blobstore and the bs_dev that the blobstore
resides on. Esnap clone blobs perform reads from other bs_devs and
require per-thread, per-bs_dev channels.

This commit augments struct spdk_bs_channel with a tree containing
channels for the external snapshot bs_devs. The tree is indexed by blob
ID. These "esnap channels" are lazily created on the first read from an
external snapshot via each bs_channel. They are removed as bs_channels
are destroyed and blobs are closed.

Change-Id: I97aebe5a2f3584bfbf3a10ede8f3128448d30d6e
Signed-off-by: Mike Gerdts <mgerdts@nvidia.com>
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/14974
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Community-CI: Mellanox Build Bot
Reviewed-by: Ben Walker <benjamin.walker@intel.com>

show more ...

blob: pass bs context with esnap_bs_dev_create

When a blobstore consumer creates or loads a blobstore, it should be
able to set a per-blobstore context pointer that will be passed back to
the consum

blob: pass bs context with esnap_bs_dev_create

When a blobstore consumer creates or loads a blobstore, it should be
able to set a per-blobstore context pointer that will be passed back to
the consumer via bs->esnap_bs_dev_create().

Signed-off-by: Mike Gerdts <mgerdts@nvidia.com>
Change-Id: I59c0ebe21eaf65c3d79a4ac3469715283f56313a
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/14970
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Community-CI: Mellanox Build Bot
Reviewed-by: Ben Walker <benjamin.walker@intel.com>

show more ...

blob: clones of external snapshots

This is the beginning of support for external snapshots. An external
snapshot is a read-only blobstore device (struct spdk_bs_dev) that can
be used as a blob's bac

blob: clones of external snapshots

This is the beginning of support for external snapshots. An external
snapshot is a read-only blobstore device (struct spdk_bs_dev) that can
be used as a blob's back device. Normally a blob will have no back
device (a normal blob), a zeroes back device (a thin provisioned blob),
or a blob back device (a clone blob). When a blob has an external
snapshot ("esnap") as its back device, it is called an esnap clone.

With this patch, esnap clones can be created but they are not yet
useful. Subsequent patches in the series will plumb the IO path, enable
various features, and allow lvol bdevs to be esnap clones.

Signed-off-by: Mike Gerdts <mgerdts@nvidia.com>
Change-Id: I29206b628a2b03b6386a88532565e228df988e0e
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/14969
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Shuhei Matsumoto <smatsumoto@nvidia.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>

show more ...

blobstore: convert used_lock to spinlock

Convert bs->used_lock to a spinlock. This is being done to help with
the debugging and fixing of a race that has led to a failed assertion in
bs_claim_md_pa

blobstore: convert used_lock to spinlock

Convert bs->used_lock to a spinlock. This is being done to help with
the debugging and fixing of a race that has led to a failed assertion in
bs_claim_md_page.

Signed-off-by: Mike Gerdts <mgerdts@nvidia.com>
Change-Id: I11b80096de022f79a217c65d787ee57ca54240f9
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/15952
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Aleksey Marchuk <alexeymar@nvidia.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Shuhei Matsumoto <smatsumoto@nvidia.com>
Community-CI: Mellanox Build Bot

show more ...

blobstore: rename used_clusters_mutex to used_lock

The bs->used_clusters_mutex protects used_md_pages, used_clusters, and
num_free_clusters. A more generic name is appropraite. The next patch in
thi

blobstore: rename used_clusters_mutex to used_lock

The bs->used_clusters_mutex protects used_md_pages, used_clusters, and
num_free_clusters. A more generic name is appropraite. The next patch in
this series will convert it from a mutex to a spinlock and having
"mutex" or "spin" in the name is of little help to maintainers, so a
more generic name is used.

Signed-off-by: Mike Gerdts <mgerdts@nvidia.com>
Change-Id: I5ce7b85b84fdec2a0c5d2ac959e0109e1d80c7f5
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/15981
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Aleksey Marchuk <alexeymar@nvidia.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Shuhei Matsumoto <smatsumoto@nvidia.com>
Community-CI: Mellanox Build Bot

show more ...

update Intel copyright notices

per Intel policy to include file commit date using git cmd
below. The policy does not apply to non-Intel (C) notices.

git log --follow -C90% --format=%ad --date defa

update Intel copyright notices

per Intel policy to include file commit date using git cmd
below. The policy does not apply to non-Intel (C) notices.

git log --follow -C90% --format=%ad --date default <file> | tail -1

and then pull just the 4 digit year from the result.

Intel copyrights were not added to files where Intel either had
no contribution ot the contribution lacked substance (ie license
header updates, formatting changes, etc). Contribution date used
"--follow -C95%" to get the most accurate date.

Note that several files in this patch didn't end the license/(c)
block with a blank comment line so these were added as the vast
majority of files do have this last blank line. Simply there for
consistency.

Signed-off-by: paul luse <paul.e.luse@intel.com>
Change-Id: Id5b7ce4f658fe87132f14139ead58d6e285c04d4
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/15192
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Community-CI: Mellanox Build Bot

show more ...

blobstore: Add function to get io_unit per cluster

This function returns the number of io_units per cluster

Signed-off-by: Damiano Cipriani <damiano.cipriani@suse.com>
Change-Id: I8f33d24a63876a0a9

blobstore: Add function to get io_unit per cluster

This function returns the number of io_units per cluster

Signed-off-by: Damiano Cipriani <damiano.cipriani@suse.com>
Change-Id: I8f33d24a63876a0a918830b9eeaa69a91ff21193
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/14431
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Shuhei Matsumoto <smatsumoto@nvidia.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Konrad Sztyber <konrad.sztyber@intel.com>
Community-CI: Mellanox Build Bot

show more ...

blob: add assert that cluster_sz > 0

Avoids divide-by-zero scanbuild warning on Fedora36.

Fixes issue #2667.

Signed-off-by: Jim Harris <james.r.harris@intel.com>
Change-Id: Ib2793c793725e8bb8ba25f

blob: add assert that cluster_sz > 0

Avoids divide-by-zero scanbuild warning on Fedora36.

Fixes issue #2667.

Signed-off-by: Jim Harris <james.r.harris@intel.com>
Change-Id: Ib2793c793725e8bb8ba25fb779ffc14334929da0
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/14238
Community-CI: Mellanox Build Bot
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Dong Yi <dongx.yi@intel.com>
Reviewed-by: Aleksey Marchuk <alexeymar@nvidia.com>

show more ...

Replace most BSD 3-clause license text with SPDX identifier.

Many open source projects have moved to using SPDX identifiers
to specify license information, reducing the amount of
boilerplate code in

Replace most BSD 3-clause license text with SPDX identifier.

Many open source projects have moved to using SPDX identifiers
to specify license information, reducing the amount of
boilerplate code in every source file. This patch replaces
the bulk of SPDK .c, .cpp and Makefiles with the BSD-3-Clause
identifier.

Almost all of these files share the exact same license text,
and this patch only modifies the files that contain the
most common license text. There can be slight variations
because the third clause contains company names - most say
"Intel Corporation", but there are instances for Nvidia,
Samsung, Eideticom and even "the copyright holder".

Used a bash script to automate replacement of the license text
with SPDX identifier which is checked into scripts/spdx.sh.

Signed-off-by: Jim Harris <james.r.harris@intel.com>
Change-Id: Iaa88ab5e92ea471691dc298cfe41ebfb5d169780
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/12904
Community-CI: Broadcom CI <spdk-ci.pdl@broadcom.com>
Community-CI: Mellanox Build Bot
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Aleksey Marchuk <alexeymar@nvidia.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: Dong Yi <dongx.yi@intel.com>
Reviewed-by: Konrad Sztyber <konrad.sztyber@intel.com>
Reviewed-by: Paul Luse <paul.e.luse@intel.com>
Reviewed-by: <qun.wan@intel.com>

show more ...

blobstore: Preallocate md_page for new cluster

When a new cluster is added to a thin provisioned blob,
md_page is allocated to update extents in base dev
This memory allocation reduces perfromance,

blobstore: Preallocate md_page for new cluster

When a new cluster is added to a thin provisioned blob,
md_page is allocated to update extents in base dev
This memory allocation reduces perfromance, it can
take 250usec - 1 msec on ARM platform.

Since we may have only 1 outstainding cluster
allocation per io_channel, we can preallcoate md_page
on each channel and remove dynamic memory allocation.

With this change blob_write_extent_page() expects
that md_page is given by the caller. Sicne this function
is also used during snapshot deletion, this patch also
updates this process. Now we allocate a single page
and reuse it for each extent in the snapshot.

Signed-off-by: Alexey Marchuk <alexeymar@mellanox.com>
Change-Id: I815a4c8c69bd38d8eff4f45c088e5d05215b9e57
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/12129
Community-CI: Broadcom CI <spdk-ci.pdl@broadcom.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>

show more ...

blob: remove unused inline functions

bs_back_dev_lba_to_io_unit() and bs_num_pages_to_cluster_boundary() are
unused inline functions. The last consumer (by the earlier _spdk_* name)
was removed in c

blob: remove unused inline functions

bs_back_dev_lba_to_io_unit() and bs_num_pages_to_cluster_boundary() are
unused inline functions. The last consumer (by the earlier _spdk_* name)
was removed in commit 6609b776.

Change-Id: Ib1babfed8002fb44451b337aa0db66c15a6805d2
Signed-off-by: Mike Gerdts <mgerdts@nvidia.com>
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/11561
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Aleksey Marchuk <alexeymar@mellanox.com>

show more ...

blob: fix spelling, white space, grammar

Signed-off-by: Mike Gerdts <mgerdts@nvidia.com>
Change-Id: I236c8a1c7f1ae4b0afd0d20175a1a2a647dba758
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+

blob: fix spelling, white space, grammar

Signed-off-by: Mike Gerdts <mgerdts@nvidia.com>
Change-Id: I236c8a1c7f1ae4b0afd0d20175a1a2a647dba758
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/11265
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Community-CI: Broadcom CI <spdk-ci.pdl@broadcom.com>
Community-CI: Mellanox Build Bot
Reviewed-by: Aleksey Marchuk <alexeymar@mellanox.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Shuhei Matsumoto <smatsumoto@nvidia.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Paul Luse <paul.e.luse@intel.com>

show more ...

blobstore: Use RB_TREE to do blob lookup

If blobs held in a blobstore are opened a lot, lookup
by RB_TREE will be much more efficient.

Change-Id: I7075b95c597a958e7bb10890f803191309532021
Signed-of

blobstore: Use RB_TREE to do blob lookup

If blobs held in a blobstore are opened a lot, lookup
by RB_TREE will be much more efficient.

Change-Id: I7075b95c597a958e7bb10890f803191309532021
Signed-off-by: Liu Xiaodong <xiaodong.liu@intel.com>
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/10917
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Community-CI: Broadcom CI <spdk-ci.pdl@broadcom.com>
Community-CI: Mellanox Build Bot

show more ...

spelling: lib

Part of #2256

* accessible
* activation
* additional
* allocate
* association
* attempt
* barrier
* broadcast
* buffer
* calculate
* cases
* channel
* children
* command
* completion

spelling: lib

Part of #2256

* accessible
* activation
* additional
* allocate
* association
* attempt
* barrier
* broadcast
* buffer
* calculate
* cases
* channel
* children
* command
* completion
* connect
* copied
* currently
* descriptor
* destroy
* detachment
* doesn't
* enqueueing
* exceeds
* execution
* extended
* fallback
* finalize
* first
* handling
* hugepages
* ignored
* implementation
* in_capsule
* initialization
* initialized
* initializing
* initiator
* negotiated
* notification
* occurred
* original
* outstanding
* partially
* partition
* processing
* receive
* received
* receiving
* redirected
* regions
* request
* requested
* response
* retrieved
* running
* satisfied
* should
* snapshot
* status
* succeeds
* successfully
* supplied
* those
* transferred
* translate
* triggering
* unregister
* unsupported
* urlsafe
* virtqueue
* volumes
* workaround
* zeroed

Change-Id: I569218754bd9d332ba517d4a61ad23d29eedfd0c
Signed-off-by: Josh Soref <jsoref@gmail.com>
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/10405
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>

show more ...

util: Add macro SPDK_SIZEOF_MEMBER to get size of a member of a struct

We find a few files to get the size of a member of a struct. How to
do it is a little complex. So add a macro to do it will be

util: Add macro SPDK_SIZEOF_MEMBER to get size of a member of a struct

We find a few files to get the size of a member of a struct. How to
do it is a little complex. So add a macro to do it will be helpful
to read the current code and develop new features.

lib/dif had used member_size() internally but Linux use sizeof_member()
as the macro. Besides, SPDK have used upper case letters for similar
macros, SPDK_CONTAINEROF() and SPDK_COUNTOF(). Hence spdk_member_size()
may be good but propose SPDK_SIZEOF_MEMBER() as the macro.

Signed-off-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Change-Id: I2179c845a3b75fb71aa039075cc4dfd30617b898
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/8738
Reviewed-by: Konrad Sztyber <konrad.sztyber@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Community-CI: Broadcom CI <spdk-ci.pdl@broadcom.com>

show more ...

lib/blob: complete multiple persists

When blob persist starts, there can already be multiple
of such requests pending. It is possible to complete
a set of persists at once, if blob state after their

lib/blob: complete multiple persists

When blob persist starts, there can already be multiple
of such requests pending. It is possible to complete
a set of persists at once, if blob state after their
execution would be the same. This is the case when
persists are already pending when a particular persist
request is started.

This patch implements such mechanism by introducing
persists_to_complete queue, containing entries that
were previously queued up before starting the current
persist request. If there are any entries in this queue,
further requests are put into pending_persists.

When first request from persists_to_complete is persisted,
completions are issued for all requests on that queue at once.
If at that point there are any new entries on pending_persists,
all of them are put into persists_to_complete. Persist process is started
again with the first request from that queue.

Signed-off-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Change-Id: I10063e55d6f821b1863de016d3148da6a719a422
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/7643
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>

show more ...

blob: switch to bit_pool for tracking used_clusters

We still need to be able to explicitly set specific
bits in the cluster array during initialization and
loading (especially recovery), so we use a

blob: switch to bit_pool for tracking used_clusters

We still need to be able to explicitly set specific
bits in the cluster array during initialization and
loading (especially recovery), so we use a bit_array
during load, and then convert it to a bit_pool just
before calling the user's cmopletion callback.

This gives a roughly 300% improvement over baseline
on a benchmark which does continuous resize operations.
The benefit is primarily from saving the lowest free
bit rather than having to always start at bit 0. We
may be able to further improve this by saving extents
in the bit pool as well, although after this patch,
the benchmark shows other hot spots different from the
bit search.

Signed-off-by: Jim Harris <james.r.harris@intel.com>
Change-Id: Idb1d75d8348bc50560b1f42d49dbe4d79d024619

Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/3975
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Paul Luse <paul.e.luse@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>

show more ...

blob: Add a bitmask for quickly checking which blobs are open

This can speed up the check for whether a blob is already open
significantly.

Signed-off-by: Ben Walker <benjamin.walker@intel.com>
Cha

blob: Add a bitmask for quickly checking which blobs are open

This can speed up the check for whether a blob is already open
significantly.

Signed-off-by: Ben Walker <benjamin.walker@intel.com>
Change-Id: If32b0b1f168fcdb58e61df6281d7b7520725a195
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/2781
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Community-CI: Mellanox Build Bot
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Reviewed-by: Paul Luse <paul.e.luse@intel.com>

show more ...

lib/blob: remove _spdk prefix from all functions.

Signed-off-by: Seth Howell <seth.howell@intel.com>
Change-Id: Idb33816e5b66266987845172c27c87667ac0a596
Reviewed-on: https://review.spdk.io/gerrit/c

lib/blob: remove _spdk prefix from all functions.

Signed-off-by: Seth Howell <seth.howell@intel.com>
Change-Id: Idb33816e5b66266987845172c27c87667ac0a596
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/2437
Community-CI: Mellanox Build Bot
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Aleksey Marchuk <alexeymar@mellanox.com>

show more ...