blobfs/fuse: put FUSE code in module blobfs_bdev

FUSE operations in blobfs_fuse.c are extracted from
test/blobfs/fuse/fuse.c to blobfs_fuse.c in module blobfs_bdev.
And it is extended to create one

blobfs/fuse: put FUSE code in module blobfs_bdev

FUSE operations in blobfs_fuse.c are extracted from
test/blobfs/fuse/fuse.c to blobfs_fuse.c in module blobfs_bdev.
And it is extended to create one new thread dedicatedly for one
FUSE mountpoint to handle FUSE requests by blobfs sync API.

spdk_blobfs_bdev_mount is implemented as the export API. So
related code can be utilized by other modules/apps.

Now test/blobfs/fuse/fuse.c is much simplified with function
spdk_blobfs_bdev_mount.

Change-Id: Iefa16977fabbae2008c8f65fe1b69d650b6fd18d
Signed-off-by: Xiaodong Liu <xiaodong.liu@intel.com>
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/469347
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>

show more ...

blobfs: add blobfs_create RPC as mkfs

Following test/blobfs/mkfs case, add it as one RPC
method to let build a new blobfs on given block
device.

Change-Id: I0ffbb1add95dfbc8655e0238ed6f3cd519dd945b

blobfs: add blobfs_create RPC as mkfs

Following test/blobfs/mkfs case, add it as one RPC
method to let build a new blobfs on given block
device.

Change-Id: I0ffbb1add95dfbc8655e0238ed6f3cd519dd945b
Signed-off-by: Xiaodong Liu <xiaodong.liu@intel.com>
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/466485
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>

show more ...

blobfs: add blobfs_detect RPC

Add module blobfs_bdev as a general module to simplify
the operations of blobfs on bdev. Then its functions
can be utilized by other libraries or apps.

blobfs_detect c

blobfs: add blobfs_detect RPC

Add module blobfs_bdev as a general module to simplify
the operations of blobfs on bdev. Then its functions
can be utilized by other libraries or apps.

blobfs_detect can be used to detect whether there is
one blobfs on given bdev.

Change-Id: Ib425574816061dc945fb652b539f791a44097a43
Signed-off-by: Xiaodong Liu <xiaodong.liu@intel.com>
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/466486
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Paul Luse <paul.e.luse@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>

show more ...

UT/blobfs: add a append write over blob boundary UT

Change-Id: Ifc50920ae316ba76719c14d40615a23930457fc8
Signed-off-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-on: https://review.gerrithub.

UT/blobfs: add a append write over blob boundary UT

Change-Id: Ifc50920ae316ba76719c14d40615a23930457fc8
Signed-off-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/464083
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>

show more ...

UT/blobfs: add asynchronous Writev/Readv unit tests

Change-Id: I8c0961e78afff367f6fc8b1a97d5ec5c635e3224
Signed-off-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-on: https://review.gerrithub.

UT/blobfs: add asynchronous Writev/Readv unit tests

Change-Id: I8c0961e78afff367f6fc8b1a97d5ec5c635e3224
Signed-off-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/457455
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Reviewed-by: Darek Stojaczyk <dariusz.stojaczyk@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>

show more ...

UT/blobfs: add asynchronous Write/Read unit tests

Change-Id: I4a64e7b883274f3bcd5d4a7ae151156d61acae01
Signed-off-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-on: https://review.gerrithub.io

UT/blobfs: add asynchronous Write/Read unit tests

Change-Id: I4a64e7b883274f3bcd5d4a7ae151156d61acae01
Signed-off-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/457080
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>

show more ...

ut/blobfs: fix blobfs_sync_ut hang issue when testing file_length

we use this statement:
while (g_file->length_flushed != buf_length) {} in
file_length function.

It means that in this test case, le

ut/blobfs: fix blobfs_sync_ut hang issue when testing file_length

we use this statement:
while (g_file->length_flushed != buf_length) {} in
file_length function.

It means that in this test case, length_flushed are
accessed by two different threads, so better to
use another new variable length_flused with volatile
before the variable definition. Then our ut will not hang.

Change-Id: I6152a4ba3f27f0fad1c8c2baa71324a36a2fb9e8
Signed-off-by: Ziye Yang <ziye.yang@intel.com>
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/456580
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>

show more ...

blobfs: don't flush a partial buffer with no sync pending

We flush a cache buffer once it's filled. When the write
for that cache buffer has completed, we look to see if
there's more data to flush.

blobfs: don't flush a partial buffer with no sync pending

We flush a cache buffer once it's filled. When the write
for that cache buffer has completed, we look to see if
there's more data to flush. Currently if there's *any*
more data to flush, we will flush it, even if it's not
a full buffer.

That can hurt performance though. Ideally we only want
to flush a partial buffer if there's been an explicit
sync operation that requires that partial buffer to be
flushed. Otherwise we will end up writing the partial
buffer to disk, and then come back and write that data
again later when the buffer is full.

Add a new unit test to test for this condition. This
patch breaks one of the existing unit tests which was
designed specifically around a RocksDB failure condition.
Change that file_length unit test to now write exactly
one CACHE_BUFFER, which still tests the general logic
making sure that we don't confuse the amount of data
flushed with the value written to the file's length
xattr.

Signed-off-by: Jim Harris <james.r.harris@intel.com>
Change-Id: I83795fb45afe854b38648d0e0c1a7928219307a2

Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/455698
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Ziye Yang <ziye.yang@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>

show more ...

blobfs: ensure length xattr is written even if all data is flushed

Data can get implicitly flushed as cache buffers are filled. But
the length xattr is only written in response to a sync or close
o

blobfs: ensure length xattr is written even if all data is flushed

Data can get implicitly flushed as cache buffers are filled. But
the length xattr is only written in response to a sync or close
operation. So we cannot just look at the amount of data flushed,
and ignore the sync operation if all of the data written has been
flushed - we still need to write the length xattr.

This also adds a unit test which reproduces the original problem.

Fixes issue #297.

Signed-off-by: Jim Harris <james.r.harris@intel.com>
Change-Id: Icca6ef4d1544f72e9bc31c4ee77d26b4b7f0cce4

Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/455692
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Ziye Yang <ziye.yang@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>

show more ...

UT/blobfs: add rename UT to blobfs_sync_ut

Change-Id: I17c34a02cbffce4078ef82cfc191394a7440ba52
Signed-off-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-on: https://review.gerrithub.io/c/spdk

UT/blobfs: add rename UT to blobfs_sync_ut

Change-Id: I17c34a02cbffce4078ef82cfc191394a7440ba52
Signed-off-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/455151
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>

show more ...

UT/blobfs: add data verification test to blobfs_sync_ut

Change-Id: If61ea8fcc96b02e9b6af9c2bb5bc703d4e2f46bf
Signed-off-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-on: https://review.gerrit

UT/blobfs: add data verification test to blobfs_sync_ut

Change-Id: If61ea8fcc96b02e9b6af9c2bb5bc703d4e2f46bf
Signed-off-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/455145
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>

show more ...

thread: Add a mechanism to exit a lightweight thread

Lightweight threads may now be exited by calling
spdk_thread_exit() within the thread. The framework
polling the thread can release the resources

thread: Add a mechanism to exit a lightweight thread

Lightweight threads may now be exited by calling
spdk_thread_exit() within the thread. The framework
polling the thread can release the resources associated
with that lightweight thread by calling
spdk_thread_destroy().

Change-Id: I6586b9d22556b3874fb113ce5402c6b1f371786e
Signed-off-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/455319
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Seth Howell <seth.howell5141@gmail.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: Darek Stojaczyk <dariusz.stojaczyk@intel.com>

show more ...

blobfs: remove trace short names

They were removed from the trace API and BlobFS doesn't
compile right now.

Change-Id: Ie98339f19353e4db901af551c706f57c18b89fc5
Signed-off-by: Darek Stojaczyk <dari

blobfs: remove trace short names

They were removed from the trace API and BlobFS doesn't
compile right now.

Change-Id: Ie98339f19353e4db901af551c706f57c18b89fc5
Signed-off-by: Darek Stojaczyk <dariusz.stojaczyk@intel.com>
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/452863
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>

show more ...

blobfs: add trace events

For now, just do events for open, close and start/end
for writing the length xattr. These will be useful for
debugging the intermittent RocksDB failures.

Note these events

blobfs: add trace events

For now, just do events for open, close and start/end
for writing the length xattr. These will be useful for
debugging the intermittent RocksDB failures.

Note these events may only be recorded on the polling
SPDK thread. SPDK events cannot be recorded on
non-SPDK threads (i.e. RocksDB user or background threads).

Signed-off-by: Jim Harris <james.r.harris@intel.com>
Change-Id: I72fb302db23b399f583f56194c237340587f3027

Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/452262
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: Ziye Yang <ziye.yang@intel.com>
Reviewed-by: Darek Stojaczyk <dariusz.stojaczyk@intel.com>

show more ...

thread: Add cpumask parameter to spdk_thread_create

Allow users to optionally specify an affinity mask when
creating a thread. This isn't currently used, but we want
the API to be in its final form

thread: Add cpumask parameter to spdk_thread_create

Allow users to optionally specify an affinity mask when
creating a thread. This isn't currently used, but we want
the API to be in its final form for the next release.

Change-Id: I7bd05e921ece6d8d5f61775bd14286f6a58f267f
Signed-off-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/451683
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>

show more ...

blobfs: Change channel type for synchronous operations

In the next step, this won't be a channel at all.

Change-Id: Ia8fe4da5b0b283e8dfc5c6477b84cfdd346d89a0
Signed-off-by: Ben Walker <benjamin.wal

blobfs: Change channel type for synchronous operations

In the next step, this won't be a channel at all.

Change-Id: Ia8fe4da5b0b283e8dfc5c6477b84cfdd346d89a0
Signed-off-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/449464
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>

show more ...

blobfs: Simplify sync unit test

Rely on the available threading abstractions to make this
a bit simpler.

Change-Id: If8f028092c057637ff167d2ec7faa3dce009af61
Signed-off-by: Ben Walker <benjamin.wal

blobfs: Simplify sync unit test

Rely on the available threading abstractions to make this
a bit simpler.

Change-Id: If8f028092c057637ff167d2ec7faa3dce009af61
Signed-off-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-on: https://review.gerrithub.io/c/spdk/spdk/+/449463
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>

show more ...

thread: Optionally allow the current time to be passed to
spdk_thread_poll()

This is an optimization if the calling function already knows the
current time.

Change-Id: I1645e08e7475ba6345a44e0f9d4b

thread: Optionally allow the current time to be passed to
spdk_thread_poll()

This is an optimization if the calling function already knows the
current time.

Change-Id: I1645e08e7475ba6345a44e0f9d4b297a79f6c3c2
Signed-off-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-on: https://review.gerrithub.io/c/443634
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Darek Stojaczyk <dariusz.stojaczyk@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>

show more ...

test/unit: remove duplicate mk/spdk.common.mk includes

mk/spdk.unittest.mk already includes mk/spdk.common.mk, so it's
not needed. This also fixes an issue where touching an included
.mk file would

test/unit: remove duplicate mk/spdk.common.mk includes

mk/spdk.unittest.mk already includes mk/spdk.common.mk, so it's
not needed. This also fixes an issue where touching an included
.mk file would not trigger unit tests to rebuild if they had
this duplicated mk/spdk.common.mk include.

Signed-off-by: Jim Harris <james.r.harris@intel.com>
Change-Id: I41a04eb77ce468849cb9b53bd1f76df6fec06e46

Reviewed-on: https://review.gerrithub.io/c/443980
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Reviewed-by: Changpeng Liu <changpeng.liu@intel.com>
Reviewed-by: Darek Stojaczyk <dariusz.stojaczyk@intel.com>

show more ...

thread: Simplify two unit tests with nested poll calls

Now that spdk_thread_poll can be called from within
a thread, some of the unit tests can be a bit simpler.

Change-Id: Ib7b92c3f59862e5c128f5e0

thread: Simplify two unit tests with nested poll calls

Now that spdk_thread_poll can be called from within
a thread, some of the unit tests can be a bit simpler.

Change-Id: Ib7b92c3f59862e5c128f5e0cc8a08f65e2b89e93
Signed-off-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-on: https://review.gerrithub.io/c/441160
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Darek Stojaczyk <dariusz.stojaczyk@intel.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Chandler-Test-Pool: SPDK Automated Test System <sys_sgsw@intel.com>

show more ...

thread: Rename spdk_free_thread to spdk_thread_exit

This name more closely resembles pthread_exit, which is a
closer analogy to how the new threading library works.

Change-Id: I68b04509f3ff8e94b868

thread: Rename spdk_free_thread to spdk_thread_exit

This name more closely resembles pthread_exit, which is a
closer analogy to how the new threading library works.

Change-Id: I68b04509f3ff8e94b8688804a7e5661155a3ecd1
Signed-off-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-on: https://review.gerrithub.io/c/440597
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Reviewed-by: Darek Stojaczyk <dariusz.stojaczyk@intel.com>
Chandler-Test-Pool: SPDK Automated Test System <sys_sgsw@intel.com>

show more ...

thread: Rename spdk_allocate_thread to spdk_thread_create

This mirrors pthread_create, which works more closely
to the new style where SPDK libraries can spawn their
own threads.

Change-Id: Ic524c4

thread: Rename spdk_allocate_thread to spdk_thread_create

This mirrors pthread_create, which works more closely
to the new style where SPDK libraries can spawn their
own threads.

Change-Id: Ic524c4c35bcf7c1611e4f261ebb64b98ac5a5a1b
Signed-off-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-on: https://review.gerrithub.io/c/440596
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Reviewed-by: Darek Stojaczyk <dariusz.stojaczyk@intel.com>

show more ...

thread: spdk_free_thread now takes a thread parameter

Instead of implicitly grabbing the thread from the thread
local variable, make it explicit.

Change-Id: I733fad06181439e12b1e71a4829b84e7b64e246

thread: spdk_free_thread now takes a thread parameter

Instead of implicitly grabbing the thread from the thread
local variable, make it explicit.

Change-Id: I733fad06181439e12b1e71a4829b84e7b64e2468
Signed-off-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-on: https://review.gerrithub.io/c/440595
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Reviewed-by: Darek Stojaczyk <dariusz.stojaczyk@intel.com>
Chandler-Test-Pool: SPDK Automated Test System <sys_sgsw@intel.com>

show more ...

thread: Eliminate function pointers in spdk_allocate_thread

These are no longer used by anything.

Change-Id: I0db6bc88e4dc945ff4f64df2ac410e1d00a669c1
Signed-off-by: Ben Walker <benjamin.walker@int

thread: Eliminate function pointers in spdk_allocate_thread

These are no longer used by anything.

Change-Id: I0db6bc88e4dc945ff4f64df2ac410e1d00a669c1
Signed-off-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-on: https://review.gerrithub.io/c/437601
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Chandler-Test-Pool: SPDK Automated Test System <sys_sgsw@intel.com>
Reviewed-by: Darek Stojaczyk <dariusz.stojaczyk@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>

show more ...

test: Use ut_multithread framework in blobfs sync test

Change-Id: I5a15f3af773aad23e0479ad94c4544a2a5f9ea50
Signed-off-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-on: https://review.gerrithu

test: Use ut_multithread framework in blobfs sync test

Change-Id: I5a15f3af773aad23e0479ad94c4544a2a5f9ea50
Signed-off-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-on: https://review.gerrithub.io/c/422470
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Shuhei Matsumoto <shuhei.matsumoto.xt@hitachi.com>
Reviewed-by: Darek Stojaczyk <dariusz.stojaczyk@intel.com>
Chandler-Test-Pool: SPDK Automated Test System <sys_sgsw@intel.com>

show more ...