thread: remove pool parameter from spdk_iobuf_for_each_entry

We always want to iterate across the entire iobuf_channel, so just
make the iterator do exactly that, rather than requiring the caller
to

thread: remove pool parameter from spdk_iobuf_for_each_entry

We always want to iterate across the entire iobuf_channel, so just
make the iterator do exactly that, rather than requiring the caller
to have to iterate both of the pools itself.

This both simplifies the calling logic, and prepares for upcoming
changes which will support multiple NUMA node caches in one channel.

Signed-off-by: Jim Harris <jim.harris@samsung.com>
Change-Id: Ieed144671e7ee7cb4d7b7b28e803ea3cae641fee
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/24515
Reviewed-by: Aleksey Marchuk <alexeymar@nvidia.com>
Community-CI: Broadcom CI <spdk-ci.pdl@broadcom.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Ben Walker <ben@nvidia.com>
Community-CI: Mellanox Build Bot
Reviewed-by: Konrad Sztyber <konrad.sztyber@intel.com>
Community-CI: Community CI Samsung <spdk.community.ci.samsung@gmail.com>

show more ...

test/unit: add poller_get_stats unit test

Change-Id: Ibf94c19643ba2ae33998dd3b6e5fad174872c9c6
Signed-off-by: Anisa Su <anisa.su@samsung.com>
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+

test/unit: add poller_get_stats unit test

Change-Id: Ibf94c19643ba2ae33998dd3b6e5fad174872c9c6
Signed-off-by: Anisa Su <anisa.su@samsung.com>
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/24526
Reviewed-by: Shuhei Matsumoto <smatsumoto@nvidia.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <jim.harris@samsung.com>
Community-CI: Mellanox Build Bot

show more ...

test/unit: add poller_get_period_ticks unit test

Change-Id: I76f9f517dd2e16fd9091735575985910ff42cbad
Signed-off-by: Anisa Su <anisa.su@samsung.com>
Reviewed-on: https://review.spdk.io/gerrit/c/spdk

test/unit: add poller_get_period_ticks unit test

Change-Id: I76f9f517dd2e16fd9091735575985910ff42cbad
Signed-off-by: Anisa Su <anisa.su@samsung.com>
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/24525
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <jim.harris@samsung.com>
Community-CI: Mellanox Build Bot
Reviewed-by: Shuhei Matsumoto <smatsumoto@nvidia.com>

show more ...

test/unit: add poller_get_state_str unit test

Change-Id: I469afb87a0626fa70701f2b0ab57811584093952
Signed-off-by: Anisa Su <anisa.su@samsung.com>
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/sp

test/unit: add poller_get_state_str unit test

Change-Id: I469afb87a0626fa70701f2b0ab57811584093952
Signed-off-by: Anisa Su <anisa.su@samsung.com>
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/24524
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Community-CI: Mellanox Build Bot
Reviewed-by: Shuhei Matsumoto <smatsumoto@nvidia.com>
Reviewed-by: Jim Harris <jim.harris@samsung.com>

show more ...

test/unit: add poller_get_id unit test

Change-Id: I2e040a19b9358e33934bafa314bb0f9a5725f550
Signed-off-by: Anisa Su <anisa.su@samsung.com>
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/24

test/unit: add poller_get_id unit test

Change-Id: I2e040a19b9358e33934bafa314bb0f9a5725f550
Signed-off-by: Anisa Su <anisa.su@samsung.com>
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/24523
Reviewed-by: Shuhei Matsumoto <smatsumoto@nvidia.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <jim.harris@samsung.com>
Community-CI: Mellanox Build Bot

show more ...

test/unit: add poller_get_name unit test

Change-Id: I1f8a7439221e81632349ab1db28fb05aee711ea7
Signed-off-by: Anisa Su <anisa.su@samsung.com>
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/

test/unit: add poller_get_name unit test

Change-Id: I1f8a7439221e81632349ab1db28fb05aee711ea7
Signed-off-by: Anisa Su <anisa.su@samsung.com>
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/24522
Community-CI: Mellanox Build Bot
Reviewed-by: Jim Harris <jim.harris@samsung.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Shuhei Matsumoto <smatsumoto@nvidia.com>

show more ...

iobuf: prioritize already waiting entries

If an iobuf entry was waiting for a buffer, requested another one in its
iobuf_get_cb, and got placed back onto the wait queue, it'll now be put
at the head

iobuf: prioritize already waiting entries

If an iobuf entry was waiting for a buffer, requested another one in its
iobuf_get_cb, and got placed back onto the wait queue, it'll now be put
at the head of that queue.

This ensures that requests requiring multiple iobufs to execute will be
processed in the order they originally requested the buffers, which
should also ensure that we won't end up with a deadlock.

Signed-off-by: Konrad Sztyber <konrad.sztyber@intel.com>
Change-Id: I58d4a23bd5e73fc84d92a261f7a09c808e48992e
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/24237
Reviewed-by: Jacek Kalwas <jacek.kalwas@intel.com>
Community-CI: Mellanox Build Bot
Reviewed-by: Krzysztof Goreczny <krzysztof.goreczny@dell.com>
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <jim.harris@samsung.com>

show more ...

thread: Don't move to EXITED if for_each_channel is in progress

There was a race between spdk_thread_exit() and spdk_for_each_channel().
A SPDK thread should not move to EXITED until spdk_for_each_c

thread: Don't move to EXITED if for_each_channel is in progress

There was a race between spdk_thread_exit() and spdk_for_each_channel().
A SPDK thread should not move to EXITED until spdk_for_each_channel()
completes especially if the SPDK thread is the caller.
Realize this adding a counter to spdk_thread structure and using it.
spdk_for_each_thread() has the same issue. Hence, fix
spdk_for_each_thread() together.
Verify this by simple unit test cases.

Fixes #3201

Signed-off-by: Shuhei Matsumoto <smatsumoto@nvidia.com>
Change-Id: If6be50cf3204d13f71596bbbb84f4de0f9adfcec
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/20788
Reviewed-by: Konrad Sztyber <konrad.sztyber@intel.com>
Community-CI: Mellanox Build Bot
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <jim.harris@samsung.com>

show more ...

test/unit: use spdk_ut_run_tests()

Replaced direct calls to the CUnit's functions to run the tests with
spdk_ut_run_tests(). That way, each test will have the ability to run
a specific test case.

test/unit: use spdk_ut_run_tests()

Replaced direct calls to the CUnit's functions to run the tests with
spdk_ut_run_tests(). That way, each test will have the ability to run
a specific test case.

The blob.c unit test wasn't changed, because it runs all tests multiple
times with different parameter combinations, so it cannot be easily
converted. In the future, each such combination could be split into a
separate test suite, which would make it compatible with
spdk_ut_run_tests().

Signed-off-by: Konrad Sztyber <konrad.sztyber@intel.com>
Change-Id: I4463f808f89844e9bf32b5b31eda197c5d729d1d
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/19288
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Community-CI: Mellanox Build Bot
Reviewed-by: Aleksey Marchuk <alexeymar@nvidia.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>

show more ...

test/unit: move spdk_cunit.h to include/spdk_internal

It'll make it easier to include this file outside of unit tests.

Signed-off-by: Konrad Sztyber <konrad.sztyber@intel.com>
Change-Id: I171ddb864

test/unit: move spdk_cunit.h to include/spdk_internal

It'll make it easier to include this file outside of unit tests.

Signed-off-by: Konrad Sztyber <konrad.sztyber@intel.com>
Change-Id: I171ddb8649f67b5786f08647560e2907603d0574
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/19284
Reviewed-by: Tomasz Zawadzki <tomasz.zawadzki@intel.com>
Community-CI: Mellanox Build Bot
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Aleksey Marchuk <alexeymar@nvidia.com>

show more ...

thread: iobuf now grabs from the central pool in batches

This is much more efficient, but it is a slight behavior change. See the
unit test change in this patch.

Change-Id: I5eaa49a5bf7a0131d599258

thread: iobuf now grabs from the central pool in batches

This is much more efficient, but it is a slight behavior change. See the
unit test change in this patch.

Change-Id: I5eaa49a5bf7a0131d59925806c47aaf03b7c207e
Signed-off-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/16280
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Aleksey Marchuk <alexeymar@nvidia.com>
Community-CI: Mellanox Build Bot
Reviewed-by: Jim Harris <james.r.harris@intel.com>

show more ...

thread: iobuf now allocates 4k aligned buffers

The buffers returned by the iobuf pool are now always 4k aligned.
This will reduce complexity elsewhere (lib/nvmf).

This also removes an incorrect de

thread: iobuf now allocates 4k aligned buffers

The buffers returned by the iobuf pool are now always 4k aligned.
This will reduce complexity elsewhere (lib/nvmf).

This also removes an incorrect dependency on bdev.h.

Change-Id: I1d0d9c79309f9962419d96a6a7a396108b476015
Signed-off-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/16224
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Community-CI: Mellanox Build Bot
Reviewed-by: Aleksey Marchuk <alexeymar@nvidia.com>

show more ...

thread: Move iobuf code to a separate compilation unit.

This makes it much easier to mock this code in unit tests without having
to mock up the entire thread library.

Change-Id: Ic3d9cb826ae71af780

thread: Move iobuf code to a separate compilation unit.

This makes it much easier to mock this code in unit tests without having
to mock up the entire thread library.

Change-Id: Ic3d9cb826ae71af780a06f88669c37cef2c9a4ae
Signed-off-by: Ben Walker <benjamin.walker@intel.com>
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/16173
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Community-CI: Mellanox Build Bot
Reviewed-by: Aleksey Marchuk <alexeymar@nvidia.com>

show more ...

lib/thread: iobuf get/put functions will not add offset

When a buffer is get, it does not need to reserve the space
for tailq header.

Signed-off-by: MengjinWu <mengjin.wu@intel.com>
Change-Id: I0aa

lib/thread: iobuf get/put functions will not add offset

When a buffer is get, it does not need to reserve the space
for tailq header.

Signed-off-by: MengjinWu <mengjin.wu@intel.com>
Change-Id: I0aa2d77739fbb86a6e2df1c00a772aff1cb7c6e4
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/16181
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Konrad Sztyber <konrad.sztyber@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>

show more ...

thread: cache a number of iobuf buffers on each channel

Users can now specify a number of small/large buffers to be cached on
each iobuf channel. Previously, we relied on the cache of the
underlyin

thread: cache a number of iobuf buffers on each channel

Users can now specify a number of small/large buffers to be cached on
each iobuf channel. Previously, we relied on the cache of the
underlying spdk_mempool, which has per-core caches. However, since iobuf
channels are tied to a module and an SPDK thread, each module and each
thread is now guaranteed to have a number of buffers available, so it
won't be starved by other modules/threads.

Signed-off-by: Konrad Sztyber <konrad.sztyber@intel.com>
Change-Id: I1e29fe29f78a13de371ab21d3e40bf55fbc9c639
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/15634
Reviewed-by: Aleksey Marchuk <alexeymar@nvidia.com>
Reviewed-by: Shuhei Matsumoto <smatsumoto@nvidia.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Reviewed-by: Ben Walker <benjamin.walker@intel.com>
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Community-CI: Mellanox Build Bot

show more ...

thread: introduce iobuf buffer pools

The idea behind "iobuf" is to have a single place for allocating data
buffers across different libraries. That way, each library won't need
to allocate its own

thread: introduce iobuf buffer pools

The idea behind "iobuf" is to have a single place for allocating data
buffers across different libraries. That way, each library won't need
to allocate its own mempools, therefore decreasing the memory footprint
of the whole application.

There are two reasons for putting these kind of functions in the thread
library. Firstly, the code is pretty small, so it doesn't make sense to
create a new library. Secondly, it relies on the IO channel abstraction,
so users will need to pull in the thread library anyway.

It's very much inspired by the way bdev layer handles data buffers (much
of the code was directly copied over). There are two global mempools,
one for small and one for large buffers, and per-thread queues that hold
requests waiting for a buffer. The main difference is that we also need
to track which module requested a buffer in order to allow users to
iterate over its pending requests.

The usage is fairly simple:

```
/* Embed spdk_iobuf_channel into an existing IO channel */
struct foo_channel {
...
struct spdk_iobuf_channel iobuf;
};

/* Embed spdk_iobuf_entry into objects that will request buffers */
struct foo_object {
...
struct spdk_iobuf_entry entry;
};

/* Register the module as iobuf user */
spdk_iobuf_register_module("foo");

/* Initialize iobuf channel in foo_channel's create cb */
spdk_iobuf_channel_init(&foo_channel->iobuf, "foo", 0, 0);

/* Finally, request a buffer... */
buf = spdk_iobuf_get(&foo_channel->iobuf, length,
&foo_objet.entry, buf_get_cb);

...

/* ...and release it */
spdk_iobuf_put(&foo_channel->iobuf, buf, length);

```

Signed-off-by: Konrad Sztyber <konrad.sztyber@intel.com>
Change-Id: Ifaa6934c03ed6587ddba972198e606921bd85008
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/15326
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Community-CI: Mellanox Build Bot
Reviewed-by: Shuhei Matsumoto <smatsumoto@nvidia.com>
Reviewed-by: Aleksey Marchuk <alexeymar@nvidia.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>

show more ...

thread: SPDK spinlocks

This introduces an enhanced spinlock that adds safeguards compared to
the default pthread_spinlock_t. In particular:

- A pthread_spinlock_t is still used, but additional erro

thread: SPDK spinlocks

This introduces an enhanced spinlock that adds safeguards compared to
the default pthread_spinlock_t. In particular:

- A pthread_spinlock_t is still used, but additional error checking is
performed to ensure there is no undefined behavior on relock,
unlocking when not the owner, or destoying a locked lock.
- The SPDK concurrency model allows an SPDK thread to be migrated
between pthreads. Releasing a pthread spinlock on a different thread
from where it is taken is undefined behavior. If an SPDK spinlock is
held at a time that a time when a poller or message returns control to
thread_poll(), the program will abort.
- SPDK spinlocks can only be obtained from an SPDK thread.

Signed-off-by: Mike Gerdts <mgerdts@nvidia.com>
Change-Id: I6dd6493ab5f5532ae69e20654546405a507eb594
Reviewed-on: https://review.spdk.io/gerrit/c/spdk/spdk/+/15277
Tested-by: SPDK CI Jenkins <sys_sgci@intel.com>
Reviewed-by: Aleksey Marchuk <alexeymar@nvidia.com>
Reviewed-by: Shuhei Matsumoto <smatsumoto@nvidia.com>
Reviewed-by: Jim Harris <james.r.harris@intel.com>
Community-CI: Mellanox Build Bot

show more ...