lib: remove pthread.h from includes

The header files should have the minimum embedded includes.
The file pthread.h can logically be removed from
rte_per_lcore.h and rte_ethdev_core.h files.

Signed-

lib: remove pthread.h from includes

The header files should have the minimum embedded includes.
The file pthread.h can logically be removed from
rte_per_lcore.h and rte_ethdev_core.h files.

Signed-off-by: Thomas Monjalon <thomas@monjalon.net>
Acked-by: Morten Brørup <mb@smartsharesystems.com>
Acked-by: Tyler Retzlaff <roretzla@linux.microsoft.com>
Acked-by: Ajit Khaparde <ajit.khaparde@broadcom.com>
Acked-by: Rosen Xu <rosen.xu@intel.com>
Acked-by: Konstantin Ananyev <konstantin.v.ananyev@yandex.ru>

show more ...

common/mlx5: fix shared mempool subscription

MLX5 PMD counted each mempool subscribe invocation. The PMD expected
that the mempool subscription will be deleted after the mempool
counter dropped to 0

common/mlx5: fix shared mempool subscription

MLX5 PMD counted each mempool subscribe invocation. The PMD expected
that the mempool subscription will be deleted after the mempool
counter dropped to 0. However, current PMD design unsubscribes mempool
callbacks only once.
As the result, the PMD destroyed mlx5_common_device but kept
shared RX subscription callback. EAL tried to activate that callback
and crashed.

The patch removes mempool subscriptions counter.
The PMD registers mempool subscription once only. An attempt
to register existing subscription returns EEXIST.
Also, the PMD expects to remove subscription when mempool unsubscribe
was activated.

Fixes: 8ad97e4b3215 ("common/mlx5: fix multi-process mempool registration")
Cc: stable@dpdk.org

Signed-off-by: Gregory Etelson <getelson@nvidia.com>
Acked-by: Matan Azrad <matan@nvidia.com>

show more ...

common/mlx5: fix null check in devargs parsing

The common MLX5 probe function parses first the devargs and save them in
a dictionary.
It gets structure called "devargs" as a member of EAL device con

common/mlx5: fix null check in devargs parsing

The common MLX5 probe function parses first the devargs and save them in
a dictionary.
It gets structure called "devargs" as a member of EAL device containing
all needed information.

When "devargs" structure is invalid, the function avoids parsing it.
However, when it valid but its field "args" is invalid, the function
tries to parse it and dereference to NULL pointer.

This patch adds check to avoid this NULL dereferencing.

Fixes: a729d2f093e9 ("common/mlx5: refactor devargs management")
Cc: stable@dpdk.org

Signed-off-by: Michael Baum <michaelba@nvidia.com>
Acked-by: Matan Azrad <matan@nvidia.com>

show more ...

common/mlx5: fix multi-process mempool registration

The `mp_cb_registered` flag shared between all processes
was used to ensure that for any IB device (MLX5 common device)
mempool event callback was

common/mlx5: fix multi-process mempool registration

The `mp_cb_registered` flag shared between all processes
was used to ensure that for any IB device (MLX5 common device)
mempool event callback was registered only once
and mempools that had been existing before the device start
were traversed only once to register them.
Since mempool callback registrations have become process-private,
callback registration must be done by every process.
The flag can no longer reflect the state for any single process.
Replace it with a registration counter to track
when no more callbacks are registered for the device in any process.
It is sufficient to only register pre-existing mempools
in the primary process because it is the one that starts the device.

Fixes: 690b2a88c2f7 ("common/mlx5: add mempool registration facilities")
Cc: stable@dpdk.org

Signed-off-by: Dmitry Kozlyuk <dkozlyuk@nvidia.com>
Acked-by: Viacheslav Ovsiienko <viacheslavo@nvidia.com>

show more ...

common/mlx5: fix non-expandable global MR cache

The number of memory regions (MR) that MLX5 PMD can use
was limited by 512 per IB device, the size of the global MR cache
that was fixed at compile ti

common/mlx5: fix non-expandable global MR cache

The number of memory regions (MR) that MLX5 PMD can use
was limited by 512 per IB device, the size of the global MR cache
that was fixed at compile time.
The cache allows to search MR LKey by address efficiently,
therefore it is the last place searched on data path
(skipped is the global MR database which would be slow).
If the application logic caused the PMD to create more than 512 MRs,
which can be the case with external memory,
those MRs would never be found on data path
and later cause a HW failure.

The cache size was fixed because at the time of overflow
the EAL memory hotplug lock may be held,
prohibiting to allocate a larger cache
(it must reside in DPDK memory for multi-process support).
This patch adds logic to release the necessary locks,
extend the cache, and repeat the attempt to insert new entries.

`mlx5_mr_btree` structure had `overflow` field
that was set when a cache (not only the global one)
could not accept new entries.
However, it was only checked for the global cache,
because caches of upper layers were dynamically expandable.
With the global cache size limitation removed, this field is not needed.
Cache size was previously limited by 16-bit indices.
Use the space in the structure previously field by `overflow` field
to extend indices to 32 bits.
With this patch, it is the HW and RAM that limit the number of MRs.

Fixes: 974f1e7ef146 ("net/mlx5: add new memory region support")
Cc: stable@dpdk.org

Signed-off-by: Dmitry Kozlyuk <dkozlyuk@nvidia.com>
Acked-by: Matan Azrad <matan@nvidia.com>

show more ...

common/mlx5: fix default devargs initialization

Device arguments list is provided along with its identifier as part of
EAL arguments.
The arguments specified in the list are taken from it, and the r

common/mlx5: fix default devargs initialization

Device arguments list is provided along with its identifier as part of
EAL arguments.
The arguments specified in the list are taken from it, and the rest is
initialized to the default values.

When no list is provided at all, all arguments should have been
initialized to their default values. However, they are mistakenly
initialized to zero which may be a valid value for some.

This patch initializes the default values before checking whether
arguments have been specified.

Bugzilla ID: 945
Fixes: a729d2f093e9 ("common/mlx5: refactor devargs management")

Signed-off-by: Michael Baum <michaelba@nvidia.com>
Acked-by: Matan Azrad <matan@nvidia.com>

show more ...

common/mlx5: support remote PD and CTX

Add option to probe common device using import CTX/PD functions instead
of create functions.
This option requires accepting the context FD and the PD handle as

common/mlx5: support remote PD and CTX

Add option to probe common device using import CTX/PD functions instead
of create functions.
This option requires accepting the context FD and the PD handle as
devargs.

This sharing can be useful for applications that use PMD for only some
operations. For example, an app that generates queues itself and uses
PMD just to configure flow rules.

Signed-off-by: Michael Baum <michaelba@nvidia.com>
Acked-by: Matan Azrad <matan@nvidia.com>

show more ...

common/mlx5: update doorbell mapping parameter name

The "tx_db_nc" devarg forces doorbell register mapping to non-cached
region eliminating the extra write memory barrier. This argument was
used in

common/mlx5: update doorbell mapping parameter name

The "tx_db_nc" devarg forces doorbell register mapping to non-cached
region eliminating the extra write memory barrier. This argument was
used in creating the UAR for Tx and thus affected its performance.

Recently [1] its use has been extended to all UAR creation in all mlx5
drivers, and now its name is no longer so accurate.

This patch changes its name to "sq_db_nc" to suit any send queue that
uses it. The old name will still work for backward compatibility.

[1] commit 5dfa003db53f ("common/mlx5: fix post doorbell barrier")

Signed-off-by: Michael Baum <michaelba@nvidia.com>
Reviewed-by: Raslan Darawsheh <rasland@nvidia.com>
Acked-by: Viacheslav Ovsiienko <viacheslavo@nvidia.com>

show more ...

common/mlx5: refactor devargs management

Improve the devargs handling in two aspects:
- Parse the devargs string only once.
- Return error and report for unknown keys.

The common driver parses on

common/mlx5: refactor devargs management

Improve the devargs handling in two aspects:
- Parse the devargs string only once.
- Return error and report for unknown keys.

The common driver parses once the devargs string into a dictionary, then
provides it to all the drivers' probe. Each driver updates within it
which keys it has used, then common driver receives the updated
dictionary and reports about unknown devargs.

Signed-off-by: Michael Baum <michaelba@nvidia.com>
Acked-by: Matan Azrad <matan@nvidia.com>

show more ...

common/mlx5: check common devargs in probing again

MLX5 common driver supports probing again in two scenarios:
- Add new driver under existing device. common probe function gets
it in devargs, t

common/mlx5: check common devargs in probing again

MLX5 common driver supports probing again in two scenarios:
- Add new driver under existing device. common probe function gets
it in devargs, then calls the requested driver's probe function
(regardless of the driver's own support in probing again) with the
existing device as parameter.
- Transfer the probing again support of the drivers themselves
(currently only net). In this scenario, the existing device is sent
as a parameter to the existing driver's probe too.

In both cases it gets a new set of arguments that do not necessarily
match the configured arguments in the existing device.
Some of the arguments belong to the configuration of the existing
device, so they can't be updated in the probing again. On the other
hand, there are arguments that belong to a specific driver or specific
port and might get a new value in the probing again.
The user might generate any argument he wants in probing again, but when
he generates arguments belonging to the common device configuration, it
does not affect.

This patch adds an explicit check for the devargs belonging to the
common device configuration. If there is no match to the existing
configuration, it returns an error.

Signed-off-by: Michael Baum <michaelba@nvidia.com>
Acked-by: Matan Azrad <matan@nvidia.com>

show more ...

common/mlx5: fix probing failure code

While probing the device with unsupported class, the process should
fail because no appropriate driver was found. After traversing all
the drivers, an error val

common/mlx5: fix probing failure code

While probing the device with unsupported class, the process should
fail because no appropriate driver was found. After traversing all
the drivers, an error value should be returned for the case.

In the previous implementation, zero value indicating probing success
was wrongly returned.

Fixes: ad435d320473 ("common/mlx5: add bus-agnostic layer")
Cc: stable@dpdk.org

Signed-off-by: Bing Zhao <bingz@nvidia.com>
Acked-by: Viacheslav Ovsiienko <viacheslavo@nvidia.com>

show more ...

common/mlx5: fix missing validation in devargs parsing

The rte_kvargs_parse function parses the arguments
"key=value,key=value,..." string and return an allocated structure that
contains a key/value

common/mlx5: fix missing validation in devargs parsing

The rte_kvargs_parse function parses the arguments
"key=value,key=value,..." string and return an allocated structure that
contains a key/value list.
It enables also to send a key without value and updates the values in
the following ways:
- "key=value,key,..." - value is updated as NULL.
- "key=value,key=,..." - value is updated as "" (empty string).

Mlx5 PMDs use this function to parse, but they don't support key without
value. They send the value as an argument to strtol function.
When strtol gets NULL as a parameter it cause a crash, when it gets ""
(empty string) it returns 0.

Adds a check that will prevent an argument in these formats, and returns
an error for it.

Fixes: 85209924039c ("common/mlx5: share memory related devargs")
Cc: stable@dpdk.org

Signed-off-by: Michael Baum <michaelba@nvidia.com>
Acked-by: Matan Azrad <matan@nvidia.com>

show more ...

common/mlx5: fix error handling in multi-class probe

The common drivers_probe function calls in a loop to all probe functions
for classes requested by the user. After it manages to probe them all,
i

common/mlx5: fix error handling in multi-class probe

The common drivers_probe function calls in a loop to all probe functions
for classes requested by the user. After it manages to probe them all,
it updates this on the device in the "classes_loaded" field.

If one of them fails, all those probed to it are remove using the
drivers_remove function. However, this function only releases the
classes in the "classes_loaded" field on the given device and misses the
newly probed classes.

This patch removes the condition from the release function, and ensures
that the caller function sends a more accurate parameter.

Fixes: 8a41f4deccc3 ("common/mlx5: introduce layer for multiple class drivers")
Cc: stable@dpdk.org

Signed-off-by: Michael Baum <michaelba@nvidia.com>
Acked-by: Matan Azrad <matan@nvidia.com>

show more ...

fix spelling in comments and strings

The tool comes from https://github.com/jsoref

Signed-off-by: Josh Soref <jsoref@gmail.com>
Signed-off-by: Thomas Monjalon <thomas@monjalon.net>

common/mlx5: fix mempool registration

Mempool registration was not correctly processing
mempools with RTE_PKTMBUF_F_PINEND_EXT_BUF flag set
("pinned mempools" for short), because it is not known
at

common/mlx5: fix mempool registration

Mempool registration was not correctly processing
mempools with RTE_PKTMBUF_F_PINEND_EXT_BUF flag set
("pinned mempools" for short), because it is not known
at registration time whether the mempool is a pktmbuf one,
and its elements may not yet be initialized to analyze them.
Attempts had been made to recognize such pools,
but there was no robust solution, only the owner of a mempool
(the application or a device) knows its type.
This patch extends common/mlx5 registration code
to accept a hint that the mempool is a pinned one
and uses this capability from net/mlx5 driver.

1. Remove all code assuming pktmbuf pool type
or trying to recognize the type of a pool.
2. Register pinned mempools used for Rx
and their external memory on port start.
Populate the MR cache with all their MRs.
3. Change Tx slow path logic as follows:
3.1. Search the mempool database for a memory region (MR)
by the mbuf pool and its buffer address.
3.2. If not MR for the address is found for the mempool,
and the mempool contains only pinned external buffers,
perform the mempool registration of the mempool
and its external pinned memory.
3.3. Fall back to using page-based MRs in other cases
(for example, a buffer with externally attached memory,
but not from a pinned mempool).

Fixes: 690b2a88c2f7 ("common/mlx5: add mempool registration facilities")
Fixes: fec28ca0e3a9 ("net/mlx5: support mempool registration")

Signed-off-by: Dmitry Kozlyuk <dkozlyuk@nvidia.com>
Reviewed-by: Matan Azrad <matan@nvidia.com>
Reviewed-by: Viacheslav Ovsiienko <viacheslavo@nvidia.com>

show more ...

common/mlx5: fix MPRQ mempool registration

Mempool registration code had a wrong assumption that it is always
dealing with packet mempools and always called rte_pktmbuf_priv_flags(),
which returned

common/mlx5: fix MPRQ mempool registration

Mempool registration code had a wrong assumption that it is always
dealing with packet mempools and always called rte_pktmbuf_priv_flags(),
which returned a random value for different types of mempools.
In particular, it could consider MPRQ mempools as having externally
pinned buffers, which is wrong.
Packet mempools cannot be reliably recognized, but it is sufficient to
check that the mempool is not a packet one, so it cannot have externally
pinned buffers.
Compare mempool private data size to that of packet mempools to check.

Fixes: 690b2a88c2f7 ("common/mlx5: add mempool registration facilities")
Fixes: fec28ca0e3a9 ("net/mlx5: support mempool registration")

Signed-off-by: Dmitry Kozlyuk <dkozlyuk@nvidia.com>
Acked-by: Viacheslav Ovsiienko <viacheslavo@nvidia.com>

show more ...

common/mlx5: fix external memory pool registration

Registration of packet mempools with RTE_PKTMBUF_POOL_PINNED_EXT_MEM
was performed incorrectly: after population of such mempool chunks
only contai

common/mlx5: fix external memory pool registration

Registration of packet mempools with RTE_PKTMBUF_POOL_PINNED_EXT_MEM
was performed incorrectly: after population of such mempool chunks
only contain memory for rte_mbuf structures, while pointers to actual
external memory are not yet filled. MR LKeys could not be obtained
for external memory addresses of such mempools. Rx datapath assumes
all used mempools are registered and does not fallback to dynamic
MR creation in such case, so no packets could be received.

Skip registration of extmem pools on population because it is useless.
If used for Rx, they are registered at port start.
During registration, recognize such pools, inspect their mbufs
and recover the pages they reside in.

While MRs for these pages may already be created by rte_dev_dma_map(),
they are not reused to avoid synchronization on Rx datapath
in case these MRs are changed in the database.

Fixes: 690b2a88c2f7 ("common/mlx5: add mempool registration facilities")

Signed-off-by: Dmitry Kozlyuk <dkozlyuk@nvidia.com>
Reviewed-by: Matan Azrad <matan@nvidia.com>
Reviewed-by: Viacheslav Ovsiienko <viacheslavo@nvidia.com>

show more ...

common/mlx5: fix post doorbell barrier

The rdma-core library can map doorbell register in two ways, depending
on the environment variable "MLX5_SHUT_UP_BF":

- as regular cached memory, the variab

common/mlx5: fix post doorbell barrier

The rdma-core library can map doorbell register in two ways, depending
on the environment variable "MLX5_SHUT_UP_BF":

- as regular cached memory, the variable is either missing or set to
zero. This type of mapping may cause the significant doorbell
register writing latency and requires an explicit memory write
barrier to mitigate this issue and prevent write combining.

- as non-cached memory, the variable is present and set to not "0"
value. This type of mapping may cause performance impact under
heavy loading conditions but the explicit write memory barrier is
not required and it may improve core performance.

The UAR creation function maps a doorbell in one of the above ways
according to the system. In run time, it always adds an explicit memory
barrier after writing to.
In cases where the doorbell was mapped as non-cached memory, the
explicit memory barrier is unnecessary and may impair performance.

The commit [1] solved this problem for a Tx queue. In run time, it
checks the mapping type and provides the memory barrier after writing to
a Tx doorbell register if it is needed. The mapping type is extracted
directly from the uar_mmap_offset field in the queue properties.

This patch shares this code between the drivers and extends the above
solution for each of them.

[1] commit 8409a28573d3
("net/mlx5: control transmit doorbell register mapping")

Fixes: f8c97babc9f4 ("compress/mlx5: add data-path functions")
Fixes: 8e196c08ab53 ("crypto/mlx5: support enqueue/dequeue operations")
Fixes: 4d4e245ad637 ("regex/mlx5: support enqueue")
Cc: stable@dpdk.org

Signed-off-by: Michael Baum <michaelba@nvidia.com>
Reviewed-by: Viacheslav Ovsiienko <viacheslavo@nvidia.com>
Acked-by: Matan Azrad <matan@nvidia.com>

show more ...

common/mlx5: fix doorbell mapping configuration

UAR mapping type can be affected by the devarg tx_db_nc, which can cause
setting the environment variable MLX5_SHUT_UP_BF.
So, the MLX5_SHUT_UP_BF val

common/mlx5: fix doorbell mapping configuration

UAR mapping type can be affected by the devarg tx_db_nc, which can cause
setting the environment variable MLX5_SHUT_UP_BF.
So, the MLX5_SHUT_UP_BF value and the UAR mapping parameter affect the
UAR cache mode.

Wrongly, the devarg was considered for the MLX5_SHUT_UP_BF but not for
the UAR mapping parameter in all the drivers except the net.

Take the tx_db_nc devarg into account for all the drivers.

Fixes: ca1418ce3910 ("common/mlx5: share device context object")
Cc: stable@dpdk.org

Signed-off-by: Michael Baum <michaelba@nvidia.com>
Reviewed-by: Viacheslav Ovsiienko <viacheslavo@nvidia.com>
Acked-by: Matan Azrad <matan@nvidia.com>

show more ...

common/mlx5: fix UAR allocation diagnostics messages

Depending on kernel capabilities and rdma-core version the mapping of
UAR (User Access Region) of desired memory caching type (non-cached or
writ

common/mlx5: fix UAR allocation diagnostics messages

Depending on kernel capabilities and rdma-core version the mapping of
UAR (User Access Region) of desired memory caching type (non-cached or
write combining) might fail. The PMD implements the flexible strategy
of UAR mapping, alternating the type of caching to succeed.

During this process the failure diagnostics messages are emitted.
These messages are merely diagnostics ones and the logging level should
be adjusted to DEBUG.

Fixes: 9cc0e99c81ab0 ("common/mlx5: share UAR allocation routine")
Cc: stable@dpdk.org

Signed-off-by: Michael Baum <michaelba@nvidia.com>
Reviewed-by: Viacheslav Ovsiienko <viacheslavo@nvidia.com>
Acked-by: Matan Azrad <matan@nvidia.com>

show more ...

common/mlx5: remove unreachable branch in UAR allocation

The User Access Region (UAR) provides access to the hardware resources
like Doorbell Register from userspace.
It means the resources should b

common/mlx5: remove unreachable branch in UAR allocation

The User Access Region (UAR) provides access to the hardware resources
like Doorbell Register from userspace.
It means the resources should be mapped by the kernel to some virtual
address range. There two types of memory mapping are supported by mlx5
kernel driver:

MLX5DV_UAR_ALLOC_TYPE_NC - non-cached, all writes promoted directly to
hardware.
MLX5DV_UAR_ALLOC_TYPE_BF - "BlueFlame", all writes might be cached by
CPU, and will be flushed to hardware
explicitly with memory barriers.

The supported mapping types depend on the platform (x86/ARM/etc), kernel
version, driver version, virtualization environment (hypervisor), etc.

In UAR allocation, if the system supports the allocation with non-cached
mapping, the first attempt is performed with MLX5DV_UAR_ALLOC_TYPE_NC.
Then, if this fails, the next attempt is done with
MLX5DV_UAR_ALLOC_TYPE_BF.

However, the function adds a condition for the case where the first
attempt was performed with MLX5DV_UAR_ALLOC_TYPE_BF, a condition that is
unattainable since the first attempt was always performed with
MLX5DV_UAR_ALLOC_TYPE_NC.

Remove the unreachable code.

Fixes: 9cc0e99c81ab0 ("common/mlx5: share UAR allocation routine")
Cc: stable@dpdk.org

Signed-off-by: Michael Baum <michaelba@nvidia.com>
Reviewed-by: Viacheslav Ovsiienko <viacheslavo@nvidia.com>
Acked-by: Matan Azrad <matan@nvidia.com>

show more ...

common/mlx5: make multi-process MR management port-agnostic

In the multi-process mechanism, there are things that the secondary
process does not perform itself but asks the primary process to perfor

common/mlx5: make multi-process MR management port-agnostic

In the multi-process mechanism, there are things that the secondary
process does not perform itself but asks the primary process to perform
for it.
There is a special API for communication between the processes that
receives parameters necessary for the specific action required as well
as a special structure called mp_id that contains the port number of the
processes through which the initial process finds the relevant ETH
device for the processes.

One of the operations performed through this mechanism is the creation
of a memory region, where the secondary process sends the virtual
address as a parameter and the mp_id structure with the port number
inside it.
However, once the memory area management is shared between the drivers
and either port number or ETH device is no longer relevant to them, it
seems unnecessary to continue communicating between the processes
through the mp_id variable.

In this patch we will remove the use of the above structure for all MR
management, and add to the specific parameter of operations a pointer to
the common device that contains everything needed to create/register MR.

Fixes: 9f1d636f3ef08 ("common/mlx5: share MR management")

Signed-off-by: Michael Baum <michaelba@nvidia.com>
Reviewed-by: Viacheslav Ovsiienko <viacheslavo@nvidia.com>
Reviewed-by: Dmitry Kozlyuk <dkozlyuk@nvidia.com>
Acked-by: Matan Azrad <matan@nvidia.com>

show more ...

common/mlx5: share MR mempool registration

Expand the use of mempool registration to MR management for other
drivers.

Signed-off-by: Michael Baum <michaelba@nvidia.com>
Acked-by: Matan Azrad <matan

common/mlx5: share MR mempool registration

Expand the use of mempool registration to MR management for other
drivers.

Signed-off-by: Michael Baum <michaelba@nvidia.com>
Acked-by: Matan Azrad <matan@nvidia.com>

show more ...

common/mlx5: support device DMA map and unmap

Since MR management has moved to the common area, there is no longer a
need for the DMA map and unmap function for each driver.
This patch share those f

common/mlx5: support device DMA map and unmap

Since MR management has moved to the common area, there is no longer a
need for the DMA map and unmap function for each driver.
This patch share those functions. For most drivers it supports these
operations for the first time.

Signed-off-by: Michael Baum <michaelba@nvidia.com>
Acked-by: Matan Azrad <matan@nvidia.com>

show more ...

common/mlx5: share MR management

Add global shared MR cache as a field of common device structure.
Move MR management to use this global cache for all drivers.

Signed-off-by: Michael Baum <michaelb

common/mlx5: share MR management

Add global shared MR cache as a field of common device structure.
Move MR management to use this global cache for all drivers.

Signed-off-by: Michael Baum <michaelba@nvidia.com>
Acked-by: Matan Azrad <matan@nvidia.com>

show more ...