dev: hide driver object

Make rte_driver opaque for non internal users.
This will make extending this object possible without breaking the ABI.

Introduce a new driver header and move rte_driver defi

dev: hide driver object

Make rte_driver opaque for non internal users.
This will make extending this object possible without breaking the ABI.

Introduce a new driver header and move rte_driver definition.
Update drivers and library to use the internal header.

Some applications may have been dereferencing rte_driver objects, mark
this object's accessors as stable.

Signed-off-by: David Marchand <david.marchand@redhat.com>
Acked-by: Bruce Richardson <bruce.richardson@intel.com>
Acked-by: Jay Jayatheerthan <jay.jayatheerthan@intel.com>
Acked-by: Ajit Khaparde <ajit.khaparde@broadcom.com>
Acked-by: Akhil Goyal <gakhil@marvell.com>
Acked-by: Abhinandan Gujjar <abhinandan.gujjar@intel.com>

show more ...

common/mlx5: fix redundant field in MR control structure

Inside the MR control structure there is a pointer to the common device.
This pointer enables access to the global cache as well as hardware

common/mlx5: fix redundant field in MR control structure

Inside the MR control structure there is a pointer to the common device.
This pointer enables access to the global cache as well as hardware
objects that may be required in case a new MR needs to be created.

The purpose of adding this pointer into the MR control structure was to
avoid its transfer as a parameter to all the functions of searching MR
in the caches.
However, adding it to this structure increased the Rx and Tx data-path
structures, all the fields that followed it were slightly moved away
which caused to a reduction in performance.

This patch removes the pointer from the structure. It can be accessed
through the "dev_gen_ptr" existing field using the "container_of"
operator.

Fixes: 334ed198ab4d ("common/mlx5: remove redundant parameter in MR search")

Signed-off-by: Michael Baum <michaelba@nvidia.com>
Acked-by: Matan Azrad <matan@nvidia.com>

show more ...

common/mlx5: fix queue size in DevX queue pair creation

The number of WQEBBs was provided to QP create, and QP size was calculated
by multiplying the number of WQEBBs by 64, which is the send WQE si

common/mlx5: fix queue size in DevX queue pair creation

The number of WQEBBs was provided to QP create, and QP size was calculated
by multiplying the number of WQEBBs by 64, which is the send WQE size.
When creating RQ in the QP (i.e., vdpa driver), the queue size was bigger
because the receive WQE size is 16.
Provide queue size to QP create instead of the number of WQEBBs.

Fixes: f9213ab12cf9 ("common/mlx5: share DevX queue pair operations")

Signed-off-by: Raja Zidane <rzidane@nvidia.com>
Acked-by: Matan Azrad <matan@nvidia.com>

show more ...

crypto/mlx5: fix queue size configuration

The DevX interface for QP creation expects the number of WQEBBs.
Wrongly, the number of descriptors was provided to the QP creation.
In addition, the QP siz

crypto/mlx5: fix queue size configuration

The DevX interface for QP creation expects the number of WQEBBs.
Wrongly, the number of descriptors was provided to the QP creation.
In addition, the QP size must be a power of 2 what was not guaranteed.
Provide the number of WQEBBs to the QP creation API.
Round up the SQ size to a power of 2.
Rename (sq/rq)_size to num_of_(send/receive)_wqes.

Fixes: 6152534e211e ("crypto/mlx5: support queue pairs operations")
Cc: stable@dpdk.org

Signed-off-by: Raja Zidane <rzidane@nvidia.com>
Acked-by: Matan Azrad <matan@nvidia.com>
Acked-by: Tal Shnaiderman <talshn@nvidia.com>

show more ...

common/mlx5: fix post doorbell barrier

The rdma-core library can map doorbell register in two ways, depending
on the environment variable "MLX5_SHUT_UP_BF":

- as regular cached memory, the variab

common/mlx5: fix post doorbell barrier

The rdma-core library can map doorbell register in two ways, depending
on the environment variable "MLX5_SHUT_UP_BF":

- as regular cached memory, the variable is either missing or set to
zero. This type of mapping may cause the significant doorbell
register writing latency and requires an explicit memory write
barrier to mitigate this issue and prevent write combining.

- as non-cached memory, the variable is present and set to not "0"
value. This type of mapping may cause performance impact under
heavy loading conditions but the explicit write memory barrier is
not required and it may improve core performance.

The UAR creation function maps a doorbell in one of the above ways
according to the system. In run time, it always adds an explicit memory
barrier after writing to.
In cases where the doorbell was mapped as non-cached memory, the
explicit memory barrier is unnecessary and may impair performance.

The commit [1] solved this problem for a Tx queue. In run time, it
checks the mapping type and provides the memory barrier after writing to
a Tx doorbell register if it is needed. The mapping type is extracted
directly from the uar_mmap_offset field in the queue properties.

This patch shares this code between the drivers and extends the above
solution for each of them.

[1] commit 8409a28573d3
("net/mlx5: control transmit doorbell register mapping")

Fixes: f8c97babc9f4 ("compress/mlx5: add data-path functions")
Fixes: 8e196c08ab53 ("crypto/mlx5: support enqueue/dequeue operations")
Fixes: 4d4e245ad637 ("regex/mlx5: support enqueue")
Cc: stable@dpdk.org

Signed-off-by: Michael Baum <michaelba@nvidia.com>
Reviewed-by: Viacheslav Ovsiienko <viacheslavo@nvidia.com>
Acked-by: Matan Azrad <matan@nvidia.com>

show more ...

common/mlx5: remove redundant parameter in MR search

Memory region management has recently been shared between drivers,
including the search for caches in the data plane.
The initial search in the l

common/mlx5: remove redundant parameter in MR search

Memory region management has recently been shared between drivers,
including the search for caches in the data plane.
The initial search in the local linear cache of the queue, usually
yields a result and one should not continue searching in the next level
caches.

The function that searches in the local cache gets the pointer to a
device as a parameter, that is not necessary for its operation
but for subsequent searches (which, as mentioned, usually do not
happen).
Transferring the device to a function and maintaining it, takes some
time and causes some impact on performance.

Add the pointer to the device as a field of the mr_ctrl structure. The
field will be updated during control path and will be used only when
needed in the search.

Fixes: fc59a1ec556b ("common/mlx5: share MR mempool registration")

Signed-off-by: Michael Baum <michaelba@nvidia.com>
Reviewed-by: Viacheslav Ovsiienko <viacheslavo@nvidia.com>
Reviewed-by: Dmitry Kozlyuk <dkozlyuk@nvidia.com>
Acked-by: Matan Azrad <matan@nvidia.com>

show more ...

regex/mlx5: prevent double setup of queue pair

When mlx5_regex_qp_setup() is called, make sure
the provided QP is not already setup.

Signed-off-by: Ady Agbarih <adypodoman@gmail.com>
Acked-by: Ori

regex/mlx5: prevent double setup of queue pair

When mlx5_regex_qp_setup() is called, make sure
the provided QP is not already setup.

Signed-off-by: Ady Agbarih <adypodoman@gmail.com>
Acked-by: Ori Kam <orika@nvidia.com>

show more ...

regex/mlx5: remove RXP CSR file

The mlx5_rxp_csrs.h file has been deprecated as
its contents has now been moved to FW.

Signed-off-by: Francis Kelly <fkelly@nvidia.com>
Acked-by: Ori Kam <orika@nvid

regex/mlx5: remove RXP CSR file

The mlx5_rxp_csrs.h file has been deprecated as
its contents has now been moved to FW.

Signed-off-by: Francis Kelly <fkelly@nvidia.com>
Acked-by: Ori Kam <orika@nvidia.com>

show more ...

regex/mlx5: add cleanup on stop

When stopping the device we should release all
data allocated.

After rte_regexdev_configure(), the QPs are pre-allocated,
and will be configured only in rte_regexdev

regex/mlx5: add cleanup on stop

When stopping the device we should release all
data allocated.

After rte_regexdev_configure(), the QPs are pre-allocated,
and will be configured only in rte_regexdev_queue_pair_setup().
That's why the QP jobs array initialization is checked
before attempting to destroy the QP.

Signed-off-by: Ori Kam <orika@nvidia.com>
Signed-off-by: Ady Agbarih <adypodoman@gmail.com>

show more ...

common/mlx5: share MR management

Add global shared MR cache as a field of common device structure.
Move MR management to use this global cache for all drivers.

Signed-off-by: Michael Baum <michaelb

common/mlx5: share MR management

Add global shared MR cache as a field of common device structure.
Move MR management to use this global cache for all drivers.

Signed-off-by: Michael Baum <michaelba@nvidia.com>
Acked-by: Matan Azrad <matan@nvidia.com>

show more ...

common/mlx5: add MR control initialization

Add function for MR control structure initialization.
This function include:
- btree initialization.
- dev_gen_ptr initialization.

Signed-off-by: Michae

common/mlx5: add MR control initialization

Add function for MR control structure initialization.
This function include:
- btree initialization.
- dev_gen_ptr initialization.

Signed-off-by: Michael Baum <michaelba@nvidia.com>
Acked-by: Matan Azrad <matan@nvidia.com>

show more ...

common/mlx5: share HCA capabilities handle

Add HCA attributes structure as a field of device config structure.
It query in common probing, and updates the timestamp format fields.

Each driver use H

common/mlx5: share HCA capabilities handle

Add HCA attributes structure as a field of device config structure.
It query in common probing, and updates the timestamp format fields.

Each driver use HCA attributes from common device config structure,
instead of query it for itself.

Signed-off-by: Michael Baum <michaelba@nvidia.com>
Acked-by: Matan Azrad <matan@nvidia.com>

show more ...

common/mlx5: share protection domain object

Create shared Protection Domain in common area and add it and its PDN as
fields of common device structure.

Use this Protection Domain in all drivers and

common/mlx5: share protection domain object

Create shared Protection Domain in common area and add it and its PDN as
fields of common device structure.

Use this Protection Domain in all drivers and remove the PD and PDN
fields from their private structure.

Signed-off-by: Michael Baum <michaelba@nvidia.com>
Acked-by: Matan Azrad <matan@nvidia.com>

show more ...

common/mlx5: share device context object

Create shared context device in common area and add it as a field of
common device.
Use this context device in all drivers and remove the ctx field from
thei

common/mlx5: share device context object

Create shared context device in common area and add it as a field of
common device.
Use this context device in all drivers and remove the ctx field from
their private structure.

Signed-off-by: Michael Baum <michaelba@nvidia.com>
Acked-by: Matan Azrad <matan@nvidia.com>

show more ...

regex/mlx5: refactor HW queue objects

The mlx5 PMD for regex class uses an MMO WQE operated by the GGA engine
in BF devices.
Currently, all the MMO WQEs are managed by the SQ object.
Starting from B

regex/mlx5: refactor HW queue objects

The mlx5 PMD for regex class uses an MMO WQE operated by the GGA engine
in BF devices.
Currently, all the MMO WQEs are managed by the SQ object.
Starting from BF3, the queue of the MMO WQEs should be connected to the
GGA engine using a new configuration, MMO, that will be supported only
in the QP object.
The FW introduced new capabilities to define whether the MMO
configuration should be configured for the GGA queue.
Replace all the GGA queue objects to QP, set MMO configuration according
to the new FW capabilities.

Signed-off-by: Raja Zidane <rzidane@nvidia.com>
Acked-by: Matan Azrad <matan@nvidia.com>

show more ...

regex/mlx5: fix memory region unregistration

The issue can cause illegal physical address access while a huge-page A
is released and huge-page B is allocated on the same virtual address.
The old MR

regex/mlx5: fix memory region unregistration

The issue can cause illegal physical address access while a huge-page A
is released and huge-page B is allocated on the same virtual address.
The old MR can be matched using the virtual address of huge-page B but
the HW will access the physical address of huge-page A which is no more
part of the DPDK process.

Register a driver callback for memory event in order to free out all the
MRs of memory that is going to be freed from the DPDK process.

Fixes: cda883bbb655 ("regex/mlx5: add dynamic memory registration to datapath")
Cc: stable@dpdk.org

Signed-off-by: Michael Baum <michaelba@nvidia.com>
Acked-by: Ori Kam <orika@nvidia.com>

show more ...

regex/mlx5: add data path scattered mbuf process

UMR (User-Mode Memory Registration) WQE can present data buffers
scattered within multiple mbufs with single indirect mkey. Take
advantage of the UMR

regex/mlx5: add data path scattered mbuf process

UMR (User-Mode Memory Registration) WQE can present data buffers
scattered within multiple mbufs with single indirect mkey. Take
advantage of the UMR WQE, scattered mbuf in one operation can be
presented to an indirect mkey. The RegEx which only accepts one
mkey can now process the whole scattered mbuf in one operation.

The maximum scattered mbuf can be supported in one UMR WQE is now
defined as 64. The mbufs from multiple operations can be combined
into one UMR WQE as well if there is enough space in the KLM array,
since the operations can address their own mbuf's content by the
mkey's address and length. However, one operation's scattered mbuf's
can't be placed in two different UMR WQE's KLM array, if the UMR
WQE's KLM does not has enough free space for one operation, the
extra UMR WQE will be engaged.

In case the UMR WQE's indirect mkey will be over wrapped by the SQ's
WQE move, the mkey's index used by the UMR WQE should be the index
of last the RegEX WQE in the operations. As one operation consumes
one WQE set, build the RegEx WQE by reverse helps address the mkey
more efficiently. Once the operations in one burst consumes multiple
mkeys, when the mkey KLM array is full, the reverse WQE set index
will always be the last of the new mkey's for the new UMR WQE.

In GGA mode, the SQ WQE's memory layout becomes UMR/NOP and RegEx
WQE by interleave. The UMR and RegEx WQE can be called as WQE set.
The SQ's pi and ci will also be increased as WQE set not as WQE.

For operations don't have scattered mbuf, uses the mbuf's mkey directly,
the WQE set combination is NOP + RegEx.
For operations have scattered mbuf but share the UMR WQE with others,
the WQE set combination is NOP + RegEx.
For operations complete the UMR WQE, the WQE set combination is UMR +
RegEx.

Signed-off-by: John Hurley <jhurley@nvidia.com>
Signed-off-by: Suanming Mou <suanmingm@nvidia.com>
Acked-by: Ori Kam <orika@nvidia.com>

show more ...

regex/mlx5: support timestamp format

This patch adds support for the timestamp format settings for
the receive and send queues. If the firmware version x.30.1000
or above is installed and the NIC ti

regex/mlx5: support timestamp format

This patch adds support for the timestamp format settings for
the receive and send queues. If the firmware version x.30.1000
or above is installed and the NIC timestamps are configured
with the real-time format, the default zero values for newly
added fields cause the queue creation to fail.

The patch queries the timestamp formats supported by the hardware
and sets the configuration values in queue context accordingly.

Fixes: 92f2c6a30fe0 ("regex/mlx5: add send queue")
Cc: stable@dpdk.org

Signed-off-by: Viacheslav Ovsiienko <viacheslavo@nvidia.com>
Acked-by: Matan Azrad <matan@nvidia.com>
Acked-by: Ori Kam <orika@nvidia.com>

show more ...

regex/mlx5: move DevX SQ creation to common

Using common function for DevX SQ creation.

Signed-off-by: Michael Baum <michaelba@nvidia.com>
Acked-by: Matan Azrad <matan@nvidia.com>

regex/mlx5: move DevX CQ creation to common

Using common function for DevX CQ creation.

Signed-off-by: Michael Baum <michaelba@nvidia.com>
Acked-by: Matan Azrad <matan@nvidia.com>

regex/mlx5: fix leak on queue setup failure

In regex QP setup, the PMD creates some SQ objects.

When SQ object creation is failed, the previous SQ objects memory were
not freed what caused a memory

regex/mlx5: fix leak on queue setup failure

In regex QP setup, the PMD creates some SQ objects.

When SQ object creation is failed, the previous SQ objects memory were
not freed what caused a memory leak.

Free them.

Fixes: 54fa1f6a67d7 ("regex/mlx5: add teardown for fastpath buffers")

Signed-off-by: Michael Baum <michaelba@nvidia.com>
Acked-by: Ori Kam <orika@nvidia.com>

show more ...

regex/mlx5: add dynamic memory registration to datapath

Currently job data is being copied to pre-registered buffer.
To avoid memcpy on the datapath, use dynamic memory registration.

This change wi

regex/mlx5: add dynamic memory registration to datapath

Currently job data is being copied to pre-registered buffer.
To avoid memcpy on the datapath, use dynamic memory registration.

This change will reduce latency when sending regex jobs. The first few
jobs may have high latency due to registration, but assuming all
following mbufs will arrive from the same mempool/hugepage, there will
be no further memory registration.

Signed-off-by: Yuval Avnery <yuvalav@nvidia.com>
Acked-by: Ori Kam <orika@nvidia.com>

show more ...

regex/mlx5: add teardown for fastpath buffers

Added missing code to free Input/Output buffers and memory
registration.
Also added calls to this code in case of error in the qp setup
procedure.
The r

regex/mlx5: add teardown for fastpath buffers

Added missing code to free Input/Output buffers and memory
registration.
Also added calls to this code in case of error in the qp setup
procedure.
The rollback code itself did not handle rollback properly
and did not check return value from the fastpath setup.

Signed-off-by: Yuval Avnery <yuvalav@mellanox.com>
Acked-by: Ori Kam <orika@mellanox.com>

show more ...

regex/mlx5: support dequeue

Implement dequeue function for the regex API.

Signed-off-by: Yuval Avnery <yuvalav@mellanox.com>
Acked-by: Ori Kam <orika@mellanox.com>

regex/mlx5: support enqueue

Will look for a free SQ to send the job on.
doorbell will be given when sq is full, or no more jobs on the burst.

Signed-off-by: Yuval Avnery <yuvalav@mellanox.com>
Acke

regex/mlx5: support enqueue

Will look for a free SQ to send the job on.
doorbell will be given when sq is full, or no more jobs on the burst.

Signed-off-by: Yuval Avnery <yuvalav@mellanox.com>
Acked-by: Ori Kam <orika@mellanox.com>

show more ...