ethdev: move RSS expansion code to mlx5 driver

Patch [1] added support for RSS flow expansion.
It was added in ethdev for public use, but until now it is used only
by MLX5 PMD.
To allow local change

ethdev: move RSS expansion code to mlx5 driver

Patch [1] added support for RSS flow expansion.
It was added in ethdev for public use, but until now it is used only
by MLX5 PMD.
To allow local changes in this code, this patch removes it from ethdev
and moves it to MLX5 PMD file.

[1] commit 4ed05fcd441b ("ethdev: add flow API to expand RSS flows")

Signed-off-by: Dekel Peled <dekelp@nvidia.com>
Acked-by: Matan Azrad <matan@nvidia.com>
Reviewed-by: Ferruh Yigit <ferruh.yigit@intel.com>

show more ...

eal: remove deprecated coherent IO memory barriers

Since the 20.08 release deprecated rte_cio_*mb APIs because these APIs
provide the same functionality as rte_io_*mb APIs on all platforms, so
remov

eal: remove deprecated coherent IO memory barriers

Since the 20.08 release deprecated rte_cio_*mb APIs because these APIs
provide the same functionality as rte_io_*mb APIs on all platforms, so
remove them and use rte_io_*mb instead.

Signed-off-by: Phil Yang <phil.yang@arm.com>
Signed-off-by: Joyce Kong <joyce.kong@arm.com>
Reviewed-by: Ruifeng Wang <ruifeng.wang@arm.com>
Reviewed-by: Honnappa Nagarahalli <honnappa.nagarahalli@arm.com>
Acked-by: David Marchand <david.marchand@redhat.com>

show more ...

ethdev: remove underscore prefix from internal API

'_rte_eth_dev_callback_process()' & '_rte_eth_dev_reset()' internal APIs
has unconventional underscore ('_') prefix.
Although this is not documente

ethdev: remove underscore prefix from internal API

'_rte_eth_dev_callback_process()' & '_rte_eth_dev_reset()' internal APIs
has unconventional underscore ('_') prefix.
Although this is not documented most probably this is to mark them as
internal. Since we have '__rte_internal' flag to mark this, removing '_'
from API names.

For '_rte_eth_dev_reset()', there is already a public API named
'rte_eth_dev_reset()', so renaming '_rte_eth_dev_reset()' to
'rte_eth_dev_internal_reset'.

Signed-off-by: Ferruh Yigit <ferruh.yigit@intel.com>
Acked-by: Andrew Rybchenko <arybchenko@solarflare.com>
Acked-by: David Marchand <david.marchand@redhat.com>
Acked-by: Sachin Saxena <sachin.saxena@nxp.com>

show more ...

ethdev: move inline device operations

This patch is a preparation to hide the 'struct eth_dev_ops' from
applications by moving some device operations from 'struct eth_dev_ops'
to 'struct rte_eth_dev

ethdev: move inline device operations

This patch is a preparation to hide the 'struct eth_dev_ops' from
applications by moving some device operations from 'struct eth_dev_ops'
to 'struct rte_eth_dev'.

Mentioned ethdev APIs are in the data path and implemented as inline
because of performance reasons.

Exposing 'struct eth_dev_ops' to applications is bad because it is a
contract between ethdev and PMDs, not really needs to be known by
applications, also changes in the struct causing ABI breakages which
shouldn't.

To be able to both keep APIs inline and hide the 'struct eth_dev_ops',
moving device operations used in ethdev inline APIs to 'struct
rte_eth_dev' to the same level with Rx/Tx burst functions.

The list of dev_ops moved:
eth_rx_queue_count_t rx_queue_count;
eth_rx_descriptor_done_t rx_descriptor_done;
eth_rx_descriptor_status_t rx_descriptor_status;
eth_tx_descriptor_status_t tx_descriptor_status;

Signed-off-by: Ferruh Yigit <ferruh.yigit@intel.com>
Reviewed-by: Andrew Rybchenko <arybchenko@solarflare.com>
Acked-by: David Marchand <david.marchand@redhat.com>
Acked-by: Sachin Saxena <sachin.saxena@nxp.com>

show more ...

net/mlx5: rename constant conflicting with Windows

Enumerated variable REG_NONE (defined in mlx5_prm.h) is in conflict with
Windows definition (winnt.h): #define REG_NONE ( 0ul ) // No value type
To

net/mlx5: rename constant conflicting with Windows

Enumerated variable REG_NONE (defined in mlx5_prm.h) is in conflict with
Windows definition (winnt.h): #define REG_NONE ( 0ul ) // No value type
To enable mlx5 PMD Windows compilation - rename REG_NONE as REG_NON.

Signed-off-by: Ophir Munk <ophirmu@mellanox.com>
Acked-by: Matan Azrad <matan@mellanox.com>

show more ...

net/mlx5: remove ineffective increment in hairpin split

The flow_hairpin_split function defines a pointer called addr that
points to the list of items.
When the function wants to progress in the lis

net/mlx5: remove ineffective increment in hairpin split

The flow_hairpin_split function defines a pointer called addr that
points to the list of items.
When the function wants to progress in the list, it adds the size of an
item to the pointer.

At the end of the function, it precedes the pointer one more time even
though it is not used afterwards. In fact, this line is unaffected and
the operation of the function would have been no different without it.

Remove the line where the pointer is preceded unnecessarily.

Fixes: d85c7b5ea59f ("net/mlx5: split hairpin flows")
Cc: stable@dpdk.org

Signed-off-by: Michael Baum <michaelba@mellanox.com>
Acked-by: Matan Azrad <matan@mellanox.com>

show more ...

net/mlx5: fix initialization of steering registers

The mlx5_flow_action_copy_mreg structure contains a field called src
type enum modify_reg, similarly the mlx5_rte_flow_item_tag field
contains a fi

net/mlx5: fix initialization of steering registers

The mlx5_flow_action_copy_mreg structure contains a field called src
type enum modify_reg, similarly the mlx5_rte_flow_item_tag field
contains a field called id type enum modify_reg.
The enum modify_reg variable represents different registers in the
system and it also has a field called REG_NONE whose value is 0 which
means that the register does not exist.

The flow_mreg_add_copy_action function sets a variable of struct
mlx5_flow_action_copy_mreg type, and initializes the src field to be 0.
Similarly the flow_create_split_metadata function sets a variable of
struct mlx5_rte_flow_item_tag type and initializes the id field to be 0.
In both functions, they initialize a enum modify_reg type variable with
an int type value while modify_reg has an appropriate field for that
value (REG_NONE).

Replace assigning 0 with REG_NONE in both functions.

Fixes: dd3c774f6ffb ("net/mlx5: add metadata register copy table")
Fixes: 71e254bc0294 ("net/mlx5: split Rx flows to provide metadata copy")
Cc: stable@dpdk.org

Signed-off-by: Michael Baum <michaelba@mellanox.com>
Acked-by: Matan Azrad <matan@mellanox.com>

show more ...

net/mlx5: fix counter query

Currently, the counter query requires the counter ID should start
with 4 aligned. In none-batch mode, the counter pool might have the
chance to get the counter ID not 4 a

net/mlx5: fix counter query

Currently, the counter query requires the counter ID should start
with 4 aligned. In none-batch mode, the counter pool might have the
chance to get the counter ID not 4 aligned. In this case, the counter
should be skipped, or the query will be failed.

Skip the counter with ID not 4 aligned as the first counter in the
none-batch count pool to avoid invalid counter query. Once having
new min_dcs ID in the poll less than the skipped counters, the
skipped counters will be returned to the pool free list to use.

Fixes: 5382d28c2110 ("net/mlx5: accelerate DV flow counter transactions")
Cc: stable@dpdk.org

Signed-off-by: Suanming Mou <suanmingm@mellanox.com>
Acked-by: Matan Azrad <matan@mellanox.com>

show more ...

net/mlx5: fix VLAN push action on hairpin queue

Push VLAN action is allowed on Tx only, same as encap action.
Flow rules for hairpin queue are created on Rx, and split
by PMD to Rx and Tx rules, acc

net/mlx5: fix VLAN push action on hairpin queue

Push VLAN action is allowed on Tx only, same as encap action.
Flow rules for hairpin queue are created on Rx, and split
by PMD to Rx and Tx rules, according to the above limitation.
In current implementation the encap action is split to Tx rule.
This patch adds the same handling for push-vlan action, as well as
its complementing actions set-vlan-vid and set-vlan-pcp.

Fixes: d85c7b5ea59f ("net/mlx5: split hairpin flows")
Cc: stable@dpdk.org

Signed-off-by: Dekel Peled <dekelp@mellanox.com>
Acked-by: Matan Azrad <matan@mellanox.com>

show more ...

common/mlx5: remove inclusion of Verbs header files

Several source files include Verbs header files as in (1). These source
files will not compile under non-Linux operating systems. This commit
remo

common/mlx5: remove inclusion of Verbs header files

Several source files include Verbs header files as in (1). These source
files will not compile under non-Linux operating systems. This commit
removes this inclusion in two cases:

Case 1: There is no usage of ibv_* or mlx5dv_* symbols in the source
file so the inclusion in (1) can be safely removed.

Case 2: Verbs symbols are used. Please note the inclusion in (1) already
appears in file linux/mlx5_glue.h (which represents the interface
to the rdma-core library). Therefore, replace (1) in the source file
with (2). Under non-Linux operating systems - file mlx5_glue.h will not
include (1).

(1)
#include <infiniband/verbs.h>
#include <infiniband/mlx5dv.h>

(2)
#include <mlx5_glue.h>

Signed-off-by: Ophir Munk <ophirmu@mellanox.com>
Acked-by: Matan Azrad <matan@mellanox.com>

show more ...

net/mlx5: move flow priority discovery to Verbs file

Function calls mlx5_flow_adjust_priority() and
mlx5_flow_discover_priorities() are Verbs based. Move them from file
mlx5_flow.c to file mlx5_flow

net/mlx5: move flow priority discovery to Verbs file

Function calls mlx5_flow_adjust_priority() and
mlx5_flow_discover_priorities() are Verbs based. Move them from file
mlx5_flow.c to file mlx5_flow_verbs.c

Signed-off-by: Ophir Munk <ophirmu@mellanox.com>
Acked-by: Matan Azrad <matan@mellanox.com>

show more ...

net/mlx5: convert control path memory to unified malloc

This commit allocates the control path memory from unified malloc
function.

The objects be changed:

1. hlist;
2. rss key;
3. vlan vmwa;
4. i

net/mlx5: convert control path memory to unified malloc

This commit allocates the control path memory from unified malloc
function.

The objects be changed:

1. hlist;
2. rss key;
3. vlan vmwa;
4. indexed pool;
5. fdir objects;
6. meter profile;
7. flow counter pool;
8. hrxq and indirect table;
9. flow object cache resources;
10. temporary resources in flow create;

Signed-off-by: Suanming Mou <suanmingm@mellanox.com>
Acked-by: Matan Azrad <matan@mellanox.com>

show more ...

net/mlx5: add flow validation of eCPRI header

When creating a flow with eCPRI header item, the validation of it is
mandatory. The detailed limitations are listed below:
1. Over Ether / VLAN, ether

net/mlx5: add flow validation of eCPRI header

When creating a flow with eCPRI header item, the validation of it is
mandatory. The detailed limitations are listed below:
1. Over Ether / VLAN, ethertype must be 0xAEFE.
2. No tunnel support is described in the specification now.
3. L3 layer is only supported when L4 is UDP, see #4.
4. Over TCP is not supported from the specification, and over UDP
is not supported right now.
5. Concatenation indicator matching is not supported now.
6. No need to check the revision.
7. Only type field in the common header is mandatory, and one byte
should be matched integrally.
8. Fields in the message payload header are optional.
9. Only messages with type #0, #2 and #5 are supported now.

Some limitations are only from software right now, because there is
no need to support all the message types and variants of protocol
stack listed in the specification.

Signed-off-by: Bing Zhao <bingz@mellanox.com>
Acked-by: Viacheslav Ovsiienko <viacheslavo@mellanox.com>

show more ...

net/mlx5: add OS specific flow type selection

In current implementation the flow type (DV/Verbs) is selected
using dedicated function flow_get_drv_type().

This patch adds OS specific function mlx5_

net/mlx5: add OS specific flow type selection

In current implementation the flow type (DV/Verbs) is selected
using dedicated function flow_get_drv_type().

This patch adds OS specific function mlx5_flow_os_get_type(), to
allow OS specific flow type selection.
The new function is called by flow_get_drv_type(), and if it returns a
valid value (DV/Verbs) no more logic is required.
Otherwise the existing logic is executed.

Signed-off-by: Dekel Peled <dekelp@mellanox.com>
Acked-by: Matan Azrad <matan@mellanox.com>

show more ...

net/mlx5: add default miss action to flow engine

The new action is an internal mlx5 action that will call
the rdma-core function MLX5DV_FLOW_ACTION_DEFAULT_MISS.

The default miss action will be use

net/mlx5: add default miss action to flow engine

The new action is an internal mlx5 action that will call
the rdma-core function MLX5DV_FLOW_ACTION_DEFAULT_MISS.

The default miss action will be used when a bond is
configured to allow traffic related to the bond to
be managed in the kernel.

Signed-off-by: Shiri Kuzin <shirik@mellanox.com>
Acked-by: Matan Azrad <matan@mellanox.com>

show more ...

net/mlx5: optimize free counter lookup

Currently, when allocate a new counter, it needs loop the whole
container pool list to get a free counter.

In the case with millions of counters allocated, an

net/mlx5: optimize free counter lookup

Currently, when allocate a new counter, it needs loop the whole
container pool list to get a free counter.

In the case with millions of counters allocated, and all the pools
are empty, allocate the new counter will still need to loop the
whole container pool list first, then allocate a new pool to get a
free counter. It wastes the cycles during the pool list traversal.

Add a global free counter list in the container helps to get the free
counters more efficiently.

Signed-off-by: Suanming Mou <suanmingm@mellanox.com>
Acked-by: Matan Azrad <matan@mellanox.com>

show more ...

net/mlx5: refactor device operations for Linux

There are three types of eth_dev_ops: primary, secondary and isolate.
Their function calls assignments are moved from common file
mlx5.c to the Linux s

net/mlx5: refactor device operations for Linux

There are three types of eth_dev_ops: primary, secondary and isolate.
Their function calls assignments are moved from common file
mlx5.c to the Linux specific file linux/mlx5_os.c.

Signed-off-by: Ophir Munk <ophirmu@mellanox.com>
Acked-by: Matan Azrad <matan@mellanox.com>

show more ...

net/mlx5: rename ib in names

Renames in this commit:
mlx5_ibv_list -> mlx5_dev_ctx_list
mlx5_alloc_shared_ibctx -> mlx5_alloc_shared_dev_ctx
mlx5_free_shared_ibctx -> mlx5_free_shared_dev_ctx
mlx5_i

net/mlx5: rename ib in names

Renames in this commit:
mlx5_ibv_list -> mlx5_dev_ctx_list
mlx5_alloc_shared_ibctx -> mlx5_alloc_shared_dev_ctx
mlx5_free_shared_ibctx -> mlx5_free_shared_dev_ctx
mlx5_ibv_shared_port -> mlx5_dev_shared_port
ibv_port -> dev_port

Signed-off-by: Ophir Munk <ophirmu@mellanox.com>
Acked-by: Matan Azrad <matan@mellanox.com>

show more ...

net/mlx5: rename Verbs shared object

Replace all 'mlx5_ibv_shared' appearances with 'mlx5_dev_ctx_shared'.

Signed-off-by: Ophir Munk <ophirmu@mellanox.com>
Acked-by: Matan Azrad <matan@mellanox.com>

net/mlx5: fix unreachable MPLS error path

The mlx5_flow_validate_item_mpls function checks MPLS item validation.
It first checks if the device supports MPLS, it is done using the ifdef
condition tha

net/mlx5: fix unreachable MPLS error path

The mlx5_flow_validate_item_mpls function checks MPLS item validation.
It first checks if the device supports MPLS, it is done using the ifdef
condition that if it fails to skip to endif and return the appropriate
error.

When MPLS is supported, the preprocessor will copy the body of the
function ending with return 0 followed by the lines that report MPLS
support.
In fact, these lines are unreachable because before them the function
returns 0 and in any case they are unnecessary.

Replace the endif by else and move endif to the end of the
function.

Fixes: 23c1d42c7138 ("net/mlx5: split flow validation to dedicated function")
Cc: stable@dpdk.org

Signed-off-by: Michael Baum <michaelba@mellanox.com>
Acked-by: Matan Azrad <matan@mellanox.com>

show more ...

net/mlx5: fix flow counter container resize

The design of counter container resize used double buffer algorithm in
order to synchronize between the query thread to the control thread.
When the contr

net/mlx5: fix flow counter container resize

The design of counter container resize used double buffer algorithm in
order to synchronize between the query thread to the control thread.
When the control thread detected resize need, it created new bigger
buffer for the counter pools in a new container and change the container
index atomically.
In case the query thread had not detect the previous resize before a new
one need was detected by the control thread, the control thread returned
EAGAIN to the flow creation API used a COUNT action.

The rte_flow API doesn't allow unblocked commands and doesn't expect to
get EAGAIN error type.

So, when a lot of flows were created between 2 different periodic
queries, 2 different resizes might try to be created and caused EAGAIN
error.
This behavior may blame flow creations.

Change the synchronization way to use lock instead of double buffer
algorithm.

The critical section of this lock is very small, so flow insertion
rate should not be decreased.

Fixes: ebbac312e448 ("net/mlx5: resize a full counter container")
Cc: stable@dpdk.org

Signed-off-by: Matan Azrad <matan@mellanox.com>
Acked-by: Viacheslav Ovsiienko <viacheslavo@mellanox.com>

show more ...

net/mlx5: fix meter color register consideration

The mlx5_flow_get_reg_id() function translates tag ID to register
from the registers that are supported and available for use. The
user does not know

net/mlx5: fix meter color register consideration

The mlx5_flow_get_reg_id() function translates tag ID to register
from the registers that are supported and available for use. The
user does not know which register is available at a time and therefore
there is an array that represents mapping to the available registers.
Usually the free registers are continuous in the flow_mreg_c array but
sometimes the mtr_color_reg register is between them and it must be
skipped and the next register returned, in which case the function
returns the mapping of the next entity in the array.

When the function reads from the next entity in the array, it does not
check whether such an entity exists and in some situation invalid access
to memory occurs beyond the array boundaries.

So, when all the registers are valid from HW perspective and the meter
color register is not the default, the tag id 5 causes an out of bound
access.

Validate registers availability when meter color register is not the
default.

Coverity issue: 146355
Fixes: 792e749e92d5 ("net/mlx5: fix register usage in meter")
Cc: stable@dpdk.org

Signed-off-by: Michael Baum <michaelba@mellanox.com>
Acked-by: Matan Azrad <matan@mellanox.com>

show more ...

net/mlx5: fix gcc 10 enum-conversion warning

gcc 10.0.1 reports warnings when using mlx5_rte_flow enums
with rte_flow type enums. For example:

../drivers/net/mlx5/mlx5_flow.c: In function ‘flow_hai

net/mlx5: fix gcc 10 enum-conversion warning

gcc 10.0.1 reports warnings when using mlx5_rte_flow enums
with rte_flow type enums. For example:

../drivers/net/mlx5/mlx5_flow.c: In function ‘flow_hairpin_split’:
../drivers/net/mlx5/mlx5_flow.c:3406:19:
warning: implicit conversion from ‘enum mlx5_rtedflow_action_type’ to
‘enum rte_flow_action_type’ [-Wenum-conversion]
3406 | tag_action->type = MLX5_RTE_FLOW_ACTION_TYPE_TAG;
| ^
../drivers/net/mlx5/mlx5_flow.c:3419:13:
warning: implicit conversion from ‘enum mlx5_rte_flow_item_type’
to ‘enum rte_flow_item_type’ [-Wenum-conversion]
3419 | item->type = MLX5_RTE_FLOW_ITEM_TYPE_TAG;
| ^

Fix by casting to the correct enum.

Signed-off-by: Kevin Traynor <ktraynor@redhat.com>
Acked-by: Viacheslav Ovsiienko <viacheslavo@mellanox.com>

show more ...

net/mlx5: support flow aging

Currently, there is no flow aging check and age-out event callback
mechanism for mlx5 driver, this patch implements it. It's included:
- Splitting the current counter co

net/mlx5: support flow aging

Currently, there is no flow aging check and age-out event callback
mechanism for mlx5 driver, this patch implements it. It's included:
- Splitting the current counter container to aged or no-aged container
since reducing memory consumption. Aged container will allocate extra
memory to save the aging parameter from user configuration.
- Aging check and age-out event callback mechanism based on current
counter. When a flow be checked aged-out, RTE_ETH_EVENT_FLOW_AGED
event will be triggered to applications.
- Implement the new API: rte_flow_get_aged_flows, applications can use
this API to get aged flows.

Signed-off-by: Dong Zhou <dongz@mellanox.com>
Acked-by: Matan Azrad <matan@mellanox.com>

show more ...

net/mlx5: fix RSS enablement

PMD create some default control rules with RSS action
if it's not isolated mode.

However whether default control rules need to do RSS or not should be
controlled by dev

net/mlx5: fix RSS enablement

PMD create some default control rules with RSS action
if it's not isolated mode.

However whether default control rules need to do RSS or not should be
controlled by device configuration, the mq_mode of rxmode configuration
in specific.

In another word, only when mq_mode is configured with ETH_MQ_RX_RSS_FLAG
set, then RSS is needed for default rules.

Fixes: c64ccc0eca2f ("mlx5: fix overwritten RSS configuration")
Cc: stable@dpdk.org

Signed-off-by: Xiaoyu Min <jackmin@mellanox.com>
Signed-off-by: Viacheslav Ovsiienko <viacheslavo@mellanox.com>
Acked-by: Viacheslav Ovsiienko <viacheslavo@mellanox.com>
Acked-by: Matan Azrad <matan@mellanox.com>

show more ...