net/mlx5: add basic flow queue operation

The HW steering uses async queue-based flow rules management
mechanism. The matcher and part of the actions have been
prepared during flow table creation. So

net/mlx5: add basic flow queue operation

The HW steering uses async queue-based flow rules management
mechanism. The matcher and part of the actions have been
prepared during flow table creation. Some remaining actions
will be constructed during flow creation if needed.

A flow postpone attribute bit describes if flow management
should be applied to the HW directly. An extra push function
is provided to force push all the cached flows to the HW.

Once the flow has been applied to the HW, the pull function
will be called to get the queued creation/destruction flows.

The DR rule flow memory is represented in PMD layer instead
of allocating from HW steering layer. While destroying the
flow, the flow rule memory can only be freed after the CQE
received.

The HW queue job descriptor is currently introduced to convey
the flow information and operation type between the flow
insertion/destruction in the pull function.

This commit adds the basic flow queue operation for:
rte_flow_async_create();
rte_flow_async_destroy();
rte_flow_push();
rte_flow_pull();

Signed-off-by: Suanming Mou <suanmingm@nvidia.com>
Acked-by: Viacheslav Ovsiienko <viacheslavo@nvidia.com>

show more ...

net/mlx5: add table management

Flow table is a group of flows with the same matching criteria
and the same actions defined for them. The table defines rules
that have the same matching fields but wi

net/mlx5: add table management

Flow table is a group of flows with the same matching criteria
and the same actions defined for them. The table defines rules
that have the same matching fields but with different matching
values. For example, matching on 5 tuple, the table will be
(IPv4 source + IPv4 dest + s_port + d_port + next_proto)
while the values for each rule will be different.

The templates' relevant matching criteria and action instances
will be created in the table creation and saved in the table.
As table attributes indicate the supported flow number, the flow
memory will also be allocated at the same time.

This commit adds the table management functions.

Signed-off-by: Suanming Mou <suanmingm@nvidia.com>
Acked-by: Viacheslav Ovsiienko <viacheslavo@nvidia.com>

show more ...

net/mlx5: add action template management

The action template holds a list of action types that will be
used together on the same rule. The template's actions instances
will be created only when the

net/mlx5: add action template management

The action template holds a list of action types that will be
used together on the same rule. The template's actions instances
will be created only when the template bind to the dedicated
group. And the created actions will be saved to each individual
group in order for best performance. The actions in a group will
not be shared with each other unless shared actions are specified.

This commit adds the action template management which stores the
flow action template.

Signed-off-by: Suanming Mou <suanmingm@nvidia.com>
Acked-by: Viacheslav Ovsiienko <viacheslavo@nvidia.com>

show more ...

net/mlx5: add pattern template management

The pattern template defines flows that have the same matching
fields but with different matching values.
For example, matching on 5 tuple TCP flow, the tem

net/mlx5: add pattern template management

The pattern template defines flows that have the same matching
fields but with different matching values.
For example, matching on 5 tuple TCP flow, the template will be
(eth(null) + IPv4(source + dest) + TCP(s_port + d_port) while
the values for each rule will be different.

Due to the pattern template can be used in different domains, the
items will only be cached in pattern template create stage, while
the template is bound to a dedicated table, the HW criteria will
be created and saved to the table. The pattern templates can be
used by multiple tables. But different tables create the same
criteria and will not share the matcher between each other in order
to have better performance.

This commit adds pattern template management.

Signed-off-by: Suanming Mou <suanmingm@nvidia.com>
Acked-by: Viacheslav Ovsiienko <viacheslavo@nvidia.com>

show more ...

net/mlx5: add port flow configuration

The hardware steering is backend to support rte_flow_async API in
mlx5 PMD. The port configuration function creates the queues and
needed flow management resour

net/mlx5: add port flow configuration

The hardware steering is backend to support rte_flow_async API in
mlx5 PMD. The port configuration function creates the queues and
needed flow management resources.

The PMD layer configuration function allocates the queues' context
and per-queue job descriptor pool. The job descriptor pool size
is equal to the queue size, and the job descriptors will be popped
from pool with LIFO strategy to convey the flow information during
flow insertion/destruction. Then, while polling the queued operation
result, the flow information will be extracted from the job descriptor
and the descriptor will be pushed back to the LIFO pool.

The commit creates the flow port queues and the job descriptor pools.

Signed-off-by: Suanming Mou <suanmingm@nvidia.com>
Acked-by: Viacheslav Ovsiienko <viacheslavo@nvidia.com>

show more ...

net/mlx5: introduce hardware steering enable routine

The new hardware steering engine relies on using dedicated steering WQEs
instead of writing to the low-level steering table entries directly.
In

net/mlx5: introduce hardware steering enable routine

The new hardware steering engine relies on using dedicated steering WQEs
instead of writing to the low-level steering table entries directly.
In the first implementation the hardware steering engine supports the
new queue based Flow API, the existing synchronous non-queue based Flow
API is not supported.

A new dv_flow_en value 2 is added to manage mlx5 PMD steering engine:

dv_flow_en rte_flow API rte_flow_async API
------------------------------------------------
0 support not support
1 support not support
2 not support support

This commit introduces the extra dv_flow_en = 2 to specify the new
flow initialize and manage operation routine.

Signed-off-by: Suanming Mou <suanmingm@nvidia.com>
Acked-by: Viacheslav Ovsiienko <viacheslavo@nvidia.com>

show more ...

net/mlx5: introduce hardware steering operation

The Connect-X steering is a lookup hardware mechanism that accesses flow
tables, matches packets to the rules, and performs specified actions.
Histori

net/mlx5: introduce hardware steering operation

The Connect-X steering is a lookup hardware mechanism that accesses flow
tables, matches packets to the rules, and performs specified actions.
Historically, mlx5 PMD implements several software engines to manage
steering hardware facility:

- FW Steering - Verbs/Direct Verbs, uses FW calls to manage flows
- SW Steering - DevX/mlx5dv, uses WQEs to access table memory directly

However, there are still some disadvantages:

- performance is limited, we should invoke firmware either to
manage the entire flow, or to handle some internal steering objects

- organizing and preparing flow infrastructure (actions, matchers,
groups, etc.) on the flow inserting is sure to cause slow flow
insertion

- security, exposing the low-level steering entries directly to the
userspace may cause security risks

A new hardware WQE based steering operation with codename "HW Steering"
is going to be introduced to get rid of the security risks. And it will
take advantage of the recently new introduced async queue-based rte_flow
APIs to prepare everything in advance to achieve high insertion rate.

In this new HW steering engine, the original SW steering rte_flow API
will not be supported in the first implementation, only the new async
queue-based flow operations is going to be supported. A new steering
mode parameter for dv_flow_en will be introduced and user will be
able to engage the new steering engine.

This commit adds the basic driver operation.

Signed-off-by: Suanming Mou <suanmingm@nvidia.com>
Acked-by: Viacheslav Ovsiienko <viacheslavo@nvidia.com>

show more ...

net/mlx5: refactor getting counter action pointer

Previously API flow_dv_query_count_ptr is defined to get counter's
action pointer. This DV function is directly called and the better
way is by the

net/mlx5: refactor getting counter action pointer

Previously API flow_dv_query_count_ptr is defined to get counter's
action pointer. This DV function is directly called and the better
way is by the callback.

Add one arg in API mlx5_counter_query and the related callback
counter_query. The added arg is for counter's action pointer.

Signed-off-by: Haifei Luo <haifeil@nvidia.com>
Acked-by: Viacheslav Ovsiienko <viacheslavo@nvidia.com>

show more ...

net/mlx5: fix meter sub-policy creation

If meter policy action was RSS, the correct items were not provided
for sub-policy creation.

This fixes the issue by providing original items in meter split,

net/mlx5: fix meter sub-policy creation

If meter policy action was RSS, the correct items were not provided
for sub-policy creation.

This fixes the issue by providing original items in meter split, so
the sub-policy creation gets the correct items.

Fixes: 3c481324baf3 ("net/mlx5: fix meter flow direction check")
Cc: stable@dpdk.org

Signed-off-by: Shun Hao <shunh@nvidia.com>
Acked-by: Viacheslav Ovsiienko <viacheslavo@nvidia.com>

show more ...

net/mlx5: add shared device context config structure

Add configuration structure for shared device context. This structure
contains all configurations coming from devargs which oriented to
device. I

net/mlx5: add shared device context config structure

Add configuration structure for shared device context. This structure
contains all configurations coming from devargs which oriented to
device. It is a field of shared device context (SH) structure, and is
updated once in mlx5_alloc_shared_dev_ctx() function.
This structure cannot be changed when probing again, so add function to
prevent it. The mlx5_probe_again_args_validate() function creates a
temporary IB context configure structure according to new devargs
attached in probing again, then checks the match between the temporary
structure and the existing IB context configure structure.

Signed-off-by: Michael Baum <michaelba@nvidia.com>
Acked-by: Matan Azrad <matan@nvidia.com>

show more ...

net/mlx5: concentrate all device configurations

Move all device configure to be performed by mlx5_os_cap_config()
function instead of the spawn function.
In addition move all relevant fields from ml

net/mlx5: concentrate all device configurations

Move all device configure to be performed by mlx5_os_cap_config()
function instead of the spawn function.
In addition move all relevant fields from mlx5_dev_config structure to
mlx5_dev_cap.

Signed-off-by: Michael Baum <michaelba@nvidia.com>
Acked-by: Matan Azrad <matan@nvidia.com>

show more ...

net/mlx5: remove DevX flag duplication

The sharing device context structure has a field named "devx" which
indicates if DevX is supported.
The common configure structure has also field named "devx"

net/mlx5: remove DevX flag duplication

The sharing device context structure has a field named "devx" which
indicates if DevX is supported.
The common configure structure has also field named "devx" with the same
meaning.

There is no need for this duplication, because there is a reference to
the common structure from within the sharing device context structure.

This patch removes it from sharing device context structure and uses the
common config structure instead.

Signed-off-by: Michael Baum <michaelba@nvidia.com>
Acked-by: Matan Azrad <matan@nvidia.com>

show more ...

net/mlx5: remove HCA attribute structure duplication

The HCA attribute structure is field of net configure structure.
It is also field of common configure structure.

There is no need for this dupli

net/mlx5: remove HCA attribute structure duplication

The HCA attribute structure is field of net configure structure.
It is also field of common configure structure.

There is no need for this duplication, because there is a reference to
the common structure from within the net structures.

This patch removes it from net configure structure and uses the common
config structure instead.

Signed-off-by: Michael Baum <michaelba@nvidia.com>
Acked-by: Matan Azrad <matan@nvidia.com>

show more ...

remove unnecessary null checks

Functions like free, rte_free, and rte_mempool_free
already handle NULL pointer so the checks here are not necessary.

Remove redundant NULL pointer checks before free

remove unnecessary null checks

Functions like free, rte_free, and rte_mempool_free
already handle NULL pointer so the checks here are not necessary.

Remove redundant NULL pointer checks before free functions
found by nullfree.cocci

Signed-off-by: Stephen Hemminger <stephen@networkplumber.org>

show more ...

net/mlx5: fix mark enabling for Rx

To optimize datapath, the mlx5 pmd checked for mark action on flow
creation, and flagged possible destination rxqs (through queue/RSS
actions), then it enabled the

net/mlx5: fix mark enabling for Rx

To optimize datapath, the mlx5 pmd checked for mark action on flow
creation, and flagged possible destination rxqs (through queue/RSS
actions), then it enabled the mark action logic only for flagged rxqs.

Mark action didn't work if no queue/rss action was in the same flow,
even when the user use multi-group logic to manage the flows.
So, if mark action is performed in group X and the packet is moved to
group Y > X when the packet is forwarded to Rx queues, SW did not get
the mark ID to the mbuf.

Flag Rx datapath to report mark action for any queue when the driver
detects the first mark action after dev_start operation.

Fixes: 8e61555657b2 ("net/mlx5: fix shared RSS and mark actions combination")
Cc: stable@dpdk.org

Signed-off-by: Raja Zidane <rzidane@nvidia.com>
Acked-by: Matan Azrad <matan@nvidia.com>

show more ...

net/mlx5: fix RSS expansion with explicit next protocol

The PMD RSS expansion scheme by default compiles flow rules for all
flow item types that may branch out from a stub supplied
by application.
F

net/mlx5: fix RSS expansion with explicit next protocol

The PMD RSS expansion scheme by default compiles flow rules for all
flow item types that may branch out from a stub supplied
by application.
For example,
ETH can lead to VLAN, IPv4 or IPv6.
IPv4 can lead to UDP, TCP, IPv4 or IPv6.

If application explicitly specified next protocol type, expansion must
use that option only and not create flows with other protocol types.

The PMD ignored explicit next protocol values in GRE and VXLAN-GPE.

The patch updates RSS expansion for GRE and VXLAN-GPE with explicit
next protocol settings.

Fixes: c7870bfe09dc ("ethdev: move RSS expansion code to mlx5 driver")
Cc: stable@dpdk.org

Signed-off-by: Gregory Etelson <getelson@nvidia.com>
Acked-by: Matan Azrad <matan@nvidia.com>

show more ...

fix spelling in comments and strings

The tool comes from https://github.com/jsoref

Signed-off-by: Josh Soref <jsoref@gmail.com>
Signed-off-by: Thomas Monjalon <thomas@monjalon.net>

net/mlx5: fix flow mark with sampling and metering

If there are sample action and the meter action in the same flow,
mlx5 PMD performs several levels of splitting. For example, sampling
feature spli

net/mlx5: fix flow mark with sampling and metering

If there are sample action and the meter action in the same flow,
mlx5 PMD performs several levels of splitting. For example, sampling
feature splits the original flow into prefix subflow with sample action,
and suffix subflow with the rest of actions. Then, metering feature
splits the sampling suffix subflow into its own meter subflows.
If mark action was added before the sample and meter action, the
flow mark flag was kept in the sample subflows but reset on
handling the metering split, causing the flow mark value missed.

This patch keeps the flow mark flag of previous subflow, and then
the following meter subflows handle the flow mark correctly.

Fixes: 9ade91dfe85d ("net/mlx5: fix group value of sample suffix flow")
Cc: stable@dpdk.org

Signed-off-by: Jiawei Wang <jiaweiw@nvidia.com>
Acked-by: Viacheslav Ovsiienko <viacheslavo@nvidia.com>

show more ...

net/mlx5: fix mismatch metadata flow with meter action

The mlx5 PMD introduced the table id attribute to allow multiple
flow tables on the same table level for flow metering, there can be
multiple f

net/mlx5: fix mismatch metadata flow with meter action

The mlx5 PMD introduced the table id attribute to allow multiple
flow tables on the same table level for flow metering, there can be
multiple flow table objects with the same table level but different
table ids.

If the extended metadata mode is enabled, all flows containing
destination Queue/RSS actions are split into two subflows - prefix one
jumps to the MLX5_FLOW_MREG_CP_TABLE_GROUP flow table to copy
MARK action data, and suffix one to perform the destination Queue/RSS
action. The table_id for the jump in the metadata split prefix flow
is always 0.

If flow itself was the metering split suffix subflow the table id was
set to 1 in the flow split structure and the metadata split suffix
subflow was created in the table with wrong table id, causing the
metadata suffix flow mismatch.

This patch resets the table id to 0 while creating the metadata
suffix flows.

Fixes: 51ec04dc7bcf ("net/mlx5: connect meter policy to created flows")
Cc: stable@dpdk.org

Signed-off-by: Jiawei Wang <jiaweiw@nvidia.com>
Acked-by: Viacheslav Ovsiienko <viacheslavo@nvidia.com>

show more ...

net/mlx5: fix metadata and meter split shared tag

In the metadata flow split, PMD created the prefix subflow
with removed Queue or RSS action and appended the set tag and
copy table jump actions. If

net/mlx5: fix metadata and meter split shared tag

In the metadata flow split, PMD created the prefix subflow
with removed Queue or RSS action and appended the set tag and
copy table jump actions. If the flow being split for metadata
was the meter prefix subflow, the driver supposed to share the same
meter split tag action for the metadata split flow. There was the wrong
check for preceding meter split tag action, causing append with metadata
split set tag action and resulting the meter suffix subflow was missed
due to tag value mismatch.

This patch adds the checking before copying into extend action list,
to make sure the correct shared tag is used.

Fixes: 8d72fa668964 ("net/mlx5: share tag between meter and metadata")
Cc: stable@dpdk.org

Signed-off-by: Jiawei Wang <jiaweiw@nvidia.com>
Acked-by: Viacheslav Ovsiienko <viacheslavo@nvidia.com>

show more ...

net/mlx5: fix RSS expansion scheme for GRE header

RFC-2784 allows any valid Ethernet type in GRE protocol type field.

Add Ethernet to GRE RSS expansion.

Fixes: f4b901a46aec ("net/mlx5: add flow GR

net/mlx5: fix RSS expansion scheme for GRE header

RFC-2784 allows any valid Ethernet type in GRE protocol type field.

Add Ethernet to GRE RSS expansion.

Fixes: f4b901a46aec ("net/mlx5: add flow GRE item")
Cc: stable@dpdk.org

Signed-off-by: Gregory Etelson <getelson@nvidia.com>
Acked-by: Viacheslav Ovsiienko <viacheslavo@nvidia.com>

show more ...

net/mlx5: add Ethernet header to GENEVE RSS expansion

RFC-8926 allows inner Ethernet header after GENEVE tunnel.

Current GENEVE RSS expansion created IPv4 and IPv6 paths only.

The patch adds Ether

net/mlx5: add Ethernet header to GENEVE RSS expansion

RFC-8926 allows inner Ethernet header after GENEVE tunnel.

Current GENEVE RSS expansion created IPv4 and IPv6 paths only.

The patch adds Ethernet to RSS expansion scheme.

Cc: stable@dpdk.org

Signed-off-by: Gregory Etelson <getelson@nvidia.com>
Acked-by: Viacheslav Ovsiienko <viacheslavo@nvidia.com>

show more ...

net/mlx5: fix tag ID conflict with sample action

For the flows containing sample action, the tag action was added
implicitly to store the unique flow index into metadata register in the
split prefix

net/mlx5: fix tag ID conflict with sample action

For the flows containing sample action, the tag action was added
implicitly to store the unique flow index into metadata register in the
split prefix subflow, and then match on this index in the split suffix
subflow. The metadata register for flow index of sample split subflows
was also used to store application metadata TAG 0 item, this might cause
TAG 0 corruption in the flows with sample actions.

This patch uses the same metadata register C index as used for
ASO action since it's reserved and not used directly by the application,
and adds the checking in validation to make sure not to conflict
with ASO CT in the same flow.

Fixes: b4c0ddbfcc58 ("net/mlx5: split sample flow into two sub-flows")
Cc: stable@dpdk.org

Signed-off-by: Jiawei Wang <jiaweiw@nvidia.com>
Acked-by: Viacheslav Ovsiienko <viacheslavo@nvidia.com>

show more ...

net/mlx5: fix tunnel offload validation

Tunnel offload API allows the application to restore packet to
its original form if the chain of flows is missed after DECAP action.

MLX5 PMD provides tunnel

net/mlx5: fix tunnel offload validation

Tunnel offload API allows the application to restore packet to
its original form if the chain of flows is missed after DECAP action.

MLX5 PMD provides tunnel offload support only if DV API was enabled.

The patch verifies DV availability before processing with
tunnel offload tasks.

Fixes: 4ec6360de37d ("net/mlx5: implement tunnel offload")
Cc: stable@dpdk.org

Signed-off-by: Gregory Etelson <getelson@nvidia.com>
Acked-by: Viacheslav Ovsiienko <viacheslavo@nvidia.com>

show more ...

net/mlx5: workaround MR creation for flow counter

Due to kernel driver / FW issues in direct MKEY creation using the DevX
API, this patch replaces the counter MR creation to use wrapped mkey
API.

F

net/mlx5: workaround MR creation for flow counter

Due to kernel driver / FW issues in direct MKEY creation using the DevX
API, this patch replaces the counter MR creation to use wrapped mkey
API.

Fixes: 5382d28c2110 ("net/mlx5: accelerate DV flow counter transactions")
Cc: stable@dpdk.org

Signed-off-by: Michael Baum <michaelba@nvidia.com>
Signed-off-by: Matan Azrad <matan@nvidia.com>

show more ...