net/virtio: remove reference to virtqueue in vrings

Vrings are part of the virtqueues, so we don't need
to have a pointer to it in Vrings descriptions.

Instead, let's just subtract from its offset

net/virtio: remove reference to virtqueue in vrings

Vrings are part of the virtqueues, so we don't need
to have a pointer to it in Vrings descriptions.

Instead, let's just subtract from its offset to
calculate virtqueue address.

Signed-off-by: Maxime Coquelin <maxime.coquelin@redhat.com>
Reviewed-by: Chenbo Xia <chenbo.xia@intel.com>
Reviewed-by: David Marchand <david.marchand@redhat.com>
Tested-by: Balazs Nemeth <bnemeth@redhat.com>

show more ...

net/virtio: remove last PCI refs in non-PCI code

This patch finalizes the bus isolation part of this
refactoring.

Signed-off-by: Maxime Coquelin <maxime.coquelin@redhat.com>
Reviewed-by: Chenbo Xia

net/virtio: remove last PCI refs in non-PCI code

This patch finalizes the bus isolation part of this
refactoring.

Signed-off-by: Maxime Coquelin <maxime.coquelin@redhat.com>
Reviewed-by: Chenbo Xia <chenbo.xia@intel.com>
Reviewed-by: David Marchand <david.marchand@redhat.com>

show more ...

net/virtio: move features definition to generic header

This patch moves all the Virtio definition to the generic
header. It also renames some helpers to no more reference
PCI.

Signed-off-by: Maxime

net/virtio: move features definition to generic header

This patch moves all the Virtio definition to the generic
header. It also renames some helpers to no more reference
PCI.

Signed-off-by: Maxime Coquelin <maxime.coquelin@redhat.com>
Reviewed-by: Chenbo Xia <chenbo.xia@intel.com>

show more ...

net/virtio: force IOVA as VA mode for virtio-user

At least Vhost-user backend of Virtio-user PMD requires
IOVA as VA mode. Until now, it was implemented as a hack
by forcing to use mbuf's buf_addr f

net/virtio: force IOVA as VA mode for virtio-user

At least Vhost-user backend of Virtio-user PMD requires
IOVA as VA mode. Until now, it was implemented as a hack
by forcing to use mbuf's buf_addr field instead of buf_iova.

This patch removes all this logic and just fails probing
if IOVA as VA mode is not selected. It simplifies the
code overall, and removes some bus-specific logic from
generic virtio_ethdev.c.

Signed-off-by: Maxime Coquelin <maxime.coquelin@redhat.com>
Reviewed-by: Chenbo Xia <chenbo.xia@intel.com>

show more ...

ethdev: make driver-only headers private

The rte_ethdev_driver.h, rte_ethdev_vdev.h and rte_ethdev_pci.h files are
for drivers only and should be a private to DPDK and not installed.

Signed-off-by:

ethdev: make driver-only headers private

The rte_ethdev_driver.h, rte_ethdev_vdev.h and rte_ethdev_pci.h files are
for drivers only and should be a private to DPDK and not installed.

Signed-off-by: Bruce Richardson <bruce.richardson@intel.com>
Reviewed-by: Maxime Coquelin <maxime.coquelin@redhat.com>
Acked-by: Thomas Monjalon <thomas@monjalon.net>
Acked-by: Steven Webster <steven.webster@windriver.com>

show more ...

net/virtio: separate AVX Rx/Tx

Split out AVX instruction based virtio packed ring Rx and Tx
implementation to a separate file.

Signed-off-by: Joyce Kong <joyce.kong@arm.com>
Reviewed-by: Ruifeng Wa

net/virtio: separate AVX Rx/Tx

Split out AVX instruction based virtio packed ring Rx and Tx
implementation to a separate file.

Signed-off-by: Joyce Kong <joyce.kong@arm.com>
Reviewed-by: Ruifeng Wang <ruifeng.wang@arm.com>
Reviewed-by: Maxime Coquelin <maxime.coquelin@redhat.com>

show more ...

net/virtio: check raw checksum failure

rte_raw_cksum_mbuf can fail, so we should check to see if it
has. If so, return with an error.

Fixes: 96cb6711939e ("net/virtio: support Rx checksum offload")

net/virtio: check raw checksum failure

rte_raw_cksum_mbuf can fail, so we should check to see if it
has. If so, return with an error.

Fixes: 96cb6711939e ("net/virtio: support Rx checksum offload")
Cc: stable@dpdk.org

Signed-off-by: Chas Williams <3chas3@gmail.com>
Reviewed-by: Maxime Coquelin <maxime.coquelin@redhat.com>

show more ...

net/virtio: fix indirect descriptors in packed datapaths

Like split ring, packed ring will utilize indirect ring
elements when queuing mbufs need multiple descriptors.
Thus each packet will take onl

net/virtio: fix indirect descriptors in packed datapaths

Like split ring, packed ring will utilize indirect ring
elements when queuing mbufs need multiple descriptors.
Thus each packet will take only one slot when having
multiple segments.

Fixes: 892dc798fa9c ("net/virtio: implement Tx path for packed queues")
Cc: stable@dpdk.org

Signed-off-by: Marvin Liu <yong.liu@intel.com>
Reviewed-by: Maxime Coquelin <maxime.coquelin@redhat.com>

show more ...

eal: remove deprecated coherent IO memory barriers

Since the 20.08 release deprecated rte_cio_*mb APIs because these APIs
provide the same functionality as rte_io_*mb APIs on all platforms, so
remov

eal: remove deprecated coherent IO memory barriers

Since the 20.08 release deprecated rte_cio_*mb APIs because these APIs
provide the same functionality as rte_io_*mb APIs on all platforms, so
remove them and use rte_io_*mb instead.

Signed-off-by: Phil Yang <phil.yang@arm.com>
Signed-off-by: Joyce Kong <joyce.kong@arm.com>
Reviewed-by: Ruifeng Wang <ruifeng.wang@arm.com>
Reviewed-by: Honnappa Nagarahalli <honnappa.nagarahalli@arm.com>
Acked-by: David Marchand <david.marchand@redhat.com>

show more ...

virtio: use one way barrier for split vring used index

In case VIRTIO_F_ORDER_PLATFORM(36) is not negotiated, then the frontend
and backend are assumed to be implemented in software, that is they ca

virtio: use one way barrier for split vring used index

In case VIRTIO_F_ORDER_PLATFORM(36) is not negotiated, then the frontend
and backend are assumed to be implemented in software, that is they can
run on identical CPUs in an SMP configuration.
Thus a weak form of memory barriers like rte_smp_r/wmb, other than
rte_cio_r/wmb, is sufficient for this case(vq->hw->weak_barriers == 1)
and yields better performance.
For the above case, this patch helps yielding even better performance
by replacing the two-way barriers with C11 one-way barriers for used
index in split ring.

Signed-off-by: Joyce Kong <joyce.kong@arm.com>
Reviewed-by: Gavin Hu <gavin.hu@arm.com>
Reviewed-by: Maxime Coquelin <maxime.coquelin@redhat.com>

show more ...

net/virtio: add vectorized packed ring Tx

Optimize packed ring Tx path like Rx path. Split Tx path into batch and
single Tx functions. Batch function is further optimized by AVX512
instructions.

Si

net/virtio: add vectorized packed ring Tx

Optimize packed ring Tx path like Rx path. Split Tx path into batch and
single Tx functions. Batch function is further optimized by AVX512
instructions.

Signed-off-by: Marvin Liu <yong.liu@intel.com>
Reviewed-by: Maxime Coquelin <maxime.coquelin@redhat.com>

show more ...

net/virtio: add vectorized packed ring Rx

Optimize packed ring Rx path with SIMD instructions. Solution of
optimization is pretty like vhost, is that split path into batch and
single functions. Batc

net/virtio: add vectorized packed ring Rx

Optimize packed ring Rx path with SIMD instructions. Solution of
optimization is pretty like vhost, is that split path into batch and
single functions. Batch function is further optimized by AVX512
instructions.

Signed-off-by: Marvin Liu <yong.liu@intel.com>
Reviewed-by: Maxime Coquelin <maxime.coquelin@redhat.com>

show more ...

net/virtio: reuse packed ring functions

Move offload, xmit cleanup and packed xmit enqueue function to header
file. These functions will be reused by packed ring vectorized path.

Signed-off-by: Mar

net/virtio: reuse packed ring functions

Move offload, xmit cleanup and packed xmit enqueue function to header
file. These functions will be reused by packed ring vectorized path.

Signed-off-by: Marvin Liu <yong.liu@intel.com>
Reviewed-by: Maxime Coquelin <maxime.coquelin@redhat.com>

show more ...

net/virtio: add parameter to enable vectorized path

Previously, virtio split ring vectorized path was enabled by default.
This is not suitable for everyone because that path does not follow
virtio s

net/virtio: add parameter to enable vectorized path

Previously, virtio split ring vectorized path was enabled by default.
This is not suitable for everyone because that path does not follow
virtio spec. Add new devarg for virtio vectorized path selection. By
default vectorized path is disabled.

Signed-off-by: Marvin Liu <yong.liu@intel.com>
Reviewed-by: Maxime Coquelin <maxime.coquelin@redhat.com>

show more ...

net/virtio: enable in-order feature if negotiated

Ring initialization is different when inorder feature negotiated. This
action should dependent on negotiated feature bits.

Signed-off-by: Marvin Li

net/virtio: enable in-order feature if negotiated

Ring initialization is different when inorder feature negotiated. This
action should dependent on negotiated feature bits.

Signed-off-by: Marvin Liu <yong.liu@intel.com>
Reviewed-by: Maxime Coquelin <maxime.coquelin@redhat.com>

show more ...

net/virtio: add Rx free threshold setting

Introduce free threshold setting in Rx queue, its default value is 32.
Limit the threshold size to multiple of four as only vectorized packed
Rx function wi

net/virtio: add Rx free threshold setting

Introduce free threshold setting in Rx queue, its default value is 32.
Limit the threshold size to multiple of four as only vectorized packed
Rx function will utilize it. Virtio driver will rearm Rx queue when
more than rx_free_thresh descs were dequeued.

Signed-off-by: Marvin Liu <yong.liu@intel.com>
Reviewed-by: Maxime Coquelin <maxime.coquelin@redhat.com>

show more ...

net/virtio: do not use PMD log type

Virtio driver has its own logtype and should not use legacy
PMD logtype.

Fixes: 32c118fd0059 ("virtio: free mbuf's with threshold")
Fixes: e5f456a98d3c ("net/vir

net/virtio: do not use PMD log type

Virtio driver has its own logtype and should not use legacy
PMD logtype.

Fixes: 32c118fd0059 ("virtio: free mbuf's with threshold")
Fixes: e5f456a98d3c ("net/virtio: support in-order Rx and Tx")
Fixes: 1c8489da561b ("net/virtio-user: fix multi-process support")
Cc: stable@dpdk.org

Signed-off-by: Stephen Hemminger <stephen@networkplumber.org>
Reviewed-by: Ferruh Yigit <ferruh.yigit@intel.com>

show more ...

net/virtio: fix Tx checksum offloads

Missing parenthesis around expression before type cast to struct
virtio_net_hdr pointer makes the arithmetic to be in
sizeof(struct virtio_net_hdr) units.

Use r

net/virtio: fix Tx checksum offloads

Missing parenthesis around expression before type cast to struct
virtio_net_hdr pointer makes the arithmetic to be in
sizeof(struct virtio_net_hdr) units.

Use rte_pktmbuf_mtod_offset() to fix the problem.

Type of head_size is changed to signed since some compilers bark
on unary minus applied to unsigned.

Fixes: 1ae55ad38e5e ("net/virtio: fix mbuf data and packet length mismatch")
Cc: stable@dpdk.org

Signed-off-by: Andrew Rybchenko <arybchenko@solarflare.com>
Reviewed-by: Tiwei Bie <tiwei.bie@intel.com>
Reviewed-by: Maxime Coquelin <maxime.coquelin@redhat.com>

show more ...

net/virtio: reject deferred Tx start

Deferred start Tx queue is not supported by the driver.

Fixes: 0748be2cf9a2 ("ethdev: queue start and stop")
Cc: stable@dpdk.org

Signed-off-by: Dilshod Urazov

net/virtio: reject deferred Tx start

Deferred start Tx queue is not supported by the driver.

Fixes: 0748be2cf9a2 ("ethdev: queue start and stop")
Cc: stable@dpdk.org

Signed-off-by: Dilshod Urazov <dilshod.urazov@oktetlabs.ru>
Signed-off-by: Andrew Rybchenko <arybchenko@solarflare.com>
Reviewed-by: Tiwei Bie <tiwei.bie@intel.com>
Reviewed-by: Maxime Coquelin <maxime.coquelin@redhat.com>

show more ...

net/virtio: reject deferred Rx start

Deferred start Rx queue is not supported by the driver.

Fixes: 0748be2cf9a2 ("ethdev: queue start and stop")
Cc: stable@dpdk.org

Signed-off-by: Dilshod Urazov

net/virtio: reject deferred Rx start

Deferred start Rx queue is not supported by the driver.

Fixes: 0748be2cf9a2 ("ethdev: queue start and stop")
Cc: stable@dpdk.org

Signed-off-by: Dilshod Urazov <dilshod.urazov@oktetlabs.ru>
Signed-off-by: Andrew Rybchenko <arybchenko@solarflare.com>
Reviewed-by: Maxime Coquelin <maxime.coquelin@redhat.com>

show more ...

net/virtio: fix descriptor addressed in Tx

Previous fix removes usage of rte_pktmbuf_prepend() to get pointer
to virtio net header which changes mbuf data_off and data_len.
Size of virtio net header

net/virtio: fix descriptor addressed in Tx

Previous fix removes usage of rte_pktmbuf_prepend() to get pointer
to virtio net header which changes mbuf data_off and data_len.
Size of virtio net header is added to segment length when Tx descriptor
is composed, but segment address (calculated using data_off) is not
adjusted to take size of virtio net header into account.

Fixes: 1ae55ad38e5e ("net/virtio: fix mbuf data and packet length mismatch")
Cc: stable@dpdk.org

Signed-off-by: Andrew Rybchenko <arybchenko@solarflare.com>
Reviewed-by: Tiwei Bie <tiwei.bie@intel.com>

show more ...

net/virtio: improve perf via one-way barriers on used flag

In case VIRTIO_F_ORDER_PLATFORM(36) is not negotiated, then the frontend
and backend are assumed to be implemented in software, that is the

net/virtio: improve perf via one-way barriers on used flag

In case VIRTIO_F_ORDER_PLATFORM(36) is not negotiated, then the frontend
and backend are assumed to be implemented in software, that is they can
run on identical CPUs in an SMP configuration.
Thus a weak form of memory barriers like rte_smp_r/wmb, other than
rte_cio_r/wmb, is sufficient for this case(vq->hw->weak_barriers == 1)
and yields better performance.
For the above case, this patch helps yielding even better performance
by replacing the two-way barriers with C11 one-way barriers for used
flags in packed ring.

Signed-off-by: Joyce Kong <joyce.kong@arm.com>
Reviewed-by: Gavin Hu <gavin.hu@arm.com>
Reviewed-by: Phil Yang <phil.yang@arm.com>
Reviewed-by: Maxime Coquelin <maxime.coquelin@redhat.com>

show more ...

net/virtio: improve perf via one-way barrier on avail flag

In case VIRTIO_F_ORDER_PLATFORM(36) is not negotiated, then the frontend
and backend are assumed to be implemented in software, that is the

net/virtio: improve perf via one-way barrier on avail flag

In case VIRTIO_F_ORDER_PLATFORM(36) is not negotiated, then the frontend
and backend are assumed to be implemented in software, that is they can
run on identical CPUs in an SMP configuration.
Thus a weak form of memory barriers like rte_smp_r/wmb, other than
rte_cio_r/wmb, is sufficient for this case(vq->hw->weak_barriers == 1)
and yields better performance.
For the above case, this patch helps yielding even better performance
by replacing the two-way barriers with C11 one-way barriers for avail
flags in packed ring.

Meanwhile, a read barrier is required to ensure ordering between
descriptor's flags and content reads [1]. With C11, load-acquire can
enforce the ordering instead of rmb barrier.

[1] https://patchwork.dpdk.org/patch/49109/

Signed-off-by: Joyce Kong <joyce.kong@arm.com>
Reviewed-by: Gavin Hu <gavin.hu@arm.com>
Reviewed-by: Phil Yang <phil.yang@arm.com>
Reviewed-by: Maxime Coquelin <maxime.coquelin@redhat.com>

show more ...

net/virtio: fix mbuf data and packet length mismatch

If reserve virtio header room by function rte_pktmbuf_prepend, both
segment data length and packet length of mbuf will be increased.
Data length

net/virtio: fix mbuf data and packet length mismatch

If reserve virtio header room by function rte_pktmbuf_prepend, both
segment data length and packet length of mbuf will be increased.
Data length will be equal to descriptor length, while packet length
should be decreased as virtio-net header won't be taken into packet.
Thus will cause mismatch in mbuf structure. Fix this issue by access
mbuf data directly and increase descriptor length if it is needed.

Fixes: 58169a9c8153 ("net/virtio: support Tx checksum offload")
Fixes: 892dc798fa9c ("net/virtio: implement Tx path for packed queues")
Fixes: 4905ed3a523f ("net/virtio: optimize Tx enqueue for packed ring")
Fixes: e5f456a98d3c ("net/virtio: support in-order Rx and Tx")
Cc: stable@dpdk.org

Reported-by: Stephen Hemminger <stephen@networkplumber.org>
Signed-off-by: Marvin Liu <yong.liu@intel.com>
Reviewed-by: Maxime Coquelin <maxime.coquelin@redhat.com>

show more ...

net/virtio: fix Rx stats with vectorized functions

With vectorized functions, only the rx stats for number of packets is
incremented.
Update also the other statistics.
Performance impact is about 2%

net/virtio: fix Rx stats with vectorized functions

With vectorized functions, only the rx stats for number of packets is
incremented.
Update also the other statistics.
Performance impact is about 2%

Fixes: fc3d66212fed ("virtio: add vector Rx")
Cc: stable@dpdk.org

Signed-off-by: Thibaut Collet <thibaut.collet@6wind.com>
Reviewed-by: Maxime Coquelin <maxime.coquelin@redhat.com>

show more ...