virtio: Prefer 1.x over 0.9

Make virtio 1.x the default if the hypervisor offers both 0.9 and 1.x.
Version 1.x is required for feature bits >= 32, as in vio(4) multiqueue.

Also be more compliant to

virtio: Prefer 1.x over 0.9

Make virtio 1.x the default if the hypervisor offers both 0.9 and 1.x.
Version 1.x is required for feature bits >= 32, as in vio(4) multiqueue.

Also be more compliant to the virtio 1.x standard, which says "Drivers
MUST match any PCI Revision ID value".

tested by bluhm@

show more ...

whitespace fixes

vio: Enable multiqueue

We use a single interrupt vector for every rx/tx queue pair. With config
and control queue vectors, we need N+2 vectors for N queues. If
multi-queue is not available, the old

vio: Enable multiqueue

We use a single interrupt vector for every rx/tx queue pair. With config
and control queue vectors, we need N+2 vectors for N queues. If
multi-queue is not available, the old scheme is used with either one
vector per virtqueue or one vector for all queues.

* virtio: Add API to establish interrupts on specific cpus in child
drivers. Also make virtio_pci_setup_msix return proper errno.

* virtio_pci: Increase max number of MSIX vectors

* vio: Configure multiple queues and allocate proper interrupts.

OK bluhm@

show more ...

virtio: Refactor attach logic

virtio 1.x requires that all queue setup, including the queue interrupt
vector, is done before setting the queue_enable register to 1. This
conflicts with how we do thi

virtio: Refactor attach logic

virtio 1.x requires that all queue setup, including the queue interrupt
vector, is done before setting the queue_enable register to 1. This
conflicts with how we do things right now:

* We implicitly make queue setup in virtio_alloc_vq(), which is called
from the child driver attach functions. This also sets queue_enable=1.

* Later, we allocate the interrupts and set the queue interrupt vectors
in the second half of the virtio transport attach functions.

This is a violation of a MUST from the standard and causes problems with
some hypervisors, in particular those that have no virtio 0.9 support,
which has no such ordering requirements.

To fix this:

* Move the interrupt allocation to a new virtio_attach_finish() function.
This does all queue setup, including the interrupt vectors.

* Don't call virtio_setup_queue() in virtio_alloc_vq() anymore.

* We can also move the setting of the DRIVER_OK flag into this function.
virtio_attach_finish() must be called before using any virtqueue or
writing any virtio config register.

While there,

* also streamline the attach error handling in all drivers.

* skip initially setting sc_config_change to NULL, the softc is
initialized to 0.

ok jan@
tested by bluhm@

show more ...

vio: Unlock, switch to qstart function

Run without kernel lock. Use the network stack functions used for
multiqueue, but still only run on one queue.

Add a virtio interface for an interrupt barrier

vio: Unlock, switch to qstart function

Run without kernel lock. Use the network stack functions used for
multiqueue, but still only run on one queue.

Add a virtio interface for an interrupt barrier.

This is the reverted diff plus a missing chunk.
Tested by dtucker, bluhm, sf

show more ...

Revert "vio: Unlock"

This causes some crashes. Revert for now

ok sf@

vio: Unlock, switch to qstart function

Run without kernel lock. Use the network stack functions used for
multiqueue, but still only run on one queue.

Add a virtio interface for an interrupt barrier

vio: Unlock, switch to qstart function

Run without kernel lock. Use the network stack functions used for
multiqueue, but still only run on one queue.

Add a virtio interface for an interrupt barrier.

ok dlg@

show more ...

virtio_pci: Negotiate ACCESS_PLATTFORM feature

Accepting VIRTIO_F_ACCESS_PLATFORM is required for SEV on KVM/qemu.

We always use the standard pci access mechanisms of the
architecture/platform we a

virtio_pci: Negotiate ACCESS_PLATTFORM feature

Accepting VIRTIO_F_ACCESS_PLATFORM is required for SEV on KVM/qemu.

We always use the standard pci access mechanisms of the
architecture/platform we are running on. Therefore we should negotiate
the VIRTIO_F_ACCESS_PLATFORM feature if it is offered. Strictly
speaking we should bypass any existing IOMMUs if the host does not offer
this feature, but this no regression and can be fixed later.

feedback from kettenis@

show more ...

virtio_pci: Fix off-by-one in interrupt setup

This was introduced by "virtio: Move interrupt setup into separate
function".

ok jan@

virtio: Move interrupt setup into separate function

Put the MSIX vector into struct virtqueue and create a transport
specific function that feeds the vectors to the device. This will allow
child dev

virtio: Move interrupt setup into separate function

Put the MSIX vector into struct virtqueue and create a transport
specific function that feeds the vectors to the device. This will allow
child devices to influence which vectors are used for which virtqueues.
This will be used by multi-queue vio(4) to route corresponding rx/tx
queue interrupts to the same cpu.

The setup_intrs() function also sets the config interrupt MSIX vector
which fixes a bug that virtio_pci_set_msix_config_vector() would not
be called after a device reset.

OK bluhm@

show more ...

virtio_pci: Improve interrupt names

Make interrupt strings according to the child device. This gives names
like vio0:1, vioblk0 instead of virtio0, virtio1. Also allocate array
of interrupt handler

virtio_pci: Improve interrupt names

Make interrupt strings according to the child device. This gives names
like vio0:1, vioblk0 instead of virtio0, virtio1. Also allocate array
of interrupt handlers dynamically. The current size will be too small
for vio multi-queue.

OK bluhm@

show more ...

constify struct virtio_ops

OK bluhm@

virtio: Introduce dedicated attach args

Instead of abusing virtio_softc as attach args, create a separate
struct. Use it to pass the number of available interrupts. This will be
useful for vio(4) mu

virtio: Introduce dedicated attach args

Instead of abusing virtio_softc as attach args, create a separate
struct. Use it to pass the number of available interrupts. This will be
useful for vio(4) multi-queue support.

ok jan@

show more ...

return type on a dedicated line when declaring functions
ok mglocker@

vio: Fix signal handling and locking in sysctl path

Commits f0b002d01d5 "Release the netlock when sleeping for control
messages in in vioioctl()" and 126b881f71 "Insert a workaround for
per-ifp ioct

vio: Fix signal handling and locking in sysctl path

Commits f0b002d01d5 "Release the netlock when sleeping for control
messages in in vioioctl()" and 126b881f71 "Insert a workaround for
per-ifp ioctl being called w/o NET_LOCK()." in vio(4) fixed a deadlock
but may cause a crash with a protection fault trap if addresses are
added/removed concurrently.

The actual issue is that signals are not handled correctly while
sleeping. After a signal, there is a race condition where sc_ctrl_inuse
is first set to FREE and then the interrupt handler sets it to DONE,
causing a hang in the next vio_wait_ctrl() call.

To fix it:

* Revert the NET_LOCK unlocking work-around.

* Remove PCATCH from the sleep call when we wait for control queue,
avoiding the race with vio_ctrleof(). To ensure that we don't hang
forever, use a 5 second timeout.

* If the timeout is hit, or if the hypervisor has set the
DEVICE_NEEDS_RESET status bit, do not try to use the control queue
until the next ifconfig down/up which resets the device.

* In order to allow reading the device status from device drivers, add a
new interface to the virtio transport drivers.

* Avoid a crash if there is outgoing traffic while doing ifconfig down.

OK bluhm@

show more ...

vio(4): poll device status after issuing device reset.

The virtio spec says a driver "should" wait for a device to report
a clear device status after performing a reset. In some hypervisors,
this do

vio(4): poll device status after issuing device reset.

The virtio spec says a driver "should" wait for a device to report
a clear device status after performing a reset. In some hypervisors,
this doesn't matter as the vcpu's io instruction emulation and
virtio network device emulation happen serially in the same thread.
In hypervisors like vmd(8), device reset happens asynchronously and
the driver can't assume the device is ready.

This race condition results in mbuf pool corruption, causing panics.

Bug reported and reproduced by bluhm@. Root cause found and diff
from sf@. ok dv@ and committed on sf@'s behalf with his permission.

show more ...

The per-VQ MSI-X interrupt handler needs to sync DMA mappings in the
same way that the shared interrupt handler does. This is one of the
requirements of virtio_dequeue(), as specified in its comment

The per-VQ MSI-X interrupt handler needs to sync DMA mappings in the
same way that the shared interrupt handler does. This is one of the
requirements of virtio_dequeue(), as specified in its comment above.

Without the DMA sync, it will not see a new entry on the ring and
return. Since the interrupt is edge-triggered there won't be another
one and we'll get stuck.

ok dv@

show more ...

Fix off-by-one in the MSI-X interrupt establish loop that always tried to
establish one more interrupt than would be needed for per-VQ IRQs. This
meant even though there were enough MSI-X vectors ava

Fix off-by-one in the MSI-X interrupt establish loop that always tried to
establish one more interrupt than would be needed for per-VQ IRQs. This
meant even though there were enough MSI-X vectors available this path could
fail, roll back previously established interrupts and switch to shared IRQs
as a fallback.

ok dv@

show more ...

virtio: Set DRIVER_OK earlier

The DRIVER_OK bit must be set before using any virt-queues. To allow
virtio device drivers to use the virt-queues in their attach functions,
set the bit there and not i

virtio: Set DRIVER_OK earlier

The DRIVER_OK bit must be set before using any virt-queues. To allow
virtio device drivers to use the virt-queues in their attach functions,
set the bit there and not in the virtio transport attach function. Only
vioscsi and viogpu really need this, but let's only have one standard
way to do this.

Noticed because of hangs with vioscsi on qemu/windows and in the Oracle
cloud. With much debugging help by Aaron Mason.

Also revert vioscsi.c 1.31 "Temporarily workaround double calls into
vioscsi_req_done()"

ok krw@

show more ...

remove duplicate includes
ok deraadt@ miod@ krw@

Constify struct cfattach.

Make virtio(4) less restrictive on the type of BAR it supports for
legacy versions. The current version of Parallels on M1 seems to
not provide the I/O BAR that we expect, and reducing our expectati

Make virtio(4) less restrictive on the type of BAR it supports for
legacy versions. The current version of Parallels on M1 seems to
not provide the I/O BAR that we expect, and reducing our expectations
seems to be help.

ok kettenis@

show more ...

Restrict MSI override to i386 and amd64. On other architectures we can
trust the flag set by the PCI host bridge driver and there are cases
where it isn't set because MSIs aren't implemented (for ex

Restrict MSI override to i386 and amd64. On other architectures we can
trust the flag set by the PCI host bridge driver and there are cases
where it isn't set because MSIs aren't implemented (for example on riscv64).

ok patrick@, sf@

show more ...

fix virtio_pci on 32bit archs

bus_space_read/write_8 do not exist there, use two 32bit operations.

Support virtio 1.0 for virtio_pci

virtio 1.0 for virtio_mmio it not yet implemented, but 0.9 devices
continue to work.