1.\" $NetBSD: nvme.4,v 1.7 2016/10/08 17:46:06 jdolecek Exp $ 2.\" $OpenBSD: nvme.4,v 1.2 2016/04/14 11:53:37 jmc Exp $ 3.\" 4.\" Copyright (c) 2016 David Gwynne <dlg@openbsd.org> 5.\" 6.\" Permission to use, copy, modify, and distribute this software for any 7.\" purpose with or without fee is hereby granted, provided that the above 8.\" copyright notice and this permission notice appear in all copies. 9.\" 10.\" THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL WARRANTIES 11.\" WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF 12.\" MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR 13.\" ANY SPECIAL, DIRECT, INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES 14.\" WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN 15.\" ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF 16.\" OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE. 17.\" 18.Dd October 8, 2016 19.Dt NVME 4 20.Os 21.Sh NAME 22.Nm nvme 23.Nd Non-Volatile Memory Host Controller Interface 24.Sh SYNOPSIS 25.Cd "nvme* at pci? dev ? function ?" 26.Sh DESCRIPTION 27The 28.Nm 29driver provides support for NVMe, or NVM Express, 30storage controllers conforming to the 31Non-Volatile Memory Host Controller Interface specification. 32Controllers complying to specification version 1.1 and 1.2 are known to work. 33Other versions should work too for normal operation with the exception of some 34pass-through commands. 35.Pp 36The driver supports the following features: 37.Bl -bullet -compact -offset indent 38.It 39controller and namespace configuration and management using 40.Xr nvmectl 8 41.It 42highly parallel I/O using per-CPU I/O queues 43.It 44PCI MSI/MSI-X attachment, and INTx for legacy systems 45.El 46.Pp 47On systems supporting MSI/MSI-X, the 48.Nm 49driver uses per-CPU IO queue pairs for lockless and highly parallelized I/O. 50Interrupt handlers are scheduled on distinct CPUs. 51The driver allocates as many interrupt vectors as available, up to number 52of CPUs + 1. 53MSI supports up to 32 interrupt vectors within the system, 54MSI-X can have up to 2k. 55Each I/O queue pair has a separate command circular buffer. 56The 57.Nm 58specification allows up to 64k commands per queue, the driver currently allocates 591024 items per queue by default. 60Command submissions are done always on the current CPU, the command completion 61interrupt is handled on the CPU corresponding to the I/O queue ID 62- first I/O queue on CPU0, second I/O queue on CPU1, etc. 63Admin queue command completion is not tied to any CPU, it's handled by 64any CPU. 65To keep lock contention to minimum, it is recommended to keep this assignment, 66even though it is possible to reassign the interrupt handlers differently 67using 68.Xr intrctl 8 . 69The driver also uses soft interrupts to process command completions, in order to 70increase the total system I/O capacity and throughput. 71.Pp 72On systems without MSI, the driver uses a single HW interrupt handler for 73both admin and standard I/O commands. 74Command submissions are done on the current CPU, the command completion interrupt 75is handled on any available CPU. 76This leads to some lock contention, especially on command ccbs. 77Also, command completion handling must be done within the HW interrupt 78handler. 79.Sh FILES 80.Bl -tag -width /dev/nvmeX -compact 81.It Pa /dev/nvme* 82nvme device special files used by 83.Xr nvmectl 8 . 84.El 85.Sh SEE ALSO 86.Xr intro 4 , 87.Xr ld 4 , 88.Xr pci 4 , 89.Xr intrctl 8 , 90.Xr MAKEDEV 8 , 91.Xr nvmectl 8 92.Rs 93.%A NVM Express, Inc. 94.%T "NVM Express \- scalable, efficient, and industry standard" 95.%D 2016-06-12 96.%U http://nvmexpress.org/ 97.Re 98.Rs 99.%A NVM Express, Inc. 100.%T "NVM Express Revision 1.2.1" 101.%D 2016-06-05 102.%U http://www.nvmexpress.org/wp-content/uploads/NVM_Express_1_2_1_Gold_20160603.pdf 103.Re 104.Sh HISTORY 105The 106.Nm 107driver first appeared in 108.Ox 6.0 109and in 110.Nx 8.0 . 111.Sh AUTHORS 112.An -nosplit 113The 114.Nm 115driver was written by 116.An David Gwynne 117.Aq Mt dlg@openbsd.org 118for 119.Ox 120and ported to 121.Nx 122by 123.An NONAKA Kimihiro 124.Aq Mt nonaka@NetBSD.org . 125.An Jaromir Dolecek 126.Aq Mt jdolecek@NetBSD.org 127contributed to making this driver MPSAFE. 128.Sh NOTES 129At least some 130.Tn Intel 131.Nm 132adapter cards are known to require 133.Tn PCIe 134Generation 3 slot. 135Such cards do not even probe when plugged 136into older generation slot. 137.Pp 138The driver also attaches and works fine for emulated 139.Nm 140devices under QEMU and 141.Tn Oracle 142.Tn VirtualBox . 143However, there seems to be some broken interaction between 144.Tn VirtualBox 1455.1.6 146and the driver, the emulated 147.Nm 148controller responds to commands only the first time it is attached, 149after reboot or module reload it stops responding. 150The virtual machine must be completely powered off 151(or even killed) to fix this. 152