1.\" $NetBSD: nvme.4,v 1.10 2017/04/27 17:32:26 jdolecek Exp $ 2.\" $OpenBSD: nvme.4,v 1.2 2016/04/14 11:53:37 jmc Exp $ 3.\" 4.\" Copyright (c) 2016 David Gwynne <dlg@openbsd.org> 5.\" 6.\" Permission to use, copy, modify, and distribute this software for any 7.\" purpose with or without fee is hereby granted, provided that the above 8.\" copyright notice and this permission notice appear in all copies. 9.\" 10.\" THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL WARRANTIES 11.\" WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF 12.\" MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR 13.\" ANY SPECIAL, DIRECT, INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES 14.\" WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN 15.\" ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF 16.\" OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE. 17.\" 18.Dd April 27, 2017 19.Dt NVME 4 20.Os 21.Sh NAME 22.Nm nvme 23.Nd Non-Volatile Memory Host Controller Interface 24.Sh SYNOPSIS 25.Cd "nvme* at pci? dev ? function ?" 26.Sh DESCRIPTION 27The 28.Nm 29driver provides support for NVMe, or NVM Express, 30storage controllers conforming to the 31Non-Volatile Memory Host Controller Interface specification. 32Controllers complying to specification version 1.1 and 1.2 are known to work. 33Other versions should work too for normal operation with the exception of some 34pass-through commands. 35.Pp 36The driver supports the following features: 37.Bl -bullet -compact -offset indent 38.It 39controller and namespace configuration and management using 40.Xr nvmectl 8 41.It 42highly parallel I/O using per-CPU I/O queues 43.It 44PCI MSI/MSI-X attachment, and INTx for legacy systems 45.El 46.Pp 47On systems supporting MSI/MSI-X, the 48.Nm 49driver uses per-CPU IO queue pairs for lockless and highly parallelized I/O. 50Interrupt handlers are scheduled on distinct CPUs. 51The driver allocates as many interrupt vectors as available, up to number 52of CPUs + 1. 53MSI supports up to 32 interrupt vectors within the system, 54MSI-X can have up to 2k. 55Each I/O queue pair has a separate command circular buffer. 56The 57.Nm 58specification allows up to 64k commands per queue, the driver currently 59allocates 1024 entries per queue, or controller maximum, whatever is smaller. 60Command submissions are done always on the current CPU, the command completion 61interrupt is handled on the CPU corresponding to the I/O queue ID 62- first I/O queue on CPU0, second I/O queue on CPU1, etc. 63Admin queue command completion is handled by CPU0 by default. 64To keep lock contention to minimum, it is recommended to keep this assignment, 65even though it is possible to reassign the interrupt handlers differently 66using 67.Xr intrctl 8 . 68.Pp 69On systems without MSI, the driver uses a single HW interrupt handler for 70both admin and standard I/O commands. 71Command submissions are done on the current CPU, the command completion 72interrupt is handled on CPU0 by default. 73This leads to some lock contention, especially on command ccbs. 74.Pp 75The driver offloads command completion processing to soft interrupt, 76in order to increase the total system I/O capacity and throughput. 77.Sh FILES 78.Bl -tag -width /dev/nvmeX -compact 79.It Pa /dev/nvme* 80nvme device special files used by 81.Xr nvmectl 8 . 82.El 83.Sh SEE ALSO 84.Xr intro 4 , 85.Xr ld 4 , 86.Xr pci 4 , 87.Xr intrctl 8 , 88.Xr MAKEDEV 8 , 89.Xr nvmectl 8 90.Rs 91.%A NVM Express, Inc. 92.%T "NVM Express \- scalable, efficient, and industry standard" 93.%D 2016-06-12 94.%U http://nvmexpress.org/ 95.Re 96.Rs 97.%A NVM Express, Inc. 98.%T "NVM Express Revision 1.2.1" 99.%D 2016-06-05 100.%U http://www.nvmexpress.org/wp-content/uploads/NVM_Express_1_2_1_Gold_20160603.pdf 101.Re 102.Sh HISTORY 103The 104.Nm 105driver first appeared in 106.Ox 6.0 107and in 108.Nx 8.0 . 109.Sh AUTHORS 110.An -nosplit 111The 112.Nm 113driver was written by 114.An David Gwynne 115.Aq Mt dlg@openbsd.org 116for 117.Ox 118and ported to 119.Nx 120by 121.An NONAKA Kimihiro 122.Aq Mt nonaka@NetBSD.org . 123.An Jaromir Dolecek 124.Aq Mt jdolecek@NetBSD.org 125contributed to making this driver MPSAFE. 126.Sh NOTES 127At least some 128.Tn Intel 129.Nm 130adapter cards are known to require 131.Tn PCIe 132Generation 3 slot. 133Such cards do not even probe when plugged 134into older generation slot. 135.Pp 136The driver was also tested and confirmed working fine for emulated 137.Nm 138devices under QEMU 2.8.0 and 139.Tn Oracle 140.Tn VirtualBox 1415.1.20. 142