1# Changelog 2 3## v20.07: (Upcoming Release) 4 5### nvmf 6 7The NVMe-oF target no longer supports connecting scheduling configuration and instead 8always uses what was previously called "transport" scheduling. 9 10`spdk_nvmf_tgt_accept` no longer takes a function pointer as an argument. New connections 11are automatically assigned to poll groups by the underlying transport. Further, 12`spdk_nvmf_transport_ops` has changed such that the accept function pointer no longer 13takes a function pointer as an argument. Instead, transports should call 14`spdk_nvmf_tgt_new_qpair` whenever they previously would have called that callback. 15 16### nvme 17 18Add `opts_size` in `spdk_nvme_ctrlr_opts` structure in order to solve the compatiblity issue 19for different ABI version. 20 21### RPC 22 23Command line parameters `-r` and `--rpc-socket` will longer accept TCP ports. RPC server 24must now be started on a Unix domain socket. Exposing RPC on the network, as well as providing 25proper authentication (if needed) is now a responsibility of the user. 26 27### build 28 29The fio plugins now compile to `build/fio` and are named `spdk_bdev` and `spdk_nvme`. 30Existing fio configuration files will need to be updated. 31 32### accel 33 34A new API was added `spdk_accel_get_capabilities` that allows applications to 35query the capabilities of the currently enabled accel engine back-end. 36 37A new capability, CRC-32C, was added via `spdk_accel_submit_crc32c`. 38 39The software accel engine implemenation has added support for CRC-32C. 40 41A new capability, compare, was added via `spdk_accel_submit_compare`. 42 43The software accel engine implemenation has added support for compare. 44 45### dpdk 46 47Updated DPDK submodule to DPDK 19.11.2, which includes fixes for DPDK vulnerabilities: 48CVE-2020-10722, CVE-2020-10723, CVE-2020-10724, CVE-2020-10725, CVE-2020-10724. 49 50### idxd 51 52IDXD engine support for CRC-32C has been added. 53 54### rdma 55 56A new `rdma` library has been added. It is an abstraction layer over different RDMA providers. 57Two providers are available - verbs (used by default when RDMA is enabled or enabled explicitly 58using --with-rdma=verbs) and mlx5 Direct Verbs aka DV (enabled by --with-rdma=mlx5_dv). 59Using mlx5_dv requires libmlx5 installed on the system. 60 61### rpc 62 63Parameter `-p` or `--max-qpairs-per-ctrlr` of `nvmf_create_transport` RPC command accepted by the 64rpc.py script is deprecated, new parameter `-m` or `--max-io-qpairs-per-ctrlr` is added. 65 66Parameter `max_qpairs_per_ctrlr` of `nvmf_create_transport` RPC command accepted by the NVMF target 67is deprecated, new parameter `max_io_qpairs_per_ctrlr` is added. 68 69Added `sock_impl_get_options` and `sock_impl_set_options` RPC methods. 70 71### sock 72 73Added `spdk_sock_impl_get_opts` and `spdk_sock_impl_set_opts` functions to set/get socket layer configuration 74options. Options can be set independently for each implementation. 75 76Added `recv_buf_size` and 'send_buf_size' socket layer options. They are used only in posix implementation. 77 78Added `uring` based socket implementation, the code is located in module/sock/uring. This feature is only 79available in Linux which requires kernel version is greater than 5.4.3. Currently, our CI pool added the uring 80based socket tests for iSCSI target and also the tests for SPDK NVMe-oF tcp transport. 81 82### vhost 83 84The function `spdk_vhost_blk_get_dev` has been removed. 85 86## v20.04: 87 88IDXD engine support for compare has been added. 89 90### configuration 91 92Legacy INI style configuration for SPDK applications has been deprecated and will be 93removed in future release. 94Please switch to JSON-RPC configuration files and/or RPC driven run-time configuration. 95 96### copy 97 98The copy engine library, modules and public APIs have been renamed. Use of the word `copy` 99has been replaced with the word `accel` short for accelerator in preparation for adding new 100capabilities in the future. Additionally, APIs for what was previously called the `memcpy` 101engine have been renamed to identify the engine as a software accelerator. 102 103### crypto 104 105Support for AES_XTS was added for the QAT polled mode driver (pmd). The create RPC 106`bdev_crypto_create` has 2 new optional parameters: cipher and key2. Cipher can be either 107AES_CBC (default) or AES_XTS. AES_XTS is only valid when using the QAT polled mode driver. 108The key2 parameter is the second key required for AES_XTS. 109 110### event 111 112Reactors now accumulate CPU stats and they are retrieved by the RPC `framework_get_reactors`. 113 114### idxd 115 116IDXD support was added in the form of a low level library that can directly 117utilize the IDXD offload engine as well as an accel plug-in module allowing for use 118with the generic accel framework API. IDXD is the first in a family of offload 119engines that share the same interface, specifically DSA is added here. More info 120can be found here: https://01.org/blogs/2019/introducing-intel-data-streaming-accelerator 121 122Much of the implementation models IOAT, however the low level interfaces are very 123different. The RPC to enable IDXD requires a configuration number as well. The 124code includes two pre-defined configurations of IDXD groups/work queues/engines. A future 125version will provide an interface to allow for setting of individual configuration 126parameters. 127 128IDXD is not yet available so this feature should be considered experimental. It will 129be built up with additional documentation as an ongoing activity. 130 131### iscsi 132 133The iSCSI target now creates a lightweight thread per poll group instead of assuming a pool 134of lightweight threads already exist at start up time. A poll group is a collection of 135unrelated iSCSI connections. Each poll group is only accessed from the associated 136lightweight thread. 137 138### ftl 139 140Several changes have been made to the `spdk_ftl_conf`, `spdk_ftl_dev_init_ops`, and 141`spdk_ftl_attrs` structs. Please see `include/spdk/ftl.h` for more details. 142 143### miscellaneous 144 145The `--json-ignore-init-errors` command line parameter has been added to ignore 146initialization errors on JSON config load. 147 148The public header file io_channel.h has been removed. Please use thread.h which has the 149exact same API. 150 151### nvme 152 153Exported internal nvme_ctrlr_cmd_security_receive/send() APIs as public APIs with "the spdk_" 154prefix. 155 156Added `priority` field in `spdk_nvme_transport_id`, this field is used to specify the priority 157of the NVMe-oF connection, and currently it is used for NVMe-oF tcp connection. 158 159A new poll group API has been added to allow for pooling of nvme qpairs across a single 160entity which can be polled for completions. This new API consists of the `spdk_nvme_poll_group` 161family of functions. As a result of this new API, all NVMe transports are expected to implement 162several poll group related functions. 163 164A new flag, `create_only`, has been added to the `spdk_nvme_io_qpair_opts` structure. This flag 165allows a user to call `spdk_nvme_ctrlr_get_default_io_qpair` without also connecting the qpair 166within the context of that call. 167 168As a result of the `create_only` flag, two new API functions, `spdk_nvme_ctrlr_connect_io_qpair` 169and `spdk_nvme_ctrlr_disconnect_io_qpair`, have been added to facilitate connecting newly created 170qpairs (for example, after they have been added to a poll group) and disconnecting qpairs without 171destroying them (for example to disconnect a qpair before migrating it to a new poll group and 172reconnecting it). 173 174The functions `spdk_nvme_ctrlr_alloc_cmb_io_buffer` and `spdk_nvme_ctrlr_free_cmb_io_buffer` 175have been changed to `spdk_nvme_ctrlr_map_cmb` and `spdk_nvme_ctrlr_unmap_cmb` respectively. 176 177An additional function, `spdk_nvme_ctrlr_reserve_cmb`, has been added to facilitate reserving 178the entire size of the controller memory buffer for data transfer. 179 180### nvme_cuse 181 182`spdk_nvme_cuse_get_ctrlr_name` now takes two additional parameters, `char *name` which 183stores the pointer to the controller name, and `size_t *size` which stores the length of 184the name. The return type has also been changed from char * to int. 185 186`spdk_nvme_cuse_get_ns_name` now takes two additional parameters, `char *name` which 187stores the pointer to the namespace name, and `size_t *size` which stores the length of 188the name. The return type has also been changed from char * to int. 189 190### nvme_opal 191 192Several public OPAL structure definitions have been changed since the last release. 193 194### nvmf 195 196`spdk_nvmf_poll_group_destroy()` is now asynchronous and accepts a completion callback. 197 198The NVMe-oF target now creates a lightweight thread per poll group instead of assuming a pool 199of lightweight threads already exist at start up time. A poll group is a collection of 200unrelated NVMe-oF connections. Each poll group is only accessed from the associated 201lightweight thread. 202 203A new struct, `spdk_nvmf_subsystem_listener`, has been added to encapsulate the subsystem specific 204nature of a listener object. 205 206`spdk_nvmf_tgt_listen` no longer accepts a callback function or argument. It also returns an 207int to indicate the status of the listen call. 208 209The execution of `spdk_nvme_poll_group_destroy` is now asynchronous and the function accepts 210a cb_fn and cb_arg to call upon completion. 211 212The execution of `spdk_nvmf_subsystem_add_listener` is now asynchronous and the function accepts 213a cb_fn and cb_arg to call upon completion. 214 215The `nvmf_transport.h` header has been made public to allow custom NVMe-oF transports to integrate 216with NVMe-oF libraries without using internal APIs. 217 218### ocf 219 220Updated the OCF submodule to OCF v20.03 221 222New version of OCF comes with API changes and bug fixes 223 224### rpc 225 226A new RPC `thread_set_cpumask` has been added to set the cpumask of the thread 227to the specified value. 228 229A new RPC `thread_get_pollers` has been added to retrieve pollers from SPDK threads. 230 231A new RPC `thread_get_io_channels` has been added to retrieve I/O channels from SPDK threads. 232 233A new RPC `bdev_rbd_resize` has been added to resize Ceph RBD bdevs. 234 235### sock 236 237The `spdk_sock_set_priority` function has been removed since the feature to set the sock priority 238will be contained in two new functions, i.e., `spdk_sock_listen_ext` and `spdk_sock_connect_ext`. 239Users may now specify the priority of the socket in the opts that they want to use. 240 241### spdk_top 242 243A new application, `spdk_top`, has been added which allows users to monitor resource consumption 244by a running SPDK application. More information on this application can be found in 245`app/spdk_top/README`. 246 247### thread 248 249A new function `spdk_thread_lib_init_ext` has been added, and the function 250`spdk_thread_lib_init` has been deprecated. The user of `spdk_thread_lib_init_ext` is 251expected to implement both functions `spdk_thread_op_fn` and `spdk_thread_op_supported_fn`. 252`spdk_thread_op_supported_fn` is called to check whether the SPDK thread operation 253is supported. `spdk_thread_op_fn` is called to execute the SPDK thread operation. 254Current SPDK operation types are `SPDK_THREAD_OP_NEW` and `SPDK_THREAD_OP_RESCHED`. 255The operation `SPDK_THREAD_OP_NEW` is called each time a new thread is created. 256The operation `SPDK_THREAD_OP_RESCHED` is called when SPDK thread needs to be rescheduled. 257 258A unique ID has been added for each created SPDK thread, it is retrieved by a new function 259`spdk_thread_get_id`, and the SPDK thread which has the specific ID is got by 260a new function `spdk_thread_get_by_id`. 261 262A new function `spdk_thread_cpumask` has been added to set the current thread's cpumask 263to the specified value. The function requires the operation `SPDK_THREAD_OP_RESCHED` 264is supported. 265 266A new function `spdk_poller_register_named` has been added to set arbitrary name to the 267created poller. If NULL, the name is set to the pointer of the poller function. 268 269The function `spdk_thread_poll` now measures run time per call correctly on multiple SPDK 270threads configuration, and a new function `spdk_thread_get_last_tsc` has been added to use together. 271 272Voluntary termination of SPDK thread has been supported by refining the functions `spdk_thread_exit` 273and `spdk_thread_poll`. 274 275### util 276 277New functions `spdk_sn32_lt` and `spdk_sn32_gt` have been added. They compare two sequence 278numbers based on serial number arithmetic. 279 280### vhost 281 282Poll groups per session have been replaced by SPDK threads per vhost controller. 283 284### vmd 285 286A new function, `spdk_vmd_fini`, has been added. It releases all resources acquired by the VMD 287library through the `spdk_vmd_init` call. 288 289## v20.01 290 291### bdev 292 293A new function, `spdk_bdev_set_timeout`, has been added to set per descriptor I/O timeouts. 294 295A new class of functions `spdk_bdev_compare*`, have been added to allow native bdev support 296of block comparisons and compare-and-write. 297 298A new class of bdev events, `SPDK_BDEV_EVENT_MEDIA_MANAGEMENT`, has been added to allow bdevs 299which expose raw media to alert all I/O channels of pending media management events. 300 301A new API was added `spdk_bdev_io_get_aux_buf` allowing the caller to request 302an auxiliary buffer for its own private use. The API is used in the same manner that 303`spdk_bdev_io_get_buf` is used and the length of the buffer is always the same as the 304bdev_io primary buffer. 'spdk_bdev_io_put_aux_buf' frees the allocated auxiliary 305buffer. 306 307### blobfs 308 309Added boolean return value for function spdk_fs_set_cache_size to indicate its operation result. 310 311Added `blobfs_set_cache_size` RPC method to set cache size for blobstore filesystem. 312 313### blobstore 314 315Added new `use_extent_table` option to `spdk_blob_opts` for creating blobs with Extent Table descriptor. 316Using this metadata format, dramatically decreases number of writes required to persist each cluster allocation 317for thin provisioned blobs. Extent Table descriptor is enabled by default. 318See the [Blobstore Programmer's Guide](https://spdk.io/doc/blob.html#blob_pg_cluster_layout) for more details. 319 320### dpdk 321 322Updated DPDK submodule to DPDK 19.11. 323 324### env_dpdk 325 326`spdk_env_dpdk_post_init` now takes a boolean, `legacy_mem`, as an argument. 327 328A new function, `spdk_env_dpdk_dump_mem_stats`, prints information about the memory consumed by DPDK to a file specified by 329the user. A new utility, `scripts/dpdk_mem_info.py`, wraps this function and prints the output in an easy to read way. 330 331### event 332 333The functions `spdk_reactor_enable_framework_monitor_context_switch()` and 334`spdk_reactor_framework_monitor_context_switch_enabled()` have been changed to 335`spdk_framework_enable_context_switch_monitor()` and 336`spdk_framework_context_switch_monitor_enabled()`, respectively. 337 338### ftl 339 340All NVMe dependencies were removed from ftl library. 341From now ftl library is using bdev_zone API. 342bdev_ftl becomes virtual bdev. 343 344`ctrlr` and `trid` fields in `spdk_ftl_dev_init_opts` structure were replaced by 345`base_bdev_desc`. 346 347`bdev_ftl_create` RPC method `trtype` and `traddr` parameters were replaced by `base_bdev` 348parameter. 349 350`spdk_ftl_module_init` and `spdk_ftl_module_fini` functions were removed. 351 352`spdk_ftl_punit_range` and `ftl_module_init_opts` structures were removed. 353 354### isa-l 355 356Updated ISA-L submodule to commit f3993f5c0b6911 which includes implementation and 357optimization for aarch64. 358 359Enabled ISA-L on aarch64 by default in addition to x86. 360 361### nvme 362 363`delayed_pcie_doorbell` parameter in `spdk_nvme_io_qpair_opts` was renamed to `delay_cmd_submit` 364to allow reuse in other transports. 365 366Added RDMA WR batching to NVMf RDMA initiator. Send and receive WRs are chained together 367and posted with a single call to ibv_post_send(receive) in the next call to qpair completion 368processing function. Batching is controlled by 'delay_cmd_submit' qpair option. 369 370The NVMe-oF initiator now supports plugging out of tree NVMe-oF transports. In order 371to facilitate this feature, several small API changes have been made: 372 373The `spdk_nvme_transport_id` struct now contains a trstring member used to identify the transport. 374A new function, `spdk_nvme_transport_available_by_name`, has been added. 375A function table, `spdk_nvme_transport_ops`, and macro, `SPDK_NVME_TRANSPORT_REGISTER`, have been added which 376enable registering out of tree transports. 377 378A new function, `spdk_nvme_ns_supports_compare`, allows a user to check whether a given namespace supports the compare 379operation. 380 381A new family of functions, `spdk_nvme_ns_compare*`, give the user access to submitting compare commands to NVMe namespaces. 382 383A new function, `spdk_nvme_ctrlr_cmd_get_log_page_ext`, gives users more granular control over the command dwords sent in 384log page requests. 385 386### nvmf 387 388Add SockPriority option in [Transport] section, this can be used for NVMe-oF target 389on TCP transport to set sock priority for the incomming TCP connections. 390 391The NVMe-oF target now supports plugging out of tree NVMe-oF transports. In order 392to facilitate this feature, several small API changes have been made: 393 394The `spdk_nvme_transport_id` struct now contains a trstring member used to identify the transport. 395`spdk_nvmf_tgt_get_transport`, `spdk_nvmf_transport_opts_init`, and `spdk_nvmf_transport_create` all have been 396modified to take a string. 397A function table, `spdk_nvmf_transport_ops`, and macro, `SPDK_NVMF_TRANSPORT_REGISTER`, have been added which 398enable registering out of tree transports. 399 400Add `spdk_nvmf_tgt_stop_listen()` that can be used to stop listening for 401incoming connections for specified target and trid. Listener is not stopped 402implicitly upon destruction of a subsystem any more. 403 404A custom NVMe admin command handler has been added which allows the user to use the real drive 405attributes from one of the target NVMe drives when reporting drive attributes to the initiator. 406This handler can be enabled via the `nvmf_set_config` RPC. 407Note: In a future version of SPDK, this handler will be enabled by default. 408 409The SPDK target and initiator both now include compare-and-write functionality with one caveat. If using the RDMA transport, 410the target expects the initiator to send both the compare command and write command either with, or without inline data. The 411SPDK initiator currently respects this requirement, but this note is included as a flag for other initiators attempting 412compatibility with this version of SPDK. 413 414The `spdk_nvme_ctrlr_opts` struct has been extended with new field `transport_ack_timeout` which allows 415to configure transport ACK timeout. Applicable for RDMA transport only. 416 417### rpc 418 419A new RPC, `bdev_zone_block_create`, enables creating an emulated zoned bdev on top of a standard block device. 420 421A new RPC, `bdev_ocssd_create`, enables creating an emulated zoned bdev on top of an Open Channel SSD. 422 423A new RPC, `blobfs_set_cache_size`, enables managing blobfs cache size. 424 425A new RPC, `env_dpdk_get_mem_stats`, has been added to facilitate reading DPDK related memory 426consumption stats. Please see the env_dpdk section above for more details. 427 428A new RPC, `framework_get_reactors`, has been added to retrieve a list of all reactors. 429 430`bdev_ftl_create` now takes a `base_bdev` argument in lieu of `trtype`, `traddr`, and `punits`. 431 432`bdev_nvme_set_options` now allows users to disable I/O submission batching with the `-d` flag 433 434`bdev_nvme_cuse_register` now accepts a `name` parameter. 435 436`bdev_uring_create` now takes arguments for `bdev_name` and `block_size` 437 438`nvmf_set_config` now takes an argument to enable passthru of identify commands to base NVMe devices. 439Please see the nvmf section above for more details. 440 441### scsi 442 443`spdk_scsi_lun_get_dif_ctx` now takes an additional argument of type `spdk_scsi_task`. 444 445### sock 446 447Added spdk_sock_writev_async for performing asynchronous writes to sockets. This call will 448never return EAGAIN, instead queueing internally until the data has all been sent. This can 449simplify many code flows that create pollers to continue attempting to flush writes 450on sockets. 451 452Added `impl_name` parameter in spdk_sock_listen and spdk_sock_connect functions. Users may now 453specify the sock layer implementation they'd prefer to use. Valid implementations are currently 454"vpp" and "posix" and NULL, where NULL results in the previous behavior of the functions. 455 456### thread 457 458`spdk_thread_send_msg` now returns int indicating if the message was successfully 459sent. 460 461A new function `spdk_thread_send_critical_msg`, has been added to support sending a single message from 462a context that may be interrupted, e.g. a signal handler. 463 464Two new functions, `spdk_poller_pause`, and `spdk_poller_resume`, have been added to give greater control 465of pollers to the application owner. 466 467### util 468 469`spdk_pipe`, a new utility for buffering data from sockets or files for parsing 470has been added. The public API is available at `include/spdk/pipe.h`. 471 472### bdev 473 474Added spdk_bdev_io_get_nvme_fused_status function for translating bdev_io status to NVMe status 475code for fused compare-and-write operation. 476 477Added spdk_bdev_get_acwu function for getting block device atomic compare and write unit size. 478 479## v19.10 480 481### rpc 482 483Many of SPDK's RPCs were renamed to be more consistent and intuitive in this release. 484The old names will continue to function, but will display a deprecation warning. 485 486Added optional parameters '--arbitration-burst' and '--low/medium/high-priority-weight' to 487'bdev_nvme_set_options' RPC method. 488 489Added optional parameter '--md-size' to 'construct_null_bdev' RPC method. 490 491Added optional parameters '--dif-type' and '--dif-is-head-of-md' to 'construct_null_bdev' 492RPC method. 493 494Added `blobfs_detect` RPC method to detect whether a blobfs exists on given bdev. 495 496Added `blobfs_create` RPC method to build blobfs on given bdev. 497 498Added `blobfs_mount` RPC method to mount blobfs on given bdev to a host path by FUSE. 499Then on the host path, user can directly do some file operations which will be mapped 500to blobfs. 501 502### bdev 503 504Added new parameter `cdw0` to `spdk_bdev_io_complete_nvme_status()` and 505`spdk_bdev_io_get_nvme_status()` that allows setting/getting 506the NVMe completion queue DW0 entry. This allows vendor specific IO commands 507to return commmand specific completion info back to the initiator. 508 509Added `spdk_bdev_get_write_unit_size()` function for retrieving required number 510of logical blocks for write operation. 511 512New zone-related fields were added to the result of the `get_bdevs` RPC call: 513 514 - `zoned`: indicates whether the device is zoned or a regular 515 block device 516 - `zone_size`: number of blocks in a single zone 517 - `max_open_zones`: maximum number of open zones 518 - `optimal_open_zones`: optimal number of open zones 519 520The `zoned` field is a boolean and is always present, while the rest is only available for zoned 521bdevs. 522 523A new `spdk_bdev_open_ext` function has been added and `spdk_bdev_open` function has been deprecated. 524The new open function introduces requirement to provide callback function that will be called by 525asynchronous event such as bdev removal. `spdk_bdev_open_ext` function takes bdev name as 526an argument instead of bdev structure to avoid a race condition that can happen when the bdev 527is being removed between a call to get its structure based on a name and actually openning it. 528 529New 'resize' event has been added to notify about change of block count property of block device. 530Event is delivered only if block device was opened with `spdk_bdev_open_ext` function. 531 532### bdev zone 533 534Added new public header for zoned bdev. Zoned bdev is an extension 535of the bdev interface. 536 537`spdk_bdev_get_zone_size()`, `spdk_bdev_get_max_open_zones()`, `spdk_bdev_get_optimal_open_zones()` 538APIs were added for retrieving zoned device information. 539`spdk_bdev_get_zone_info()` API was added for retrieving information about zones in zoned 540device. 541Added `spdk_bdev_zone_management()` API for changing zone state. 542`spdk_bdev_zone_append()` and `spdk_bdev_zone_append_with_md()` APIs were added for 543appending data to a zone. 544Added `spdk_bdev_io_get_append location()` function for retrieving append location for I/O. 545Added `spdk_bdev_is_zoned()` function for checking if bdev supports zoned namespace semantics. 546 547### bdev opal 548 549EXPERIMENTAL: A new opal bdev has been added to support management of 550NVMe self-encrypting drives through the Opal specification. Users can 551create opal bdevs from an NVMe namespace bdev, if the controller 552containing that namespace supports Opal. Currently this is only 553supported for namespace ID=1. The following RPCs have been added to 554support Opal: `bdev_nvme_opal_init`, `bdev_nvme_opal_revert`, 555`bdev_opal_create`, `bdev_opal_delete`, `bdev_opal_get_info`, 556`bdev_opal_new_user`, `bdev_opal_set_lock_state`. 557It does not yet support recreating the opal bdevs after application restart. 558This bdev module should be considered very experimental, and the RPCs may 559change significantly in future releases. 560 561### delay bdev 562 563The `bdev_delay_update_latency` has been added to allow users to update 564a latency value for a given delay bdev. 565 566### compress bdev 567 568A new RPC `bdev_compress_get_orphans` has been added to list compress bdevs 569that were not loaded due to a missing pm metadata file. In this state they 570can only be deleted. 571 572### null bdev 573 574Metadata support has been added to Null bdev module. 575 576Protection information support has been added to Null bdev module. 577 578### nvme 579 580Added `no_shn_notification` to NVMe controller initialization options, users can enable 581it for NVMe controllers. When the option is enabled, the controller will not do the 582shutdown process and just disable the controller, users can start their application 583later again to initialize the controller to the ready state. 584 585A controller flag `SPDK_NVME_CTRLR_WRR_SUPPORTED` was added to indicate the controller 586can support weighted round robin arbitration feature with submission queue. 587 588Added `arbitration_burst` option for arbitration feature, and added three 589`low/medium/high_priority_weight` options for weighted round robin arbitration. 590 591Added `spdk_nvme_ns_cmd_write_uncorrectable`. 592 593Added new error handling and reporting functionality. This includes several 594new API functions to facilitate applications recovering when a qpair or 595controller fails. 596 597`spdk_nvme_ctrlr_reconnect_io_qpair` - Reconnects a failed I/O qpair. 598`spdk_nvme_ctrlr_set_trid` - Sets the trid of an existing controller. Can be used to 599change the trid for failover cases. 600`spdk_nvme_ctrlr_is_failed` - Returns the failed state of a controller. 601`spdk_nvme_ctrlr_fail` - Forces a controller into a failed state. 602 603Modified the return behavior of several API functions to better indicate to 604applications when a qpair is failed. This list of functions includes: 605 606`spdk_nvme_qpair_process_completions` 607`spdk_nvme_ns_cmd_*` 608`spdk_nvme_ctrlr_process_admin_completions` 609`spdk_nvme_ctrlr_cmd_*` 610 611These functions now return -ENXIO when the qpair or controller on which they 612operate is failed. 613 614EXPERIMENTAL: Added NVMe character device support to allow to create NVMe device nodes in Linux 615kernel for controller as well as for namespace and process ioctl requests as usual 616from linux environment. 617 618### nvmf 619 620The `spdk_nvmf_tgt_create` function now accepts an object of type `spdk_nvmf_target_opts` 621as its only parameter. This new structure contains the max_subsystems parameter previously 622passed into that function. 623 624A new public API function `spdk_nvmf_get_tgt` has been added which allows users to 625retrieve a pointer to an `spdk_nvmf_tgt` object by supplying its name. In the special 626case where an RPC or application only creates a single target, this function can accept 627a null name parameter and will return the only available target. 628 629The majority of the NVMe-oF RPCs now accept an optional tgt_name parameter. This will 630allow those RPCs to work with applications that create more than one target. 631 632Three new NVMe-oF RPCs have been added `nvmf_create_target`, `nvmf_delete_target`, and 633`nvmf_get_targets`. These new RPCs provide a basic interface for managing multiple target 634objects. In SPDK the target object defines a unique discovery service. As of this release, 635these RPCs are not intended to be used with the in-tree SPDK target applications, spdk_tgt and 636nvmf_tgt, which use a single, global target structure. As such, they are not included in scripts/rpc.py 637 638Three new header functions have also been added to help deal with multiple targets. 639`spdk_nvmf_tgt_get_name` takes a target pointer as an argument and returns its human readable name. 640`spdk_nvmf_get_first_target` takes no arguments and returns the first target in the global list. 641`spdk_nvmf_get_next_tgt` takes a target pointer as an argument and returns the next one in the global list. 642 643The `spdk_nvmf_tgt_accept` takes additional argument allowing to pass arbitrary context 644information to the `new_qpair` callback. This will simplify the code when having multiple 645nvmf targets or when retrieving the context information from globals is not suitable. 646 647### blobstore 648 649A new `spdk_bdev_create_bs_dev_from_desc` function has been added and `spdk_bdev_create_bs_dev` 650function has been deprecated. 651The new create function can cowork with `spdk_bdev_open_ext` function, which provides callback 652function that will be called by asynchronous event such as bdev removal. 653 654### blobfs_bdev 655 656A new blobfs module `bdev` has been added to simplify the operations of blobfs on bdev. 657 658Function `spdk_blobfs_bdev_detect` is added to detect whether blobfs exists on the given block device. 659 660Function `spdk_blobfs_bdev_create` is added to create a blobfs on the given block device. 661 662Function `spdk_blobfs_bdev_mount` is added to mount a blobfs on the given block device to 663a host path by FUSE. Then, a new thread is created dedicatedly for one mountpoint to handle 664FUSE request by blobfs API. 665 666### build 667 668Option to build FUSE components into blobfs_bdev module for mounting a blobfs filesystem. 669It requires the installation of libfuse3. By default, it is disabled. And it will be 670enabled if run `./configure` with `--with-fuse` option. 671 672### iSCSI 673 674Portals may no longer be associated with a cpumask. The scheduling of 675connections is moving to a more dynamic model. 676 677An new RPC `iscsi_portal_group_set_auth` has been added to set CHAP authentication 678for discovery sessions specific for the existing iSCSI portal group. This RPC overwrites 679the setting by the global parameters for the iSCSI portal group. 680 681### socket 682 683Added `spdk_sock_is_connected` to check whether the socket is currently connected. 684`spdk_sock_group_poll` now returns number of events on success. 685 686### env 687 688Added `spdk_pci_device_unclaim()` function to cleanup pci claim file. 689 690### event 691 692`framework_start_init` RPC no longer stops the application on error during 693initialization. 694 695### DPDK 696 697Updated DPDK submodule to DPDK 19.08. 698 699### ocf 700 701Updated OCF submodule to OCF v19.06 702 703Along with update, new cache mode 'write only' was added. 704 705New cache modes added to use via RPC, wi - write invalidate and wa - write around. 706 707New version of OCF provides fully asynchronous management API. 708 709## v19.07 710 711### ftl 712 713EXPERIMENTAL: Added ability to mirror writes to persistent write buffer cache 714to allow for recovery from dirty shutdown event. 715 716Added handling of Asynchronous Nand Management Events (ANM). 717 718### vmd 719 720EXPERIMENTAL: Added Intel Volume Management Device (VMD) driver. VMD is an integrated 721controller inside the CPU PCIe root complex. It enables virtual HBAs for the connected 722NVMe SSDs. `spdk_vmd_init()` enumerates NVMe SSDs behind VMD device and hook them into 723SPDK PCI subsystem. `spdk_nvme_probe()` or `spdk_nvme_connect()` can be used to connect 724NVMe driver to the device located at the given transport ID. 725 726To obtain transport ID of NVMe SSD behind VMD `spdk_lspci` can be used. 727 728Current implementation does not support hotplug. 729 730### blobfs 731 732Blobfs file asynchronous operations were added to public APIs. 733 734### util 735 736A new file API `spdk_posix_file_load` was added to load file content into a data buffer. 737 738New APIs `spdk_dif_ctx_set_data_offset`, `spdk_dif_verify_stream`, 739`spdk_dif_get_range_with_md`, `spdk_dif_get_length_with_md` have been added, 740and existing APIs `spdk_dif_ctx_init`, `spdk_dif_set_md_interleave_iovs`, and 741`spdk_dif_generate_stream` have been refined to insert or strip DIF by stream 742fasion with any alignment. 743 744New APIs `spdk_dif_ctx_set_remapped_init_ref_tag`, `spdk_dif_remap_ref_tag`, 745and `spdk_dix_remap_ref_tag` have been added to remap DIF reference tag. 746 747New APIs `spdk_dif_update_crc32c` and `spdk_dif_update_crc32c_stream` have been 748added to compute CRC-32C checksum for extended LBA payload. 749 750Bdevperf and bdevio applications now support starting tests with application specific 751RPCs. Please see helper Python scripts in their respective directories. 752This is a move towards simpler RPC-only configuration for all main 753and auxiliary applications. 754 755Legacy INI style configuration for SPDK applications will become deprecated in SPDK 19.10, 756and removed in SPDK 20.01. Please consider moving to JSON-RPC configuration files and/or 757RPC driven run-time configuration. 758 759### nvmf 760 761EXPERIMENTAL: A Fibre Channel transport that supports Broadcom HBAs has been 762added. This depends on the FC HBA driver at 763https://github.com/ecdufcdrvr/bcmufctdrvr. See [the documentation](https://spdk.io/doc/nvmf.html#nvmf_fc_transport) 764for more information. 765 766Persistent reservation emulation has been added to the NVMe-oF target. Persistent reservation 767state is stored in a JSON file on the local filesystem between target restart. To support this, 768an optional parameter to the RPC method `nvmf_subsystem_add_ns` called `--ptpl-file` was added. 769This allows the user to specify which file to store the persistent reservation state in. Note 770that this is done per namespace. 771 772The c2h success optimization under which a command capsule response is not sent 773for reads is turned on by default. A config knob was added to allow disabling 774the optimization. This will mostly be used for integration testing with 5.0.x kernels 775while some compatibility fixes make their way down the pipeline for 5.1.x kernels. 776 777The sock priority setting of the TCP connection owned by the tcp transport is added. It is 778used to optimize the TCP connection performance under designated traffic classes. And the 779priority is used to differeniate the sock priority between SPDK NVMe-oF TCP target application 780and other TCP based applications. 781 782Shared receive queue can now be disabled even for NICs that support it using the 783`nvmf_create_transport` RPC method parameter `no_srq`. The actual use of a shared 784receive queue is predicated on hardware support when this flag is not used. 785 786spdk_nvmf_get_optimal_poll_group was added, which is used to return the optimal 787poll group for the qpair. And `ConnectionScheduler` configuration is added into the 788[Nvmf] section in etc/spdk/nvmf.conf.in to demonstrate how to configure the connection 789scheduling strategy among different spdk threads. 790 791Added infrastructure to retrieve global and per poll group NVMf statistics. 792 793DIF strip and insert is now supported for TCP transport. When it is enabled, DIF 794setting is not exposed to the NVMe-oF initiator, and DIF is attached into data 795for write I/O and stripped from data for read I/O. 796 797Added a field `dif_insert_or_strip` to struct spdk_nvmf_transport_opts, and 798updated the related rpc function nvmf_create_transport to make this 799configurable parameter available to users. The `dif_insert_or_strip` is relevant 800for TCP transport for now and used to configure the DIF strip and insert. 801 802Added infrastructure to retrieve NVMf transport statistics. 803 804### notify 805 806The function `spdk_notify_get_types()` and `spdk_notify_get_events()` were 807renamed to `spdk_notify_foreach_type()` and `spdk_notify_foreach_event()`, 808respectively. And update type name of callback accordingly. 809 810### bdev 811 812The format of the data returned by the get_bdevs_iostat RPC has changed to 813make it easier to parse. It now returns an object with a "ticks" object 814and "bdevs" array with the per-bdev statistics. 815 816A new bdev module `delay` has been added which simulates a drive latency when placed 817on top of a Null bdev. This module is intended only for testing and can be created using 818the new RPC `bdev_delay_create`. That RPC takes the name of the underlying bdev as well 819as average and p99 latency arguments for both read and write operations. Average latency is 820defined as a value close to what you would expect a perf tool such as FIO to report back as 821the mean latency of all I/O submitted to the drive. p99 latency is defined as the value one 822would expect the drive to see the slowest 1% of I/O report. For underlying drives with already 823significant latency, the latency values provided to the drive will be additive. This should be 824taken into account if trying to achieve an artificial latency on top of an nvme drive or aio device. 825 826DIF reference tag remapping is now supported for partition type virtual bdev 827modules. When using partition type virtual bdevs, block address space is 828remapped during I/O processing and DIF reference tag is remapped accordingly. 829 830Added spdk_bdev_*_with_md() functions allowing for IO with metadata being transferred in 831separate buffer. To check support for separatate metadata, use spdk_bdev_is_md_separate(). 832 833All bdevs now have a UUID. For devices whose backing hardware does not provide a UUID, 834one is automatically generated. Across runs of SPDK, bdevs whose UUID is automatically 835generated may change. 836 837A new virtual bdev module `compress` has been added to provide compression services on top of 838a thinly provisioned logical volume. See documentation for complete details. 839 840### nvme 841 842Added an optional parameter `--io-queue-requests` to RPC `set_bdev_nvme_options`, which 843can be used to change the number of requests allocated for one NVMe I/O queue. For 844very big I/O size, e.g. 128MiB, with this option user will not get an error due to 845limited requests in NVMe driver layer. 846 847Added spdk_nvme_ctrlr_get_transport_id() to get the transport ID from a 848previously attached controller. 849 850Nvme Opal library spdk_opal_cmd deprecated. Adding seperate command APIs. 851NVMe Opal library add support for activating locking SP which will make the transaction 852from "Manufactured-Inactive" state to "Manufactured" state. Upon successfully invoking 853of this method, lock and unlock features will be enabled. 854NVMe Opal library add support for locking/unlocking range and list locking range info. 855NVMe opal library add support for multiuser. Admin can enable user and add user to specific 856locking range and the user can lock/unlock his range. 857 858Added spdk_nvme_ctrlr_io_cmd_raw_no_payload_build() allowing a caller to pass 859a completely formed command to an NVMe submission queue (buffer addresses and all). 860This is supported on the PCIe transport only. 861 862Added spdk_nvme_get_ctrlr_registers() to return a pointer to the virtual address 863of the NVMe controller registers. This is supported on the PCIe transport only. 864 865Added additional options to the spdk_nvme_ctrlr_alloc_qpair() option parameter 866structure to allow caller to override the virtual and optionally physical address 867of the submission and completion queue pair to be created. This is supported on 868the PCIe transport only. 869 870Added `disable_error_logging` to struct spdk_nvme_ctrlr_opts, which disables 871logging of failed requests. By default logging is enabled. 872 873Added spdk_nvme_qpair_print_command(), spdk_nvme_qpair_print_completion() and 874spdk_nvme_cpl_get_status_string(). Allowing for easier display of error messages. 875 876Added support for NVMe Sanitize command. 877 878### env 879 880The parameter `free_space` has been added to spdk_ring_enqueue() to wait when 881the ring is almost full and resume when there is enough space available in 882the ring. 883 884A new API `spdk_mempool_lookup` has been added to lookup the memory pool created 885by the primary process. 886 887Added spdk_pci_get_first_device() and spdk_pci_get_next_device() to allow 888iterating over PCI devices detected by SPDK. Because of this, all SPDK APIs 889to attach/detach PCI devices are no longer thread safe. They are now meant to 890be called from only a single thread only, the same only that called spdk_env_init(). 891This applies to the newly added APIs as well. 892 893### vpp 894 895SPDK now supports VPP version 19.04.2, up from VPP 18.01. 896 897VPP socket abstraction now uses VPP Session API, instead of VLC (VPP Communications Library). 898This allows for better control over sessions and queues. 899Please see VPP documentation for more details: 900[VPP Host Stack](https://wiki.fd.io/view/VPP/HostStack) 901 902### sock 903 904Add spdk_sock_get_optimal_sock_group(), which returns the optimal sock group for 905this socket. When a socket is created, it is often assigned to a sock group using 906spdk_sock_group_add_sock so that a set of sockets can be polled more efficiently. 907For some network devices, it is optimal to assign particular sockets to specific 908sock groups. This API is intended to provide the user with that information. 909 910spdk_sock_group_get_ctx() was added to return the context of the spdk_sock_group. 911spdk_sock_group_create() is updated to allow input the user provided ctx. 912spdk_sock_set_priority() is added to set the priority of the socket. 913 914### rpc 915 916Added thread_get_stats RPC method to retrieve existing statistics. 917 918Added nvmf_get_stats RPC method to retrieve NVMf susbsystem statistics. 919 920Response buffers for RPC requests are now always pre-allocated, which implies 921that all spdk_jsonrpc_begin_result() calls always succeed and return a valid 922buffer for JSON response. RPC calls no longer need to check if the buffer is 923non-NULL. 924 925Added SPDK_RPC_REGISTER_ALIAS_DEPRECATED to help with deprecation process when 926renaming existing RPC. First time a deprecated alias is used, it will print 927a warning message. 928 929RPC `get_rpc_methods` was renamed `rpc_get_methods`. The old name is still usable, 930but is now deprecated. 931 932### blobstore 933 934A snapshot can now be deleted if there is only a single clone on top of it. 935 936### build 937 938Preliminary support for cross compilation is now available. Targeting an older 939CPU on the same architecture using your native compiler can be accomplished by 940using the `--target-arch` option to `configure` as follows: 941 942~~~ 943./configure --target-arch=broadwell 944~~~ 945 946Additionally, some support for cross-compiling to other architectures has been 947added via the `--cross-prefix` argument to `configure`. To cross-compile, set CC 948and CXX to the cross compilers, then run configure as follows: 949 950~~~ 951./configure --target-arch=aarm64 --cross-prefix=aarch64-linux-gnu 952~~~ 953 954### vhost 955 956A security vulnerability has been identified and fixed in SPDK Vhost-SCSI target. 957A malicious client (e.g. a virtual machine) could send a carefully prepared, 958invalid I/O request to crash the entire SPDK process. All users of SPDK Vhost-SCSI 959target are strongly recommended to update. All SPDK versions < 19.07 are affected. 960 961By default, SPDK will now rely on upstream DPDK's rte_vhost instead of its fork 962located inside SPDK repo. The internal fork is still kept around to support older 963DPDK versions, but is considered legacy and will be eventually removed. 964 965`configure` will now automatically use the upstream rte_vhost if the used DPDK 966version is >= 19.05. 967 968spdk_vhost_init() is now asynchronous and accepts a completion callback. 969 970### iscsi target 971 972A security vulnerability has been identified and fixed in SPDK iSCSI target. 973A malicious client (e.g. an iSCSI initiator) could send a carefully prepared, 974invalid I/O request to crash the entire SPDK process. All users of SPDK iSCSI 975target are strongly recommended to update. All SPDK versions < 19.07 are affected. 976 977### thread 978 979Exposed spdk_set_thread() in order for applications to associate 980with SPDK thread when necessary. 981 982Added spdk_thread_destroy() to allow framework polling the thread to 983release resources associated with that thread. 984 985## v19.04 986 987### nvme 988 989Added asynchronous probe support. New APIs spdk_nvme_probe_async(), 990spdk_nvme_connect_async() and spdk_nvme_probe_poll_async() were added to enable 991this feature. spdk_nvme_probe_async() and spdk_nvme_connect_async() return a 992context associated with the specified controllers. Users then call 993spdk_nvme_probe_poll_async() until it returns 0, indicating that the operation 994completed. 995 996A new qpair creation option, delay_pcie_doorbell, was added. This can be passed 997to spdk_nvme_alloc_io_qpair(). This makes the I/O submission functions, such as 998spdk_nvme_ns_writev(), skip ringing the submission queue doorbell. Instead the 999doorbell will be rung as necessary inside spdk_nvme_qpair_process_completions(). 1000This can result in significantly fewer MMIO writes to the doorbell register 1001under heavy load, greatly improving performance. 1002 1003spdk_nvme_ctrlr_get_regs_cmbsz() was added to report the size of the controller 1004memory buffer, if available. 1005 1006spdk_nvme_ctrlr_get_flags() was added to return controller feature 1007flags. Two flags are currently tracked: 1008SPDK_NVME_CTRLR_SGL_SUPPORTED 1009SPDK_NVME_CTRLR_SECURITY_SEND_RECV_SUPPORTED 1010 1011The NVMe hotplug poller is now able to detach devices hot-removed from the system 1012via `/sys/bus/pci/devices/<bdf>/remove` and `/sys/bus/pci/devices/<bdf>/driver/unbind`. 1013 1014Opal support was added for scan, take ownership, revert TPer, and dumping device 1015info. The nvme_manage tool can be used to perform these operations. The public 1016API functions are spdk_nvme_ctrlr_security_receive() and 1017spdk_nvme_ctrlr_security_send(). This module should be considered experimental 1018pending additional features and tests. 1019 1020The NVMe-oF initiator is now able to transfer IO whose size is more than 128KiB 1021if the corresponding NVMe-oF target allows. 1022 1023### raid 1024 1025Added new strip_size_kb rpc param on create to replace the more ambiguous 1026strip_size. The strip_size rpc param is deprecated. 1027 1028Changed the raid bdev product_name from "Pooled Device" to "Raid Volume" 1029 1030### thread 1031 1032Added spdk_thread_has_pollers() function to verify if there are any registered 1033pollers to be run on the thread. Added spdk_thread_is_idle() function to check 1034if there are any scheduled operations to be performed on the thread at given 1035time. 1036 1037spdk_thread_create() now takes an optional CPU affinity mask that is passed to 1038the scheduler. 1039 1040spdk_thread_lib_init() now takes an optional context size. For each thread 1041created, an additional region of memory of this size will be allocated. A 1042pointer to this region of memory can be obtained by calling 1043spdk_thread_get_ctx(). The inverse operation is also available via 1044spdk_thread_get_from_ctx(). 1045 1046spdk_thread_poll() now optionally accepts the current time, in ticks. This can 1047be used to avoid some calls to spdk_get_ticks() internally. 1048 1049spdk_app_start() now only accepts a single context argument. 1050 1051### bdev 1052 1053An new API `spdk_bdev_get_data_block_size` has been added to get size of data 1054block except for metadata. 1055 1056spdk_vbdev_register() has been deprecated. spdk_bdev_register() should be used 1057instead. 1058 1059A mechanism for acquiring and releasing data buffers from bdev modules, used 1060to perform zero copy operations, was added. 1061 1062New APIs spdk_bdev_get_md_size(), spdk_bdev_is_md_interleaved(), spdk_bdev_get_dif_type(), 1063spdk_bdev_is_dif_head_of_md(), and spdk_bdev_is_dif_check_enabled() have been 1064added to get metadata and DIF settings. 1065 1066Bdevs claimed by the `examine_config` callback will be now further examined in the 1067`examine_disk` callback. 1068 1069spdk_bdev_io_get_io_channel() was added as a convenient way to get an io_channel 1070from a bdev_io. 1071 1072### ocf 1073 1074Updated OCF submodule to OCF v19.3.2 1075 1076Added support for many-to-one configuration for OCF bdev. 1077Multiple core devices can now be cached on single cache device. 1078 1079Added persistent metadata support, allowing to restore cache state after shutdown. 1080During start of SPDK application, the devices are examined and if OCF metadata 1081is present - appropriate OCF bdevs will be recreated. 1082 1083Added Write-Back mode support. In this mode, data is first written to 1084caching device and periodically synchronized to the core devices. 1085Dirty data is saved as persistent metadata on cache device, 1086allowing for safe restore during application restart. 1087For more details please see OCF documentation: 1088[OpenCAS cache configuration](https://open-cas.github.io/cache_configuration.html) 1089 1090### NVMe-oF Target 1091 1092Support for per-device shared receive queues in the RDMA transport has been added. 1093It is enabled by default for any device that supports it. 1094 1095The size of a shared receive queue is defined by transport configuration file parameter 1096`MaxSRQDepth` and `nvmf_create_transport` RPC method parameter `max_srq_depth`. 1097Default size is 4096. 1098 1099Add model number as parameter to construct_nvmf_subsystem (-d option), 1100rather than using hardcoded define. 1101 1102DIF passthrough feature has been added. DIF setting of the allocated bdevs is 1103exposed to the NVMe-oF initiator and data with DIF from the NVMe-oF initiator is 1104passed through to the allocated bdevs. 1105 1106### env 1107 1108The `phys_addr` parameter in spdk_malloc() and spdk_zmalloc() has been deprecated. 1109For retrieving physical addresses, spdk_vtophys() should be used instead. 1110 1111spdk_realloc() has been added to reallocate DMA/shared memory. 1112 1113spdk_pci_device_is_removed() has been added to let the upper-layer SPDK drivers know 1114that device has a pending external hotremove request. 1115 1116spdk_env_fini() and spdk_env_dpdk_post_fini() were added to release any resources 1117allocated by spdk_env_init() or spdk_env_dpdk_post_init() respectively. It is expected 1118that common usage of those functions is to call them just before terminating the process. 1119 1120Added spdk_iommu_is_enabled() to report if SPDK application is using IOMMU for DMA. 1121 1122### DPDK 1123 1124Dropped support for DPDK 17.07 and earlier, which SPDK won't even compile with right now. 1125 1126Updated DPDK submodule to DPDK 19.02. 1127 1128### rpc 1129 1130New `get_spdk_version` RPC method is introduced to get version info of the running SPDK application. 1131 1132The `start_nbd_disk` RPC method now take nbd_device as an optional parameter. If nbd_device 1133is specified, use that specified nbd device. If it's not specified, pick available one. 1134 1135### iSCSI target 1136 1137DIF strip and insert is now supported. DIF settings are not exposed to the iSCSI initiator. 1138DIF is attached into data for write I/O and stripped from data for read I/O. 1139 1140### vhost 1141 1142Added experimental support for running with the external, upstream rte_vhost library. 1143This can be enabled by configuring SPDK with an `--without-internal-vhost-lib` flag. 1144The minimum supported rte_vhost version (DPDK version) is 19.05-rc1. 1145 1146As a result of fuzz testing, a lot of data races in vhost-scsi LUN hotplug path were identified and 1147fixed. Those data races could have potentially resulted in SPDK crashes, RPC hangs, or memory leaks 1148if Vhost-SCSI LUN hotplug RPCs were executed while connected VMs were in the middle of restarting. 1149 1150The SCSI target id in `add_vhost_scsi_lun` RPC is now optional. If `-1` is passed, the first 1151unoccupied target id will be used. 1152 1153### AIO 1154 1155AIO bdev module can now reap I/O completions directly from userspace, significantly improving 1156the overall performance. 1157 1158### blobfs 1159 1160Synchronous IO operations no longer use spdk_io_channel, but instead use 1161spdk_fs_thread_ctx. The behavior is otherwise identical. 1162 1163### OCF 1164 1165Added support for caching multiple bdevs using a single bdev as a cache. 1166 1167### notify 1168 1169Added the notify library that provides a high performance local event bus 1170between libraries. Example usage was added to bdev module, which reports 1171notifications for added and removed bdevs via RPC. 1172 1173### sock 1174 1175Added new API spdk_sock_readv() to the sock library for performing vectored 1176reads. 1177 1178### event 1179 1180The function spdk_subsystem_init() no longer requires spdk_event as an argument. 1181 1182Changed API of spdk_subsystem_config_json() to no longer be asynchronous. 1183 1184### io_uring 1185 1186A bdev module that issues I/O to kernel block devices using the new io_uring Linux kernel 1187API was added. This module requires liburing. 1188 1189### build 1190 1191Options to easily compile with profile guided optimization have been added to 1192`configure`. To use profile guided optimization with SPDK, run 1193`./configure --with-pgo-capture`, build SPDK, then run a workload of your 1194choosing. Then, simply run `./configure --with-pgo-enable` and recompile to 1195build using the generated profile data. Profile guided optimization can yield 1196very large performance improvements, especially on GCC 8 and clang 7. This may 1197be combined with link time optimization which has been available under the 1198`--enable-lto` configure option for several releases. 1199 1200### compression bdev/reduce library 1201 1202Added "reduce" block compression scheme based on using SSDs for storing 1203compressed blocks of storage and presistent memory for metadata. Please see 1204[compression](https://spdk.io/doc/bdev.html) for more details. 1205 1206## v19.01 1207 1208### ocf bdev 1209 1210New virtual bdev module based on [Open CAS Framework](https://open-cas.github.io/) has been added. 1211This module allows for the use of one bdev to act as a high performance cache in front of another bdev. 1212Please see [documentation](https://spdk.io/doc/bdev.html#bdev_config_cas) for more details. 1213Only write through mode is currently supported and this feature is considered experimental. 1214 1215### event framework 1216 1217For `spdk_app_parse_args`, add return value to the callback which parses application 1218specific command line parameters to protect SPDK applications from crashing by invalid 1219values from user input. 1220 1221By default, all SPDK applications will now reserve all hugepages at runtime. The pre-reserved 1222memory size can be still set with `-s` or `--mem-size` option, although the default value 1223was reduced down to 0. 1224 1225A custom hugetlbfs directory can now be specified via spdk_app_opts. 1226This can be used to configure hugepages with different sizes, a different size limit, 1227or different access permissions than the system's default hugepage pool. 1228SPDK applications can specify a custom hugetlbfs mount with the `--huge-dir` option. 1229 1230### environment 1231 1232spdk_vtophys() has been refactored to accept length of the translated region as a new 1233parameter. The function will now update that parameter with the largest possible value 1234for which the memory is contiguous in the physical memory address space. 1235 1236The following functions were removed: 1237 1238 - spdk_pci_nvme_device_attach() 1239 - spdk_pci_nvme_enumerate() 1240 - spdk_pci_ioat_device_attach() 1241 - spdk_pci_ioat_enumerate() 1242 - spdk_pci_virtio_device_attach() 1243 - spdk_pci_virtio_enumerate() 1244 1245They were replaced with generic spdk_pci_device_attach() and spdk_pci_enumerate() which 1246require a new spdk_pci_driver object to be provided. It can be one of the following: 1247 1248 - spdk_pci_nvme_get_driver() 1249 - spdk_pci_ioat_get_driver() 1250 - spdk_pci_virtio_get_driver() 1251 1252spdk_pci_hook_device() and spdk_pci_unhook_device() were added. Those allow adding a virtual 1253spdk_pci_device into the SPDK PCI subsystem. A virtual device calls provided callbacks for 1254each BAR mapping request or PCI config access. It's attachable with spdk_pci_device_attach() 1255or spdk_pci_enumerate() like any other device. 1256 1257A new spdk_pause() function was added to pause CPU execution for an implementation specific 1258amount of time. Quoting from DPDK function this is based on: "This call is intended for 1259tight loops which poll a shared resource or wait for an event. A short pause within the loop 1260may reduce the power consumption." 1261 1262A new public header file env_dpdk.h has been introduced, and function spdk_env_dpdk_post_init 1263is added into it. If user is using DPDK, and already called rte_eal_init, then include 1264include/spdk/env_dpdk.h, and call spdk_env_dpdk_post_init() instead of spdk_env_init. 1265 1266ISA-L has been added as an SPDK submodule. ISA-L is enabled by default on x86 architecture 1267to accelerate algorithms such as CRC for iSCSI and NVMe-oF. Users may still disable ISA-L 1268by explicitly passing --without-isal to the configure script. 1269 1270### util 1271 1272A new uuid API `spdk_uuid_copy` was added to make a copy of the source uuid. 1273 1274An new parameter `init_crc` representing the initial CRC value was added to 1275`spdk_crc16_t10dif`. The parameter can be used to calculate a CRC value spanning 1276multiple separate buffers. 1277 1278New DIF APIs were added to generate and verify DIF by byte granularity for both DIF and DIX 1279formats. Among them, DIF with copy APIs will be usable to emulate DIF operations such as DIF 1280insert and strip. 1281 1282Added `spdk_strtol` and `spdk_strtoll` to provide additional error checking around `strtol` 1283and `strtoll`. 1284 1285Added `spdk_sprintf_append_realloc` and `spdk_vsprintf_append_realloc` for appending a string 1286with automatic buffer re-allocation. 1287 1288### nvme 1289 1290Wrapper functions spdk_nvme_ctrlr_security_send() and spdk_nvme_ctrlr_security_receive() are 1291introduced to support further security protocol development. 1292 1293admin_timeout_ms was added to NVMe controller initialization options, users 1294can change the default value when probing a controller. 1295 1296Add two new fields "header_digest" and "data_digest" in struct spdk_nvme_ctrlr_opts, 1297it will be used to enable the digest support for the NVMe/TCP transport. 1298 1299Add a new TCP/IP transport(located in lib/nvme/nvme_tcp.c) in nvme driver. With 1300this new transport, it can be used to connect the NVMe-oF target with the 1301same TCP/IP support. 1302 1303Added API, spdk_nvme_ctrlr_is_discovery(), to indicate whether the ctrlr 1304arg refers to a Discovery Controller or not. 1305 1306Added an API function `spdk_nvme_host_id_parse` and corresponding object `spdk_nvme_host_id` 1307for parsing host address and host service ID arguments on a per connection basis. 1308 1309The RPC `construct_nvme_bdev` now allows a user to specify a source address and service id for the host to 1310use when connecting to the controller backing the NVMe bdev. 1311 1312### NVMe-oF Target 1313 1314The `spdk_nvmf_tgt_opts` struct has been deprecated in favor of `spdk_nvmf_transport_opts`. 1315Users will no longer be able to specify target wide I/O parameters. `spdk_nvmf_tgt_listen` 1316will also no longer implicitly initialize a transport with the default target options (since 1317there are none). Instead, a user must manually instantiate the transport with `spdk_nvmf_transport_create` 1318prior to calling `spdk_nvmf_tgt_listen`. 1319 1320Related to the previous change, the rpc `set_nvmf_target_options` has been renamed to 1321`set_nvmf_target_max_subsystems` to indicate that this is the only target option available for the user to edit. 1322 1323Added fields `num_shared_buffers` and `buf_cache_size` in struct spdk_nvmf_transport_opts, 1324and also updated the related rpc function nvmf_create_transport, to make this 1325configurable parameter available to users. The `num_shared_buffers` is used to 1326configure the shared buffer numbers of the transport used by RDMA or TCP transport. 1327`buf_cache_size` configures number of shared buffers to cache per poll group. 1328 1329### nvmf 1330 1331Add a new TCP/IP transport (located in lib/nvmf/tcp.c). With this tranport, 1332the SPDK NVMe-oF target can have a new transport, and can serve the NVMe-oF 1333protocol via TCP/IP from the host. 1334 1335Added optional mechanism to modify the RDMA transport's behavior when creating protection domains and registering memory. 1336By default, the RDMA transport will use the ibverbs library to create protection domains and register memory. 1337Using `spdk_nvme_rdma_init_hooks` will subvert that and use an existing registration. 1338 1339### bdev 1340 1341Added `enable_bdev_histogram` and `get_bdev_histogram` RPC commands to allow gathering latency data for specified bdev. 1342Please see [documentation](https://spdk.io/doc/bdev.html#rpc_bdev_histogram) for more details. 1343 1344Added `required_alignment` field to `spdk_bdev`, that specifies an alignment requirement for data buffers associated with an spdk_bdev_io. 1345Bdev layer will automatically double buffer any spdk_bdev_io that violates this alignment, before the spdk_bdev_io is submitted to the bdev module. 1346 1347On shutdown, bdev unregister now proceeds in top-down fashion, with 1348claimed bdevs skipped (these will be unregistered later, when virtual 1349bdev built on top of the respective base bdev unclaims it). This 1350allows virtual bdevs to be shut down cleanly as opposed to the 1351previous behavior that didn't differentiate between hotremove and 1352planned shutdown. 1353 1354The `delete_bdev` RPC is now deprecated. Users should instead use the specific deletion RPC 1355for the bdev type to be removed (i.e. delete_malloc_bdev). 1356 1357Added support for separate bandwidth rate limits for read and write to QoS in bdev layer. 1358 1359Bdev I/O statistics now track unmap opertations. 1360 1361### logical volumes 1362 1363Logical volume bdev can now be marked as read only using `set_read_only_lvol_bdev` RPC. 1364This allows for basing clones on top of lvol_bdev without first creating a snapshot. 1365 1366Added option to change method for data erasure when deleting lvol or resizing down. 1367Default of unmapping clusters can now be changed to writing zeroes or no operation. 1368 1369Added option to change method for erasing data region on lvol store creation. 1370Default of unmapping can now be changed to writing zeroes or no operation. 1371 1372### log 1373 1374"trace flags" are now referred to as "log flags" in the SPDK log API. The 1375set_trace_flag, clear_trace_flag and get_trace_flags RPCs are now deprecated, 1376and set_log_flag, clear_log_flag and get_log_flags RPCs have been added. 1377 1378### trace 1379 1380New `trace_record` application was added. It can be used to poll spdk trace shm file and 1381append any new trace entries into another specified file. This can help retain those entries 1382that would otherwise be overwritten in the shm file. See 1383[Capturing sufficient trace events](https://spdk.io/doc/nvmf_tgt_tracepoints.html#capture_trace_events) 1384for more details. 1385 1386Number of trace entries in circular buffer per lcore can now be assigned by starting SPDK app 1387with argument "--num-trace-entries <NUM>" provided. 1388 1389New `get_tpoint_group_mask` RPC was added to get current tpoint_group_mask, and 1390each tpoint group status. 1391New `enable_tpoint_group` and `disable_tpoint_group` RPC were added to enable or 1392disable a specific tpoint group. 1393 1394### ftl 1395 1396EXPERIMENTAL: Added basic flash translation layer module allowing for using Open Channel SSDs as 1397block devices. The module is split into the library (located in lib/ftl) and bdev_ftl 1398(lib/bdev/ftl). See the [documentation](https://spdk.io/doc/ftl.html) for more details. 1399 1400### vhost 1401 1402A security vulnerability has been identified and fixed in the SPDK vhost target. A malicious 1403vhost client (i.e. virtual machine) could carefully construct a circular descriptor chain which 1404would result in a partial denial of service in the SPDK vhost target. These types of descriptor 1405chains are now properly detected by the vhost target. All SPDK vhost users serving untrusted 1406vhost clients are strongly recommended to upgrade. (Reported by Dima Stepanov and Evgeny 1407Yakovlev.) 1408 1409Vhost SCSI and Vhost Block devices can now accept multiple connections on the same socket file. 1410Each connection (internally called a vhost session) will have access to the same storage, but 1411will use different virtqueues, different features and possibly different memory. 1412 1413### vhost scsi 1414 1415SCSI target hotremove can now be performed even without the VIRTIO_SCSI_F_HOTPLUG feature negotiated. 1416Regardless of VIRTIO_SCSI_F_HOTPLUG support, the hotremoval will be still reported through SCSI sense codes. 1417 1418### DPDK 1419 1420DPDK submodule was updated to DPDK 18.11. Note that SPDK does not fully leverage the new 1421multi-process device hotplug yet and continues to work the same way it always did. 1422 1423Dropped support for DPDK 16.07 and earlier, which SPDK won't even compile with right now. 1424 1425### RPC 1426 1427The following RPC commands deprecated in the previous release are now removed: 1428 1429 - construct_virtio_user_scsi_bdev 1430 - construct_virtio_pci_scsi_bdev 1431 - construct_virtio_user_blk_bdev 1432 - construct_virtio_pci_blk_bdev 1433 - remove_virtio_scsi_bdev 1434 - construct_nvmf_subsystem 1435 1436### Miscellaneous 1437 1438The configure options `--with-raid` and `--without-raid` that were deprecated in the previous 1439release are now removed. 1440 1441### nbd 1442 1443Starting nbd using `spdk_nbd_start` is now performed asynchronously. 1444 1445### net framework 1446 1447Net framework initialization and finish is now done asynchronously. 1448 1449### rpc 1450 1451Added `spdk_rpc_is_method_allowed` function for checking whether method is permitted in a given state. 1452Added `spdk_rpc_get_state` to check current state of RPC server. 1453RPC `wait_subsystem_init` has been added to allow clients to block untill all subsystems are initialized. 1454 1455### json rpc 1456 1457JSON RPC client is now running in non-blocking mode. Requests are sent and received during spdk_jsonrpc_client_poll. 1458JSON RPC server can now recieve a callback on connection termination or server shutdown using `spdk_jsonrpc_conn_add_close_cb` 1459and `spdk_jsonrpc_conn_del_close_cb`. 1460 1461## v18.10 1462 1463### nvme 1464 1465spdk_nvme_ctrlr_cmd_security_send() and spdk_nvme_ctrlr_cmd_security_receive() 1466were added to support sending or receiving security protocol data to or from 1467nvme controller. 1468 1469spdk_nvme_ns_get_extended_sector_size() was added. This function includes 1470the metadata size per sector (if any). spdk_nvme_ns_get_sector_size() still 1471returns only the data size per sector, not including metadata. 1472 1473New `send_nvme_cmd` RPC was added to allow sending NVMe commands directly to NVMe controller. 1474See the [send_nvme_cmd](http://spdk.io/doc/jsonrpc.html#rpc_send_nvme_cmd) documentation 1475for more details. 1476 1477### Build System 1478 1479New `configure` options, `--with-shared` and `--without-shared` 1480[default], provide the capability to build, or not, SPDK shared libraries. 1481This includes the single SPDK shared lib encompassing all of the SPDK 1482static libs as well as individual SPDK shared libs corresponding to 1483each of the SPDK static ones. Although the production of the shared 1484libs conforms with conventional version naming practices, such naming 1485does not at this time confer any SPDK ABI compatibility claims. 1486 1487### bdev 1488 1489spdk_bdev_alias_del_all() was added to delete all alias from block device. 1490 1491A new virtual bdev module has been added to perform at rest data encryption using the DPDK CryptoDev 1492Framework. The module initially uses a software AESNI CBC cipher with experimental support for the 1493Intel QAT hardware accelerator also currently implemented with support for CBC cipher. Future work 1494may include additional ciphers as well as consideration for authentication. 1495 1496The RAID virtual bdev module is now always enabled by default. The configure --with-raid and 1497--without-raid options are now ignored and deprecated and will be removed in the next release. 1498 1499Enforcement of bandwidth limits for quality of service (QoS) has been added to the bdev layer. 1500See the new [set_bdev_qos_limit](http://www.spdk.io/doc/jsonrpc.html#rpc_set_bdev_qos_limit) 1501documentation for more details. The previous set_bdev_qos_limit_iops RPC method introduced at 150218.04 release has been deprecated. The new set_bdev_qos_limit RPC method can support both 1503bandwidth and IOPS limits. 1504 1505spdk_bdev_config_json() and corresponding `get_bdevs_config` RPC was removed. 1506 1507### Environment Abstraction Layer and Event Framework 1508 1509The size parameter of spdk_mem_map_translate is now a pointer. This allows the 1510function to report back the actual size of the translation relative to the original 1511request made by the user. 1512 1513A new structure spdk_mem_map_ops has been introduced to hold memory map related 1514callbacks. This structure is now passed as the second argument of spdk_mem_map_alloc 1515in lieu of the notify callback. 1516 1517### DPDK 18.08 1518 1519The DPDK submodule has been updated to the DPDK 18.08 release. SPDK will now automatically 1520utilize DPDK's dynamic memory management with DPDK versions >= 18.05.1. 1521 1522Hugepages can be still reserved with `[-s|--mem-size <size>]` option at application startup, 1523but once we use them all up, instead of failing user allocations with -ENOMEM, we'll try 1524to dynamically reserve even more. This allows starting SPDK with `--mem-size 0` and using 1525only as many hugepages as it is really needed. 1526 1527Due to this change, the memory buffers returned by `spdk_*malloc()` are no longer guaranteed 1528to be physically contiguous. 1529 1530### I/OAT 1531 1532I/OAT driver can now reinitialize I/OAT channels after encountering DMA errors. 1533 1534### iscsi target 1535 1536Parameter names of `set_iscsi_options` and `get_iscsi_global_params` RPC 1537method for CHAP authentication in discovery sessions have been changed to 1538align with `construct_target_node` RPC method. Old names are still usable 1539but will be removed in future release. 1540 1541`set_iscsi_discovery_auth` and `set_iscsi_target_node_auth` RPC methods have 1542been added to set CHAP authentication for discovery sessions and existing 1543target nodes, respectively. 1544 1545The SPDK iSCSI target supports an AuthFile which can be used to load CHAP 1546shared secrets when the iSCSI target starts. SPDK previously provided a 1547default location for this file (`/usr/local/etc/spdk/auth.conf`) if none was 1548specified. This default has been removed. Users must now explicitly specify 1549the location of this file to load CHAP shared secrets from a file, or use 1550the related iSCSI RPC methods to add them at runtime. 1551 1552### iscsi initiator 1553 1554The SPDK iSCSI initiator is no longer considered experimental and becomes 1555a first-class citizen among bdev modules. The basic usage has been briefly 1556described in the bdev user guide: [iSCSI bdev](https://spdk.io/doc/bdev.html#bdev_config_iscsi) 1557 1558### Miscellaneous 1559 1560The SPDK application framework can now parse long name command line parameters. 1561Most single-character parameters have a long name equivalent now. See the 1562[Command Line Parameters](https://spdk.io/doc/app_overview.html) documentation 1563for details or use the `--help` command line parameter to list all available 1564params. 1565 1566bdevperf `-s` param (io size) was renamed to `-o` as `-s` had been already 1567used by existing apps for memory size. 1568 1569bdevio can now accept all SPDK command line parameters. The config now has to 1570be provided with `-c` or `--config` param. 1571 1572The following ioat/perf and nvme/perf parameters were renamed as well: 1573 `-s` (io size) to `-o` 1574 `-d` (mem size) to `-s` 1575 1576The ReactorMask config file parameter has been deprecated. Users should 1577use the -m or --cpumask command line option to specify the CPU core mask 1578for the application. 1579 1580Default config file pathnames have been removed from iscsi_tgt, nvmf_tgt 1581and vhost. Config file pathnames may now only be specified using the 1582-c command line option. 1583 1584Users may no longer set DPDK_DIR in their environment to specify the 1585location of the DPDK installation used to build SPDK. Using DPDK_DIR 1586has not been the documented nor recommended way to specify the DPDK 1587location for several releases, but removing it ensures no unexpected 1588surprises for users who may have DPDK_DIR defined for other reasons. 1589Users should just use the "configure" script to specify the DPDK 1590location before building SPDK. 1591 1592Although we know that many developers still use Python 2 we are officially 1593switching to Python3 with requirement that all new code must be valid also 1594for Python 2 up to the EOL which is year 2020. 1595 1596Invoking interpreter explicitly is forbidden for executable scripts. There 1597is no need to use syntax like "python ./scripts/rpc.py". All executable 1598scripts must contain proper shebang pointing to the right interpreter. 1599Scripts without shebang musn't be executable. 1600 1601A Python script has been added to enable conversion of old INI config file 1602to new JSON-RPC config file format. This script can be found at 1603scripts/config_converter.py. Example how this script can be used: 1604~~~{.sh} 1605cat old_format.ini | scripts/config_converter.py > new_json_format.json 1606~~~ 1607 1608### Sock 1609 1610Two additional parameters were added to spdk_sock_get_addr() for the server 1611port and client port. These parameters are named "sport" and "cport" 1612respectively. 1613 1614### Virtio 1615 1616The following RPC commands have been deprecated: 1617 1618 - construct_virtio_user_scsi_bdev 1619 - construct_virtio_pci_scsi_bdev 1620 - construct_virtio_user_blk_bdev 1621 - construct_virtio_pci_blk_bdev 1622 - remove_virtio_scsi_bdev 1623 1624The `construct_virtio_*` ones were replaced with a single `construct_virtio_dev` 1625command that can create any type of Virtio bdev(s). `remove_virtio_scsi_bdev` 1626was replaced with `remove_virtio_bdev` that can delete both Virtio Block and SCSI 1627devices. 1628 1629### Blobfs 1630 1631spdk_file_get_id() returning unique ID for the file was added. 1632 1633### JSON 1634 1635Added jsonrpc-client C library intended for issuing RPC commands from applications. 1636 1637Added API enabling iteration over JSON object: 1638 1639 - spdk_json_find() 1640 - spdk_json_find_string() 1641 - spdk_json_find_array() 1642 - spdk_json_object_first() 1643 - spdk_json_array_first() 1644 - spdk_json_next() 1645 1646### Blobstore 1647 1648Blobstore I/O operations are now based on io_units, instead of blobstore page size. 1649The io_unit size is now the same as the underlying block device's block size. 1650Logical volumes built on a block device with 512B block size can now be used as boot devices 1651in QEMU. 1652 1653### SPDKCLI 1654 1655The SPDKCLI interactive command tool for managing SPDK is no longer considered experimental. 1656Support for the iSCSI and NVMe-oF targets has been added. 1657 1658## v18.07 1659 1660### bdev 1661 1662A new public header file bdev_module.h has been introduced to facilitate the 1663development of new bdev modules. This header includes an interface for the 1664spdk_bdev_part and spdk_bdev_part_base objects to enable the creation of 1665multiple virtual bdevs on top of a single base bdev and should act as the 1666primary API for module authors. 1667 1668spdk_bdev_get_opts() and spdk_bdev_set_opts() were added to set bdev-wide 1669options. 1670 1671A mechanism for handling out of memory condition errors (ENOMEM) returned from 1672I/O submission requests at the bdev layer has been added. See 1673spdk_bdev_queue_io_wait(). 1674 1675The spdk_bdev_get_io_stat() function now returns cumulative totals instead of 1676resetting on each call. This allows multiple callers to query I/O statistics 1677without conflicting with each other. Existing users will need to adjust their 1678code to record the previous I/O statistics to calculate the delta between calls. 1679 1680I/O queue depth tracking and samples options have been added. See 1681spdk_bdev_get_qd(), spdk_bdev_get_qd_sampling_period(), and 1682spdk_bdev_set_qd_sampling_period(). 1683 1684### RAID module 1685 1686A new bdev module called "raid" has been added as experimental module which 1687aggregates underlying NVMe bdevs and exposes a single raid bdev. Please note 1688that vhost will not work with this module because it does not yet have support 1689for multi-element io vectors. 1690 1691### Log 1692 1693The debug log component flag available on several SPDK applications has been 1694renamed from `-t` to `-L` to prevent confusion with tracepoints and to allow the 1695option to be added to tools that already use `-t` to mean something else. 1696 1697### Blobstore 1698 1699A new function, spdk_bs_dump(), has been added that dumps all of the contents of 1700a blobstore to a file pointer. This includes the metadata and is very useful for 1701debugging. 1702 1703Two new operations have been added for thin-provisioned blobs. 1704spdk_bs_inflate_blob() will allocate clusters for all thinly provisioned regions 1705of the blob and populate them with the correct data by reading from the backing 1706blob(s). spdk_bs_blob_decouple_parent() works similarly, but will only allocate 1707clusters that correspond to data in the blob's immediate parent. Clusters 1708allocated to grandparents or that aren't allocated at all will remain 1709thin-provisioned. 1710 1711### BlobFS 1712 1713Changed the return type of spdk_file_truncate() from void to int to allow the 1714propagation of `ENOMEM` errors. 1715 1716### NVMe Driver 1717 1718The new API functions spdk_nvme_qpair_add_cmd_error_injection() and 1719spdk_nvme_qpair_remove_cmd_error_injection() have been added for NVMe error 1720emulation. Users can set a specified command to fail with a particular error 1721status. 1722 1723Changed the name `timeout_sec` parameter to `timeout_us` in 1724spdk_nvme_ctrlr_register_timeout_callback(), and also changed the type from 1725uint32_t to uint64_t. This will give users more fine-grained control over the 1726timeout period. 1727 1728Basic support for Open Channel SSDs was added. See nvme_ocssd.h 1729 1730### NVMe Over Fabrics 1731 1732The spdk_nvmf_tgt_destroy() function is now asynchronous and takes a callback 1733as a parameter. 1734 1735spdk_nvmf_qpair_disconnect() was added to allow the user to disconnect qpairs. 1736 1737spdk_nvmf_subsystem_get_max_namespaces() was added to query the maximum allowed 1738number of namespaces for a given subsystem. 1739 1740### Build System 1741 1742The build system now generates a combined shared library (libspdk.so) that may 1743be used in place of the individual static libraries (libspdk_*.a). The combined 1744library includes all components of SPDK and is intended to make linking against 1745SPDK easier. The static libraries are also still provided for users that prefer 1746to link only the minimal set of components required. 1747 1748### git pre-commit and pre-push hooks 1749 1750The pre-commit hook will run `scripts/check_format.sh` and verify there are no 1751formating errors before allowing `git commit` to run. The pre-push hook runs 1752`make CONFIG_WERROR=y` with and without `CONFIG_DEBUG=y` using both the gcc and 1753clang compiler before allowing `git push` to run. Following each DEBUG build 1754`test/unit/unittest.sh` is run and verified. Results are recorded in the 1755`make.log` file. 1756 1757To enable type: 'git config core.hooksPath .githooks'. To override after 1758configuration use the `git --no-verify` flag. 1759 1760### RPC 1761 1762The `start_nbd_disk` RPC method now returns the path to the kernel NBD device node 1763rather than always returning `true`. 1764 1765### DPDK 18.05 1766 1767The DPDK submodule has been rebased on the DPDK 18.05 release. DPDK 18.05 supports 1768dynamic memory allocation, but due to some issues found after the DPDK 18.05 release, 1769that support is not enabled for SPDK 18.07. Therefore, SPDK 18.07 will continue to use 1770the legacy memory allocation model. The plan is to enable dynamic memory allocation 1771after the DPDK 18.08 release which should fix these issues. 1772 1773### Environment Abstraction Layer and Event Framework 1774 1775The spdk_mem_map_translate() function now takes a size parameter to indicate the size of 1776the memory region. This can be used by environment implementations to validate the 1777requested translation. 1778 1779The I/O Channel implementation has been moved to its own library - lib/thread. The 1780public API that was previously in spdk/io_channel.h is now in spdk/thread.h The 1781file spdk/io_channel.h remains and includes spdk/thread.h. 1782 1783spdk_reactor_get_tsc_stats was added to return interesting statistics for each 1784reactor. 1785 1786### IOAT 1787 1788IOAT for copy engine is disabled by default. It can be enabled by specifying the Enable 1789option with "Yes" in `[Ioat]` section of the configuration file. The Disable option is 1790now deprecated and will be removed in a future release. 1791 1792## v18.04: Logical Volume Snapshot/Clone, iSCSI Initiator, Bdev QoS, VPP Userspace TCP/IP 1793 1794### vhost 1795 1796The SPDK vhost-scsi, vhost-blk and vhost-nvme applications have fixes to address the 1797DPDK rte_vhost vulnerability [CVE-2018-1059](http://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2018-1059). 1798Please see this [security advisory](https://access.redhat.com/security/cve/cve-2018-1059) 1799for additional information on the DPDK vulnerability. 1800 1801Workarounds have been added to ensure vhost compatibility with QEMU 2.12. 1802 1803EXPERIMENTAL: Support for vhost-nvme has been added to the SPDK vhost target. See the 1804[vhost documentation](http://www.spdk.io/doc/vhost.html) for more details. 1805 1806### Unified Target Application 1807 1808A new unified SPDK target application, `spdk_tgt`, has been added. This application combines the 1809functionality of several existing SPDK applications, including the iSCSI target, NVMe-oF target, 1810and vhost target. The new application can be managed through the existing configuration file and 1811[JSON-RPC](http://www.spdk.io/doc/jsonrpc.html) methods. 1812 1813### Env 1814 1815spdk_mempool_get_bulk() has been added to wrap DPDK rte_mempool_get_bulk(). 1816 1817New memory management functions spdk_malloc(), spdk_zmalloc(), and spdk_free() have been added. 1818These new functions have a `flags` parameter that allows the user to specify whether the allocated 1819memory needs to be suitable for DMA and whether it should be shared across processes with the same 1820shm_id. The new functions are intended to replace spdk_dma_malloc() and related functions, which will 1821eventually be deprecated and removed. 1822 1823### Bdev 1824 1825A new optional bdev module interface function, `init_complete`, has been added to notify bdev modules 1826when the bdev subsystem initialization is complete. This may be useful for virtual bdevs that require 1827notification that the set of initialization examine() calls is complete. 1828 1829The bdev layer now allows modules to provide an optional per-bdev UUID, which can be retrieved with 1830the spdk_bdev_get_uuid() function. 1831 1832Enforcement of IOPS limits for quality of service (QoS) has been added to the bdev layer. See the 1833[set_bdev_qos_limit_iops](http://www.spdk.io/doc/jsonrpc.html#rpc_set_bdev_qos_limit_iops) documentation 1834for more details. 1835 1836### RPC 1837 1838The `[Rpc]` configuration file section, which was deprecated in v18.01, has been removed. 1839Users should switch to the `-r` command-line parameter instead. 1840 1841The JSON-RPC server implementation now allows up to 32 megabyte responses, growing as 1842needed; previously, the response was limited to 32 kilobytes. 1843 1844### SPDKCLI 1845 1846EXPERIMENTAL: New SPDKCLI interactive command tool for managing SPDK is available. 1847See the [SPDKCLI](http://www.spdk.io/doc/spdkcli.html) documentation for more details. 1848 1849### NVMe Driver 1850 1851EXPERIMENTAL: Support for WDS and RDS capable CMBs in NVMe controllers has been added. This support is 1852experimental pending a functional allocator to free and reallocate CMB buffers. 1853 1854spdk_nvme_ns_get_uuid() has been added to allow retrieval of per-namespace UUIDs when available. 1855 1856New API functions spdk_nvme_ctrlr_get_first_active_ns() and spdk_nvme_ctrlr_get_next_active_ns() 1857have been added to iterate active namespaces, as well as spdk_nvme_ctrlr_is_active_ns() to check if 1858a namespace ID is active. 1859 1860### NVMe-oF Target 1861 1862Namespaces may now be assigned unique identifiers via new optional `eui64` and `nguid` parameters 1863to the `nvmf_subsystem_add_ns` RPC method. Additionally, the NVMe-oF target automatically exposes 1864the backing bdev's UUID as the namespace UUID when available. 1865 1866spdk_nvmf_subsystem_remove_ns() is now asynchronous and requires a callback to indicate completion. 1867 1868### Blobstore 1869 1870A number of functions have been renamed: 1871 1872- spdk_bs_io_write_blob() => spdk_blob_io_write() 1873- spdk_bs_io_read_blob() => spdk_blob_io_read() 1874- spdk_bs_io_writev_blob() => spdk_blob_io_writev() 1875- spdk_bs_io_readv_blob() => spdk_blob_io_readv() 1876- spdk_bs_io_unmap_blob() => spdk_blob_io_unmap() 1877- spdk_bs_io_write_zeroes_blob() => spdk_blob_io_write_zeroes() 1878 1879The old names still exist but are deprecated. They will be removed in the v18.07 release. 1880 1881spdk_blob_resize() is now an asynchronous operation to enable resizing a blob while I/O 1882are in progress to that blob on other threads. An explicit spdk_blob_sync_md() is still 1883required to sync the updated metadata to disk. 1884 1885### Logical Volumes 1886 1887A new `destroy_lvol_bdev` RPC method to delete logical volumes has been added. 1888 1889Lvols now have their own UUIDs which replace previous LvolStoreUUID_BlobID combination. 1890 1891New Snapshot and Clone functionalities have been added. User may create Snapshots of existing Lvols 1892and Clones of existing Snapshots. 1893See the [lvol snapshots](http://www.spdk.io/doc/logical_volumes.html#lvol_snapshots) documentation 1894for more details. 1895 1896Resizing logical volumes is now supported via the `resize_lvol_bdev` RPC method. 1897 1898### Lib 1899 1900A set of changes were made in the SPDK's lib code altering 1901instances of calls to `exit()` and `abort()` to return a failure instead 1902wherever reasonably possible. 1903 1904spdk_app_start() no longer exit()'s on an internal failure, but 1905instead returns a non-zero error status. 1906 1907spdk_app_parse_args() no longer exit()'s on help, '-h', or an invalid 1908option, but instead returns SPDK_APP_PARSE_ARGS_HELP and 1909SPDK_APP_PARSE_ARGS_FAIL, respectively, and SPDK_APP_PARSE_ARGS_SUCCESS 1910on success. 1911 1912spdk_pci_get_device() has been deprecated and will be removed in SPDK v18.07. 1913 1914### I/O Channels 1915 1916The prototype for spdk_poller_fn() has been modified; it now returns a value indicating 1917whether or not the poller did any work. Existing pollers will need to be updated to 1918return a value. 1919 1920### iSCSI Target 1921 1922The SPDK iSCSI target now supports the fd.io Vector Packet Processing (VPP) framework userspace 1923TCP/IP stack. See the [iSCSI VPP documentation](http://www.spdk.io/doc/iscsi.html#vpp) for more 1924details. 1925 1926### iSCSI initiator 1927 1928An iSCSI initiator bdev module has been added to SPDK. This module should be considered 1929experimental pending additional features and tests. More details can be found in 1930lib/bdev/iscsi/README. 1931 1932### PMDK 1933 1934The persistent memory (PMDK) bdev module is now enabled using `--with-pmdk` instead of 1935`--with-nvml`. This reflects the renaming of the persistent memory library from NVML to 1936PMDK. 1937 1938### Virtio Block driver 1939 1940A userspace driver for Virtio Block devices has been added. It was built on top of the 1941[Virtio](http://www.spdk.io/doc/virtio.html) library and can be managed similarly to 1942the Virtio SCSI driver. See the 1943[Virtio Block](http://www.spdk.io/doc/bdev.html#bdev_config_virtio_blk) reference for 1944more information. 1945 1946### Virtio with 2MB hugepages 1947 1948The previous 1GB hugepage limitation has now been lifted. A new `-g` command-line option 1949enables SPDK Virtio to work with 2MB hugepages. 1950See [2MB hugepages](http://www.spdk.io/doc/virtio.html#virtio_2mb) for details. 1951 1952## v18.01: Blobstore Thin Provisioning 1953 1954### Build System 1955 1956The build system now includes a `make install` rule, including support for the common 1957`DESTDIR` and `prefix` variables as used in other build systems. Additionally, the prefix 1958may be set via the configure `--prefix` option. Example: `make install prefix=/usr`. 1959 1960### RPC 1961 1962A JSON RPC listener is now enabled by default using a UNIX domain socket at /var/run/spdk.sock. 1963A -r option command line option has been added to enable an alternative UNIX domain socket location, 1964or a TCP port in the format ip_addr:tcp_port (i.e. 127.0.0.1:5260). The Rpc configuration file 1965section is now deprecated and will be removed in the v18.04 release. 1966 1967### I/O Channels 1968 1969spdk_poller_register() and spdk_poller_unregister() were moved from the event 1970framework (include/spdk/event.h) to the I/O channel library 1971(include/spdk/io_channel.h). This allows code that doesn't depend on the event 1972framework to request registration and unregistration of pollers. 1973 1974spdk_for_each_channel() now allows asynchronous operations during iteration. 1975Instead of immediately continuing the interation upon returning from the iteration 1976callback, the user must call spdk_for_each_channel_continue() to resume iteration. 1977 1978### Block Device Abstraction Layer (bdev) 1979 1980The poller abstraction was removed from the bdev layer. There is now a general purpose 1981abstraction for pollers available in include/spdk/io_channel.h 1982 1983### Lib 1984 1985A set of changes were made in the SPDK's lib code altering, 1986instances of calls to `exit()` and `abort()` to return a failure instead 1987wherever reasonably possible. This has resulted in return type changes of 1988the API for: 1989 1990- spdk_env_init() from type `void` to `int`. 1991- spdk_mem_map_init() from type `void` to `int`. 1992 1993Applications making use of these APIs should be modified to check for 1994a non-zero return value instead of relying on them to fail without return. 1995 1996### NVMe Driver 1997 1998SPDK now supports hotplug for vfio-attached devices. But there is one thing keep in mind: 1999Only physical removal events are supported; removing devices via the sysfs `remove` file will not work. 2000 2001### NVMe-oF Target 2002 2003Subsystems are no longer tied explicitly to CPU cores. Instead, connections are handed out to the available 2004cores round-robin. The "Core" option in the configuration file has been removed. 2005 2006### Blobstore 2007 2008A number of functions have been renamed: 2009 2010- spdk_bs_md_resize_blob() => spdk_blob_resize() 2011- spdk_bs_md_sync_blob() => spdk_blob_sync_md() 2012- spdk_bs_md_close_blob() => spdk_blob_close() 2013- spdk_bs_md_get_xattr_names() => spdk_blob_get_xattr_names() 2014- spdk_bs_md_get_xattr_value() => spdk_blob_get_xattr_value() 2015- spdk_blob_md_set_xattr() => spdk_blob_set_xattr() 2016- spdk_blob_md_remove_xattr() => spdk_blob_remove_xattr() 2017- spdk_bs_md_create_blob() => spdk_bs_create_blob() 2018- spdk_bs_md_open_blob() => spdk_bs_open_blob() 2019- spdk_bs_md_delete_blob() => spdk_bs_delete_blob() 2020- spdk_bs_md_iter_first() => spdk_bs_iter_first() 2021- spdk_bs_md_iter_next() => spdk_bs_iter_next() 2022 2023The function signature of spdk_blob_close() has changed. It now takes a struct spdk_blob * argument 2024rather than struct spdk_blob **. 2025 2026The function signature of spdk_bs_iter_next() has changed. It now takes a struct spdk_blob * argument 2027rather than struct spdk_blob **. 2028 2029Thin provisioning support has been added to the blobstore. It can be enabled by setting the 2030`thin_provision` flag in struct spdk_blob_opts when calling spdk_bs_create_blob_ext(). 2031 2032### NBD device 2033 2034The NBD application (test/lib/bdev/nbd) has been removed; Same functionality can now be 2035achieved by using the test/app/bdev_svc application and start_nbd_disk RPC method. 2036See the [GPT](http://www.spdk.io/doc/bdev.html#bdev_config_gpt) documentation for more details. 2037 2038### FIO plugin 2039 2040SPDK `fio_plugin` now supports FIO 3.3. The support for previous FIO 2.21 has been dropped, 2041although it still remains to work for now. The new FIO contains huge amount of bugfixes and 2042it's recommended to do an update. 2043 2044### Virtio library 2045 2046Previously a part of the bdev_virtio module, now a separate library. Virtio is now available 2047via `spdk_internal/virtio.h` file. This is an internal interface to be used when implementing 2048new Virtio backends, namely Virtio-BLK. 2049 2050### iSCSI 2051 2052The MinConnectionIdleInterval parameter has been removed, and connections are no longer migrated 2053to an epoll/kqueue descriptor on the master core when idle. 2054 2055## v17.10: Logical Volumes 2056 2057### New dependencies 2058 2059libuuid was added as new dependency for logical volumes. 2060 2061libnuma is now required unconditionally now that the DPDK submodule has been updated to DPDK 17.08. 2062 2063### Block Device Abstraction Layer (bdev) 2064 2065An [fio](http://github.com/axboe/fio) plugin was added that can route 2066I/O to the bdev layer. See the [plugin documentation](https://github.com/spdk/spdk/tree/master/examples/bdev/fio_plugin/) 2067for more information. 2068 2069spdk_bdev_unmap() was modified to take an offset and a length in bytes as 2070arguments instead of requiring the user to provide an array of SCSI 2071unmap descriptors. This limits unmaps to a single contiguous range. 2072 2073spdk_bdev_write_zeroes() was introduced. It ensures that all specified blocks will be zeroed out. 2074If a block device doesn't natively support a write zeroes command, the bdev layer emulates it using 2075write commands. 2076 2077New API functions that accept I/O parameters in units of blocks instead of bytes 2078have been added: 2079 2080- spdk_bdev_read_blocks(), spdk_bdev_readv_blocks() 2081- spdk_bdev_write_blocks(), spdk_bdev_writev_blocks() 2082- spdk_bdev_write_zeroes_blocks() 2083- spdk_bdev_unmap_blocks() 2084 2085The bdev layer now handles temporary out-of-memory I/O failures internally by queueing the I/O to be 2086retried later. 2087 2088### Linux AIO bdev 2089 2090The AIO bdev now allows the user to override the auto-detected block size. 2091 2092### NVMe driver 2093 2094The NVMe driver now recognizes the NVMe 1.3 Namespace Optimal I/O Boundary field. 2095NVMe 1.3 devices may report an optimal I/O boundary, which the driver will take 2096into account when splitting I/O requests. 2097 2098The HotplugEnable option in `[Nvme]` sections of the configuration file is now 2099"No" by default. It was previously "Yes". 2100 2101The NVMe library now includes a spdk_nvme_ns_get_ctrlr() function which returns the 2102NVMe Controller associated with a given namespace. 2103 2104The NVMe library now allows the user to specify a host identifier when attaching 2105to a controller. The host identifier is used as part of the Reservations feature, 2106as well as in the NVMe-oF Connect command. The default host ID is also now a 2107randomly-generated UUID, and the default host NQN uses the host ID to generate 2108a UUID-based NQN. 2109 2110spdk_nvme_connect() was added to allow the user to connect directly to a single 2111NVMe or NVMe-oF controller. 2112 2113### NVMe-oF Target (nvmf_tgt) 2114 2115The NVMe-oF target no longer requires any in-capsule data buffers to run, and 2116the feature is now entirely optional. Previously, at least 4 KiB in-capsule 2117data buffers were required. 2118 2119NVMe-oF subsytems have a new configuration option, AllowAnyHost, to control 2120whether the host NQN whitelist is enforced when accepting new connections. 2121If no Host options have been specified and AllowAnyHost is disabled, the 2122connection will be denied; this is a behavior change from previous releases, 2123which allowed any host NQN to connect if the Host list was empty. 2124AllowAnyHost is disabled by default. 2125 2126NVMe-oF namespaces may now be assigned arbitrary namespace IDs, and the number 2127of namespaces per subsystem is no longer limited. 2128 2129The NVMe-oF target now supports the Write Zeroes command. 2130 2131### Environment Abstraction Layer 2132 2133A new default value, SPDK_MEMPOOL_DEFAULT_CACHE_SIZE, was added to provide 2134additional clarity when constructing spdk_mempools. Previously, -1 could be 2135passed and the library would choose a reasonable default, but this new value 2136makes it explicit that the default is being used. 2137 2138### Blobstore 2139 2140The blobstore super block now contains a bstype field to identify the type of the blobstore. 2141Existing code should be updated to fill out bstype when calling spdk_bs_init() and spdk_bs_load(). 2142 2143spdk_bs_destroy() was added to allow destroying blobstore on device 2144with an initialized blobstore. 2145 2146spdk_bs_io_readv_blob() and spdk_bs_io_writev_blob() were added to enable 2147scattered payloads. 2148 2149A CLI tool for blobstore has been added, allowing basic operations through either command 2150line or shell interface. See the [blobcli](https://github.com/spdk/spdk/tree/master/examples/blob/cli) 2151documentation for more details. 2152 2153### Event Framework 2154 2155The ability to set a thread name, previously only used by the reactor code, is 2156now part of the spdk_thread_allocate() API. Users may specify a thread name 2157which will show up in tools like `gdb`. 2158 2159### Log 2160 2161The spdk_trace_dump() function now takes a new parameter to allow the caller to 2162specify an output file handle (stdout or stderr, for example). 2163 2164### Logical Volumes 2165 2166Logical volumes library built on top of SPDK blobstore has been added. 2167It is possible to create logical volumes on top of other devices using RPC. 2168 2169See the [logical volumes](http://www.spdk.io/doc/logical_volumes.html) documentation for more information. 2170 2171### Persistent Memory 2172 2173A new persistent memory bdev type has been added. 2174The persistent memory block device is built on top of [libpmemblk](http://pmem.io/nvml/libpmemblk/). 2175It is possible to create pmem devices on top of pmem pool files using RPC. 2176 2177See the [Pmem Block Device](http://www.spdk.io/doc/bdev.html#bdev_config_pmem) documentation for more information. 2178 2179### Virtio SCSI driver 2180 2181A userspace driver for Virtio SCSI devices has been added. 2182The driver is capable of creating block devices on top of LUNs exposed by another SPDK vhost-scsi application. 2183 2184See the [Virtio SCSI](http://www.spdk.io/doc/virtio.html) documentation and [Getting Started](http://www.spdk.io/doc/bdev.html#bdev_config_virtio_scsi) guide for more information. 2185 2186### Vhost target 2187 2188The vhost target application now supports live migration between QEMU instances. 2189 2190## v17.07: Build system improvements, userspace vhost-blk target, and GPT bdev 2191 2192### Build System 2193 2194A `configure` script has been added to simplify the build configuration process. 2195The existing CONFIG file and `make CONFIG_...` options are also still supported. 2196Run `./configure --help` for information about available configuration options. 2197 2198A DPDK submodule has been added to make building SPDK easier. If no `--with-dpdk` 2199option is specified to configure, the SPDK build system will automatically build a 2200known-good configuration of DPDK with the minimal options enabled. See the Building 2201section of README.md for more information. 2202 2203A [Vagrant](https://www.vagrantup.com/) setup has been added to make it easier to 2204develop and use SPDK on systems without suitable NVMe hardware. See the Vagrant 2205section of README.md for more information. 2206 2207### Userspace vhost-blk target 2208 2209The vhost library and example app have been updated to support the vhost-blk 2210protocol in addition to the existing vhost-scsi protocol. 2211See the [vhost documentation](http://www.spdk.io/doc/vhost.html) for more details. 2212 2213### Block device abstraction layer (bdev) 2214 2215A GPT virtual block device has been added, which automatically exposes GPT partitions 2216with a special SPDK-specific partition type as bdevs. 2217See the [GPT bdev documentation](http://www.spdk.io/doc/bdev.md#bdev_config_gpt) for 2218more information. 2219 2220### NVMe driver 2221 2222The NVMe driver has been updated to support recent Intel SSDs, including the Intel® 2223Optane™ SSD DC P4800X series. 2224 2225A workaround has been added for devices that failed to recognize register writes 2226during controller reset. 2227 2228The NVMe driver now allocates request tracking objects on a per-queue basis. The 2229number of requests allowed on an I/O queue may be set during `spdk_nvme_probe()` by 2230modifying `io_queue_requests` in the opts structure. 2231 2232The SPDK NVMe `fio_plugin` has been updated to support multiple threads (`numjobs`). 2233 2234spdk_nvme_ctrlr_alloc_io_qpair() has been modified to allow the user to override 2235controller-level options for each individual I/O queue pair. 2236Existing callers with qprio == 0 can be updated to: 2237~~~ 2238... = spdk_nvme_ctrlr_alloc_io_qpair(ctrlr, NULL, 0); 2239~~~ 2240Callers that need to specify a non-default qprio should be updated to: 2241~~~ 2242struct spdk_nvme_io_qpair_opts opts; 2243spdk_nvme_ctrlr_get_default_io_qpair_opts(ctrlr, &opts, sizeof(opts)); 2244opts.qprio = SPDK_NVME_QPRIO_...; 2245... = spdk_nvme_ctrlr_alloc_io_qpair(ctrlr, &opts, sizeof(opts)); 2246~~~ 2247 2248### Environment Abstraction Layer 2249 2250The environment abstraction layer has been updated to include several new functions 2251in order to wrap additional DPDK functionality. See `include/spdk/env.h` for the 2252current set of functions. 2253 2254### SPDK Performance Analysis with Intel® VTune™ Amplifier 2255 2256Support for SPDK performance analysis has been added to Intel® VTune™ Amplifier 2018. 2257 2258This analysis provides: 2259 2260- I/O performance monitoring (calculating standard I/O metrics like IOPS, throughput, etc.) 2261- Tuning insights on the interplay of I/O and compute devices by estimating how many cores 2262 would be reasonable to provide for SPDK to keep up with a current storage workload. 2263 2264See the VTune Amplifier documentation for more information. 2265 2266## v17.03: Blobstore and userspace vhost-scsi target 2267 2268### Blobstore and BlobFS 2269 2270The blobstore is a persistent, power-fail safe block allocator designed to be 2271used as the local storage system backing a higher-level storage service. 2272See the [blobstore documentation](http://www.spdk.io/doc/blob.html) for more details. 2273 2274BlobFS adds basic filesystem functionality like filenames on top of the blobstore. 2275This release also includes a RocksDB Env implementation using BlobFS in place of the 2276kernel filesystem. 2277See the [BlobFS documentation](http://www.spdk.io/doc/blobfs.html) for more details. 2278 2279### Userspace vhost-scsi target 2280 2281A userspace implementation of the QEMU vhost-scsi protocol has been added. 2282The vhost target is capable of exporting SPDK bdevs to QEMU-based VMs as virtio devices. 2283See the [vhost documentation](http://www.spdk.io/doc/vhost.html) for more details. 2284 2285### Event framework 2286 2287The overhead of the main reactor event loop was reduced by optimizing the number of 2288calls to spdk_get_ticks() per iteration. 2289 2290### NVMe library 2291 2292The NVMe library will now automatically split readv/writev requests with scatter-gather 2293lists that do not map to valid PRP lists when the NVMe controller does not natively 2294support SGLs. 2295 2296The `identify` and `perf` NVMe examples were modified to add a consistent format for 2297specifying remote NVMe over Fabrics devices via the `-r` option. 2298This is implemented using the new `spdk_nvme_transport_id_parse()` function. 2299 2300### iSCSI Target 2301 2302The [Nvme] section of the configuration file was modified to remove the `BDF` directive 2303and replace it with a `TransportID` directive. Both local (PCIe) and remote (NVMe-oF) 2304devices can now be specified as the backing block device. A script to generate an 2305entire [Nvme] section based on the local NVMe devices attached was added at 2306`scripts/gen_nvme.sh`. 2307 2308### NVMe-oF Target 2309 2310The [Nvme] section of the configuration file was modified to remove the `BDF` directive 2311and replace it with a `TransportID` directive. Both local (PCIe) and remote (NVMe-oF) 2312devices can now be specified as the backing block device. A script to generate an 2313entire [Nvme] section based on the local NVMe devices attached was added at 2314`scripts/gen_nvme.sh`. 2315 2316## v16.12: NVMe over Fabrics host, hotplug, and multi-process 2317 2318### NVMe library 2319 2320The NVMe library has been changed to create its own request memory pool rather than 2321requiring the user to initialize the global `request_mempool` variable. Apps can be 2322updated by simply removing the initialization of `request_mempool`. Since the NVMe 2323library user no longer needs to know the size of the internal NVMe request 2324structure to create the pool, the `spdk_nvme_request_size()` function was also removed. 2325 2326The `spdk_nvme_ns_cmd_deallocate()` function was renamed and extended to become 2327`spdk_nvme_ns_cmd_dataset_management()`, which allows access to all of the NVMe 2328Dataset Management command's parameters. Existing callers can be updated to use 2329`spdk_nvme_ns_cmd_dataset_management()` with `SPDK_NVME_DSM_ATTR_DEALLOCATE` as the 2330`type` parameter. 2331 2332The NVMe library SGL callback prototype has been changed to return virtual addresses 2333rather than physical addresses. Callers of `spdk_nvme_ns_cmd_readv()` and 2334`spdk_nvme_ns_cmd_writev()` must update their `next_sge_fn` callbacks to match. 2335 2336The NVMe library now supports NVMe over Fabrics devices in addition to the existing 2337support for local PCIe-attached NVMe devices. For an example of how to enable 2338NVMe over Fabrics support in an application, see `examples/nvme/identify` and 2339`examples/nvme/perf`. 2340 2341Hot insert/remove support for NVMe devices has been added. To enable NVMe hotplug 2342support, an application should call the `spdk_nvme_probe()` function on a regular 2343basis to probe for new devices (reported via the existing `probe_cb` callback) and 2344removed devices (reported via a new `remove_cb` callback). Hotplug is currently 2345only supported on Linux with the `uio_pci_generic` driver, and newly-added NVMe 2346devices must be bound to `uio_pci_generic` by an external script or tool. 2347 2348Multiple processes may now coordinate and use a single NVMe device simultaneously 2349using [DPDK Multi-process Support](http://dpdk.org/doc/guides/prog_guide/multi_proc_support.html). 2350 2351### NVMe over Fabrics target (`nvmf_tgt`) 2352 2353The `nvmf_tgt` configuration file format has been updated significantly to enable 2354new features. See the example configuration file `etc/spdk/nvmf.conf.in` for 2355more details on the new and changed options. 2356 2357The NVMe over Fabrics target now supports virtual mode subsystems, which allow the 2358user to export devices from the SPDK block device abstraction layer as NVMe over 2359Fabrics subsystems. Direct mode (raw NVMe device access) is also still supported, 2360and a single `nvmf_tgt` may export both types of subsystems simultaneously. 2361 2362### Block device abstraction layer (bdev) 2363 2364The bdev layer now supports scatter/gather read and write I/O APIs, and the NVMe 2365blockdev driver has been updated to support scatter/gather. Apps can use the 2366new scatter/gather support via the `spdk_bdev_readv()` and `spdk_bdev_writev()` 2367functions. 2368 2369The bdev status returned from each I/O has been extended to pass through NVMe 2370or SCSI status codes directly in cases where the underlying device can provide 2371a more specific status code. 2372 2373A Ceph RBD (RADOS Block Device) blockdev driver has been added. This allows the 2374`iscsi_tgt` and `nvmf_tgt` apps to export Ceph RBD volumes as iSCSI LUNs or 2375NVMe namespaces. 2376 2377### General changes 2378 2379`libpciaccess` has been removed as a dependency and DPDK PCI enumeration is 2380used instead. Prior to DPDK 16.07 enumeration by class code was not supported, 2381so for earlier DPDK versions, only Intel SSD DC P3x00 devices will be discovered 2382by the NVMe library. 2383 2384The `env` environment abstraction library has been introduced, and a default 2385DPDK-based implementation is provided as part of SPDK. The goal of the `env` 2386layer is to enable use of alternate user-mode memory allocation and PCI access 2387libraries. See `doc/porting.md` for more details. 2388 2389The build process has been modified to produce all of the library files in the 2390`build/lib` directory. This is intended to simplify the use of SPDK from external 2391projects, which can now link to SPDK libraries by adding the `build/lib` directory 2392to the library path via `-L` and linking the SPDK libraries by name (for example, 2393`-lspdk_nvme -lspdk_log -lspdk_util`). 2394 2395`nvmf_tgt` and `iscsi_tgt` now have a JSON-RPC interface, which allows the user 2396to query and modify the configuration at runtime. The RPC service is disabled by 2397default, since it currently does not provide any authentication or security 2398mechanisms; it should only be enabled on systems with controlled user access 2399behind a firewall. An example RPC client implemented in Python is provided in 2400`scripts/rpc.py`. 2401 2402## v16.08: iSCSI target, NVMe over Fabrics maturity 2403 2404This release adds a userspace iSCSI target. The iSCSI target is capable of exporting 2405NVMe devices over a network using the iSCSI protocol. The application is located 2406in app/iscsi_tgt and a documented configuration file can be found at etc/spdk/spdk.conf.in. 2407 2408This release also significantly improves the existing NVMe over Fabrics target. 2409 2410 - The configuration file format was changed, which will require updates to 2411 any existing nvmf.conf files (see `etc/spdk/nvmf.conf.in`): 2412 - `SubsystemGroup` was renamed to `Subsystem`. 2413 - `AuthFile` was removed (it was unimplemented). 2414 - `nvmf_tgt` was updated to correctly recognize NQN (NVMe Qualified Names) 2415 when naming subsystems. The default node name was changed to reflect this; 2416 it is now "nqn.2016-06.io.spdk". 2417 - `Port` and `Host` sections were merged into the `Subsystem` section 2418 - Global options to control max queue depth, number of queues, max I/O 2419 size, and max in-capsule data size were added. 2420 - The Nvme section was removed. Now a list of devices is specified by 2421 bus/device/function directly in the Subsystem section. 2422 - Subsystems now have a Mode, which can be Direct or Virtual. This is an attempt 2423 to future-proof the interface, so the only mode supported by this release 2424 is "Direct". 2425 - Many bug fixes and cleanups were applied to the `nvmf_tgt` app and library. 2426 - The target now supports discovery. 2427 2428This release also adds one new feature and provides some better examples and tools 2429for the NVMe driver. 2430 2431 - The Weighted Round Robin arbitration method is now supported. This allows 2432 the user to specify different priorities on a per-I/O-queue basis. To 2433 enable WRR, set the `arb_mechanism` field during `spdk_nvme_probe()`. 2434 - A simplified "Hello World" example was added to show the proper way to use 2435 the NVMe library API; see `examples/nvme/hello_world/hello_world.c`. 2436 - A test for measuring software overhead was added. See `test/lib/nvme/overhead`. 2437 2438## v16.06: NVMf userspace target 2439 2440This release adds a userspace NVMf (NVMe over Fabrics) target, conforming to the 2441newly-released NVMf 1.0/NVMe 1.2.1 specification. The NVMf target exports NVMe 2442devices from a host machine over the network via RDMA. Currently, the target is 2443limited to directly exporting physical NVMe devices, and the discovery subsystem 2444is not supported. 2445 2446This release includes a general API cleanup, including renaming all declarations 2447in public headers to include a `spdk` prefix to prevent namespace clashes with 2448user code. 2449 2450- NVMe 2451 - The `nvme_attach()` API was reworked into a new probe/attach model, which 2452 moves device detection into the NVMe library. The new API also allows 2453 parallel initialization of NVMe controllers, providing a major reduction in 2454 startup time when using multiple controllers. 2455 - I/O queue allocation was changed to be explicit in the API. Each function 2456 that generates I/O requests now takes a queue pair (`spdk_nvme_qpair *`) 2457 argument, and I/O queues may be allocated using 2458 `spdk_nvme_ctrlr_alloc_io_qpair()`. This allows more flexible assignment of 2459 queue pairs than the previous model, which only allowed a single queue 2460 per thread and limited the total number of I/O queues to the lowest number 2461 supported on any attached controller. 2462 - Added support for the Write Zeroes command. 2463 - `examples/nvme/perf` can now report I/O command latency from the 2464 the controller's viewpoint using the Intel vendor-specific read/write latency 2465 log page. 2466 - Added namespace reservation command support, which can be used to coordinate 2467 sharing of a namespace between multiple hosts. 2468 - Added hardware SGL support, which enables use of scattered buffers that 2469 don't conform to the PRP list alignment and length requirements on supported 2470 NVMe controllers. 2471 - Added end-to-end data protection support, including the ability to write and 2472 read metadata in extended LBA (metadata appended to each block of data in the 2473 buffer) and separate metadata buffer modes. 2474 See `spdk_nvme_ns_cmd_write_with_md()` and `spdk_nvme_ns_cmd_read_with_md()` 2475 for details. 2476- IOAT 2477 - The DMA block fill feature is now exposed via the `ioat_submit_fill()` 2478 function. This is functionally similar to `memset()`, except the memory is 2479 filled with an 8-byte repeating pattern instead of a single byte like memset. 2480- PCI 2481 - Added support for using DPDK for PCI device mapping in addition to the 2482 existing libpciaccess option. Using the DPDK PCI support also allows use of 2483 the Linux VFIO driver model, which means that SPDK userspace drivers will work 2484 with the IOMMU enabled. Additionally, SPDK applications may be run as an 2485 unprivileged user with access restricted to a specific set of PCIe devices. 2486 - The PCI library API was made more generic to abstract away differences 2487 between the underlying PCI access implementations. 2488 2489## v1.2.0: IOAT user-space driver 2490 2491This release adds a user-space driver with support for the Intel I/O Acceleration Technology (I/OAT, also known as "Crystal Beach") DMA offload engine. 2492 2493- IOAT 2494 - New user-space driver supporting DMA memory copy offload 2495 - Example programs `ioat/perf` and `ioat/verify` 2496 - Kernel-mode DMA engine test driver `kperf` for performance comparison 2497- NVMe 2498 - Per-I/O flags for Force Unit Access (FUA) and Limited Retry 2499 - Public API for retrieving log pages 2500 - Reservation register/acquire/release/report command support 2501 - Scattered payload support - an alternate API to provide I/O buffers via a sequence of callbacks 2502 - Declarations and `nvme/identify` support for Intel SSD DC P3700 series vendor-specific log pages and features 2503- Updated to support DPDK 2.2.0 2504 2505## v1.0.0: NVMe user-space driver 2506 2507This is the initial open source release of the Storage Performance Development Kit (SPDK). 2508 2509Features: 2510 2511- NVMe user-space driver 2512- NVMe example programs 2513 - `examples/nvme/perf` tests performance (IOPS) using the NVMe user-space driver 2514 - `examples/nvme/identify` displays NVMe controller information in a human-readable format 2515- Linux and FreeBSD support 2516