1# Changelog 2 3## v21.10: (Upcoming Release) 4 5### idxd 6 7Added `spdk_idxd_get_socket` to query the socket that the idxd device 8is on. 9 10### nvmf 11 12Added `oncs` to `struct spdk_nvmf_ctrlr_data` so that the transport layer 13can decide support RESERVATION feature or not. 14 15An `opts_size` element was added in the `spdk_nvmf_ns_opts` structure to solve the 16ABI compatibility issue between different SPDK version. An new option `anagrpid` was 17added in the `spdk_nvmf_ns_opts` structure. 18 19An new parameter `anagrpid` was added to the RPC `nvmf_subsystem_add_ns`. 20 21An new parameter `anagrpid` was added to the RPC `nvmf_subsystem_listener_set_ana_state`. 22 23`spdk_nvmf_subsystem_destroy` is now can be asynchronous, it accepts a callback and callback argument. 24 25### bdev 26 27New API `spdk_bdev_get_memory_domains` has been added, it allows to get SPDK memory domains used by bdev. 28 29New API functions `spdk_bdev_readv_blocks_ext` and `spdk_bdev_writev_blocks_ext` have been added. 30These functions accept `spdk_bdev_ext_io_opts` structure with extended IO request 31options, e.g. DMA memory domain which describes data that may belong to another memory domain and 32can't be accessed directly. 33 34Added `async_fini_start` to allow bdev modules to complete the `fini_start` asynchronously, 35with new `spdk_bdev_module_fini_start_done` API. 36 37Deprecated `spdk_bdev_module_finish_done()` API, which will be removed in SPDK 22.01. 38Bdev modules should use `spdk_bdev_module_fini_done()` instead. 39 40The `ocssd` bdev has been removed. The Open Channel specification has been largely superceded by 41zoned namespaces, few if any devices have been brough to market, and there is little reason to 42continue to support this. OCSSD support in the nvme driver will remain for now. 43 44### dma 45 46A new library, lib/dma, has been added. This library provides the necessary infrastructure for 47handling systems and devices with multiple memory domains. For example, a PCIe add-in card with an 48SoC may be running SPDK on the SoC. That SoC has its own local memory, but SPDK may be controlling 49devices that can also access the host system memory. This library provides infrastructure to enumerate 50the memory domains and request hardware perform DMA transfers between them. 51 52### event 53 54Added the `disable_signal_handlers` flag to the `spdk_app_opts` struct. 55 56### log 57 58Added API `spdk_log_to_syslog_level` to return syslog level based on SPDK's 59log level. 60 61### nvme 62 63Added new functions `spdk_nvme_zns_set_zone_desc_ext` and `spdk_nvme_zns_ext_report_zones` 64to set zone descriptor extension and to get extended zone report respectively. 65 66New API `spdk_nvme_ctrlr_get_memory_domains` has been added, it allows to get SPDK memory domains used by nvme controller. 67 68New API functions `spdk_nvme_ns_cmd_readv_ext` and `spdk_nvme_ns_cmd_writev_ext` 69have been added. These functions accept `spdk_nvme_ns_cmd_ext_io_opts` structure with extended IO request 70options, e.g. DMA memory domain which describes data that may belong to another memory domain and 71can't be accessed directly. 72 73Added a new function `spdk_nvme_ctrlr_get_regs_bpinfo` to get boot partition info of a controller. 74Added new functions `spdk_nvme_ctrlr_write_boot_partition`, 75`spdk_nvme_ctrlr_read_boot_partition_start` and `spdk_nvme_ctrlr_read_boot_partition_poll` 76to write and read the boot partitions of a controller. 77 78Added `spdk_nvme_ctrlr_get_opts` to retrieve the current controller options. 79 80### dpdk 81 82Updated DPDK submodule to DPDK 21.08. 83 84### scsi 85 86New functions, `spdk_scsi_dev_get_first_lun` and `spdk_scsi_dev_get_next_lun` 87have been added to iterate LUNs of a SCSI device. 88 89### util 90 91The `spdk_fd_group_add` API now takes a `name` parameter. 92 93## v21.07 94 95### accel_fw 96 97Added API `spdk_accel_submit_copy_crc32c` to perform a CRC32C while copying data. 98 99Added API `spdk_accel_batch_prep_copy_crc32c` to batch CRC32C + copy commands. 100 101Added API `spdk_accel_submit_copy_crc32cv` to submit chained CRC32C + copy commands. 102 103### bdev 104 105Change `spdk_bdev_read_blocks_with_md` arg offset definition from int64_t to uint64_t. 106 107Red-black tree has been used to organize the bdev names and aliases uniformly 108to provide faster lookup. 109 110Removed ZCOPY emulation: The bdev module can be checked to see if it supports ZCOPY 111and if not supported then use existing READ/WRITE commands. 112 113Added iov to spdk_bdev_zcopy_start. 114 115### dpdk 116 117Updated DPDK submodule to DPDK 21.05. 118 119### idxd 120 121Remove the `probe_cb` parameter in `spdk_idxd_probe` function. And remove the definition 122of `spdk_idxd_probe_cb` function pointer. It should be implemented in idxd_user.c. 123 124Added API `spdk_idxd_submit_copy_crc32c` to perform a CRC32C while copying data. 125 126Added API `spdk_idxd_batch_prep_copy_crc32c` to prepare a batch operation to perform 127a CRC32C while copying data. 128 129We have userspace idxd driver under accel_engine library (module/accel/idxd/accel_engine). 130And kernel idxd driver usage is also added under the accel_engine library. This approach 131implemented a wrapper library to use IDXD device by leveraging the kernel DSA driver in 132SPDK idxd library (lib/idxd). Then users can leverage the RPC later to configure how to 133use the DSA device by user space driver or kernel driver. 134 135### init 136 137Added new `init` library that initializes the SPDK subsystems, which previously was 138internal to application framework. That functionality has been made public and for 139use by applications which don't leverage SPDK's application framework. 140 141### iscsi 142 143New parameters, `pdu_pool_size`, `immediate_data_pool_size`, and `data_out_pool_size`, 144were added to the RPC `iscsi_set_options` to run iSCSI target with varying amount of 145available memory. 146 147### json 148 149Added API `spdk_json_write_named_uint128` and `spdk_json_write_uint128` to perform 150the uint128 related data. 151 152### net 153 154Removed deprecated `net` library. 155 156Added support for ZCOPY. ZCOPY is used in preference to READ and WRITE if the 157bdev module supports ZCOPY and the bdev module has ZCOPY enabled. 158 159### nvme 160 161`spdk_nvme_map_prps` and `spdk_nvme_map_cmd` were moved to nvmf/vfio-user as internal APIs 162as vfio-user is the only user for the above two APIs. 163 164Added a new function `spdk_nvme_ns_cmd_copy` to submit a Simple Copy Command to a Namespace. 165 166Update the `spdk_nvme_generic_command_status_code` structure with new status code 167according to the definition in NVMe 1.4 spec. 168 169`spdk_nvme_ctrlr_get_default_ctrlr_opts` now sets `use_cmb_sqs` to false. This means 170that if a controller has a CMB and supports SQs in the CMB, SPDK will not use 171the CMB for SQs by default - the user must set `use_cmb_sqs` to true in 172the `spdk_nvme_ctrlr_opts` structure prior to controller attach. 173 174Add a new function `spdk_nvme_detach_poll` to simplify a common use case to continue 175polling until all detachments complete. 176 177Added new argument `timeout_admin_us` to `spdk_nvme_ctrlr_register_timeout_callback` so callers 178can specify a different timeout for admin commands vs. io commands. 179 180An existing function `spdk_nvme_detach_async` was updated to add one or more detachments 181to an active context while it is being polled. 182 183Rename a variable in the member `cmic` of the struct `spdk_nvme_ctrlr_data` from 184`multi_host` to `multi_ctrlr`. The variable means that the NVM subsystem may have two or 185more controllers if set to 1. However `multi_host` had indicated a particular use case 186such that the NVM subsystem is used by multiple hosts. 187 188A new option `disable_read_ana_log_page` was added to struct `spdk_nvme_ctrlr_opts` to disable 189reading ANA log page. The upper layer is expected to read ANA log page instead if `true`. 190The default value is `false`. 191 192New APIs, `spdk_nvme_ctrlr_reset_async` and `spdk_nvme_ctrlr_reset_poll_async`, have been added to 193reset a controller asynchronously. 194 195New RPC `bdev_nvme_reset_controller` was added, to reset an NVMe controller. 196 197Added `spdk_nvme_ns_get_nguid` function to get NGUID for the given namespace. 198 199Added `spdk_nvme_ctrlr_is_fabrics` function to indicate whether a ctrlr handle 200is associated with a fabrics controller. 201 202### nvmf 203 204Added `min_cntlid` and `max_cntlid` to `nvmf_create_subsystem` to limit the controller ID range. 205Added `spdk_nvmf_subsystem_get_min_cntlid` and `spdk_nvmf_subsystem_get_max_cntlid` to request those values. 206 207A new parameter, `poll_groups_mask` was added to the nvmf_set_config RPC that allows specifying 208a subset of cores for the nvmf poll groups. This helps to avoid imbalances when some cores are 209busy with periodic timer tasks that run very frequently. 210 211`spdk_nvmf_request_get_buffers_multi` API is removed. 212 213Added the `nvmf_set_crdt` RPC for setting command retry delay times. 214 215Expanded `spdk_nvmf_poll_group_stat` with current qpair count statistics. 216 217Removed following deprecated APIs: 218 219- `spdk_nvmf_poll_group_get_stat` (function in `nvmf.h`), 220- `spdk_nvmf_transport_poll_group_get_stat` (function in `nvmf.h`), 221- `spdk_nvmf_transport_poll_group_free_stat`(function in `nvmf.h`), 222- `spdk_nvmf_rdma_device_stat` (struct in `nvmf.h`), 223- `spdk_nvmf_transport_poll_group_stat` (struct in `nvmf.h`), 224- `poll_group_get_stat` (transport op in `nvmf_transport.h`), 225- `poll_group_free_stat` (transport op in `nvmf_transport.h`). 226 227### rpc 228 229New RPC `bdev_rbd_register_cluster` and `bdev_rbd_unregister_cluster` was added, it allows to create 230and delete the rados object cluster, then users can choose the cluster to create related rbd 231device. 232 233Revised `bdev_rbd_create` parameter, it allows to use an optional parameter `--cluster-name` 234to create a rbd bdev with an already registered Rados Cluster Object. 235 236New RPC `bdev_rbd_get_clusters_info` was added, it allows to get the info of the registered 237Rados Cluster names. 238 239New optional parameter, `timeout_admin_us`, added to the bdev_nvme_set_options RPC. 240 241Revised a parameter `--stripe-size_kb` to `--stripe-size-kb` of `bdev_raid_create` method 242provided in `scripts/rpc.py` for consistency. 243 244An new optional parameter `config_kernel_mode` was added to the RPC `idxd_scan_accel_engine`, 245and this is used to enable using the kernel mode IDXD driver. 246 247Removed deprecated parameter `enable-zerocopy-send` of RPC `sock_impl_set_options`, 248use `enable-zerocopy-send-server` or `enable-zerocopy-send-client` instead. 249Removed deprecated parameter `disable-zerocopy-send` of RPC `sock_impl_set_options`, 250use `disable-zerocopy-send-server` or `disable-zerocopy-send-client` instead. 251 252### rpm 253 254Removed deprecated `pkg/spdk.spec`, please use `rpmbuild/spdk.spec` instead. 255See [RPM documentation](https://spdk.io/doc/rpm.html) for more details. 256 257### thread 258 259Red-black tree has been used for timed pollers to provide faster insertion and deletion 260and for io_devices to provide faster lookup. 261 262Removed `spdk_io_channel` structure from public header and moved it to thread_internal.h. 263 264### trace 265 266Added `spdk_trace_register_description_ext` function to register variable number of tracepoint arguments. 267 268Added ability to chain multiple trace entries together to extend the size of the argument buffer. 269 270### util 271 272Red-black tree macros has been added by using the macros provided by the FreeBSD operating system 273under the same BSD license. 274 275Add an new macro `SPDK_SIZEOF_MEMBER` to get the size of a member of a struct. 276 277`spdk_crc32c_iov_update` function was added to support calculating the crc32c of the iovs. 278 279Added zipf random number generator with power law probability distribution. 280When applied to performance testing of block devices, it will select blocks over 281the full range of LBAs, but will more frequently select lower-numbered LBAs. 282 283## v21.04 284 285### accel 286 287Two new accelerated crc32 functions `spdk_accel_submit_crc32cv` and 288`spdk_accel_batch_prep_crc32cv` are added in order to provide the 289chained accelerated CRC32 computation support. 290 291### bdev 292 293For `bdev_ocssd_create` RPC, the optional parameter `range` was removed. 294Only one OCSSD bdev can be created for one OCSSD namespace. 295 296Removed the `spdk_bdev_open` from bdev library API. 297Removed the `spdk_vbdev_register` and `spdk_bdev_part_base_construct` from bdev module API. 298Removed the `config_text` function for bdev modules to report legacy config. 299 300Added `spdk_bdev_get_max_active_zones` API to display maximum number active zones of a given bdev. 301 302Added `spdk_bdev_get_max_zone_append_size` API to display maximum zone append data transfer size. 303 304### bdev_nvme 305 306Added support for zoned namespaces. 307 308### blobstore 309 310Removed the `spdk_bdev_create_bs_dev_from_desc` and `spdk_bdev_create_bs_dev` API. 311 312### env 313 314Added `spdk_pci_device_allow` API to allow applications to add PCI addresses to 315the allowed list after the application has started. 316 317Removed the `pci_whitelist`, `pci_blacklist` and `master_core` members of struct `spdk_env_opts`. 318 319Added hotplug support based on uevent in `pci_event.c`. Added usage of this functionality in 320nvme, virtio-scsi and virtio_blk libraries. Please see the new API `spdk_pci_event_listen`, 321`spdk_pci_get_event`, `spdk_pci_register_error_handler` and `spdk_pci_unregister_error_handler`. 322 323### event 324 325Removed the `config_file`, `max_delay_us`, `pci_whitelist` 326and `pci_blacklist` members of struct `spdk_app_opts`. 327 328### idxd 329 330A new API `spdk_idxd_device_needs_rebalance` was added so that users of the library 331can know whether they need to rebalance the flow control for the channel 332that was just added/removed. This is based on how the low level library 333shares devices amongst channels. 334 335The API `spdk_idxd_reconfigure_chan` had the `num_channels` removed as this 336is now tracked in the library. The app makes use the new API above to 337determine whether to rebalance or not. This applies to `spdk_idxd_configure_chan` 338as well. 339 340The API `spdk_idxd_put_channel` now returns the rebalance state for the 341underlying device. 342 343### iscsi 344 345A security vulnerability has been identified and fixed in the SPDK iSCSI target. 346A TEXT PDU with no data, but CONTINUE flag set, would result in a NULL pointer dereference 347and crash the SPDK iSCSI target process. All users of the SPDK iSCSI target 348are recommended to update. All SPDK versions <= v21.01 are affected. 349 350### net 351 352The net library is deprecated and will be removed in the 21.07 release. 353 354### nvme 355 356Added a new function `spdk_nvme_ctrlr_get_regs_pmrcap` to get the PMR capabilities. 357 358Added an accelerated table pointer in `spdk_nvme_poll_group` 359which can be used to provide the accelerated functions by users with 360hardware engine, such as crc32c accelerated function. 361 362Added new functions `spdk_nvme_ctrlr_get_pmrsz`, `spdk_nvme_ctrlr_enable_pmr`, 363`spdk_nvme_ctrlr_disable_pmr`, `spdk_nvme_ctrlr_map_pmr` and `spdk_nvme_ctrlr_unmap_pmr`. 364 365Added NVMe transport operations to enable, disable, map and unmap the PMR. 366 367Added `spdk_nvme_qpair_get_optimal_poll_group` function and `qpair_get_optimal_poll_group` 368function pointer in spdk_nvmf_transport_ops structure in order to add the qpair to the most 369suitable polling group. 370 371Added OPTPERF and namespace optimal performance fields to nvme_spec.h. 372 373Added `spdk_nvme_set_hotplug_filter` API to allow applications to choose which 374hot-inserted SSDs should be probed. This is useful for use cases where multiple 375independent SPDK processes are running on one node. The filter function can 376then be implemented in these processes to decide which SSDs to probe based on 377the new SSD's PCI address. 378 379New functions `spdk_nvme_poll_group_get_stats` and `spdk_nvme_poll_group_free_stats` 380were added. These functions allow to get transport statistics per NVME poll group. 381 382Added `spdk_nvme_map_cmd` API to map the NVMe command with SGL cases. 383 384Added support for vector variant of ZNS zone append commands with new API 385`spdk_nvme_zns_zone_appendv` and `spdk_nvme_zns_zone_appendv_with_md`. 386 387Added `spdk_nvme_zns_ns_get_max_open_zones` and `spdk_nvme_zns_ns_get_max_active_zones` API, 388to display maximum number of open and active zones of the given namespace. 389 390Added `spdk_nvme_zns_ns_get_zone_size_sectors` API to provide size of zone in number of 391sectors. 392 393Added `spdk_nvme_qpair_get_id` API to display the ID of the specified qpair. 394 395### nvmf 396 397Removed the `spdk_nvmf_tgt_listen` and `spdk_nvmf_subsystem_add_ns` API. 398 399Added new APIs: 400 401- `spdk_nvmf_poll_group_dump_stat` (function in `nvmf.h`). 402- `poll_group_dump_stat` (transport op in `nvmf_transport.h`). 403 404The following APIs have been deprecated and will be removed in SPDK 21.07: 405 406- `spdk_nvmf_poll_group_get_stat` (function in `nvmf.h`), 407- `spdk_nvmf_transport_poll_group_get_stat` (function in `nvmf.h`), 408- `spdk_nvmf_transport_poll_group_free_stat`(function in `nvmf.h`), 409- `spdk_nvmf_rdma_device_stat` (struct in `nvmf.h`), 410- `spdk_nvmf_transport_poll_group_stat` (struct in `nvmf.h`), 411- `poll_group_get_stat` (transport op in `nvmf_transport.h`), 412- `poll_group_free_stat` (transport op in `nvmf_transport.h`). 413 414See header files for details. 415 416The `trtype` field in JSON returned by `nvmf_get_stats` RPC contains now the name of the transport, 417which is the same as the type for defined transports and more informative for a custom transport. 418 419Added `hdgst` and `ddgst` parameters to `bdev_nvme_attach_controller` RPC in order change 420state of TCP header and data digest. 421 422Added `num_cqe` parameter to `nvmf_create_transport` RPC to set number of completion queues (CQ) 423for RDMA transport. Useful when CQ resize operation is not supported, for example iWARP. 424 425### ocf 426 427Updated OCF submodule to v20.12.2 428 429Added `bdev_ocf_set_cache_mode` RPC to dynamically switch cache mode of OCF bdev. 430 431### opal 432 433Removed the `spdk_opal_supported` API. 434 435### raid 436 437For `bdev_raid_create` RPC, the deprecated parameter `strip_size` was removed. 438 439### rpc 440 441New RPC `bdev_nvme_get_transport_statistics` was added, it allows to get transport statistics 442of nvme poll groups. 443 444Parameter `enable-zerocopy-send` of RPC `sock_impl_set_options` is deprecated and will be removed in SPDK 21.07, 445use `enable-zerocopy-send-server` or `enable-zerocopy-send-client` instead. 446Parameter `disable-zerocopy-send` of RPC `sock_impl_set_options` is deprecated and will be removed in SPDK 21.07, 447use `disable-zerocopy-send-server` or `disable-zerocopy-send-client` instead. 448 449Added cmd_parser.py used to parse the args from argparse. There are 450two benefit to use command parser: 451 452- Simplify the definition of rpc method. It will reduce the rpc method code. 453- Make the rpc call more versatile. User can add private args into rpc method. 454 455### rpm 456 457Added support for new RPM spec, rpmbuild/spdk.spec, which can be used for packaging the 458SPDK. The pkg/spdk.spec is considered to be deprecated and scheduled for removal in SPDK 21.07. 459See [RPM documentation](https://spdk.io/doc/rpm.html) for more details. 460 461### sock 462 463The type of `enable_placement_id` in struct `spdk_sock_impl_opts` is changed from 464bool to int. We can use RPC to configure different value of `enable_placement_id`. 465Then we can leverage SO_INCOMING_CPU to get placement_id, which aims to utilize 466CPU cache locality, enabled by setting enable_placement_id=2. 467 468A new socket placement mode called PLACEMENT_MARK has been added. Some NICs allow 469sockets to be marked using the SO_MARK socket option as a hint for which hardware 470queue they should be associated with. This mode leverages that by setting the same 471value for all sockets within a poll group. 472 473New parameters `enable_zerocopy_send_server` and `enable_zerocopy_send_client` were added 474to struct spdk_sock_impl_opts, these parameters enable or disable zero copy send for server 475and client sockets which are created using `spdk_sock_listen` and `spdk_sock_listen_ext` (server); 476`spdk_sock_connect` and `spdk_sock_connect_ext` (client) functions. Existing parameter 477`enable_zerocopy_send` enables or disables zero copy send for both server and client sockets. 478 479### thread 480 481A new API `spdk_io_channel_get_io_device` was added to get the io_device for the specified 482I/O channel. 483 484Added `spdk_thread_set_interrupt_mode` function in order to set present spdk_thread into 485interrupt mode or back to poll mode. It is valid only when thread interrupt facility is 486enabled by `spdk_interrupt_mode_enable`. 487 488Added `spdk_poller_register_interrupt` function to mark that the poller is capable of 489entering interrupt mode. Callback function will be called when the poller must transition 490into or out of interrupt mode. 491 492### virtio 493 494Added the `bdev_virtio_blk_set_hotplug` RPC for the virtio blk pci device. 495 496## v21.01 497 498### bdev 499 500An `opts_size` element was added in the `spdk_bdev_opts` structure to solve the 501ABI compatibility issue between different SPDK version. And also add `opts_size` 502parameter in spdk_bdev_get_opts function. Two fields `small_buf_pool_size` and 503`large_buf_pool_size` were added into spdk_bdev_opts, which were used to determine 504the small and large buffer pool size of the whole bdev module. 505 506A new API `spdk_bdev_wait_for_examine` was added to allow for checking state of 507examine process. Along with corresponding `bdev_wait_for_examine` RPC, which 508is now always called during `spdk_bdev_subsystem_config_json` making sure 509every bdev is ready to be used. 510 511A new API `spdk_bdev_io_get_aio_status` was added for getting the status of 512bdev_io as Linux AIO errno. Also `spdk_bdev_io_complete_aio_status` function 513and `SPDK_BDEV_IO_STATUS_AIO_ERROR` were added for bdev module to complete 514a bdev_io with Linux AIO errno. 515 516A new API `spdk_bdev_get_module_name` was added for getting block device module name. 517Also `spdk_bdev_get_module_ctx` was added to obtain a bdev module context for 518the block device opened by the specified descriptor. 519 520Added `max_segment_size` and `max_num_segments` to `spdk_bdev` structure for 521bdev modules to specify splitting requirements. 522 523### blob 524 525An `opts_size` element was added in the `spdk_bs_opts` structure to solve the 526ABI compatibility issue between different SPDK version. And also add `opts_size` 527parameter in `spdk_bs_opts_init` function. 528 529An `opts_size` element was added in the `spdk_blob_opts` structure to solve the 530ABI compatibility issue between different SPDK version. And also add `opts_size` 531parameter in `spdk_blob_opts_init` function. 532 533An `opts_size` element was added in the `spdk_blob_open_opts` structure to solve the 534ABI compatibility issue between different SPDK version. And also add `opts_size` 535parameter in `spdk_blob_open_opts_init` function. 536 537### build 538 539SPDK now generates pkg-config files to simplify the process of determining which 540libraries must be linked into an SPDK application. 541 542### dpdk 543 544Updated DPDK submodule to DPDK 20.11. 545 546Removed `--with-igb-uio-driver` configure option. Since DPDK 20.11 igb_uio driver 547was moved to separate dpdk-kmods repository. If required, users need to refer to 548dpdk-kmods repository to build the driver. 549 550### env 551 552The `pci_whitelist`, `pci_blacklist` and `master_core` members of struct `spdk_env_opts` 553have been deprecated. The new members are named `pci_allowed`, `pci_blocked` and 554`main_core` respectively. 555 556An `opts_size` element was added in the `spdk_app_opts` structure 557to solve the ABI compatibility issue between different SPDK version. An `opts_size` 558parameter is added into `spdk_app_opts_init` function. 559 560### event 561 562The `pci_whitelist` and `pci_blacklist` members of struct `spdk_app_opts` have been 563deprecated. The new members are named `pci_allowed` and `pci_blocked` respectively. 564 565The `--pci-blacklist` command line option has been deprecated, replaced with 566`--pci-blocked`. 567 568The `--pci-whitelist/-W` command line options have been deprecated, replaced with 569`--pci-allowed/-A`. 570 571Added new experimental `dynamic` scheduler that rebalances idle threads, adjusts CPU frequency 572using dpdk_governor and turns idle reactor cores to interrupt mode. Please see 573[scheduler documentation](https://www.spdk.io/doc/scheduler.html) for details. 574 575## ioat 576 577The PCI BDF whitelist option has been removed from the `ioat_scan_accel_engine` RPC. 578IOAT PCI functions can still be allowed or denied using SPDK application command 579line options. 580 581### isa-l 582 583Updated ISA-L submodule to v2.30.0. 584 585### json 586 587A new API `spdk_jsonrpc_send_bool_response` was added to allow sending response for 588writing json bool results into one function. 589Update API `bdev_nvme_set_options` and add a `keep_alive_timeout_ms` parameter. Now you 590can specify the `keep_alive_timeout` before creating NVMe bdev. 591 592### nbd 593 594Change the return type of function `spdk_nbd_stop` from void to int. And update the 595`spdk_nbd_fini` with two parameters to make its behavior from sync to async. 596 597### nvme 598 599Directives support was added to the NVMe driver. 600 601Two async APIs `spdk_nvme_ctrlr_cmd_directive_receive` and `spdk_nvme_ctrlr_cmd_directive_send` 602are added for Directive Send and Directive Receive command, respectively. 603 604Added a new function `spdk_nvme_ctrlr_reset_subsystem` to perform a NVMe 605subsystem reset. Note: The NVMe-oF target does not support the subsystem reset yet. 606 607Add a new function `spdk_nvme_bytes_to_numd` to transfer bytes to number of 608dwords. 609 610Added a new custom transport `SPDK_NVME_TRANSPORT_VFIOUSER` to enable NVMe 611driver running with NVMe over vfio-user target. 612 613Added the vfio-user custom transport implementation in NVMe driver which can connect 614to NVMe over vfio-user target via vfio-user transport. 615 616Added a new function `spdk_nvme_ctrlr_set_remove_cb` to remove or override 617`remove_cb` and `remove_ctx` specified when the controller was probed. 618 619Added support for ZNS zone append command with new API `spdk_nvme_zns_zone_append` and 620`spdk_nvme_zns_zone_append_with_md`. 621 622### nvmf 623 624`nvmf_fc_lld_fini` now takes callback and hence updating FC Broadcom LLD driver 625to the latest is required. 626 627FC transport supported primary tagging and VMID. 628 629Broadcom FC LLD driver and SPDK NVMe-oF FC transport consolidated one LLD API, 630`nvmf_fc_init_poller_queues` into another LLD API `nvmf_fc_init_q`. 631Hence updating Broadcom FC LLD driver to the latest is required. 632 633The functions `destroy` and `qpair_fini` in the transport interface now accept a 634`cb_fn` and `cb_arg` to call upon completion, and their execution can be asynchronous. 635 636The SPDK NVMe-oF target now supports async event notification for discovery log changes. 637This allows the initiator to create persistent connection to discovery controller and 638be notified of any discovery log changes. 639 640An `opts_size` element was added in the `spdk_nvmf_transport_opts` structure 641to solve the ABI compatiblity issue between different SPDK version. And also add 642`opts_size` parameter in `spdk_nvmf_transport_opts_init` function. 643 644Added a new custom vfio-user transport implementation in NVMe-oF which can provide 645emulated NVMe devices to QEMU and SPDK NVMe driver. 646 647Added new API `spdk_nvmf_tgt_listen_ext` that uses `spdk_nvmf_listen_opts` to allow 648passing generic options to specific transport. This functionality replaces one provided 649with `nvmf_subsystem_set_options`. Also removed `nvmf_subsystem_set_options` RPC 650that allowed transport specific options within subsystem. 651 652The `trsvcid` in `nvmf_subsystem_add_listener` RPC is now optional. 653 654Pausing a subsystem now only pauses all admin queues. This allows for the 655management of hosts and listeners, plus the addition of namespaces without a 656full subsystem pause. Additionally, the target now allows for pausing 657individual namespaces within a subsystem. To remove a namespace from a 658subsystem, only the specific namespace must be paused. I/O will continue to 659other namespaces while these operations execute. 660 661### rpc 662 663An new optional parameter `wait` was added to the RPC `iscsi_create_portal_group`, 664and an new RPC `iscsi_start_portal_group` was added. They will be used not to start 665listening on portals for a portal group until all associated target nodes are created 666at startup, otherwise some iSCSI initiators may fail to re-login when SPDK iSCSI 667target application restarts. 668 669Two optional parameter `--small-buf-pool-size` and `--large-buf-pool-size` were added 670into `bdev_set_options` function. 671 672### vhost 673 674Added optional `packed_ring_recovery` parameter to `vhost_create_blk_controller` RPC 675enabling packed ring live recovery. 676 677## v20.10 678 679### accel 680 681New APIs `spdk_accel_batch_cancel` and `spdk_idxd_batch_cancel` were added to allow 682cancelling a batch sequence. 683 684### bdev 685 686A new `spdk_bdev_part_base_construct_ext` function has been added and the 687`spdk_bdev_part_base_construct` has been deprecated. The 688`spdk_bdev_part_base_construct_ext` function takes bdev name as an argument instead 689of bdev structure to avoid a race condition that can happen when the bdev is being 690removed between a call to get its structure based on a name and actually opening it. 691 692Removed `spdk_bdev_config_text` function for bdev modules to report legacy config. 693 694### blobstore 695 696A new `spdk_bdev_create_bs_dev_ext` function has been added and `spdk_bdev_create_bs_dev_from_desc` 697function has been deprecated. 698 699### dpdk 700 701Updated DPDK submodule to DPDK 20.08. 702 703### env 704 705A new API `spdk_mem_get_fd_and_offset` was added to get file descriptor and offset for 706a given virtual address. 707 708### event 709 710Removed `spdk_subsystem_config` callback for submodules as part of legacy config removal. 711 712Removed `spdk_app_get_running_config` function that printed configuration in legacy format, 713and removed `usr1_handler` from `struct spdk_app_opts` callback that was used to call it. 714 715Added SPDK thread scheduler framework which is used to rebalance load among CPU cores. 716The scheduler implementation is pluggable and two schedulers are added first, `static` scheduler 717and `gscheduler` scheduler. 718 719`static` scheduler is used by default and does not reschedule threads and keeps the initial 720assignments. 721 722CPU frequency governor framework and implementation are also added. 723`dpdk_governor` is the first implementation and is based on rte_power library of DPDK. 724`gscheduler` scheduler utilizes the CPU frequency governor. 725 726Scheduler and governor frameworks are experimental features. 727 728### fio 729 730Bdev fio_plugin no longer supports legacy configuration files. Options `spdk_conf` and 731`spdk_json_conf` have to point to valid JSON configuration. 732 733### gpt 734 735Removed option to disable GPT probing via legacy configuration. Please use generic 736bdev functionality to replicate the behavior. See `bdev_set_options --disable-auto-examine` 737and `bdev_examine` RPC. 738 739### intel-ipsec-mb 740 741Updated intel-ipsec-mb submodule to v0.54 742 743### isa-l 744 745Updated ISA-L submodule to v2.29.0. 746 747### iscsi 748 749The iSCSI target supports iSCSI login redirection feature to achieve fault tolerance 750and load balancing particularly in scale out iSCSI target system, which runs multiple 751SPDK iSCSI targets. 752 753### json 754 755A new API `spdk_json_decode_object_relaxed` was added to allow parsing json values 756even if decoder for a given key is not found. 757 758A new API `spdk_json_free_object` was added to free memory allocated by `spdk_json_decode_object`. 759 760### log 761 762A log flag structure, `spdk_log_register_flag`, `spdk_log_get_first_flag`, 763`spdk_log_get_next_flag` and macro `SPDK_LOG_REGISTER_COMPONENT`, have been added which 764enable registering and interacting with log flags. Added `SPDK_INFOLOG`, `SPDK_DEBUGLOG` 765and `SPDK_LOGDUMP` macros to display logs for specified log flag. 766 767The log flags in the event framework `-L` and several SPDK applications has been enabled on 768release builds of SPDK. On debug builds this option will additionally set 769log print level to `SPDK_LOG_DEBUG`. 770 771A new API `spdk_log_enable_timestamps` and new RPC `log_enable_timestamps` was added to 772toggle log timestamps. 773 774### miscellaneous 775 776The contents of the log_rpc library have been moved to the event library. The log_rpc 777library now no longer exists. 778 779The contents of the app_rpc library have been moved to the event library. The app_rpc 780library now no longer exists. 781 782The contents of the bdev_rpc library have been moved to the bdev library. The app_rpc 783library now no longer exists. 784 785The bdevperf application now disables the zcopy API by default. Prior to this change, 786bdevperf enabled using the zcopy API by default which caused a performance impact of 787up to 25% on bdevs that don't natively support zcopy because the API emulates zero-copy 788by allocating a buffer. The bdevperf `-x` param was renamed to `-Z` and the default 789value changed to false. For bdevs that support zcopy, use the -Z flag to enable 790using zcopy API. 791 792### net 793 794Net framework intended for use by user-space TCP stacks was removed. 795 796### nvme 797 798Zoned Namespace Command Set support has been added to NVMe driver, see `nvme_zns.h` for details. 799 800New APIs, `spdk_nvme_detach_async` and `spdk_nvme_detach_poll_async`, have been added to 801detach multiple controllers in parallel to mitigate lengthy shutdown notification of 802a few NVMe SSDs. 803 804New APIs, `spdk_nvme_ns_get_ana_group_id` and `spdk_nvme_ns_get_ana_state`, have been 805added to get ANA group ID and ANA state of the given namespace, respectively. 806 807If no specific command set is requested (by setting the command_set member in the 808`spdk_nvme_ctrlr_opts` structure), SPDK will automatically select the most appropriate 809command set based on what the controller supports. 810 811### nvmf 812 813A new API `spdk_nvmf_subsystem_add_ns_ext` has been added and the API `spdk_nvmf_subsystem_add_ns` 814has been deprecated. 815 816The NVMe-oF target now supports Asymmetric Namespace Access (ANA) Reporting to provide 817multipath to NVMe-oF initiator. 818 819Add `no_wr_batching` parameter to `spdk_nvmf_transport_opts` struct to disable 820Work Requests batching in RDMA transport. 821 822NVMf Target transports can now parse any additional JSON params in the nvmf_create_transport RPC 823through the JSON context provided via spdk_nvmf_target_opts->transport_specific. 824 825A new RPC `nvmf_subsystem_set_options` was added to allow transport specific options within subsystem. 826 827A new API `spdk_nvmf_transport_stop_listen_async` was added to stop accepting new connections 828at the provided address. `nvmf_subsystem_remove_listener` RPC now uses this function. 829 830A new API `spdk_nvmf_subsystem_disconnect_host` was added to disconnect all connections 831originating from the provided hostnqn. `nvmf_subsystem_remove_host` RPC now uses this function. 832 833### ocf 834 835Updated OCF submodule to v20.03.1 836 837### rpc 838 839New optional parameters, `enable_placement_id` and `enable_quickack` were added to the 840`sock_impl_set_options` RPC. 841 842A new RPC `bdev_examine_bdev` was added to allow users to examine a bdev explicitly. 843It can be used only if bdev_auto_examine is set to false by the RPC `bdev_set_options`. 844 845New optional parameters `no_wr_batching` and `control_msg_num` were added to the RPC 846'nvmf_create_transport'. 847 848New RPCs, `iscsi_target_node_set_redirect` and `iscsi_target_node_request_logout`, have 849been added, and a new optional parameter `private` has been added to the RPC 850`iscsi_create_portal_group` to support iSCSI login redirection feature. 851 852A new optional parameter `ana_reporting` was added to the RPC `nvmf_create_subsystem`, and 853a new RPC `nvmf_subsystem_listener_set_ana_state` was added for ANA reporting. 854 855New RPCs, `nvmf_subsystem_get_listeners` and `nvmf_subsystem_get_qpairs`, were added to 856retrieve configuration of the NVMe-oF subsystem. 857 858### scsi 859 860Two new APIs have been added `spdk_scsi_dev_construct_ext` and 861`spdk_scsi_dev_add_lun_ext` that allow the upper layer(e.g. vhost-scsi) to 862receive the notification when the scsi bdev has been resized. 863 864The `spdk_scsi_dev_construct` and `spdk_scsi_dev_add_lun` eventually may be 865deprecated and removed. 866 867### sock 868 869The `enable_placement_id` field was added in the `struct spdk_sock_impl_opts` to 870make the placement_id feature configurable by users. The default setting is 871not enabled. 872 873The `enable_quick_ack` field was added in the `struct spdk_sock_impl_opts` to enable 874or disable quick ack for the POSIX sock module. The default setting is not enabled. 875 876### thread 877 878`fd_group` is applied to support interrupt mode. 879New APIs were added to support an experimental interrupt mode. This allows modules or 880libraries to selectively register file descriptors that the spdk_thread can wait on, 881as an alternative to polling. In v20.10, this functionality is enabled in a very small 882subset of SPDK libraries and modules. 883 884### util 885 886A new utility named `fd_group` was add. It is now 887implemented by epoll on Linux platform. It can be used by 888spdk_thread and reactor to implement interrupt mode. 889 890Added `spdk_bit_pool` utility, see `bit_pool.h` for details. 891 892### vhost 893 894SPDK has switched to DPDK's rte_vhost library since 19.07 release, removed the internal 895rte_vhost library which is used for DPDK older than 19.05, removed the experimental vhost 896nvme target which depends on the internal rte_vhost library. 897 898### vpp 899 900Removed socket abstraction for VPP. 901 902## v20.07: SPDK CSI driver, new accel_fw commands, I/O abort support 903 904spdk_json_free_object() was added to free memory allocated by spdk_json_decode_object(). 905 906### accel 907 908A new API was added `spdk_accel_get_capabilities` that allows applications to 909query the capabilities of the currently enabled accel engine back-end. 910 911A new capability, CRC-32C, was added via `spdk_accel_submit_crc32c`. 912 913The software accel engine implemenation has added support for CRC-32C. 914 915A new capability, compare, was added via `spdk_accel_submit_compare`. 916 917The software accel engine implemenation has added support for compare. 918 919Several APIs were added to `accel_engine.h` to support batched submission 920of operations. 921 922Several APIs were added to `accel_engine.h` to support dualcast operations. 923 924### accel_fw 925 926The accel_fw was updated to no longer require the app to allocate an 927accel_task on its behalf. All public APIs now take a callback arg as 928the parameter that used to be the accel_task. 929 930The accel_fw API `spdk_accel_task_size` was removed as it is no longer 931required. 932 933The accel_fw was updated to support compare, dualcast, crc32c. 934 935The accel_fw introduced batching support for all commands in all plug-ins. 936See docs for detailed information. 937 938### bdev 939 940A new API `spdk_bdev_abort` has been added to submit abort requests to abort all I/Os 941whose callback context match to the bdev on the given channel. 942 943### build 944 945The fio plugins now compile to `build/fio` and are named `spdk_bdev` and `spdk_nvme`. 946Existing fio configuration files will need to be updated. 947 948### dpdk 949 950Updated DPDK submodule to DPDK 20.05. 951 952### env 953 954Several new APIs have been added to provide greater flexibility in registering and 955accessing polled mode PCI drivers. See `env.h` for more details. 956 957### idxd 958 959The idxd library and plug-in module for the accel_fw were updated to support 960all accel_fw commands as well as batching. Batching is supported both 961through the library and the plug-in module. 962 963IDXD engine support for CRC-32C has been added. 964 965### ioat 966 967A new API `spdk_ioat_get_max_descriptors` was added. 968 969### nvme 970 971An `opts_size`element was added in the `spdk_nvme_ctrlr_opts` structure 972to solve the ABI compatiblity issue between different SPDK version. 973 974A new API `spdk_nvme_ctrlr_cmd_abort_ext` has been added to abort previously submitted 975commands whose callback argument match. 976 977Convenience functions, `spdk_nvme_print_command` and `spdk_nvme-print_completion` were added 978to the public API. 979 980A new function, `spdk_nvmf_cuse_update_namespaces`, updates the cuse representation of an NVMe 981controller. 982 983A new function `qpair_iterate_requests` has been added to the nvme transport interface. ALl 984implementations of the transport interface will have to implement that function. 985 986### nvmf 987 988The NVMe-oF target no longer supports connecting scheduling configuration and instead 989always uses what was previously called "transport" scheduling. 990 991`spdk_nvmf_tgt_accept` no longer exists. The accept process now occurs automatically after 992the creation of an nvmf target and queue pairs are assigned to poll groups by the underlying 993transport. Further, `spdk_nvmf_transport_ops` has changed such that the accept function 994pointer no longer takes a function pointer as an argument. Instead, transports should call 995`spdk_nvmf_tgt_new_qpair` whenever they previously would have called that callback. 996 997The NVMe-oF target now supports aborting any submitted NVM or Admin command. Previously, 998the NVMe-oF target could abort only Asynchronous Event Request commands. 999 1000### rdma 1001 1002A new `rdma` library has been added. It is an abstraction layer over different RDMA providers. 1003Two providers are available - verbs (used by default when RDMA is enabled or enabled explicitly 1004using --with-rdma=verbs) and mlx5 Direct Verbs aka DV (enabled by --with-rdma=mlx5_dv). 1005Using mlx5_dv requires libmlx5 installed on the system. 1006 1007### rpc 1008 1009Parameter `-p` or `--max-qpairs-per-ctrlr` of `nvmf_create_transport` RPC command accepted by the 1010rpc.py script is deprecated, new parameter `-m` or `--max-io-qpairs-per-ctrlr` was added. 1011 1012Added `sock_impl_get_options` and `sock_impl_set_options` RPC methods. 1013 1014Command line parameters `-r` and `--rpc-socket` will longer accept TCP ports. RPC server 1015must now be started on a Unix domain socket. Exposing RPC on the network, as well as providing 1016proper authentication (if needed) is now a responsibility of the user. 1017 1018The `bdev_set_options` RPC has a new option, `bdev_auto_examine` to control the auto examine function 1019of bdev modules. 1020 1021New RPCs `sock_impl_get_options` and `sock_impl_set_options` been added to expose new socket features. 1022See `sock` section for more details. 1023 1024### sock 1025 1026Added `spdk_sock_impl_get_opts` and `spdk_sock_impl_set_opts` functions to set/get socket layer configuration 1027options. Options can be set independently for each implementation. 1028 1029Added `recv_buf_size` and 'send_buf_size' socket layer options. They are used only in posix implementation. 1030 1031Added `uring` based socket implementation, the code is located in module/sock/uring. This feature is only 1032available in Linux which requires kernel version is greater than 5.4.3. Currently, our CI pool added the uring 1033based socket tests for iSCSI target and also the tests for SPDK NVMe-oF tcp transport. 1034 1035Added `enable_recv_pipe` socket layer option to allow disabling of double buffering on receive. 1036New option is used only in posix implementation. 1037 1038Added `enable_zerocopy_send` socket layer option to allow disabling of zero copy flow on send. 1039New option is used only in posix implementation. 1040 1041### util 1042 1043Some previously exposed CRC32 functions have been removed from the public API - 1044`spdk_crc32_update`, `spdk_crc32_table_init`, and the `spdk_crc32_table` struct. 1045 1046### vhost 1047 1048The function `spdk_vhost_blk_get_dev` has been removed. 1049 1050## v20.04: SPDK Top, IDXD, NVMe qpair groups 1051 1052IDXD engine support for compare has been added. 1053 1054### configuration 1055 1056Legacy INI style configuration for SPDK applications has been deprecated and will be 1057removed in future release. 1058Please switch to JSON-RPC configuration files and/or RPC driven run-time configuration. 1059 1060### copy 1061 1062The copy engine library, modules and public APIs have been renamed. Use of the word `copy` 1063has been replaced with the word `accel` short for accelerator in preparation for adding new 1064capabilities in the future. Additionally, APIs for what was previously called the `memcpy` 1065engine have been renamed to identify the engine as a software accelerator. 1066 1067### crypto 1068 1069Support for AES_XTS was added for the QAT polled mode driver (pmd). The create RPC 1070`bdev_crypto_create` has 2 new optional parameters: cipher and key2. Cipher can be either 1071AES_CBC (default) or AES_XTS. AES_XTS is only valid when using the QAT polled mode driver. 1072The key2 parameter is the second key required for AES_XTS. 1073 1074### event 1075 1076Reactors now accumulate CPU stats and they are retrieved by the RPC `framework_get_reactors`. 1077 1078### idxd 1079 1080IDXD support was added in the form of a low level library that can directly 1081utilize the IDXD offload engine as well as an accel plug-in module allowing for use 1082with the generic accel framework API. IDXD is the first in a family of offload 1083engines that share the same interface, specifically DSA is added here. More info 1084can be found here: https://01.org/blogs/2019/introducing-intel-data-streaming-accelerator 1085 1086Much of the implementation models IOAT, however the low level interfaces are very 1087different. The RPC to enable IDXD requires a configuration number as well. The 1088code includes two pre-defined configurations of IDXD groups/work queues/engines. A future 1089version will provide an interface to allow for setting of individual configuration 1090parameters. 1091 1092IDXD is not yet available so this feature should be considered experimental. It will 1093be built up with additional documentation as an ongoing activity. 1094 1095### iscsi 1096 1097The iSCSI target now creates a lightweight thread per poll group instead of assuming a pool 1098of lightweight threads already exist at start up time. A poll group is a collection of 1099unrelated iSCSI connections. Each poll group is only accessed from the associated 1100lightweight thread. 1101 1102### ftl 1103 1104Several changes have been made to the `spdk_ftl_conf`, `spdk_ftl_dev_init_ops`, and 1105`spdk_ftl_attrs` structs. Please see `include/spdk/ftl.h` for more details. 1106 1107### miscellaneous 1108 1109The `--json-ignore-init-errors` command line parameter has been added to ignore 1110initialization errors on JSON config load. 1111 1112The public header file io_channel.h has been removed. Please use thread.h which has the 1113exact same API. 1114 1115### nvme 1116 1117Exported internal nvme_ctrlr_cmd_security_receive/send() APIs as public APIs with "the spdk_" 1118prefix. 1119 1120Added `priority` field in `spdk_nvme_transport_id`, this field is used to specify the priority 1121of the NVMe-oF connection, and currently it is used for NVMe-oF tcp connection. 1122 1123A new poll group API has been added to allow for pooling of nvme qpairs across a single 1124entity which can be polled for completions. This new API consists of the `spdk_nvme_poll_group` 1125family of functions. As a result of this new API, all NVMe transports are expected to implement 1126several poll group related functions. 1127 1128A new flag, `create_only`, has been added to the `spdk_nvme_io_qpair_opts` structure. This flag 1129allows a user to call `spdk_nvme_ctrlr_get_default_io_qpair` without also connecting the qpair 1130within the context of that call. 1131 1132As a result of the `create_only` flag, two new API functions, `spdk_nvme_ctrlr_connect_io_qpair` 1133and `spdk_nvme_ctrlr_disconnect_io_qpair`, have been added to facilitate connecting newly created 1134qpairs (for example, after they have been added to a poll group) and disconnecting qpairs without 1135destroying them (for example to disconnect a qpair before migrating it to a new poll group and 1136reconnecting it). 1137 1138The functions `spdk_nvme_ctrlr_alloc_cmb_io_buffer` and `spdk_nvme_ctrlr_free_cmb_io_buffer` 1139have been changed to `spdk_nvme_ctrlr_map_cmb` and `spdk_nvme_ctrlr_unmap_cmb` respectively. 1140 1141An additional function, `spdk_nvme_ctrlr_reserve_cmb`, has been added to facilitate reserving 1142the entire size of the controller memory buffer for data transfer. 1143 1144### nvme_cuse 1145 1146`spdk_nvme_cuse_get_ctrlr_name` now takes two additional parameters, `char *name` which 1147stores the pointer to the controller name, and `size_t *size` which stores the length of 1148the name. The return type has also been changed from char * to int. 1149 1150`spdk_nvme_cuse_get_ns_name` now takes two additional parameters, `char *name` which 1151stores the pointer to the namespace name, and `size_t *size` which stores the length of 1152the name. The return type has also been changed from char * to int. 1153 1154### nvme_opal 1155 1156Several public OPAL structure definitions have been changed since the last release. 1157 1158### nvmf 1159 1160`spdk_nvmf_poll_group_destroy()` is now asynchronous and accepts a completion callback. 1161 1162The NVMe-oF target now creates a lightweight thread per poll group instead of assuming a pool 1163of lightweight threads already exist at start up time. A poll group is a collection of 1164unrelated NVMe-oF connections. Each poll group is only accessed from the associated 1165lightweight thread. 1166 1167A new struct, `spdk_nvmf_subsystem_listener`, has been added to encapsulate the subsystem specific 1168nature of a listener object. 1169 1170`spdk_nvmf_tgt_listen` no longer accepts a callback function or argument. It also returns an 1171int to indicate the status of the listen call. 1172 1173The execution of `spdk_nvme_poll_group_destroy` is now asynchronous and the function accepts 1174a cb_fn and cb_arg to call upon completion. 1175 1176The execution of `spdk_nvmf_subsystem_add_listener` is now asynchronous and the function accepts 1177a cb_fn and cb_arg to call upon completion. 1178 1179The `nvmf_transport.h` header has been made public to allow custom NVMe-oF transports to integrate 1180with NVMe-oF libraries without using internal APIs. 1181 1182### ocf 1183 1184Updated the OCF submodule to OCF v20.03 1185 1186New version of OCF comes with API changes and bug fixes 1187 1188### rpc 1189 1190A new RPC `thread_set_cpumask` has been added to set the cpumask of the thread 1191to the specified value. 1192 1193A new RPC `thread_get_pollers` has been added to retrieve pollers from SPDK threads. 1194 1195A new RPC `thread_get_io_channels` has been added to retrieve I/O channels from SPDK threads. 1196 1197A new RPC `bdev_rbd_resize` has been added to resize Ceph RBD bdevs. 1198 1199### sock 1200 1201The `spdk_sock_set_priority` function has been removed since the feature to set the sock priority 1202will be contained in two new functions, i.e., `spdk_sock_listen_ext` and `spdk_sock_connect_ext`. 1203Users may now specify the priority of the socket in the opts that they want to use. 1204 1205### spdk_top 1206 1207A new application, `spdk_top`, has been added which allows users to monitor resource consumption 1208by a running SPDK application. More information on this application can be found in 1209`app/spdk_top/README`. 1210 1211### thread 1212 1213A new function `spdk_thread_lib_init_ext` has been added, and the function 1214`spdk_thread_lib_init` has been deprecated. The user of `spdk_thread_lib_init_ext` is 1215expected to implement both functions `spdk_thread_op_fn` and `spdk_thread_op_supported_fn`. 1216`spdk_thread_op_supported_fn` is called to check whether the SPDK thread operation 1217is supported. `spdk_thread_op_fn` is called to execute the SPDK thread operation. 1218Current SPDK operation types are `SPDK_THREAD_OP_NEW` and `SPDK_THREAD_OP_RESCHED`. 1219The operation `SPDK_THREAD_OP_NEW` is called each time a new thread is created. 1220The operation `SPDK_THREAD_OP_RESCHED` is called when SPDK thread needs to be rescheduled. 1221 1222A unique ID has been added for each created SPDK thread, it is retrieved by a new function 1223`spdk_thread_get_id`, and the SPDK thread which has the specific ID is got by 1224a new function `spdk_thread_get_by_id`. 1225 1226A new function `spdk_thread_cpumask` has been added to set the current thread's cpumask 1227to the specified value. The function requires the operation `SPDK_THREAD_OP_RESCHED` 1228is supported. 1229 1230A new function `spdk_poller_register_named` has been added to set arbitrary name to the 1231created poller. If NULL, the name is set to the pointer of the poller function. 1232 1233The function `spdk_thread_poll` now measures run time per call correctly on multiple SPDK 1234threads configuration, and a new function `spdk_thread_get_last_tsc` has been added to use together. 1235 1236Voluntary termination of SPDK thread has been supported by refining the functions `spdk_thread_exit` 1237and `spdk_thread_poll`. 1238 1239### util 1240 1241New functions `spdk_sn32_lt` and `spdk_sn32_gt` have been added. They compare two sequence 1242numbers based on serial number arithmetic. 1243 1244### vhost 1245 1246Poll groups per session have been replaced by SPDK threads per vhost controller. 1247 1248### vmd 1249 1250A new function, `spdk_vmd_fini`, has been added. It releases all resources acquired by the VMD 1251library through the `spdk_vmd_init` call. 1252 1253## v20.01: Optimized thin provisioning, FTL bdev, VMD hot plug, FUSED support 1254 1255### bdev 1256 1257A new function, `spdk_bdev_set_timeout`, has been added to set per descriptor I/O timeouts. 1258 1259A new class of functions `spdk_bdev_compare*`, have been added to allow native bdev support 1260of block comparisons and compare-and-write. 1261 1262A new class of bdev events, `SPDK_BDEV_EVENT_MEDIA_MANAGEMENT`, has been added to allow bdevs 1263which expose raw media to alert all I/O channels of pending media management events. 1264 1265A new API was added `spdk_bdev_io_get_aux_buf` allowing the caller to request 1266an auxiliary buffer for its own private use. The API is used in the same manner that 1267`spdk_bdev_io_get_buf` is used and the length of the buffer is always the same as the 1268bdev_io primary buffer. 'spdk_bdev_io_put_aux_buf' frees the allocated auxiliary 1269buffer. 1270 1271### blobfs 1272 1273Added boolean return value for function spdk_fs_set_cache_size to indicate its operation result. 1274 1275Added `blobfs_set_cache_size` RPC method to set cache size for blobstore filesystem. 1276 1277### blobstore 1278 1279Added new `use_extent_table` option to `spdk_blob_opts` for creating blobs with Extent Table descriptor. 1280Using this metadata format, dramatically decreases number of writes required to persist each cluster allocation 1281for thin provisioned blobs. Extent Table descriptor is enabled by default. 1282See the [Blobstore Programmer's Guide](https://spdk.io/doc/blob.html#blob_pg_cluster_layout) for more details. 1283 1284### dpdk 1285 1286Updated DPDK submodule to DPDK 19.11. 1287 1288### env_dpdk 1289 1290`spdk_env_dpdk_post_init` now takes a boolean, `legacy_mem`, as an argument. 1291 1292A new function, `spdk_env_dpdk_dump_mem_stats`, prints information about the memory consumed by DPDK to a file specified by 1293the user. A new utility, `scripts/dpdk_mem_info.py`, wraps this function and prints the output in an easy to read way. 1294 1295### event 1296 1297The functions `spdk_reactor_enable_framework_monitor_context_switch()` and 1298`spdk_reactor_framework_monitor_context_switch_enabled()` have been changed to 1299`spdk_framework_enable_context_switch_monitor()` and 1300`spdk_framework_context_switch_monitor_enabled()`, respectively. 1301 1302### ftl 1303 1304All NVMe dependencies were removed from ftl library. 1305From now ftl library is using bdev_zone API. 1306bdev_ftl becomes virtual bdev. 1307 1308`ctrlr` and `trid` fields in `spdk_ftl_dev_init_opts` structure were replaced by 1309`base_bdev_desc`. 1310 1311`bdev_ftl_create` RPC method `trtype` and `traddr` parameters were replaced by `base_bdev` 1312parameter. 1313 1314`spdk_ftl_module_init` and `spdk_ftl_module_fini` functions were removed. 1315 1316`spdk_ftl_punit_range` and `ftl_module_init_opts` structures were removed. 1317 1318### isa-l 1319 1320Updated ISA-L submodule to commit f3993f5c0b6911 which includes implementation and 1321optimization for aarch64. 1322 1323Enabled ISA-L on aarch64 by default in addition to x86. 1324 1325### nvme 1326 1327`delayed_pcie_doorbell` parameter in `spdk_nvme_io_qpair_opts` was renamed to `delay_cmd_submit` 1328to allow reuse in other transports. 1329 1330Added RDMA WR batching to NVMf RDMA initiator. Send and receive WRs are chained together 1331and posted with a single call to ibv_post_send(receive) in the next call to qpair completion 1332processing function. Batching is controlled by 'delay_cmd_submit' qpair option. 1333 1334The NVMe-oF initiator now supports plugging out of tree NVMe-oF transports. In order 1335to facilitate this feature, several small API changes have been made: 1336 1337The `spdk_nvme_transport_id` struct now contains a trstring member used to identify the transport. 1338A new function, `spdk_nvme_transport_available_by_name`, has been added. 1339A function table, `spdk_nvme_transport_ops`, and macro, `SPDK_NVME_TRANSPORT_REGISTER`, have been added which 1340enable registering out of tree transports. 1341 1342A new function, `spdk_nvme_ns_supports_compare`, allows a user to check whether a given namespace supports the compare 1343operation. 1344 1345A new family of functions, `spdk_nvme_ns_compare*`, give the user access to submitting compare commands to NVMe namespaces. 1346 1347A new function, `spdk_nvme_ctrlr_cmd_get_log_page_ext`, gives users more granular control over the command dwords sent in 1348log page requests. 1349 1350### nvmf 1351 1352Add SockPriority option in [Transport] section, this can be used for NVMe-oF target 1353on TCP transport to set sock priority for the incomming TCP connections. 1354 1355The NVMe-oF target now supports plugging out of tree NVMe-oF transports. In order 1356to facilitate this feature, several small API changes have been made: 1357 1358The `spdk_nvme_transport_id` struct now contains a trstring member used to identify the transport. 1359`spdk_nvmf_tgt_get_transport`, `spdk_nvmf_transport_opts_init`, and `spdk_nvmf_transport_create` all have been 1360modified to take a string. 1361A function table, `spdk_nvmf_transport_ops`, and macro, `SPDK_NVMF_TRANSPORT_REGISTER`, have been added which 1362enable registering out of tree transports. 1363 1364Add `spdk_nvmf_tgt_stop_listen()` that can be used to stop listening for 1365incoming connections for specified target and trid. Listener is not stopped 1366implicitly upon destruction of a subsystem any more. 1367 1368A custom NVMe admin command handler has been added which allows the user to use the real drive 1369attributes from one of the target NVMe drives when reporting drive attributes to the initiator. 1370This handler can be enabled via the `nvmf_set_config` RPC. 1371Note: In a future version of SPDK, this handler will be enabled by default. 1372 1373The SPDK target and initiator both now include compare-and-write functionality with one caveat. If using the RDMA transport, 1374the target expects the initiator to send both the compare command and write command either with, or without inline data. The 1375SPDK initiator currently respects this requirement, but this note is included as a flag for other initiators attempting 1376compatibility with this version of SPDK. 1377 1378The `spdk_nvme_ctrlr_opts` struct has been extended with new field `transport_ack_timeout` which allows 1379to configure transport ACK timeout. Applicable for RDMA transport only. 1380 1381### rpc 1382 1383A new RPC, `bdev_zone_block_create`, enables creating an emulated zoned bdev on top of a standard block device. 1384 1385A new RPC, `bdev_ocssd_create`, enables creating an emulated zoned bdev on top of an Open Channel SSD. 1386 1387A new RPC, `blobfs_set_cache_size`, enables managing blobfs cache size. 1388 1389A new RPC, `env_dpdk_get_mem_stats`, has been added to facilitate reading DPDK related memory 1390consumption stats. Please see the env_dpdk section above for more details. 1391 1392A new RPC, `framework_get_reactors`, has been added to retrieve a list of all reactors. 1393 1394`bdev_ftl_create` now takes a `base_bdev` argument in lieu of `trtype`, `traddr`, and `punits`. 1395 1396`bdev_nvme_set_options` now allows users to disable I/O submission batching with the `-d` flag 1397 1398`bdev_nvme_cuse_register` now accepts a `name` parameter. 1399 1400`bdev_uring_create` now takes arguments for `bdev_name` and `block_size` 1401 1402`nvmf_set_config` now takes an argument to enable passthru of identify commands to base NVMe devices. 1403Please see the nvmf section above for more details. 1404 1405### scsi 1406 1407`spdk_scsi_lun_get_dif_ctx` now takes an additional argument of type `spdk_scsi_task`. 1408 1409### sock 1410 1411Added spdk_sock_writev_async for performing asynchronous writes to sockets. This call will 1412never return EAGAIN, instead queueing internally until the data has all been sent. This can 1413simplify many code flows that create pollers to continue attempting to flush writes 1414on sockets. 1415 1416Added `impl_name` parameter in spdk_sock_listen and spdk_sock_connect functions. Users may now 1417specify the sock layer implementation they'd prefer to use. Valid implementations are currently 1418"vpp" and "posix" and NULL, where NULL results in the previous behavior of the functions. 1419 1420### thread 1421 1422`spdk_thread_send_msg` now returns int indicating if the message was successfully 1423sent. 1424 1425A new function `spdk_thread_send_critical_msg`, has been added to support sending a single message from 1426a context that may be interrupted, e.g. a signal handler. 1427 1428Two new functions, `spdk_poller_pause`, and `spdk_poller_resume`, have been added to give greater control 1429of pollers to the application owner. 1430 1431### util 1432 1433`spdk_pipe`, a new utility for buffering data from sockets or files for parsing 1434has been added. The public API is available at `include/spdk/pipe.h`. 1435 1436### bdev 1437 1438Added spdk_bdev_io_get_nvme_fused_status function for translating bdev_io status to NVMe status 1439code for fused compare-and-write operation. 1440 1441Added spdk_bdev_get_acwu function for getting block device atomic compare and write unit size. 1442 1443## v19.10: Zoned bdev API, Opal bdev, NVMe character devices 1444 1445### rpc 1446 1447Many of SPDK's RPCs were renamed to be more consistent and intuitive in this release. 1448The old names will continue to function, but will display a deprecation warning. 1449 1450Added optional parameters '--arbitration-burst' and '--low/medium/high-priority-weight' to 1451'bdev_nvme_set_options' RPC method. 1452 1453Added optional parameter '--md-size' to 'construct_null_bdev' RPC method. 1454 1455Added optional parameters '--dif-type' and '--dif-is-head-of-md' to 'construct_null_bdev' 1456RPC method. 1457 1458Added `blobfs_detect` RPC method to detect whether a blobfs exists on given bdev. 1459 1460Added `blobfs_create` RPC method to build blobfs on given bdev. 1461 1462Added `blobfs_mount` RPC method to mount blobfs on given bdev to a host path by FUSE. 1463Then on the host path, user can directly do some file operations which will be mapped 1464to blobfs. 1465 1466### bdev 1467 1468Added new parameter `cdw0` to `spdk_bdev_io_complete_nvme_status()` and 1469`spdk_bdev_io_get_nvme_status()` that allows setting/getting 1470the NVMe completion queue DW0 entry. This allows vendor specific IO commands 1471to return commmand specific completion info back to the initiator. 1472 1473Added `spdk_bdev_get_write_unit_size()` function for retrieving required number 1474of logical blocks for write operation. 1475 1476New zone-related fields were added to the result of the `get_bdevs` RPC call: 1477 1478- `zoned`: indicates whether the device is zoned or a regular 1479 block device 1480- `zone_size`: number of blocks in a single zone 1481- `max_open_zones`: maximum number of open zones 1482- `optimal_open_zones`: optimal number of open zones 1483 1484The `zoned` field is a boolean and is always present, while the rest is only available for zoned 1485bdevs. 1486 1487A new `spdk_bdev_open_ext` function has been added and `spdk_bdev_open` function has been deprecated. 1488The new open function introduces requirement to provide callback function that will be called by 1489asynchronous event such as bdev removal. `spdk_bdev_open_ext` function takes bdev name as 1490an argument instead of bdev structure to avoid a race condition that can happen when the bdev 1491is being removed between a call to get its structure based on a name and actually openning it. 1492 1493New 'resize' event has been added to notify about change of block count property of block device. 1494Event is delivered only if block device was opened with `spdk_bdev_open_ext` function. 1495 1496### bdev zone 1497 1498Added new public header for zoned bdev. Zoned bdev is an extension 1499of the bdev interface. 1500 1501`spdk_bdev_get_zone_size()`, `spdk_bdev_get_max_open_zones()`, `spdk_bdev_get_optimal_open_zones()` 1502APIs were added for retrieving zoned device information. 1503`spdk_bdev_get_zone_info()` API was added for retrieving information about zones in zoned 1504device. 1505Added `spdk_bdev_zone_management()` API for changing zone state. 1506`spdk_bdev_zone_append()` and `spdk_bdev_zone_append_with_md()` APIs were added for 1507appending data to a zone. 1508Added `spdk_bdev_io_get_append location()` function for retrieving append location for I/O. 1509Added `spdk_bdev_is_zoned()` function for checking if bdev supports zoned namespace semantics. 1510 1511### bdev opal 1512 1513EXPERIMENTAL: A new opal bdev has been added to support management of 1514NVMe self-encrypting drives through the Opal specification. Users can 1515create opal bdevs from an NVMe namespace bdev, if the controller 1516containing that namespace supports Opal. Currently this is only 1517supported for namespace ID=1. The following RPCs have been added to 1518support Opal: `bdev_nvme_opal_init`, `bdev_nvme_opal_revert`, 1519`bdev_opal_create`, `bdev_opal_delete`, `bdev_opal_get_info`, 1520`bdev_opal_new_user`, `bdev_opal_set_lock_state`. 1521It does not yet support recreating the opal bdevs after application restart. 1522This bdev module should be considered very experimental, and the RPCs may 1523change significantly in future releases. 1524 1525### delay bdev 1526 1527The `bdev_delay_update_latency` has been added to allow users to update 1528a latency value for a given delay bdev. 1529 1530### compress bdev 1531 1532A new RPC `bdev_compress_get_orphans` has been added to list compress bdevs 1533that were not loaded due to a missing pm metadata file. In this state they 1534can only be deleted. 1535 1536### null bdev 1537 1538Metadata support has been added to Null bdev module. 1539 1540Protection information support has been added to Null bdev module. 1541 1542### nvme 1543 1544Added `no_shn_notification` to NVMe controller initialization options, users can enable 1545it for NVMe controllers. When the option is enabled, the controller will not do the 1546shutdown process and just disable the controller, users can start their application 1547later again to initialize the controller to the ready state. 1548 1549A controller flag `SPDK_NVME_CTRLR_WRR_SUPPORTED` was added to indicate the controller 1550can support weighted round robin arbitration feature with submission queue. 1551 1552Added `arbitration_burst` option for arbitration feature, and added three 1553`low/medium/high_priority_weight` options for weighted round robin arbitration. 1554 1555Added `spdk_nvme_ns_cmd_write_uncorrectable`. 1556 1557Added new error handling and reporting functionality. This includes several 1558new API functions to facilitate applications recovering when a qpair or 1559controller fails. 1560 1561`spdk_nvme_ctrlr_reconnect_io_qpair` - Reconnects a failed I/O qpair. 1562`spdk_nvme_ctrlr_set_trid` - Sets the trid of an existing controller. Can be used to 1563change the trid for failover cases. 1564`spdk_nvme_ctrlr_is_failed` - Returns the failed state of a controller. 1565`spdk_nvme_ctrlr_fail` - Forces a controller into a failed state. 1566 1567Modified the return behavior of several API functions to better indicate to 1568applications when a qpair is failed. This list of functions includes: 1569 1570`spdk_nvme_qpair_process_completions` 1571`spdk_nvme_ns_cmd_*` 1572`spdk_nvme_ctrlr_process_admin_completions` 1573`spdk_nvme_ctrlr_cmd_*` 1574 1575These functions now return -ENXIO when the qpair or controller on which they 1576operate is failed. 1577 1578EXPERIMENTAL: Added NVMe character device support to allow to create NVMe device nodes in Linux 1579kernel for controller as well as for namespace and process ioctl requests as usual 1580from linux environment. 1581 1582### nvmf 1583 1584The `spdk_nvmf_tgt_create` function now accepts an object of type `spdk_nvmf_target_opts` 1585as its only parameter. This new structure contains the max_subsystems parameter previously 1586passed into that function. 1587 1588A new public API function `spdk_nvmf_get_tgt` has been added which allows users to 1589retrieve a pointer to an `spdk_nvmf_tgt` object by supplying its name. In the special 1590case where an RPC or application only creates a single target, this function can accept 1591a null name parameter and will return the only available target. 1592 1593The majority of the NVMe-oF RPCs now accept an optional tgt_name parameter. This will 1594allow those RPCs to work with applications that create more than one target. 1595 1596Three new NVMe-oF RPCs have been added `nvmf_create_target`, `nvmf_delete_target`, and 1597`nvmf_get_targets`. These new RPCs provide a basic interface for managing multiple target 1598objects. In SPDK the target object defines a unique discovery service. As of this release, 1599these RPCs are not intended to be used with the in-tree SPDK target applications, spdk_tgt and 1600nvmf_tgt, which use a single, global target structure. As such, they are not included in scripts/rpc.py 1601 1602Three new header functions have also been added to help deal with multiple targets. 1603`spdk_nvmf_tgt_get_name` takes a target pointer as an argument and returns its human readable name. 1604`spdk_nvmf_get_first_target` takes no arguments and returns the first target in the global list. 1605`spdk_nvmf_get_next_tgt` takes a target pointer as an argument and returns the next one in the global list. 1606 1607The `spdk_nvmf_tgt_accept` takes additional argument allowing to pass arbitrary context 1608information to the `new_qpair` callback. This will simplify the code when having multiple 1609nvmf targets or when retrieving the context information from globals is not suitable. 1610 1611### blobstore 1612 1613A new `spdk_bdev_create_bs_dev_from_desc` function has been added and `spdk_bdev_create_bs_dev` 1614function has been deprecated. 1615The new create function can cowork with `spdk_bdev_open_ext` function, which provides callback 1616function that will be called by asynchronous event such as bdev removal. 1617 1618### blobfs_bdev 1619 1620A new blobfs module `bdev` has been added to simplify the operations of blobfs on bdev. 1621 1622Function `spdk_blobfs_bdev_detect` is added to detect whether blobfs exists on the given block device. 1623 1624Function `spdk_blobfs_bdev_create` is added to create a blobfs on the given block device. 1625 1626Function `spdk_blobfs_bdev_mount` is added to mount a blobfs on the given block device to 1627a host path by FUSE. Then, a new thread is created dedicatedly for one mountpoint to handle 1628FUSE request by blobfs API. 1629 1630### build 1631 1632Option to build FUSE components into blobfs_bdev module for mounting a blobfs filesystem. 1633It requires the installation of libfuse3. By default, it is disabled. And it will be 1634enabled if run `./configure` with `--with-fuse` option. 1635 1636### iSCSI 1637 1638Portals may no longer be associated with a cpumask. The scheduling of 1639connections is moving to a more dynamic model. 1640 1641An new RPC `iscsi_portal_group_set_auth` has been added to set CHAP authentication 1642for discovery sessions specific for the existing iSCSI portal group. This RPC overwrites 1643the setting by the global parameters for the iSCSI portal group. 1644 1645### socket 1646 1647Added `spdk_sock_is_connected` to check whether the socket is currently connected. 1648`spdk_sock_group_poll` now returns number of events on success. 1649 1650### env 1651 1652Added `spdk_pci_device_unclaim()` function to cleanup pci claim file. 1653 1654### event 1655 1656`framework_start_init` RPC no longer stops the application on error during 1657initialization. 1658 1659### DPDK 1660 1661Updated DPDK submodule to DPDK 19.08. 1662 1663### ocf 1664 1665Updated OCF submodule to OCF v19.06 1666 1667Along with update, new cache mode 'write only' was added. 1668 1669New cache modes added to use via RPC, wi - write invalidate and wa - write around. 1670 1671New version of OCF provides fully asynchronous management API. 1672 1673## v19.07: NVMe-oF FC Transport, VMD, NVMe-oF Persistent reservations, Bdev I/O with separate metadata 1674 1675### ftl 1676 1677EXPERIMENTAL: Added ability to mirror writes to persistent write buffer cache 1678to allow for recovery from dirty shutdown event. 1679 1680Added handling of Asynchronous Nand Management Events (ANM). 1681 1682### vmd 1683 1684EXPERIMENTAL: Added Intel Volume Management Device (VMD) driver. VMD is an integrated 1685controller inside the CPU PCIe root complex. It enables virtual HBAs for the connected 1686NVMe SSDs. `spdk_vmd_init()` enumerates NVMe SSDs behind VMD device and hook them into 1687SPDK PCI subsystem. `spdk_nvme_probe()` or `spdk_nvme_connect()` can be used to connect 1688NVMe driver to the device located at the given transport ID. 1689 1690To obtain transport ID of NVMe SSD behind VMD `spdk_lspci` can be used. 1691 1692Current implementation does not support hotplug. 1693 1694### blobfs 1695 1696Blobfs file asynchronous operations were added to public APIs. 1697 1698### util 1699 1700A new file API `spdk_posix_file_load` was added to load file content into a data buffer. 1701 1702New APIs `spdk_dif_ctx_set_data_offset`, `spdk_dif_verify_stream`, 1703`spdk_dif_get_range_with_md`, `spdk_dif_get_length_with_md` have been added, 1704and existing APIs `spdk_dif_ctx_init`, `spdk_dif_set_md_interleave_iovs`, and 1705`spdk_dif_generate_stream` have been refined to insert or strip DIF by stream 1706fasion with any alignment. 1707 1708New APIs `spdk_dif_ctx_set_remapped_init_ref_tag`, `spdk_dif_remap_ref_tag`, 1709and `spdk_dix_remap_ref_tag` have been added to remap DIF reference tag. 1710 1711New APIs `spdk_dif_update_crc32c` and `spdk_dif_update_crc32c_stream` have been 1712added to compute CRC-32C checksum for extended LBA payload. 1713 1714Bdevperf and bdevio applications now support starting tests with application specific 1715RPCs. Please see helper Python scripts in their respective directories. 1716This is a move towards simpler RPC-only configuration for all main 1717and auxiliary applications. 1718 1719Legacy INI style configuration for SPDK applications will become deprecated in SPDK 19.10, 1720and removed in SPDK 20.01. Please consider moving to JSON-RPC configuration files and/or 1721RPC driven run-time configuration. 1722 1723### nvmf 1724 1725EXPERIMENTAL: A Fibre Channel transport that supports Broadcom HBAs has been 1726added. This depends on the FC HBA driver at 1727https://github.com/ecdufcdrvr/bcmufctdrvr. See [the documentation](https://spdk.io/doc/nvmf.html#nvmf_fc_transport) 1728for more information. 1729 1730Persistent reservation emulation has been added to the NVMe-oF target. Persistent reservation 1731state is stored in a JSON file on the local filesystem between target restart. To support this, 1732an optional parameter to the RPC method `nvmf_subsystem_add_ns` called `--ptpl-file` was added. 1733This allows the user to specify which file to store the persistent reservation state in. Note 1734that this is done per namespace. 1735 1736The c2h success optimization under which a command capsule response is not sent 1737for reads is turned on by default. A config knob was added to allow disabling 1738the optimization. This will mostly be used for integration testing with 5.0.x kernels 1739while some compatibility fixes make their way down the pipeline for 5.1.x kernels. 1740 1741The sock priority setting of the TCP connection owned by the tcp transport is added. It is 1742used to optimize the TCP connection performance under designated traffic classes. And the 1743priority is used to differeniate the sock priority between SPDK NVMe-oF TCP target application 1744and other TCP based applications. 1745 1746Shared receive queue can now be disabled even for NICs that support it using the 1747`nvmf_create_transport` RPC method parameter `no_srq`. The actual use of a shared 1748receive queue is predicated on hardware support when this flag is not used. 1749 1750spdk_nvmf_get_optimal_poll_group was added, which is used to return the optimal 1751poll group for the qpair. And `ConnectionScheduler` configuration is added into the 1752[Nvmf] section in etc/spdk/nvmf.conf.in to demonstrate how to configure the connection 1753scheduling strategy among different spdk threads. 1754 1755Added infrastructure to retrieve global and per poll group NVMf statistics. 1756 1757DIF strip and insert is now supported for TCP transport. When it is enabled, DIF 1758setting is not exposed to the NVMe-oF initiator, and DIF is attached into data 1759for write I/O and stripped from data for read I/O. 1760 1761Added a field `dif_insert_or_strip` to struct spdk_nvmf_transport_opts, and 1762updated the related rpc function nvmf_create_transport to make this 1763configurable parameter available to users. The `dif_insert_or_strip` is relevant 1764for TCP transport for now and used to configure the DIF strip and insert. 1765 1766Added infrastructure to retrieve NVMf transport statistics. 1767 1768### notify 1769 1770The function `spdk_notify_get_types()` and `spdk_notify_get_events()` were 1771renamed to `spdk_notify_foreach_type()` and `spdk_notify_foreach_event()`, 1772respectively. And update type name of callback accordingly. 1773 1774### bdev 1775 1776The format of the data returned by the get_bdevs_iostat RPC has changed to 1777make it easier to parse. It now returns an object with a "ticks" object 1778and "bdevs" array with the per-bdev statistics. 1779 1780A new bdev module `delay` has been added which simulates a drive latency when placed 1781on top of a Null bdev. This module is intended only for testing and can be created using 1782the new RPC `bdev_delay_create`. That RPC takes the name of the underlying bdev as well 1783as average and p99 latency arguments for both read and write operations. Average latency is 1784defined as a value close to what you would expect a perf tool such as FIO to report back as 1785the mean latency of all I/O submitted to the drive. p99 latency is defined as the value one 1786would expect the drive to see the slowest 1% of I/O report. For underlying drives with already 1787significant latency, the latency values provided to the drive will be additive. This should be 1788taken into account if trying to achieve an artificial latency on top of an nvme drive or aio device. 1789 1790DIF reference tag remapping is now supported for partition type virtual bdev 1791modules. When using partition type virtual bdevs, block address space is 1792remapped during I/O processing and DIF reference tag is remapped accordingly. 1793 1794Added spdk_bdev_*_with_md() functions allowing for IO with metadata being transferred in 1795separate buffer. To check support for separatate metadata, use spdk_bdev_is_md_separate(). 1796 1797All bdevs now have a UUID. For devices whose backing hardware does not provide a UUID, 1798one is automatically generated. Across runs of SPDK, bdevs whose UUID is automatically 1799generated may change. 1800 1801A new virtual bdev module `compress` has been added to provide compression services on top of 1802a thinly provisioned logical volume. See documentation for complete details. 1803 1804### nvme 1805 1806Added an optional parameter `--io-queue-requests` to RPC `set_bdev_nvme_options`, which 1807can be used to change the number of requests allocated for one NVMe I/O queue. For 1808very big I/O size, e.g. 128MiB, with this option user will not get an error due to 1809limited requests in NVMe driver layer. 1810 1811Added spdk_nvme_ctrlr_get_transport_id() to get the transport ID from a 1812previously attached controller. 1813 1814Nvme Opal library spdk_opal_cmd deprecated. Adding seperate command APIs. 1815NVMe Opal library add support for activating locking SP which will make the transaction 1816from "Manufactured-Inactive" state to "Manufactured" state. Upon successfully invoking 1817of this method, lock and unlock features will be enabled. 1818NVMe Opal library add support for locking/unlocking range and list locking range info. 1819NVMe opal library add support for multiuser. Admin can enable user and add user to specific 1820locking range and the user can lock/unlock his range. 1821 1822Added spdk_nvme_ctrlr_io_cmd_raw_no_payload_build() allowing a caller to pass 1823a completely formed command to an NVMe submission queue (buffer addresses and all). 1824This is supported on the PCIe transport only. 1825 1826Added spdk_nvme_get_ctrlr_registers() to return a pointer to the virtual address 1827of the NVMe controller registers. This is supported on the PCIe transport only. 1828 1829Added additional options to the spdk_nvme_ctrlr_alloc_qpair() option parameter 1830structure to allow caller to override the virtual and optionally physical address 1831of the submission and completion queue pair to be created. This is supported on 1832the PCIe transport only. 1833 1834Added `disable_error_logging` to struct spdk_nvme_ctrlr_opts, which disables 1835logging of failed requests. By default logging is enabled. 1836 1837Added spdk_nvme_qpair_print_command(), spdk_nvme_qpair_print_completion() and 1838spdk_nvme_cpl_get_status_string(). Allowing for easier display of error messages. 1839 1840Added support for NVMe Sanitize command. 1841 1842### env 1843 1844The parameter `free_space` has been added to spdk_ring_enqueue() to wait when 1845the ring is almost full and resume when there is enough space available in 1846the ring. 1847 1848A new API `spdk_mempool_lookup` has been added to lookup the memory pool created 1849by the primary process. 1850 1851Added spdk_pci_get_first_device() and spdk_pci_get_next_device() to allow 1852iterating over PCI devices detected by SPDK. Because of this, all SPDK APIs 1853to attach/detach PCI devices are no longer thread safe. They are now meant to 1854be called from only a single thread only, the same only that called spdk_env_init(). 1855This applies to the newly added APIs as well. 1856 1857### vpp 1858 1859SPDK now supports VPP version 19.04.2, up from VPP 18.01. 1860 1861VPP socket abstraction now uses VPP Session API, instead of VLC (VPP Communications Library). 1862This allows for better control over sessions and queues. 1863Please see VPP documentation for more details: 1864[VPP Host Stack](https://wiki.fd.io/view/VPP/HostStack) 1865 1866### sock 1867 1868Add spdk_sock_get_optimal_sock_group(), which returns the optimal sock group for 1869this socket. When a socket is created, it is often assigned to a sock group using 1870spdk_sock_group_add_sock so that a set of sockets can be polled more efficiently. 1871For some network devices, it is optimal to assign particular sockets to specific 1872sock groups. This API is intended to provide the user with that information. 1873 1874spdk_sock_group_get_ctx() was added to return the context of the spdk_sock_group. 1875spdk_sock_group_create() is updated to allow input the user provided ctx. 1876spdk_sock_set_priority() is added to set the priority of the socket. 1877 1878### rpc 1879 1880Added thread_get_stats RPC method to retrieve existing statistics. 1881 1882Added nvmf_get_stats RPC method to retrieve NVMf susbsystem statistics. 1883 1884Response buffers for RPC requests are now always pre-allocated, which implies 1885that all spdk_jsonrpc_begin_result() calls always succeed and return a valid 1886buffer for JSON response. RPC calls no longer need to check if the buffer is 1887non-NULL. 1888 1889Added SPDK_RPC_REGISTER_ALIAS_DEPRECATED to help with deprecation process when 1890renaming existing RPC. First time a deprecated alias is used, it will print 1891a warning message. 1892 1893RPC `get_rpc_methods` was renamed `rpc_get_methods`. The old name is still usable, 1894but is now deprecated. 1895 1896### blobstore 1897 1898A snapshot can now be deleted if there is only a single clone on top of it. 1899 1900### build 1901 1902Preliminary support for cross compilation is now available. Targeting an older 1903CPU on the same architecture using your native compiler can be accomplished by 1904using the `--target-arch` option to `configure` as follows: 1905 1906~~~bash 1907./configure --target-arch=broadwell 1908~~~ 1909 1910Additionally, some support for cross-compiling to other architectures has been 1911added via the `--cross-prefix` argument to `configure`. To cross-compile, set CC 1912and CXX to the cross compilers, then run configure as follows: 1913 1914~~~bash 1915./configure --target-arch=aarm64 --cross-prefix=aarch64-linux-gnu 1916~~~ 1917 1918### vhost 1919 1920A security vulnerability has been identified and fixed in SPDK Vhost-SCSI target. 1921A malicious client (e.g. a virtual machine) could send a carefully prepared, 1922invalid I/O request to crash the entire SPDK process. All users of SPDK Vhost-SCSI 1923target are strongly recommended to update. All SPDK versions < 19.07 are affected. 1924 1925By default, SPDK will now rely on upstream DPDK's rte_vhost instead of its fork 1926located inside SPDK repo. The internal fork is still kept around to support older 1927DPDK versions, but is considered legacy and will be eventually removed. 1928 1929`configure` will now automatically use the upstream rte_vhost if the used DPDK 1930version is >= 19.05. 1931 1932spdk_vhost_init() is now asynchronous and accepts a completion callback. 1933 1934### iscsi target 1935 1936A security vulnerability has been identified and fixed in SPDK iSCSI target. 1937A malicious client (e.g. an iSCSI initiator) could send a carefully prepared, 1938invalid I/O request to crash the entire SPDK process. All users of SPDK iSCSI 1939target are strongly recommended to update. All SPDK versions < 19.07 are affected. 1940 1941### thread 1942 1943Exposed spdk_set_thread() in order for applications to associate 1944with SPDK thread when necessary. 1945 1946Added spdk_thread_destroy() to allow framework polling the thread to 1947release resources associated with that thread. 1948 1949## v19.04: Compression bdev, Notification library, NVMe Opal support 1950 1951### nvme 1952 1953Added asynchronous probe support. New APIs spdk_nvme_probe_async(), 1954spdk_nvme_connect_async() and spdk_nvme_probe_poll_async() were added to enable 1955this feature. spdk_nvme_probe_async() and spdk_nvme_connect_async() return a 1956context associated with the specified controllers. Users then call 1957spdk_nvme_probe_poll_async() until it returns 0, indicating that the operation 1958completed. 1959 1960A new qpair creation option, delay_pcie_doorbell, was added. This can be passed 1961to spdk_nvme_alloc_io_qpair(). This makes the I/O submission functions, such as 1962spdk_nvme_ns_writev(), skip ringing the submission queue doorbell. Instead the 1963doorbell will be rung as necessary inside spdk_nvme_qpair_process_completions(). 1964This can result in significantly fewer MMIO writes to the doorbell register 1965under heavy load, greatly improving performance. 1966 1967spdk_nvme_ctrlr_get_regs_cmbsz() was added to report the size of the controller 1968memory buffer, if available. 1969 1970spdk_nvme_ctrlr_get_flags() was added to return controller feature 1971flags. Two flags are currently tracked: 1972SPDK_NVME_CTRLR_SGL_SUPPORTED 1973SPDK_NVME_CTRLR_SECURITY_SEND_RECV_SUPPORTED 1974 1975The NVMe hotplug poller is now able to detach devices hot-removed from the system 1976via `/sys/bus/pci/devices/<bdf>/remove` and `/sys/bus/pci/devices/<bdf>/driver/unbind`. 1977 1978Opal support was added for scan, take ownership, revert TPer, and dumping device 1979info. The nvme_manage tool can be used to perform these operations. The public 1980API functions are spdk_nvme_ctrlr_security_receive() and 1981spdk_nvme_ctrlr_security_send(). This module should be considered experimental 1982pending additional features and tests. 1983 1984The NVMe-oF initiator is now able to transfer IO whose size is more than 128KiB 1985if the corresponding NVMe-oF target allows. 1986 1987### raid 1988 1989Added new strip_size_kb rpc param on create to replace the more ambiguous 1990strip_size. The strip_size rpc param is deprecated. 1991 1992Changed the raid bdev product_name from "Pooled Device" to "Raid Volume" 1993 1994### thread 1995 1996Added spdk_thread_has_pollers() function to verify if there are any registered 1997pollers to be run on the thread. Added spdk_thread_is_idle() function to check 1998if there are any scheduled operations to be performed on the thread at given 1999time. 2000 2001spdk_thread_create() now takes an optional CPU affinity mask that is passed to 2002the scheduler. 2003 2004spdk_thread_lib_init() now takes an optional context size. For each thread 2005created, an additional region of memory of this size will be allocated. A 2006pointer to this region of memory can be obtained by calling 2007spdk_thread_get_ctx(). The inverse operation is also available via 2008spdk_thread_get_from_ctx(). 2009 2010spdk_thread_poll() now optionally accepts the current time, in ticks. This can 2011be used to avoid some calls to spdk_get_ticks() internally. 2012 2013spdk_app_start() now only accepts a single context argument. 2014 2015### bdev 2016 2017An new API `spdk_bdev_get_data_block_size` has been added to get size of data 2018block except for metadata. 2019 2020spdk_vbdev_register() has been deprecated. spdk_bdev_register() should be used 2021instead. 2022 2023A mechanism for acquiring and releasing data buffers from bdev modules, used 2024to perform zero copy operations, was added. 2025 2026New APIs spdk_bdev_get_md_size(), spdk_bdev_is_md_interleaved(), spdk_bdev_get_dif_type(), 2027spdk_bdev_is_dif_head_of_md(), and spdk_bdev_is_dif_check_enabled() have been 2028added to get metadata and DIF settings. 2029 2030Bdevs claimed by the `examine_config` callback will be now further examined in the 2031`examine_disk` callback. 2032 2033spdk_bdev_io_get_io_channel() was added as a convenient way to get an io_channel 2034from a bdev_io. 2035 2036### ocf 2037 2038Updated OCF submodule to OCF v19.3.2 2039 2040Added support for many-to-one configuration for OCF bdev. 2041Multiple core devices can now be cached on single cache device. 2042 2043Added persistent metadata support, allowing to restore cache state after shutdown. 2044During start of SPDK application, the devices are examined and if OCF metadata 2045is present - appropriate OCF bdevs will be recreated. 2046 2047Added Write-Back mode support. In this mode, data is first written to 2048caching device and periodically synchronized to the core devices. 2049Dirty data is saved as persistent metadata on cache device, 2050allowing for safe restore during application restart. 2051For more details please see OCF documentation: 2052[OpenCAS cache configuration](https://open-cas.github.io/cache_configuration.html) 2053 2054### NVMe-oF Target 2055 2056Support for per-device shared receive queues in the RDMA transport has been added. 2057It is enabled by default for any device that supports it. 2058 2059The size of a shared receive queue is defined by transport configuration file parameter 2060`MaxSRQDepth` and `nvmf_create_transport` RPC method parameter `max_srq_depth`. 2061Default size is 4096. 2062 2063Add model number as parameter to construct_nvmf_subsystem (-d option), 2064rather than using hardcoded define. 2065 2066DIF passthrough feature has been added. DIF setting of the allocated bdevs is 2067exposed to the NVMe-oF initiator and data with DIF from the NVMe-oF initiator is 2068passed through to the allocated bdevs. 2069 2070### env 2071 2072The `phys_addr` parameter in spdk_malloc() and spdk_zmalloc() has been deprecated. 2073For retrieving physical addresses, spdk_vtophys() should be used instead. 2074 2075spdk_realloc() has been added to reallocate DMA/shared memory. 2076 2077spdk_pci_device_is_removed() has been added to let the upper-layer SPDK drivers know 2078that device has a pending external hotremove request. 2079 2080spdk_env_fini() and spdk_env_dpdk_post_fini() were added to release any resources 2081allocated by spdk_env_init() or spdk_env_dpdk_post_init() respectively. It is expected 2082that common usage of those functions is to call them just before terminating the process. 2083 2084Added spdk_iommu_is_enabled() to report if SPDK application is using IOMMU for DMA. 2085 2086### DPDK 2087 2088Dropped support for DPDK 17.07 and earlier, which SPDK won't even compile with right now. 2089 2090Updated DPDK submodule to DPDK 19.02. 2091 2092### rpc 2093 2094New `get_spdk_version` RPC method is introduced to get version info of the running SPDK application. 2095 2096The `start_nbd_disk` RPC method now take nbd_device as an optional parameter. If nbd_device 2097is specified, use that specified nbd device. If it's not specified, pick available one. 2098 2099### iSCSI target 2100 2101DIF strip and insert is now supported. DIF settings are not exposed to the iSCSI initiator. 2102DIF is attached into data for write I/O and stripped from data for read I/O. 2103 2104### vhost 2105 2106Added experimental support for running with the external, upstream rte_vhost library. 2107This can be enabled by configuring SPDK with an `--without-internal-vhost-lib` flag. 2108The minimum supported rte_vhost version (DPDK version) is 19.05-rc1. 2109 2110As a result of fuzz testing, a lot of data races in vhost-scsi LUN hotplug path were identified and 2111fixed. Those data races could have potentially resulted in SPDK crashes, RPC hangs, or memory leaks 2112if Vhost-SCSI LUN hotplug RPCs were executed while connected VMs were in the middle of restarting. 2113 2114The SCSI target id in `add_vhost_scsi_lun` RPC is now optional. If `-1` is passed, the first 2115unoccupied target id will be used. 2116 2117### AIO 2118 2119AIO bdev module can now reap I/O completions directly from userspace, significantly improving 2120the overall performance. 2121 2122### blobfs 2123 2124Synchronous IO operations no longer use spdk_io_channel, but instead use 2125spdk_fs_thread_ctx. The behavior is otherwise identical. 2126 2127### OCF 2128 2129Added support for caching multiple bdevs using a single bdev as a cache. 2130 2131### notify 2132 2133Added the notify library that provides a high performance local event bus 2134between libraries. Example usage was added to bdev module, which reports 2135notifications for added and removed bdevs via RPC. 2136 2137### sock 2138 2139Added new API spdk_sock_readv() to the sock library for performing vectored 2140reads. 2141 2142### event 2143 2144The function spdk_subsystem_init() no longer requires spdk_event as an argument. 2145 2146Changed API of spdk_subsystem_config_json() to no longer be asynchronous. 2147 2148### io_uring 2149 2150A bdev module that issues I/O to kernel block devices using the new io_uring Linux kernel 2151API was added. This module requires liburing. 2152 2153### build 2154 2155Options to easily compile with profile guided optimization have been added to 2156`configure`. To use profile guided optimization with SPDK, run 2157`./configure --with-pgo-capture`, build SPDK, then run a workload of your 2158choosing. Then, simply run `./configure --with-pgo-enable` and recompile to 2159build using the generated profile data. Profile guided optimization can yield 2160very large performance improvements, especially on GCC 8 and clang 7. This may 2161be combined with link time optimization which has been available under the 2162`--enable-lto` configure option for several releases. 2163 2164### compression bdev/reduce library 2165 2166Added "reduce" block compression scheme based on using SSDs for storing 2167compressed blocks of storage and presistent memory for metadata. Please see 2168[compression](https://spdk.io/doc/bdev.html) for more details. 2169 2170## v19.01: NVMe-oF TCP/IP Transport, Open Channel SSD Flash Translation Layer, Caching bdev based on OCF, ISA-L Support, DIF/DIX library 2171 2172### ocf bdev 2173 2174New virtual bdev module based on [Open CAS Framework](https://open-cas.github.io/) has been added. 2175This module allows for the use of one bdev to act as a high performance cache in front of another bdev. 2176Please see [documentation](https://spdk.io/doc/bdev.html#bdev_config_cas) for more details. 2177Only write through mode is currently supported and this feature is considered experimental. 2178 2179### event framework 2180 2181For `spdk_app_parse_args`, add return value to the callback which parses application 2182specific command line parameters to protect SPDK applications from crashing by invalid 2183values from user input. 2184 2185By default, all SPDK applications will now reserve all hugepages at runtime. The pre-reserved 2186memory size can be still set with `-s` or `--mem-size` option, although the default value 2187was reduced down to 0. 2188 2189A custom hugetlbfs directory can now be specified via spdk_app_opts. 2190This can be used to configure hugepages with different sizes, a different size limit, 2191or different access permissions than the system's default hugepage pool. 2192SPDK applications can specify a custom hugetlbfs mount with the `--huge-dir` option. 2193 2194### environment 2195 2196spdk_vtophys() has been refactored to accept length of the translated region as a new 2197parameter. The function will now update that parameter with the largest possible value 2198for which the memory is contiguous in the physical memory address space. 2199 2200The following functions were removed: 2201 2202- spdk_pci_nvme_device_attach() 2203- spdk_pci_nvme_enumerate() 2204- spdk_pci_ioat_device_attach() 2205- spdk_pci_ioat_enumerate() 2206- spdk_pci_virtio_device_attach() 2207- spdk_pci_virtio_enumerate() 2208 2209They were replaced with generic spdk_pci_device_attach() and spdk_pci_enumerate() which 2210require a new spdk_pci_driver object to be provided. It can be one of the following: 2211 2212- spdk_pci_nvme_get_driver() 2213- spdk_pci_ioat_get_driver() 2214- spdk_pci_virtio_get_driver() 2215 2216spdk_pci_hook_device() and spdk_pci_unhook_device() were added. Those allow adding a virtual 2217spdk_pci_device into the SPDK PCI subsystem. A virtual device calls provided callbacks for 2218each BAR mapping request or PCI config access. It's attachable with spdk_pci_device_attach() 2219or spdk_pci_enumerate() like any other device. 2220 2221A new spdk_pause() function was added to pause CPU execution for an implementation specific 2222amount of time. Quoting from DPDK function this is based on: "This call is intended for 2223tight loops which poll a shared resource or wait for an event. A short pause within the loop 2224may reduce the power consumption." 2225 2226A new public header file env_dpdk.h has been introduced, and function spdk_env_dpdk_post_init 2227is added into it. If user is using DPDK, and already called rte_eal_init, then include 2228include/spdk/env_dpdk.h, and call spdk_env_dpdk_post_init() instead of spdk_env_init. 2229 2230ISA-L has been added as an SPDK submodule. ISA-L is enabled by default on x86 architecture 2231to accelerate algorithms such as CRC for iSCSI and NVMe-oF. Users may still disable ISA-L 2232by explicitly passing --without-isal to the configure script. 2233 2234### util 2235 2236A new uuid API `spdk_uuid_copy` was added to make a copy of the source uuid. 2237 2238An new parameter `init_crc` representing the initial CRC value was added to 2239`spdk_crc16_t10dif`. The parameter can be used to calculate a CRC value spanning 2240multiple separate buffers. 2241 2242New DIF APIs were added to generate and verify DIF by byte granularity for both DIF and DIX 2243formats. Among them, DIF with copy APIs will be usable to emulate DIF operations such as DIF 2244insert and strip. 2245 2246Added `spdk_strtol` and `spdk_strtoll` to provide additional error checking around `strtol` 2247and `strtoll`. 2248 2249Added `spdk_sprintf_append_realloc` and `spdk_vsprintf_append_realloc` for appending a string 2250with automatic buffer re-allocation. 2251 2252### nvme 2253 2254Wrapper functions spdk_nvme_ctrlr_security_send() and spdk_nvme_ctrlr_security_receive() are 2255introduced to support further security protocol development. 2256 2257admin_timeout_ms was added to NVMe controller initialization options, users 2258can change the default value when probing a controller. 2259 2260Add two new fields "header_digest" and "data_digest" in struct spdk_nvme_ctrlr_opts, 2261it will be used to enable the digest support for the NVMe/TCP transport. 2262 2263Add a new TCP/IP transport(located in lib/nvme/nvme_tcp.c) in nvme driver. With 2264this new transport, it can be used to connect the NVMe-oF target with the 2265same TCP/IP support. 2266 2267Added API, spdk_nvme_ctrlr_is_discovery(), to indicate whether the ctrlr 2268arg refers to a Discovery Controller or not. 2269 2270Added an API function `spdk_nvme_host_id_parse` and corresponding object `spdk_nvme_host_id` 2271for parsing host address and host service ID arguments on a per connection basis. 2272 2273The RPC `construct_nvme_bdev` now allows a user to specify a source address and service id for the host to 2274use when connecting to the controller backing the NVMe bdev. 2275 2276### NVMe-oF Target 2277 2278The `spdk_nvmf_tgt_opts` struct has been deprecated in favor of `spdk_nvmf_transport_opts`. 2279Users will no longer be able to specify target wide I/O parameters. `spdk_nvmf_tgt_listen` 2280will also no longer implicitly initialize a transport with the default target options (since 2281there are none). Instead, a user must manually instantiate the transport with `spdk_nvmf_transport_create` 2282prior to calling `spdk_nvmf_tgt_listen`. 2283 2284Related to the previous change, the rpc `set_nvmf_target_options` has been renamed to 2285`set_nvmf_target_max_subsystems` to indicate that this is the only target option available for the user to edit. 2286 2287Added fields `num_shared_buffers` and `buf_cache_size` in struct spdk_nvmf_transport_opts, 2288and also updated the related rpc function nvmf_create_transport, to make this 2289configurable parameter available to users. The `num_shared_buffers` is used to 2290configure the shared buffer numbers of the transport used by RDMA or TCP transport. 2291`buf_cache_size` configures number of shared buffers to cache per poll group. 2292 2293### nvmf 2294 2295Add a new TCP/IP transport (located in lib/nvmf/tcp.c). With this tranport, 2296the SPDK NVMe-oF target can have a new transport, and can serve the NVMe-oF 2297protocol via TCP/IP from the host. 2298 2299Added optional mechanism to modify the RDMA transport's behavior when creating protection domains and registering memory. 2300By default, the RDMA transport will use the ibverbs library to create protection domains and register memory. 2301Using `spdk_nvme_rdma_init_hooks` will subvert that and use an existing registration. 2302 2303### bdev 2304 2305Added `enable_bdev_histogram` and `get_bdev_histogram` RPC commands to allow gathering latency data for specified bdev. 2306Please see [documentation](https://spdk.io/doc/bdev.html#rpc_bdev_histogram) for more details. 2307 2308Added `required_alignment` field to `spdk_bdev`, that specifies an alignment requirement for data buffers associated with an spdk_bdev_io. 2309Bdev layer will automatically double buffer any spdk_bdev_io that violates this alignment, before the spdk_bdev_io is submitted to the bdev module. 2310 2311On shutdown, bdev unregister now proceeds in top-down fashion, with 2312claimed bdevs skipped (these will be unregistered later, when virtual 2313bdev built on top of the respective base bdev unclaims it). This 2314allows virtual bdevs to be shut down cleanly as opposed to the 2315previous behavior that didn't differentiate between hotremove and 2316planned shutdown. 2317 2318The `delete_bdev` RPC is now deprecated. Users should instead use the specific deletion RPC 2319for the bdev type to be removed (i.e. delete_malloc_bdev). 2320 2321Added support for separate bandwidth rate limits for read and write to QoS in bdev layer. 2322 2323Bdev I/O statistics now track unmap opertations. 2324 2325### logical volumes 2326 2327Logical volume bdev can now be marked as read only using `set_read_only_lvol_bdev` RPC. 2328This allows for basing clones on top of lvol_bdev without first creating a snapshot. 2329 2330Added option to change method for data erasure when deleting lvol or resizing down. 2331Default of unmapping clusters can now be changed to writing zeroes or no operation. 2332 2333Added option to change method for erasing data region on lvol store creation. 2334Default of unmapping can now be changed to writing zeroes or no operation. 2335 2336### log 2337 2338"trace flags" are now referred to as "log flags" in the SPDK log API. The 2339set_trace_flag, clear_trace_flag and get_trace_flags RPCs are now deprecated, 2340and set_log_flag, clear_log_flag and get_log_flags RPCs have been added. 2341 2342### trace 2343 2344New `trace_record` application was added. It can be used to poll spdk trace shm file and 2345append any new trace entries into another specified file. This can help retain those entries 2346that would otherwise be overwritten in the shm file. See 2347[Capturing sufficient trace events](https://spdk.io/doc/nvmf_tgt_tracepoints.html#capture_trace_events) 2348for more details. 2349 2350Number of trace entries in circular buffer per lcore can now be assigned by starting SPDK app 2351with argument "--num-trace-entries <NUM>" provided. 2352 2353New `get_tpoint_group_mask` RPC was added to get current tpoint_group_mask, and 2354each tpoint group status. 2355New `enable_tpoint_group` and `disable_tpoint_group` RPC were added to enable or 2356disable a specific tpoint group. 2357 2358### ftl 2359 2360EXPERIMENTAL: Added basic flash translation layer module allowing for using Open Channel SSDs as 2361block devices. The module is split into the library (located in lib/ftl) and bdev_ftl 2362(lib/bdev/ftl). See the [documentation](https://spdk.io/doc/ftl.html) for more details. 2363 2364### vhost 2365 2366A security vulnerability has been identified and fixed in the SPDK vhost target. A malicious 2367vhost client (i.e. virtual machine) could carefully construct a circular descriptor chain which 2368would result in a partial denial of service in the SPDK vhost target. These types of descriptor 2369chains are now properly detected by the vhost target. All SPDK vhost users serving untrusted 2370vhost clients are strongly recommended to upgrade. (Reported by Dima Stepanov and Evgeny 2371Yakovlev.) 2372 2373Vhost SCSI and Vhost Block devices can now accept multiple connections on the same socket file. 2374Each connection (internally called a vhost session) will have access to the same storage, but 2375will use different virtqueues, different features and possibly different memory. 2376 2377### vhost scsi 2378 2379SCSI target hotremove can now be performed even without the VIRTIO_SCSI_F_HOTPLUG feature negotiated. 2380Regardless of VIRTIO_SCSI_F_HOTPLUG support, the hotremoval will be still reported through SCSI sense codes. 2381 2382### DPDK 2383 2384DPDK submodule was updated to DPDK 18.11. Note that SPDK does not fully leverage the new 2385multi-process device hotplug yet and continues to work the same way it always did. 2386 2387Dropped support for DPDK 16.07 and earlier, which SPDK won't even compile with right now. 2388 2389### RPC 2390 2391The following RPC commands deprecated in the previous release are now removed: 2392 2393- construct_virtio_user_scsi_bdev 2394- construct_virtio_pci_scsi_bdev 2395- construct_virtio_user_blk_bdev 2396- construct_virtio_pci_blk_bdev 2397- remove_virtio_scsi_bdev 2398- construct_nvmf_subsystem 2399 2400### Miscellaneous 2401 2402The configure options `--with-raid` and `--without-raid` that were deprecated in the previous 2403release are now removed. 2404 2405### nbd 2406 2407Starting nbd using `spdk_nbd_start` is now performed asynchronously. 2408 2409### net framework 2410 2411Net framework initialization and finish is now done asynchronously. 2412 2413### rpc 2414 2415Added `spdk_rpc_is_method_allowed` function for checking whether method is permitted in a given state. 2416Added `spdk_rpc_get_state` to check current state of RPC server. 2417RPC `wait_subsystem_init` has been added to allow clients to block untill all subsystems are initialized. 2418 2419### json rpc 2420 2421JSON RPC client is now running in non-blocking mode. Requests are sent and received during spdk_jsonrpc_client_poll. 2422JSON RPC server can now recieve a callback on connection termination or server shutdown using `spdk_jsonrpc_conn_add_close_cb` 2423and `spdk_jsonrpc_conn_del_close_cb`. 2424 2425## v18.10: Dynamic memory allocation, Crypto Virtual Bdev, jsonrpc-client, SPDKCLI iSCSI and NVMe-oF support 2426 2427### nvme 2428 2429spdk_nvme_ctrlr_cmd_security_send() and spdk_nvme_ctrlr_cmd_security_receive() 2430were added to support sending or receiving security protocol data to or from 2431nvme controller. 2432 2433spdk_nvme_ns_get_extended_sector_size() was added. This function includes 2434the metadata size per sector (if any). spdk_nvme_ns_get_sector_size() still 2435returns only the data size per sector, not including metadata. 2436 2437New `send_nvme_cmd` RPC was added to allow sending NVMe commands directly to NVMe controller. 2438See the [send_nvme_cmd](http://spdk.io/doc/jsonrpc.html#rpc_send_nvme_cmd) documentation 2439for more details. 2440 2441### Build System 2442 2443New `configure` options, `--with-shared` and `--without-shared` 2444[default], provide the capability to build, or not, SPDK shared libraries. 2445This includes the single SPDK shared lib encompassing all of the SPDK 2446static libs as well as individual SPDK shared libs corresponding to 2447each of the SPDK static ones. Although the production of the shared 2448libs conforms with conventional version naming practices, such naming 2449does not at this time confer any SPDK ABI compatibility claims. 2450 2451### bdev 2452 2453spdk_bdev_alias_del_all() was added to delete all alias from block device. 2454 2455A new virtual bdev module has been added to perform at rest data encryption using the DPDK CryptoDev 2456Framework. The module initially uses a software AESNI CBC cipher with experimental support for the 2457Intel QAT hardware accelerator also currently implemented with support for CBC cipher. Future work 2458may include additional ciphers as well as consideration for authentication. 2459 2460The RAID virtual bdev module is now always enabled by default. The configure --with-raid and 2461--without-raid options are now ignored and deprecated and will be removed in the next release. 2462 2463Enforcement of bandwidth limits for quality of service (QoS) has been added to the bdev layer. 2464See the new [set_bdev_qos_limit](http://www.spdk.io/doc/jsonrpc.html#rpc_set_bdev_qos_limit) 2465documentation for more details. The previous set_bdev_qos_limit_iops RPC method introduced at 246618.04 release has been deprecated. The new set_bdev_qos_limit RPC method can support both 2467bandwidth and IOPS limits. 2468 2469spdk_bdev_config_json() and corresponding `get_bdevs_config` RPC was removed. 2470 2471### Environment Abstraction Layer and Event Framework 2472 2473The size parameter of spdk_mem_map_translate is now a pointer. This allows the 2474function to report back the actual size of the translation relative to the original 2475request made by the user. 2476 2477A new structure spdk_mem_map_ops has been introduced to hold memory map related 2478callbacks. This structure is now passed as the second argument of spdk_mem_map_alloc 2479in lieu of the notify callback. 2480 2481### DPDK 18.08 2482 2483The DPDK submodule has been updated to the DPDK 18.08 release. SPDK will now automatically 2484utilize DPDK's dynamic memory management with DPDK versions >= 18.05.1. 2485 2486Hugepages can be still reserved with `[-s|--mem-size <size>]` option at application startup, 2487but once we use them all up, instead of failing user allocations with -ENOMEM, we'll try 2488to dynamically reserve even more. This allows starting SPDK with `--mem-size 0` and using 2489only as many hugepages as it is really needed. 2490 2491Due to this change, the memory buffers returned by `spdk_*malloc()` are no longer guaranteed 2492to be physically contiguous. 2493 2494### I/OAT 2495 2496I/OAT driver can now reinitialize I/OAT channels after encountering DMA errors. 2497 2498### iscsi target 2499 2500Parameter names of `set_iscsi_options` and `get_iscsi_global_params` RPC 2501method for CHAP authentication in discovery sessions have been changed to 2502align with `construct_target_node` RPC method. Old names are still usable 2503but will be removed in future release. 2504 2505`set_iscsi_discovery_auth` and `set_iscsi_target_node_auth` RPC methods have 2506been added to set CHAP authentication for discovery sessions and existing 2507target nodes, respectively. 2508 2509The SPDK iSCSI target supports an AuthFile which can be used to load CHAP 2510shared secrets when the iSCSI target starts. SPDK previously provided a 2511default location for this file (`/usr/local/etc/spdk/auth.conf`) if none was 2512specified. This default has been removed. Users must now explicitly specify 2513the location of this file to load CHAP shared secrets from a file, or use 2514the related iSCSI RPC methods to add them at runtime. 2515 2516### iscsi initiator 2517 2518The SPDK iSCSI initiator is no longer considered experimental and becomes 2519a first-class citizen among bdev modules. The basic usage has been briefly 2520described in the bdev user guide: [iSCSI bdev](https://spdk.io/doc/bdev.html#bdev_config_iscsi) 2521 2522### Miscellaneous 2523 2524The SPDK application framework can now parse long name command line parameters. 2525Most single-character parameters have a long name equivalent now. See the 2526[Command Line Parameters](https://spdk.io/doc/app_overview.html) documentation 2527for details or use the `--help` command line parameter to list all available 2528params. 2529 2530bdevperf `-s` param (io size) was renamed to `-o` as `-s` had been already 2531used by existing apps for memory size. 2532 2533bdevio can now accept all SPDK command line parameters. The config now has to 2534be provided with `-c` or `--config` param. 2535 2536The following ioat/perf and nvme/perf parameters were renamed as well: 2537 `-s` (io size) to `-o` 2538 `-d` (mem size) to `-s` 2539 2540The ReactorMask config file parameter has been deprecated. Users should 2541use the -m or --cpumask command line option to specify the CPU core mask 2542for the application. 2543 2544Default config file pathnames have been removed from iscsi_tgt, nvmf_tgt 2545and vhost. Config file pathnames may now only be specified using the 2546-c command line option. 2547 2548Users may no longer set DPDK_DIR in their environment to specify the 2549location of the DPDK installation used to build SPDK. Using DPDK_DIR 2550has not been the documented nor recommended way to specify the DPDK 2551location for several releases, but removing it ensures no unexpected 2552surprises for users who may have DPDK_DIR defined for other reasons. 2553Users should just use the "configure" script to specify the DPDK 2554location before building SPDK. 2555 2556Although we know that many developers still use Python 2 we are officially 2557switching to Python3 with requirement that all new code must be valid also 2558for Python 2 up to the EOL which is year 2020. 2559 2560Invoking interpreter explicitly is forbidden for executable scripts. There 2561is no need to use syntax like "python ./scripts/rpc.py". All executable 2562scripts must contain proper shebang pointing to the right interpreter. 2563Scripts without shebang musn't be executable. 2564 2565A Python script has been added to enable conversion of old INI config file 2566to new JSON-RPC config file format. This script can be found at 2567scripts/config_converter.py. Example how this script can be used: 2568~~~{.sh} 2569cat old_format.ini | scripts/config_converter.py > new_json_format.json 2570~~~ 2571 2572### Sock 2573 2574Two additional parameters were added to spdk_sock_get_addr() for the server 2575port and client port. These parameters are named "sport" and "cport" 2576respectively. 2577 2578### Virtio 2579 2580The following RPC commands have been deprecated: 2581 2582- construct_virtio_user_scsi_bdev 2583- construct_virtio_pci_scsi_bdev 2584- construct_virtio_user_blk_bdev 2585- construct_virtio_pci_blk_bdev 2586- remove_virtio_scsi_bdev 2587 2588The `construct_virtio_*` ones were replaced with a single `construct_virtio_dev` 2589command that can create any type of Virtio bdev(s). `remove_virtio_scsi_bdev` 2590was replaced with `remove_virtio_bdev` that can delete both Virtio Block and SCSI 2591devices. 2592 2593### Blobfs 2594 2595spdk_file_get_id() returning unique ID for the file was added. 2596 2597### JSON 2598 2599Added jsonrpc-client C library intended for issuing RPC commands from applications. 2600 2601Added API enabling iteration over JSON object: 2602 2603- spdk_json_find() 2604- spdk_json_find_string() 2605- spdk_json_find_array() 2606- spdk_json_object_first() 2607- spdk_json_array_first() 2608- spdk_json_next() 2609 2610### Blobstore 2611 2612Blobstore I/O operations are now based on io_units, instead of blobstore page size. 2613The io_unit size is now the same as the underlying block device's block size. 2614Logical volumes built on a block device with 512B block size can now be used as boot devices 2615in QEMU. 2616 2617### SPDKCLI 2618 2619The SPDKCLI interactive command tool for managing SPDK is no longer considered experimental. 2620Support for the iSCSI and NVMe-oF targets has been added. 2621 2622## v18.07: Raid, Infrastructure Improvements, Bug Fixes 2623 2624### bdev 2625 2626A new public header file bdev_module.h has been introduced to facilitate the 2627development of new bdev modules. This header includes an interface for the 2628spdk_bdev_part and spdk_bdev_part_base objects to enable the creation of 2629multiple virtual bdevs on top of a single base bdev and should act as the 2630primary API for module authors. 2631 2632spdk_bdev_get_opts() and spdk_bdev_set_opts() were added to set bdev-wide 2633options. 2634 2635A mechanism for handling out of memory condition errors (ENOMEM) returned from 2636I/O submission requests at the bdev layer has been added. See 2637spdk_bdev_queue_io_wait(). 2638 2639The spdk_bdev_get_io_stat() function now returns cumulative totals instead of 2640resetting on each call. This allows multiple callers to query I/O statistics 2641without conflicting with each other. Existing users will need to adjust their 2642code to record the previous I/O statistics to calculate the delta between calls. 2643 2644I/O queue depth tracking and samples options have been added. See 2645spdk_bdev_get_qd(), spdk_bdev_get_qd_sampling_period(), and 2646spdk_bdev_set_qd_sampling_period(). 2647 2648### RAID module 2649 2650A new bdev module called "raid" has been added as experimental module which 2651aggregates underlying NVMe bdevs and exposes a single raid bdev. Please note 2652that vhost will not work with this module because it does not yet have support 2653for multi-element io vectors. 2654 2655### Log 2656 2657The debug log component flag available on several SPDK applications has been 2658renamed from `-t` to `-L` to prevent confusion with tracepoints and to allow the 2659option to be added to tools that already use `-t` to mean something else. 2660 2661### Blobstore 2662 2663A new function, spdk_bs_dump(), has been added that dumps all of the contents of 2664a blobstore to a file pointer. This includes the metadata and is very useful for 2665debugging. 2666 2667Two new operations have been added for thin-provisioned blobs. 2668spdk_bs_inflate_blob() will allocate clusters for all thinly provisioned regions 2669of the blob and populate them with the correct data by reading from the backing 2670blob(s). spdk_bs_blob_decouple_parent() works similarly, but will only allocate 2671clusters that correspond to data in the blob's immediate parent. Clusters 2672allocated to grandparents or that aren't allocated at all will remain 2673thin-provisioned. 2674 2675### BlobFS 2676 2677Changed the return type of spdk_file_truncate() from void to int to allow the 2678propagation of `ENOMEM` errors. 2679 2680### NVMe Driver 2681 2682The new API functions spdk_nvme_qpair_add_cmd_error_injection() and 2683spdk_nvme_qpair_remove_cmd_error_injection() have been added for NVMe error 2684emulation. Users can set a specified command to fail with a particular error 2685status. 2686 2687Changed the name `timeout_sec` parameter to `timeout_us` in 2688spdk_nvme_ctrlr_register_timeout_callback(), and also changed the type from 2689uint32_t to uint64_t. This will give users more fine-grained control over the 2690timeout period. 2691 2692Basic support for Open Channel SSDs was added. See nvme_ocssd.h 2693 2694### NVMe Over Fabrics 2695 2696The spdk_nvmf_tgt_destroy() function is now asynchronous and takes a callback 2697as a parameter. 2698 2699spdk_nvmf_qpair_disconnect() was added to allow the user to disconnect qpairs. 2700 2701spdk_nvmf_subsystem_get_max_namespaces() was added to query the maximum allowed 2702number of namespaces for a given subsystem. 2703 2704### Build System 2705 2706The build system now generates a combined shared library (libspdk.so) that may 2707be used in place of the individual static libraries (libspdk_*.a). The combined 2708library includes all components of SPDK and is intended to make linking against 2709SPDK easier. The static libraries are also still provided for users that prefer 2710to link only the minimal set of components required. 2711 2712### git pre-commit and pre-push hooks 2713 2714The pre-commit hook will run `scripts/check_format.sh` and verify there are no 2715formating errors before allowing `git commit` to run. The pre-push hook runs 2716`make CONFIG_WERROR=y` with and without `CONFIG_DEBUG=y` using both the gcc and 2717clang compiler before allowing `git push` to run. Following each DEBUG build 2718`test/unit/unittest.sh` is run and verified. Results are recorded in the 2719`make.log` file. 2720 2721To enable type: 'git config core.hooksPath .githooks'. To override after 2722configuration use the `git --no-verify` flag. 2723 2724### RPC 2725 2726The `start_nbd_disk` RPC method now returns the path to the kernel NBD device node 2727rather than always returning `true`. 2728 2729### DPDK 18.05 2730 2731The DPDK submodule has been rebased on the DPDK 18.05 release. DPDK 18.05 supports 2732dynamic memory allocation, but due to some issues found after the DPDK 18.05 release, 2733that support is not enabled for SPDK 18.07. Therefore, SPDK 18.07 will continue to use 2734the legacy memory allocation model. The plan is to enable dynamic memory allocation 2735after the DPDK 18.08 release which should fix these issues. 2736 2737### Environment Abstraction Layer and Event Framework 2738 2739The spdk_mem_map_translate() function now takes a size parameter to indicate the size of 2740the memory region. This can be used by environment implementations to validate the 2741requested translation. 2742 2743The I/O Channel implementation has been moved to its own library - lib/thread. The 2744public API that was previously in spdk/io_channel.h is now in spdk/thread.h The 2745file spdk/io_channel.h remains and includes spdk/thread.h. 2746 2747spdk_reactor_get_tsc_stats was added to return interesting statistics for each 2748reactor. 2749 2750### IOAT 2751 2752IOAT for copy engine is disabled by default. It can be enabled by specifying the Enable 2753option with "Yes" in `[Ioat]` section of the configuration file. The Disable option is 2754now deprecated and will be removed in a future release. 2755 2756## v18.04: Logical Volume Snapshot/Clone, iSCSI Initiator, Bdev QoS, VPP Userspace TCP/IP 2757 2758### vhost 2759 2760The SPDK vhost-scsi, vhost-blk and vhost-nvme applications have fixes to address the 2761DPDK rte_vhost vulnerability [CVE-2018-1059](http://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2018-1059). 2762Please see this [security advisory](https://access.redhat.com/security/cve/cve-2018-1059) 2763for additional information on the DPDK vulnerability. 2764 2765Workarounds have been added to ensure vhost compatibility with QEMU 2.12. 2766 2767EXPERIMENTAL: Support for vhost-nvme has been added to the SPDK vhost target. See the 2768[vhost documentation](http://www.spdk.io/doc/vhost.html) for more details. 2769 2770### Unified Target Application 2771 2772A new unified SPDK target application, `spdk_tgt`, has been added. This application combines the 2773functionality of several existing SPDK applications, including the iSCSI target, NVMe-oF target, 2774and vhost target. The new application can be managed through the existing configuration file and 2775[JSON-RPC](http://www.spdk.io/doc/jsonrpc.html) methods. 2776 2777### Env 2778 2779spdk_mempool_get_bulk() has been added to wrap DPDK rte_mempool_get_bulk(). 2780 2781New memory management functions spdk_malloc(), spdk_zmalloc(), and spdk_free() have been added. 2782These new functions have a `flags` parameter that allows the user to specify whether the allocated 2783memory needs to be suitable for DMA and whether it should be shared across processes with the same 2784shm_id. The new functions are intended to replace spdk_dma_malloc() and related functions, which will 2785eventually be deprecated and removed. 2786 2787### Bdev 2788 2789A new optional bdev module interface function, `init_complete`, has been added to notify bdev modules 2790when the bdev subsystem initialization is complete. This may be useful for virtual bdevs that require 2791notification that the set of initialization examine() calls is complete. 2792 2793The bdev layer now allows modules to provide an optional per-bdev UUID, which can be retrieved with 2794the spdk_bdev_get_uuid() function. 2795 2796Enforcement of IOPS limits for quality of service (QoS) has been added to the bdev layer. See the 2797[set_bdev_qos_limit_iops](http://www.spdk.io/doc/jsonrpc.html#rpc_set_bdev_qos_limit_iops) documentation 2798for more details. 2799 2800### RPC 2801 2802The `[Rpc]` configuration file section, which was deprecated in v18.01, has been removed. 2803Users should switch to the `-r` command-line parameter instead. 2804 2805The JSON-RPC server implementation now allows up to 32 megabyte responses, growing as 2806needed; previously, the response was limited to 32 kilobytes. 2807 2808### SPDKCLI 2809 2810EXPERIMENTAL: New SPDKCLI interactive command tool for managing SPDK is available. 2811See the [SPDKCLI](http://www.spdk.io/doc/spdkcli.html) documentation for more details. 2812 2813### NVMe Driver 2814 2815EXPERIMENTAL: Support for WDS and RDS capable CMBs in NVMe controllers has been added. This support is 2816experimental pending a functional allocator to free and reallocate CMB buffers. 2817 2818spdk_nvme_ns_get_uuid() has been added to allow retrieval of per-namespace UUIDs when available. 2819 2820New API functions spdk_nvme_ctrlr_get_first_active_ns() and spdk_nvme_ctrlr_get_next_active_ns() 2821have been added to iterate active namespaces, as well as spdk_nvme_ctrlr_is_active_ns() to check if 2822a namespace ID is active. 2823 2824### NVMe-oF Target 2825 2826Namespaces may now be assigned unique identifiers via new optional `eui64` and `nguid` parameters 2827to the `nvmf_subsystem_add_ns` RPC method. Additionally, the NVMe-oF target automatically exposes 2828the backing bdev's UUID as the namespace UUID when available. 2829 2830spdk_nvmf_subsystem_remove_ns() is now asynchronous and requires a callback to indicate completion. 2831 2832### Blobstore 2833 2834A number of functions have been renamed: 2835 2836- spdk_bs_io_write_blob() => spdk_blob_io_write() 2837- spdk_bs_io_read_blob() => spdk_blob_io_read() 2838- spdk_bs_io_writev_blob() => spdk_blob_io_writev() 2839- spdk_bs_io_readv_blob() => spdk_blob_io_readv() 2840- spdk_bs_io_unmap_blob() => spdk_blob_io_unmap() 2841- spdk_bs_io_write_zeroes_blob() => spdk_blob_io_write_zeroes() 2842 2843The old names still exist but are deprecated. They will be removed in the v18.07 release. 2844 2845spdk_blob_resize() is now an asynchronous operation to enable resizing a blob while I/O 2846are in progress to that blob on other threads. An explicit spdk_blob_sync_md() is still 2847required to sync the updated metadata to disk. 2848 2849### Logical Volumes 2850 2851A new `destroy_lvol_bdev` RPC method to delete logical volumes has been added. 2852 2853Lvols now have their own UUIDs which replace previous LvolStoreUUID_BlobID combination. 2854 2855New Snapshot and Clone functionalities have been added. User may create Snapshots of existing Lvols 2856and Clones of existing Snapshots. 2857See the [lvol snapshots](http://www.spdk.io/doc/logical_volumes.html#lvol_snapshots) documentation 2858for more details. 2859 2860Resizing logical volumes is now supported via the `resize_lvol_bdev` RPC method. 2861 2862### Lib 2863 2864A set of changes were made in the SPDK's lib code altering 2865instances of calls to `exit()` and `abort()` to return a failure instead 2866wherever reasonably possible. 2867 2868spdk_app_start() no longer exit()'s on an internal failure, but 2869instead returns a non-zero error status. 2870 2871spdk_app_parse_args() no longer exit()'s on help, '-h', or an invalid 2872option, but instead returns SPDK_APP_PARSE_ARGS_HELP and 2873SPDK_APP_PARSE_ARGS_FAIL, respectively, and SPDK_APP_PARSE_ARGS_SUCCESS 2874on success. 2875 2876spdk_pci_get_device() has been deprecated and will be removed in SPDK v18.07. 2877 2878### I/O Channels 2879 2880The prototype for spdk_poller_fn() has been modified; it now returns a value indicating 2881whether or not the poller did any work. Existing pollers will need to be updated to 2882return a value. 2883 2884### iSCSI Target 2885 2886The SPDK iSCSI target now supports the fd.io Vector Packet Processing (VPP) framework userspace 2887TCP/IP stack. See the [iSCSI VPP documentation](http://www.spdk.io/doc/iscsi.html#vpp) for more 2888details. 2889 2890### iSCSI initiator 2891 2892An iSCSI initiator bdev module has been added to SPDK. This module should be considered 2893experimental pending additional features and tests. More details can be found in 2894lib/bdev/iscsi/README. 2895 2896### PMDK 2897 2898The persistent memory (PMDK) bdev module is now enabled using `--with-pmdk` instead of 2899`--with-nvml`. This reflects the renaming of the persistent memory library from NVML to 2900PMDK. 2901 2902### Virtio Block driver 2903 2904A userspace driver for Virtio Block devices has been added. It was built on top of the 2905[Virtio](http://www.spdk.io/doc/virtio.html) library and can be managed similarly to 2906the Virtio SCSI driver. See the 2907[Virtio Block](http://www.spdk.io/doc/bdev.html#bdev_config_virtio_blk) reference for 2908more information. 2909 2910### Virtio with 2MB hugepages 2911 2912The previous 1GB hugepage limitation has now been lifted. A new `-g` command-line option 2913enables SPDK Virtio to work with 2MB hugepages. 2914See [2MB hugepages](http://www.spdk.io/doc/virtio.html#virtio_2mb) for details. 2915 2916## v18.01: Blobstore Thin Provisioning 2917 2918### Build System 2919 2920The build system now includes a `make install` rule, including support for the common 2921`DESTDIR` and `prefix` variables as used in other build systems. Additionally, the prefix 2922may be set via the configure `--prefix` option. Example: `make install prefix=/usr`. 2923 2924### RPC 2925 2926A JSON RPC listener is now enabled by default using a UNIX domain socket at /var/tmp/spdk.sock. 2927A -r option command line option has been added to enable an alternative UNIX domain socket location, 2928or a TCP port in the format ip_addr:tcp_port (i.e. 127.0.0.1:5260). The Rpc configuration file 2929section is now deprecated and will be removed in the v18.04 release. 2930 2931### I/O Channels 2932 2933spdk_poller_register() and spdk_poller_unregister() were moved from the event 2934framework (include/spdk/event.h) to the I/O channel library 2935(include/spdk/io_channel.h). This allows code that doesn't depend on the event 2936framework to request registration and unregistration of pollers. 2937 2938spdk_for_each_channel() now allows asynchronous operations during iteration. 2939Instead of immediately continuing the interation upon returning from the iteration 2940callback, the user must call spdk_for_each_channel_continue() to resume iteration. 2941 2942### Block Device Abstraction Layer (bdev) 2943 2944The poller abstraction was removed from the bdev layer. There is now a general purpose 2945abstraction for pollers available in include/spdk/io_channel.h 2946 2947### Lib 2948 2949A set of changes were made in the SPDK's lib code altering, 2950instances of calls to `exit()` and `abort()` to return a failure instead 2951wherever reasonably possible. This has resulted in return type changes of 2952the API for: 2953 2954- spdk_env_init() from type `void` to `int`. 2955- spdk_mem_map_init() from type `void` to `int`. 2956 2957Applications making use of these APIs should be modified to check for 2958a non-zero return value instead of relying on them to fail without return. 2959 2960### NVMe Driver 2961 2962SPDK now supports hotplug for vfio-attached devices. But there is one thing keep in mind: 2963Only physical removal events are supported; removing devices via the sysfs `remove` file will not work. 2964 2965### NVMe-oF Target 2966 2967Subsystems are no longer tied explicitly to CPU cores. Instead, connections are handed out to the available 2968cores round-robin. The "Core" option in the configuration file has been removed. 2969 2970### Blobstore 2971 2972A number of functions have been renamed: 2973 2974- spdk_bs_md_resize_blob() => spdk_blob_resize() 2975- spdk_bs_md_sync_blob() => spdk_blob_sync_md() 2976- spdk_bs_md_close_blob() => spdk_blob_close() 2977- spdk_bs_md_get_xattr_names() => spdk_blob_get_xattr_names() 2978- spdk_bs_md_get_xattr_value() => spdk_blob_get_xattr_value() 2979- spdk_blob_md_set_xattr() => spdk_blob_set_xattr() 2980- spdk_blob_md_remove_xattr() => spdk_blob_remove_xattr() 2981- spdk_bs_md_create_blob() => spdk_bs_create_blob() 2982- spdk_bs_md_open_blob() => spdk_bs_open_blob() 2983- spdk_bs_md_delete_blob() => spdk_bs_delete_blob() 2984- spdk_bs_md_iter_first() => spdk_bs_iter_first() 2985- spdk_bs_md_iter_next() => spdk_bs_iter_next() 2986 2987The function signature of spdk_blob_close() has changed. It now takes a struct spdk_blob * argument 2988rather than struct spdk_blob **. 2989 2990The function signature of spdk_bs_iter_next() has changed. It now takes a struct spdk_blob * argument 2991rather than struct spdk_blob **. 2992 2993Thin provisioning support has been added to the blobstore. It can be enabled by setting the 2994`thin_provision` flag in struct spdk_blob_opts when calling spdk_bs_create_blob_ext(). 2995 2996### NBD device 2997 2998The NBD application (test/lib/bdev/nbd) has been removed; Same functionality can now be 2999achieved by using the test/app/bdev_svc application and start_nbd_disk RPC method. 3000See the [GPT](http://www.spdk.io/doc/bdev.html#bdev_config_gpt) documentation for more details. 3001 3002### FIO plugin 3003 3004SPDK `fio_plugin` now supports FIO 3.3. The support for previous FIO 2.21 has been dropped, 3005although it still remains to work for now. The new FIO contains huge amount of bugfixes and 3006it's recommended to do an update. 3007 3008### Virtio library 3009 3010Previously a part of the bdev_virtio module, now a separate library. Virtio is now available 3011via `spdk_internal/virtio.h` file. This is an internal interface to be used when implementing 3012new Virtio backends, namely Virtio-BLK. 3013 3014### iSCSI 3015 3016The MinConnectionIdleInterval parameter has been removed, and connections are no longer migrated 3017to an epoll/kqueue descriptor on the master core when idle. 3018 3019## v17.10: Logical Volumes 3020 3021### New dependencies 3022 3023libuuid was added as new dependency for logical volumes. 3024 3025libnuma is now required unconditionally now that the DPDK submodule has been updated to DPDK 17.08. 3026 3027### Block Device Abstraction Layer (bdev) 3028 3029An [fio](http://github.com/axboe/fio) plugin was added that can route 3030I/O to the bdev layer. See the [plugin documentation](https://github.com/spdk/spdk/tree/master/examples/bdev/fio_plugin/) 3031for more information. 3032 3033spdk_bdev_unmap() was modified to take an offset and a length in bytes as 3034arguments instead of requiring the user to provide an array of SCSI 3035unmap descriptors. This limits unmaps to a single contiguous range. 3036 3037spdk_bdev_write_zeroes() was introduced. It ensures that all specified blocks will be zeroed out. 3038If a block device doesn't natively support a write zeroes command, the bdev layer emulates it using 3039write commands. 3040 3041New API functions that accept I/O parameters in units of blocks instead of bytes 3042have been added: 3043 3044- spdk_bdev_read_blocks(), spdk_bdev_readv_blocks() 3045- spdk_bdev_write_blocks(), spdk_bdev_writev_blocks() 3046- spdk_bdev_write_zeroes_blocks() 3047- spdk_bdev_unmap_blocks() 3048 3049The bdev layer now handles temporary out-of-memory I/O failures internally by queueing the I/O to be 3050retried later. 3051 3052### Linux AIO bdev 3053 3054The AIO bdev now allows the user to override the auto-detected block size. 3055 3056### NVMe driver 3057 3058The NVMe driver now recognizes the NVMe 1.3 Namespace Optimal I/O Boundary field. 3059NVMe 1.3 devices may report an optimal I/O boundary, which the driver will take 3060into account when splitting I/O requests. 3061 3062The HotplugEnable option in `[Nvme]` sections of the configuration file is now 3063"No" by default. It was previously "Yes". 3064 3065The NVMe library now includes a spdk_nvme_ns_get_ctrlr() function which returns the 3066NVMe Controller associated with a given namespace. 3067 3068The NVMe library now allows the user to specify a host identifier when attaching 3069to a controller. The host identifier is used as part of the Reservations feature, 3070as well as in the NVMe-oF Connect command. The default host ID is also now a 3071randomly-generated UUID, and the default host NQN uses the host ID to generate 3072a UUID-based NQN. 3073 3074spdk_nvme_connect() was added to allow the user to connect directly to a single 3075NVMe or NVMe-oF controller. 3076 3077### NVMe-oF Target (nvmf_tgt) 3078 3079The NVMe-oF target no longer requires any in-capsule data buffers to run, and 3080the feature is now entirely optional. Previously, at least 4 KiB in-capsule 3081data buffers were required. 3082 3083NVMe-oF subsytems have a new configuration option, AllowAnyHost, to control 3084whether the host NQN whitelist is enforced when accepting new connections. 3085If no Host options have been specified and AllowAnyHost is disabled, the 3086connection will be denied; this is a behavior change from previous releases, 3087which allowed any host NQN to connect if the Host list was empty. 3088AllowAnyHost is disabled by default. 3089 3090NVMe-oF namespaces may now be assigned arbitrary namespace IDs, and the number 3091of namespaces per subsystem is no longer limited. 3092 3093The NVMe-oF target now supports the Write Zeroes command. 3094 3095### Environment Abstraction Layer 3096 3097A new default value, SPDK_MEMPOOL_DEFAULT_CACHE_SIZE, was added to provide 3098additional clarity when constructing spdk_mempools. Previously, -1 could be 3099passed and the library would choose a reasonable default, but this new value 3100makes it explicit that the default is being used. 3101 3102### Blobstore 3103 3104The blobstore super block now contains a bstype field to identify the type of the blobstore. 3105Existing code should be updated to fill out bstype when calling spdk_bs_init() and spdk_bs_load(). 3106 3107spdk_bs_destroy() was added to allow destroying blobstore on device 3108with an initialized blobstore. 3109 3110spdk_bs_io_readv_blob() and spdk_bs_io_writev_blob() were added to enable 3111scattered payloads. 3112 3113A CLI tool for blobstore has been added, allowing basic operations through either command 3114line or shell interface. See the [blobcli](https://github.com/spdk/spdk/tree/master/examples/blob/cli) 3115documentation for more details. 3116 3117### Event Framework 3118 3119The ability to set a thread name, previously only used by the reactor code, is 3120now part of the spdk_thread_allocate() API. Users may specify a thread name 3121which will show up in tools like `gdb`. 3122 3123### Log 3124 3125The spdk_trace_dump() function now takes a new parameter to allow the caller to 3126specify an output file handle (stdout or stderr, for example). 3127 3128### Logical Volumes 3129 3130Logical volumes library built on top of SPDK blobstore has been added. 3131It is possible to create logical volumes on top of other devices using RPC. 3132 3133See the [logical volumes](http://www.spdk.io/doc/logical_volumes.html) documentation for more information. 3134 3135### Persistent Memory 3136 3137A new persistent memory bdev type has been added. 3138The persistent memory block device is built on top of [libpmemblk](http://pmem.io/nvml/libpmemblk/). 3139It is possible to create pmem devices on top of pmem pool files using RPC. 3140 3141See the [Pmem Block Device](http://www.spdk.io/doc/bdev.html#bdev_config_pmem) documentation for more information. 3142 3143### Virtio SCSI driver 3144 3145A userspace driver for Virtio SCSI devices has been added. 3146The driver is capable of creating block devices on top of LUNs exposed by another SPDK vhost-scsi application. 3147 3148See the [Virtio SCSI](http://www.spdk.io/doc/virtio.html) documentation and 3149[Getting Started](http://www.spdk.io/doc/bdev.html#bdev_config_virtio_scsi) guide for more information. 3150 3151### Vhost target 3152 3153The vhost target application now supports live migration between QEMU instances. 3154 3155## v17.07: Build system improvements, userspace vhost-blk target, and GPT bdev 3156 3157### Build System 3158 3159A `configure` script has been added to simplify the build configuration process. 3160The existing CONFIG file and `make CONFIG_...` options are also still supported. 3161Run `./configure --help` for information about available configuration options. 3162 3163A DPDK submodule has been added to make building SPDK easier. If no `--with-dpdk` 3164option is specified to configure, the SPDK build system will automatically build a 3165known-good configuration of DPDK with the minimal options enabled. See the Building 3166section of README.md for more information. 3167 3168A [Vagrant](https://www.vagrantup.com/) setup has been added to make it easier to 3169develop and use SPDK on systems without suitable NVMe hardware. See the Vagrant 3170section of README.md for more information. 3171 3172### Userspace vhost-blk target 3173 3174The vhost library and example app have been updated to support the vhost-blk 3175protocol in addition to the existing vhost-scsi protocol. 3176See the [vhost documentation](http://www.spdk.io/doc/vhost.html) for more details. 3177 3178### Block device abstraction layer (bdev) 3179 3180A GPT virtual block device has been added, which automatically exposes GPT partitions 3181with a special SPDK-specific partition type as bdevs. 3182See the [GPT bdev documentation](http://www.spdk.io/doc/bdev.md#bdev_config_gpt) for 3183more information. 3184 3185### NVMe driver 3186 3187The NVMe driver has been updated to support recent Intel SSDs, including the Intel® 3188Optane™ SSD DC P4800X series. 3189 3190A workaround has been added for devices that failed to recognize register writes 3191during controller reset. 3192 3193The NVMe driver now allocates request tracking objects on a per-queue basis. The 3194number of requests allowed on an I/O queue may be set during `spdk_nvme_probe()` by 3195modifying `io_queue_requests` in the opts structure. 3196 3197The SPDK NVMe `fio_plugin` has been updated to support multiple threads (`numjobs`). 3198 3199spdk_nvme_ctrlr_alloc_io_qpair() has been modified to allow the user to override 3200controller-level options for each individual I/O queue pair. 3201Existing callers with qprio == 0 can be updated to: 3202~~~ 3203... = spdk_nvme_ctrlr_alloc_io_qpair(ctrlr, NULL, 0); 3204~~~ 3205Callers that need to specify a non-default qprio should be updated to: 3206~~~ 3207struct spdk_nvme_io_qpair_opts opts; 3208spdk_nvme_ctrlr_get_default_io_qpair_opts(ctrlr, &opts, sizeof(opts)); 3209opts.qprio = SPDK_NVME_QPRIO_...; 3210... = spdk_nvme_ctrlr_alloc_io_qpair(ctrlr, &opts, sizeof(opts)); 3211~~~ 3212 3213### Environment Abstraction Layer 3214 3215The environment abstraction layer has been updated to include several new functions 3216in order to wrap additional DPDK functionality. See `include/spdk/env.h` for the 3217current set of functions. 3218 3219### SPDK Performance Analysis with Intel® VTune™ Amplifier 3220 3221Support for SPDK performance analysis has been added to Intel® VTune™ Amplifier 2018. 3222 3223This analysis provides: 3224 3225- I/O performance monitoring (calculating standard I/O metrics like IOPS, throughput, etc.) 3226- Tuning insights on the interplay of I/O and compute devices by estimating how many cores 3227 would be reasonable to provide for SPDK to keep up with a current storage workload. 3228 3229See the VTune Amplifier documentation for more information. 3230 3231## v17.03: Blobstore and userspace vhost-scsi target 3232 3233### Blobstore and BlobFS 3234 3235The blobstore is a persistent, power-fail safe block allocator designed to be 3236used as the local storage system backing a higher-level storage service. 3237See the [blobstore documentation](http://www.spdk.io/doc/blob.html) for more details. 3238 3239BlobFS adds basic filesystem functionality like filenames on top of the blobstore. 3240This release also includes a RocksDB Env implementation using BlobFS in place of the 3241kernel filesystem. 3242See the [BlobFS documentation](http://www.spdk.io/doc/blobfs.html) for more details. 3243 3244### Userspace vhost-scsi target 3245 3246A userspace implementation of the QEMU vhost-scsi protocol has been added. 3247The vhost target is capable of exporting SPDK bdevs to QEMU-based VMs as virtio devices. 3248See the [vhost documentation](http://www.spdk.io/doc/vhost.html) for more details. 3249 3250### Event framework 3251 3252The overhead of the main reactor event loop was reduced by optimizing the number of 3253calls to spdk_get_ticks() per iteration. 3254 3255### NVMe library 3256 3257The NVMe library will now automatically split readv/writev requests with scatter-gather 3258lists that do not map to valid PRP lists when the NVMe controller does not natively 3259support SGLs. 3260 3261The `identify` and `perf` NVMe examples were modified to add a consistent format for 3262specifying remote NVMe over Fabrics devices via the `-r` option. 3263This is implemented using the new `spdk_nvme_transport_id_parse()` function. 3264 3265### iSCSI Target 3266 3267The [Nvme] section of the configuration file was modified to remove the `BDF` directive 3268and replace it with a `TransportID` directive. Both local (PCIe) and remote (NVMe-oF) 3269devices can now be specified as the backing block device. A script to generate an 3270entire [Nvme] section based on the local NVMe devices attached was added at 3271`scripts/gen_nvme.sh`. 3272 3273### NVMe-oF Target 3274 3275The [Nvme] section of the configuration file was modified to remove the `BDF` directive 3276and replace it with a `TransportID` directive. Both local (PCIe) and remote (NVMe-oF) 3277devices can now be specified as the backing block device. A script to generate an 3278entire [Nvme] section based on the local NVMe devices attached was added at 3279`scripts/gen_nvme.sh`. 3280 3281## v16.12: NVMe over Fabrics host, hotplug, and multi-process 3282 3283### NVMe library 3284 3285The NVMe library has been changed to create its own request memory pool rather than 3286requiring the user to initialize the global `request_mempool` variable. Apps can be 3287updated by simply removing the initialization of `request_mempool`. Since the NVMe 3288library user no longer needs to know the size of the internal NVMe request 3289structure to create the pool, the `spdk_nvme_request_size()` function was also removed. 3290 3291The `spdk_nvme_ns_cmd_deallocate()` function was renamed and extended to become 3292`spdk_nvme_ns_cmd_dataset_management()`, which allows access to all of the NVMe 3293Dataset Management command's parameters. Existing callers can be updated to use 3294`spdk_nvme_ns_cmd_dataset_management()` with `SPDK_NVME_DSM_ATTR_DEALLOCATE` as the 3295`type` parameter. 3296 3297The NVMe library SGL callback prototype has been changed to return virtual addresses 3298rather than physical addresses. Callers of `spdk_nvme_ns_cmd_readv()` and 3299`spdk_nvme_ns_cmd_writev()` must update their `next_sge_fn` callbacks to match. 3300 3301The NVMe library now supports NVMe over Fabrics devices in addition to the existing 3302support for local PCIe-attached NVMe devices. For an example of how to enable 3303NVMe over Fabrics support in an application, see `examples/nvme/identify` and 3304`examples/nvme/perf`. 3305 3306Hot insert/remove support for NVMe devices has been added. To enable NVMe hotplug 3307support, an application should call the `spdk_nvme_probe()` function on a regular 3308basis to probe for new devices (reported via the existing `probe_cb` callback) and 3309removed devices (reported via a new `remove_cb` callback). Hotplug is currently 3310only supported on Linux with the `uio_pci_generic` driver, and newly-added NVMe 3311devices must be bound to `uio_pci_generic` by an external script or tool. 3312 3313Multiple processes may now coordinate and use a single NVMe device simultaneously 3314using [DPDK Multi-process Support](http://dpdk.org/doc/guides/prog_guide/multi_proc_support.html). 3315 3316### NVMe over Fabrics target (`nvmf_tgt`) 3317 3318The `nvmf_tgt` configuration file format has been updated significantly to enable 3319new features. See the example configuration file `etc/spdk/nvmf.conf.in` for 3320more details on the new and changed options. 3321 3322The NVMe over Fabrics target now supports virtual mode subsystems, which allow the 3323user to export devices from the SPDK block device abstraction layer as NVMe over 3324Fabrics subsystems. Direct mode (raw NVMe device access) is also still supported, 3325and a single `nvmf_tgt` may export both types of subsystems simultaneously. 3326 3327### Block device abstraction layer (bdev) 3328 3329The bdev layer now supports scatter/gather read and write I/O APIs, and the NVMe 3330blockdev driver has been updated to support scatter/gather. Apps can use the 3331new scatter/gather support via the `spdk_bdev_readv()` and `spdk_bdev_writev()` 3332functions. 3333 3334The bdev status returned from each I/O has been extended to pass through NVMe 3335or SCSI status codes directly in cases where the underlying device can provide 3336a more specific status code. 3337 3338A Ceph RBD (RADOS Block Device) blockdev driver has been added. This allows the 3339`iscsi_tgt` and `nvmf_tgt` apps to export Ceph RBD volumes as iSCSI LUNs or 3340NVMe namespaces. 3341 3342### General changes 3343 3344`libpciaccess` has been removed as a dependency and DPDK PCI enumeration is 3345used instead. Prior to DPDK 16.07 enumeration by class code was not supported, 3346so for earlier DPDK versions, only Intel SSD DC P3x00 devices will be discovered 3347by the NVMe library. 3348 3349The `env` environment abstraction library has been introduced, and a default 3350DPDK-based implementation is provided as part of SPDK. The goal of the `env` 3351layer is to enable use of alternate user-mode memory allocation and PCI access 3352libraries. See `doc/porting.md` for more details. 3353 3354The build process has been modified to produce all of the library files in the 3355`build/lib` directory. This is intended to simplify the use of SPDK from external 3356projects, which can now link to SPDK libraries by adding the `build/lib` directory 3357to the library path via `-L` and linking the SPDK libraries by name (for example, 3358`-lspdk_nvme -lspdk_log -lspdk_util`). 3359 3360`nvmf_tgt` and `iscsi_tgt` now have a JSON-RPC interface, which allows the user 3361to query and modify the configuration at runtime. The RPC service is disabled by 3362default, since it currently does not provide any authentication or security 3363mechanisms; it should only be enabled on systems with controlled user access 3364behind a firewall. An example RPC client implemented in Python is provided in 3365`scripts/rpc.py`. 3366 3367## v16.08: iSCSI target, NVMe over Fabrics maturity 3368 3369This release adds a userspace iSCSI target. The iSCSI target is capable of exporting 3370NVMe devices over a network using the iSCSI protocol. The application is located 3371in app/iscsi_tgt and a documented configuration file can be found at etc/spdk/spdk.conf.in. 3372 3373This release also significantly improves the existing NVMe over Fabrics target. 3374 3375- The configuration file format was changed, which will require updates to 3376 any existing nvmf.conf files (see `etc/spdk/nvmf.conf.in`): 3377 - `SubsystemGroup` was renamed to `Subsystem`. 3378 - `AuthFile` was removed (it was unimplemented). 3379 - `nvmf_tgt` was updated to correctly recognize NQN (NVMe Qualified Names) 3380 when naming subsystems. The default node name was changed to reflect this; 3381 it is now "nqn.2016-06.io.spdk". 3382 - `Port` and `Host` sections were merged into the `Subsystem` section 3383 - Global options to control max queue depth, number of queues, max I/O 3384 size, and max in-capsule data size were added. 3385 - The Nvme section was removed. Now a list of devices is specified by 3386 bus/device/function directly in the Subsystem section. 3387 - Subsystems now have a Mode, which can be Direct or Virtual. This is an attempt 3388 to future-proof the interface, so the only mode supported by this release 3389 is "Direct". 3390- Many bug fixes and cleanups were applied to the `nvmf_tgt` app and library. 3391- The target now supports discovery. 3392 3393This release also adds one new feature and provides some better examples and tools 3394for the NVMe driver. 3395 3396- The Weighted Round Robin arbitration method is now supported. This allows 3397 the user to specify different priorities on a per-I/O-queue basis. To 3398 enable WRR, set the `arb_mechanism` field during `spdk_nvme_probe()`. 3399- A simplified "Hello World" example was added to show the proper way to use 3400 the NVMe library API; see `examples/nvme/hello_world/hello_world.c`. 3401- A test for measuring software overhead was added. See `test/lib/nvme/overhead`. 3402 3403## v16.06: NVMf userspace target 3404 3405This release adds a userspace NVMf (NVMe over Fabrics) target, conforming to the 3406newly-released NVMf 1.0/NVMe 1.2.1 specification. The NVMf target exports NVMe 3407devices from a host machine over the network via RDMA. Currently, the target is 3408limited to directly exporting physical NVMe devices, and the discovery subsystem 3409is not supported. 3410 3411This release includes a general API cleanup, including renaming all declarations 3412in public headers to include a `spdk` prefix to prevent namespace clashes with 3413user code. 3414 3415- NVMe 3416 - The `nvme_attach()` API was reworked into a new probe/attach model, which 3417 moves device detection into the NVMe library. The new API also allows 3418 parallel initialization of NVMe controllers, providing a major reduction in 3419 startup time when using multiple controllers. 3420 - I/O queue allocation was changed to be explicit in the API. Each function 3421 that generates I/O requests now takes a queue pair (`spdk_nvme_qpair *`) 3422 argument, and I/O queues may be allocated using 3423 `spdk_nvme_ctrlr_alloc_io_qpair()`. This allows more flexible assignment of 3424 queue pairs than the previous model, which only allowed a single queue 3425 per thread and limited the total number of I/O queues to the lowest number 3426 supported on any attached controller. 3427 - Added support for the Write Zeroes command. 3428 - `examples/nvme/perf` can now report I/O command latency from the 3429 the controller's viewpoint using the Intel vendor-specific read/write latency 3430 log page. 3431 - Added namespace reservation command support, which can be used to coordinate 3432 sharing of a namespace between multiple hosts. 3433 - Added hardware SGL support, which enables use of scattered buffers that 3434 don't conform to the PRP list alignment and length requirements on supported 3435 NVMe controllers. 3436 - Added end-to-end data protection support, including the ability to write and 3437 read metadata in extended LBA (metadata appended to each block of data in the 3438 buffer) and separate metadata buffer modes. 3439 See `spdk_nvme_ns_cmd_write_with_md()` and `spdk_nvme_ns_cmd_read_with_md()` 3440 for details. 3441- IOAT 3442 - The DMA block fill feature is now exposed via the `ioat_submit_fill()` 3443 function. This is functionally similar to `memset()`, except the memory is 3444 filled with an 8-byte repeating pattern instead of a single byte like memset. 3445- PCI 3446 - Added support for using DPDK for PCI device mapping in addition to the 3447 existing libpciaccess option. Using the DPDK PCI support also allows use of 3448 the Linux VFIO driver model, which means that SPDK userspace drivers will work 3449 with the IOMMU enabled. Additionally, SPDK applications may be run as an 3450 unprivileged user with access restricted to a specific set of PCIe devices. 3451 - The PCI library API was made more generic to abstract away differences 3452 between the underlying PCI access implementations. 3453 3454## v1.2.0: IOAT user-space driver 3455 3456This release adds a user-space driver with support for the Intel I/O Acceleration Technology 3457(I/OAT, also known as "Crystal Beach") DMA offload engine. 3458 3459- IOAT 3460 - New user-space driver supporting DMA memory copy offload 3461 - Example programs `ioat/perf` and `ioat/verify` 3462 - Kernel-mode DMA engine test driver `kperf` for performance comparison 3463- NVMe 3464 - Per-I/O flags for Force Unit Access (FUA) and Limited Retry 3465 - Public API for retrieving log pages 3466 - Reservation register/acquire/release/report command support 3467 - Scattered payload support - an alternate API to provide I/O buffers via a sequence of callbacks 3468 - Declarations and `nvme/identify` support for Intel SSD DC P3700 series vendor-specific log pages and features 3469- Updated to support DPDK 2.2.0 3470 3471## v1.0.0: NVMe user-space driver 3472 3473This is the initial open source release of the Storage Performance Development Kit (SPDK). 3474 3475Features: 3476 3477- NVMe user-space driver 3478- NVMe example programs 3479 - `examples/nvme/perf` tests performance (IOPS) using the NVMe user-space driver 3480 - `examples/nvme/identify` displays NVMe controller information in a human-readable format 3481- Linux and FreeBSD support 3482