1# Changelog 2 3## v23.01: (Upcoming Release) 4 5### accel 6 7New library isa-l-crypto has been added, it is used by accel library in crypto operations. 8 9### bdev 10 11Both of interleaved and separated metadata are now supported by the malloc bdev module. 12 13Protection information is now supported by the malloc bdev module. 14 15A new API `spkd_bdev_part_submit_request_ext` was added to specify a custom completion callback. 16 17### scheduler 18 19Changing scheduler from dynamic back to static is no longer possible, 20as there is no way of saving original SPDK thread distribution on reactors. 21 22### bdev 23 24New APIs `spdk_bdev_for_each_channel` and `spdk_bdev_for_each_channel_continue` and 25associated function pointers were added to iterate each channel of the required bdev. 26 27The RPC `bdev_get_iostat` now allows a user to query the per channel IO statistics for 28required bdev, and displays maximum and minimum latencies and I/O error counts. 29 30New `spdk_bdev_copy_blocks` and `spdk_bdev_get_max_copy` APIs to support copy commands. 31 32A new API `spdk_bdev_io_get_submit_tsc` was added to get the submit_tsc of the bdev I/O. 33 34Bdevs will no longer have UUIDs generated based on timestamp and are responsible for 35setting this field themselves. Generation of UUIDs for NVMe bdevs may be enabled by 36running `bdev_nvme_set_options` RPC with `--generate-uuids` option. These identifiers 37are based on serial number and namespace ID and will always be the same for a given 38device. 39 40A new API `spdk_bdev_channel_get_histogram` was added to get the histogram of a specified 41channel for a bdev. 42 43Converted internal use of `pthread_mutex_t` to `struct spdk_spinlock`. Consumers of bdev 44API functions must be on an SPDK thread or the program will abort. It is now enforced 45that no internal bdev locks can be held when a poller or message goes off CPU. 46 47A new RPC `bdev_reset_iostat` was added to reset I/O statistics of bdevs. Note that if one 48consumer reset I/O statistics, it affects all other consumers. 49 50Add function pointers, `dump_device_stat_json` and `reset_device_stat` to the bdev module 51function table to display and reset I/O statistics specific for the module specific bdev 52context. 53 54### event 55 56Added core lock file mechanism to prevent the same CPU cores from being used by multiple 57SPDK application instances. 58 59Added a command line switch to disable CPU locks on SPDK startup. 60 61Added RPCs framework_enable_cpumask_locks and framework_disable_cpumask_locks to enable 62and disable CPU core locks in runtime. 63 64Added --rpcs-allowed command line option. Users can specify a comma-separated list of RPC 65names with this option to restrict allowed RPCs to only that list. 66 67### rpc 68 69Added spdk_rpc_set_allowlist to restrict allowed RPCs to the specified list. 70 71Changed `bdev_raid_get_bdevs` RPC output format to include raid_bdev details. 72 73### bdevperf 74 75Promoted the application to example to match similar programs: fio_plugin and perf. 76It can now be found inside `examples/bdev/bdevperf`. 77 78### trace 79 80New `trace_get_info` RPC was added to get name of shared memory file, list of the 81available trace point groups and mask of the available trace points for each group. 82 83### util 84 85New API `spdk_fd_group_get_epoll_event` that returns the epoll(7) event that 86caused a function callback in file descriptor group to execute. 87 88### json 89 90Added API `spdk_json_write_double` and `spdk_json_write_named_double` to allow 91for writing and decoding of the the double data type. 92 93### thread 94 95Added `spdk_thread_get_app_thread` which returns the first thread that was created using 96`spdk_thread_create`. 97 98Added `spdk_thread_is_running`. This returns `true` for a running thread, or `false` if 99its exit process has started using `spdk_thread_exit`. 100 101Added API `spdk_spin_init`, `spdk_spin_destroy`, `spdk_spin_lock`, `spdk_spin_unlock`, and 102`spdk_spin_held` to support spinlocks that are aware of the SPDK concurrency model. 103 104### log 105 106Added consistent tracking of use of deprecated behavior. Deprecations continue to be 107listed in deprecation.md and now are in subsections with headers matching a tag 108used in `SPDK_LOG_DEPRECATED()` calls. When deprecated behavior is used, these 109tags will appear in SPDK's log at the warn level. As the SPDK application exits, it 110will log a summary of how many times `SPDK_LOG_DEPRECATED()` was called for each 111tag that was logged at least once. 112 113### nvme 114 115Added `transport_tos` to `spdk_nvme_ctrlr_opts` to support setting of the "type of service" 116value in the IPv4 header. Only RDMA is supported at this time. 117 118### bdev_nvme 119 120Updated `bdev_nvme_set_options` RPC (and rpc.py) to support the new `transport_tos` parameter. 121 122## v22.09 123 124### accel 125 126Many names were changed in the accel framework to make them consistent both with themselves and 127the rest of SPDK. The primary public header file is now named `include/spdk/accel.h`. 128 129Added a new runtime RPC `accel_get_opc_assignments` to get a list of current opcode to engine 130assignments. 131 132Added a new startup RPC `accel_assign_opc` to assign/override a specific opcode to 133an engine. 134 135### bdev 136 137New RPCs `bdev_xnvme_create` and `bdev_xnvme_delete` were added to support the xNVMe bdev. 138 139A new API `spdk_bdev_for_each_bdev_io` was added to execute the function on the appropriate 140thread for each bdev_io submitted to the bdev. 141 142A new API `spdk_bdev_get_current_qd` was added to measure and return the queue depth from a 143bdev. This API is available even when queue depth sampling is disabled. 144 145Added `spdk_bdev_seek_data`, `spdk_bdev_seek_hole` and `spdk_bdev_io_get_seek_offset` functions 146that start from a given offset and seek for next data or for next hole. At this time only 147supported by lvol bdev module. 148 149A new parameter `io_drain_timeout` has been added to `spdk_bdev` structure. It controls how long 150a bdev reset must wait for IO to complete prior to issuing a reset to the underlying device. 151If there is no outstanding IO at the end of that period, the reset is skipped. 152Best defined for bdevs that share an underlying bdev, such as multiple lvol bdevs sharing 153an nvme device, to avoid unnecessarily resetting the underlying bdev and affecting other 154bdevs that are sharing it. 155Modifying this field is optional. Setting the value to 0 keeps the original behavior 156to always send resets immediately, even if there is no I/O outstanding. 157 158### blobstore 159 160Reserve space for `used_clusters` bitmap. The reserved space could be used for blobstore growing 161in the future. 162 163Added `is_zeroes` operation to `spdk_bs_dev`. It allows to detect if logical blocks are backed 164by zeroes device and do a shortcut in copy-on-write flow by excluding copy part from zeroes device. 165 166Added `spdk_blob_get_next_allocated_io_unit` and `spdk_blob_get_next_unallocated_io_unit` functions 167that start from a given offset and seek for first io_unit belonging to an allocated cluster 168or first io_unit belonging to an unallocated cluster. 169 170Added `spdk_bs_grow` function to grow blobstore size if the underlying bdev size is increased. 171This is used by lvol bdev via `bdev_lvol_grow_lvstore` RPC. 172 173### daos 174 175Added new DAOS bdev module, that creates SPDK block device on top of DAOS DFS. 176Please see [documentation](https://spdk.io/doc/bdev.html#bdev_config_daos) for more details. 177 178### env 179 180Added `spdk_pci_register_device_provider` and matching `SPDK_PCI_REGISTER_DEVICE_PROVIDER` 181functions for registering PCI device providers. That allow the VMD driver to be notified 182when users want to attach a device under a given BDF and when a device is detached. 183 184### init 185 186`spdk_subsystem_init_from_json_config` now fails if the JSON configuration file is not 187an object with an array named "subsystems". 188 189### iscsi 190 191Added `bdev_iscsi_set_options` RPC to modify options of the iSCSI bdev module. 192 193### json 194 195Added `spdk_json_find` API return errcode: EPROTOTYPE - json not enclosed in {}. 196`spdk_json_find` now returns -EPROTOTYPE instead of -ENOENT if the object parameter 197does not point to a JSON object (i.e. is not enclosed with {}). 198 199### lvol 200 201Add `num_md_pages_per_cluster_ratio` parameter to the `bdev_lvol_create_lvstore` RPC. 202Calculate `num_md_pages` from `num_md_pages_per_cluster_ratio`, and pass it to `spdk_bs_opts`. 203 204### nvme 205 206Added SPDK_NVME_TRANSPORT_CUSTOM_FABRICS to enum `spdk_nvme_transport_type` to support custom 207fabric transport. SPDK_NVME_TRANSPORT_CUSTOM was intended to be non-fabric custom transport. 208 209Added a new function `spdk_nvme_ns_cmd_verify` to submit a Verify Command to a Namespace. 210 211Added `spdk_nvme_ctrlr_disable_read_changed_ns_list_log_page` to allow an application to 212tell the driver to not read the CHANGED_NS_LIST log page in response to a NS_ATTR_CHANGED 213AEN. When called the application is required to read this log page instead to clear the 214AEN. 215 216Added `psk` field to `spdk_nvme_ctrlr_opts` struct in order to enable SSL socket implementation 217of TCP connection and set the PSK. Applicable for TCP transport only. 218 219### raid 220 221Renamed the `raid5` module to `raid5f` to reflect that it is not a traditional 222RAID5 implementation - only full stripe writes are supported, partial stripe 223writes (read-modify-write) are not. 224 225### rpc 226 227Added `psk` parameter to `bdev_nvme_attach_controller` RPC in order to enable SSL socket implementation 228of TCP connection and set the PSK. Applicable for TCP transport only. 229 230New options `enable_ktls` and `tls_version` were added to the `sock_impl_set_options` structure. 231New options `psk_key` and `psk_identity` were added to the `sock_impl_set_options` structure. 232 233Added warning message for `bdev_rbd_create`, if it is used without -c. 234`bdev_rbd_create()` API without specifying -c is deprecated and will be removed in future release. 235 236Renamed `enable_vmd` RPC to `vmd_enable` to make it consistent with our naming scheme of 237`<subsystem>_<action>`. For now, the old name is still available, but is marked as deprecated. 238 239New function `spdk_rpc_get_method_state_mask` was added to RPC library. 240 241### sma 242 243Extended `VolumeParameters` with crypto parameters allowing the user to configure crypto when 244attaching a volume to a device. This interface is now supported by all upstream device types 245(nvmf-tcp, virtio-blk, vfio-user) using `bdev_crypto`. Users must specify the crypto engine to use 246under `crypto` section in config. It is also possible to register out-of-tree crypto engines by 247inheriting from the `CryptoEngine` class. 248 249Added two new methods: `SetQos` and `GetQosCapabilities` allowing the user to configure QoS on a 250per-device or per-volume level. Not all QoS settings have to be supported by each device, so users 251can use `GetQosCapabilities` to query them for that. All upstream device types support QoS on a 252per-volume level using bdev layer's QoS mechanism. 253 254### sock 255 256Added new `ssl` based socket implementation, the code is located in `module/sock/posix`. 257For now we are using hard-coded PSK and only support TLS 1.3. 258 259### util 260 261Added new functions: `spdk_hexlify` and `spdk_unhexlify`. 262 263A new API `spdk_xor_gen` was added to generate XOR from multiple source buffers. It is going to be 264used by `raid5f` for calculating parity. 265 266### vfu_tgt 267 268Added vfu_tgt library abstraction based on libvfio-user for PCI device emulation. 269Besides the NVMe device emulation, its now possible to emulate virtio-blk and virtio-scsi 270devices. 271 272### virtio 273 274virtio-vhost-user no longer tries to support dynamic memory allocation. The vhost target does 275not support the high rate of SET_MEM_TABLE vhost messages that result from dynamic memory 276allocation, so a virtio-vhost-user device will now present an ERRLOG, assert, and skip the 277SET_MEM_TABLE vhost message if a memory notification is received outside of the normal device 278start/stop. Applications using the virtio library in vhost-user mode should now pre-allocate 279the application's memory using the -s/--mem-size option and use single shared memory file 280segments using the -g/--single-file-segments option. 281 282Added `vfio-user` transport type for virtio-blk and virtio-scsi devices. 283 284### vmd 285 286Fixed hotplug when a device is inserted in a slot in which a disk was already enumerated previously. 287Added two new RPCs: `vmd_remove_device` simulating a hotremove, and `vmd_rescan`, which rescans all 288buses managed by the VMD driver and hotplugs all newfound devices. 289 290## v22.05 291 292### sock 293 294A new option `ack_timeout` was added to the `spdk_sock_opts` structure. 295 296### util 297 298A new parameter `bounce_iovcnt` was added to `spdk_dif_generate_copy` and `spdk_dif_verify_copy`. 299The `bounce_iovcnt` is used to specify the number of bounce_iov to support multiple block-aligned 300fragment copies. 301 302A new API `spdk_copy_iovs_to_buf` and `spdk_copy_buf_to_iovs` were added to copy iovs to buf or 303copy buf to iovs. There're many cases need to use these two APIs. 304 305### bdev 306 307Removed deprecated spdk_bdev_module_finish_done(). Use spdk_bdev_module_fini_done() instead. 308 309A new API `spdk_bdev_unregister_by_name` was added to handle race conditions correctly. 310 311New APIs, `spdk_for_each_bdev` and `spdk_for_each_bdev_leaf`, were added to provide iteration 312safe for race conditions. 313 314A new RPC `bdev_nvme_get_io_paths` was added to get all active I/O paths. 315 316A new RPC `bdev_nvme_set_preferred_path` was added to set preferred I/O path for an NVMe bdev 317when in multipath mode. This RPC does not support NVMe bdevs in failover mode. 318 319A new RPC `bdev_nvme_set_multipath_policy` was added to set multipath policy of a NVMe bdev 320in multipath mode. 321 322A new option `disable_auto_failback` was added to the `bdev_nvme_set_options` RPC to disable 323automatic failback. 324 325### idxd / dsa 326 327A new parameter `flags` was added to all low level submission and preparation 328APIs to enable the caller to pass operation flags per the DSA specification. 329 330A new flag 'SPDK_IDXD_FLAG_PERSISTENT' was added to let DSA know that 331the destination is persistent. 332 333The RPC `idxd_scan_accel_engine` has been renamed to `dsa_scan_accel_engine` 334 335The RPC `iaa_scan_accel_engine` has been added. 336 337Many HW related structs/functions with the name `idxd` have been renamed `dsa` 338to more accurately represent the HW they are associated with. 339 340Two new functions were added to the library `spdk_idxd_submit_compress` and 341`spdk_idxd_submit_decompress` 342 343### accel_fw 344 345A new parameter `flags` was added to accel API. 346The APIs include: 347`spdk_accel_submit_copy` 348`spdk_accel_submit_dualcast` 349`spdk_accel_submit_fill` 350`spdk_accel_submit_copy_crc32c` 351`spdk_accel_submit_copy_crc32cv` 352`spdk_accel_submit_compress` 353`spdk_accel_submit_decompress` 354 355A new flag `ACCEL_FLAG_PERSISTENT` was added to indicate the target memory is PMEM. 356 357The API `spdk_accel_get_capabilities` has been removed. 358 359### crypto 360 361Support for AES_XTS was added for MLX5 polled mode driver (pmd). 362 363bdev_crypto_create RPC now requires hexlified 'key' and 'key2' params for all pmd drivers. 364Unhexlifying is performed during RPC command processing and the vbdev crypto module runs on 365binary keys as before. 366 367### bdev 368 369Added a timeout option to the `bdev_get_bdevs` RPC. It allows the user to specify the amount of 370time to wait until a bdev with a given name appears in the system. 371 372### bdev_nvme 373 374Added `bdev_nvme_add_error_injection` and `bdev_nvme_remove_error_injection` RPCs to add and 375remove NVMe error injections. 376 377New parameters, `ctrlr_loss_timeout_sec`, `reconnect_delay_sec`, and `fast_io_fail_timeout_sec`, are 378added to the RPC `bdev_nvme_set_options`. They can be overridden if they are given by the RPC 379`bdev_nvme_attach_controller`. 380 381### blobstore 382 383New functions `spdk_blob_io_writev_ext` and `spdk_blob_io_readv_ext` are added. The new functions accept 384`spdk_blob_ext_io_opts` structure with extended IO request options. 385 386### event 387 388Added `msg_mempool_size` parameter to `spdk_reactors_init` and `spdk_thread_lib_init_ext`. 389The size of `g_spdk_msg_mempool` can now be controlled through the same-named 390user option of `spdk_app_opts` structure. 391 392### nvme 393 394The API `spdk_nvme_ctrlr_prepare_for_reset()` was deprecated. The functionality provided by the 395`spdk_nvme_ctrlr_prepare_for_reset()` was merged into the API `spdk_nvme_ctrlr_disconnect()`. 396 397### nvmf 398 399Removed deprecated max_qpairs_per_ctrlr parameter from nvmf_create_transport RPC. Use 400max_io_qpairs_per_ctrlr instead. 401 402Deprecated the ability for hosts to connect to the discovery subsystem automatically on any 403existing listener. Users should now explicitly add listeners for the discovery subsystem. 404Host can still connect to the discovery subsystem as before, but a warning message will be 405emitted if no listener was configured for the transport ID of the incoming connection. 406 407Added adaptive interrupt feature for vfio-user transport. New parameter `disable_adaptive_irq` 408is added to the RPC `nvmf_create_transport`. 409 410### thread 411 412Added `spdk_thread_exec_msg()` API. 413 414### scheduler 415 416`framework_set_scheduler` can now be called after application initialization. 417Added callbacks to set custom parameters specific for each scheduler implementation 418and `framework_get_scheduler` to retrieve them. 419 420Added `dynamic` scheduler options: load_limit, core_limit, core_busy. Their descriptions 421are available in JSON-RPC document, in section 422[framework_set_scheduler](jsonrpc.html#rpc_framework_set_scheduler). 423 424### raid 425 426Add concat as a special raid module. The concat module could create a virtual bdev. The 427virtual bdev combines multiple underlying bdevs together. The layout of the underlying 428bdevs is one after another. The concat bdev is extendable. When the free space of the 429concat bdev is not enough, the user can deconstruct the concat bdev, then reconstruct it 430with an additional underlying bdev. 431 432### sock 433 434Allow MSG_ZEROCOPY flag to be set or not according to data size, which can be enabled and 435set by setting "zerocopy_threshold". zerocopy_threshold = 0 means disable this function; 436zerocopy_threshold > 0 means enable it and use this value as the threshold. 437 438### rpc 439 440Introduced `zerocopy_threshold` to enable zerocopy on send for server sockets according to 441data size to be flushed. 442 443## v22.01 444 445### accel 446 447The batching capability was removed. Batching is now considered an implementation 448detail of the low level drivers. 449 450### bdev 451 452The NVMe bdev module supports multipath and improved I/O error resiliency. 453 454The parameter `retry_count` of the RPC `bdev_nvme_set_options` was deprecated and will be 455removed in SPDK 22.04, and the parameter `transport_retry_count` is added and used instead. 456 457An new parameter `bdev_retry_count` is added to the RPC `bdev_nvme_set_options`. 458 459New parameters, `ctrlr_loss_timeout_sec`, `reconnect_delay_sec`, and `fast_io_fail_timeout_sec`, are 460added to the RPC `bdev_nvme_attach_controller`. 461 462An new parameter `num_io_queues` is added to `bdev_nvme_attach_controller` RPC to allow specifying amount 463of requested IO queues. 464 465Added `key_file` parameter to the `rbd_register_cluster` RPC. It is an optional parameter to 466specify a keyring file to connect to a RADOS cluster. 467 468Added `spdk_bdev_get_zone_id` API to retrieve zone_id for a certain LBA. 469 470### bdev_aio 471 472Added `bdev_aio_rescan` RPC to allow rescanning the size of aio bdev. 473 474### bdev_nvme 475 476Added discovery service to bdev_nvme module, which now can connect to a discovery controller. 477bdev_nvme will connect to all current and future subsystems in the discovery controller. 478Added `bdev_nvme_start_discovery` and `bdev_nvme_stop_discovery` RPC. 479 480### dpdk 481 482Updated DPDK submodule to DPDK 21.11. 483 484### env 485 486Added `spdk_pci_for_each_device`. 487 488Removed `spdk_pci_get_first_device` and `spdk_pci_get_next_device`. These APIs were unsafe, because 489they did not account for PCI devices being inserted or removed while the caller was using handles 490returned from these APIs. Existing users of these APIs should switch to `spdk_pci_for_each_device` 491instead. 492 493Added 3 experimental APIs to handle PCI device interrupts (`spdk_pci_device_enable_interrupt`, 494`spdk_pci_device_disable_interrupt`, `spdk_pci_device_get_interrupt_efd`). 495 496Added `framework_get_pci_devices` RPC to list PCIe devices attached to an SPDK application. 497 498### idxd 499 500Many APIs are now vectored rather than scalar, meaning they take iovecs instead of individual pointers. 501 502### json 503 504Added `spdk_json_write_bytearray` API to serialize a buffer as a hex string. 505 506### nvme 507 508API `spdk_nvme_trtype_is_fabrics` was added to return existing transport type 509is fabric or not. 510 511API `spdk_nvme_poll_group_remove` was limited to be available only for a 512disconnected qpair in the group. 513 514New APIs, `spdk_nvme_ctrlr_disconnect`, `spdk_nvme_ctrlr_reconnect_async`, and 515`spdk_nvme_ctrlr_reconnect_poll_async`, have been added to improve error recovery, and 516the existing APIs,`spdk_nvme_ctrlr_reset_async` and `spdk_nvme_ctrlr_reset_poll_async` 517were deprecated. 518 519Added `spdk_nvme_ctrlr_get_discovery_log_page` API for getting the full discovery log page 520from a discovery controller. 521 522### nvmf 523 524Added support for zero-copy operations in the NVMe-oF TCP target. It can be enabled via 525the `zcopy` parameter when creating a transport. The zero-copy operations are only used 526for requests not using in-capsule data. 527 528Added a `subsystem` parameter to `spdk_nvmf_transport_stop_listen_async`. When not NULL, 529it will only disconnect qpairs for controllers associated with the specified subsystem. 530 531Removed accept poller from transport layer. Each transport can have its own policy of 532handling new connections. To notify transport layer `spdk_nvmf_poll_group_add` and 533`spdk_nvmf_tgt_new_qpair` can be used. 534 535### scsi 536 537Structure `spdk_scsi_lun` has been extended with new member `resizing` so that SCSI layer now reports 538unit attention for disk resize. 539 540### trace 541 542Added `spdk_trace_create_tpoint_group_mask` to return tracepoint group mask from 543a tracepoint group name. 544 545Added `trace_set_tpoint_mask` and `trace_clear_tpoint_mask` RPC to allow enabling 546individual traces. 547 548### util 549 550Added `spdk_ioviter_first` and `spdk_ioviter_next` to iterate over two iovecs and 551yield pointers to common length segments. 552 553### sock 554 555A new parameter `hint` is added to `spdk_sock_get_optimal_sock_group`. It allows to suggest 556a poll group when no optimal poll group is found. 557 558## v21.10 559 560Structure `spdk_nvmf_target_opts` has been extended with new member `discovery_filter` which allows to specify 561filtering rules applied during discovery log generation. Refer to `enum spdk_nvmf_tgt_discovery_filter` for more info. 562 563### bdev 564 565New API `spdk_bdev_get_memory_domains` has been added, it allows to get SPDK memory domains used by bdev. 566 567New API functions `spdk_bdev_readv_blocks_ext` and `spdk_bdev_writev_blocks_ext` have been added. 568These functions accept `spdk_bdev_ext_io_opts` structure with extended IO request 569options, e.g. DMA memory domain which describes data that may belong to another memory domain and 570can't be accessed directly. 571 572Added `async_fini_start` to allow bdev modules to complete the `fini_start` asynchronously, 573with new `spdk_bdev_module_fini_start_done` API. 574 575Deprecated `spdk_bdev_module_finish_done()` API, which will be removed in SPDK 22.01. 576Bdev modules should use `spdk_bdev_module_fini_done()` instead. 577 578The `ocssd` bdev has been removed. The Open Channel specification has been largely superseded by 579zoned namespaces, few if any devices have been brought to market, and there is little reason to 580continue to support this. OCSSD support in the nvme driver will remain for now. 581 582### dma 583 584A new library, lib/dma, has been added. This library provides the necessary infrastructure for 585handling systems and devices with multiple memory domains. For example, a PCIe add-in card with an 586SoC may be running SPDK on the SoC. That SoC has its own local memory, but SPDK may be controlling 587devices that can also access the host system memory. This library provides infrastructure to enumerate 588the memory domains and request hardware perform DMA transfers between them. 589 590### dpdk 591 592Updated DPDK submodule to DPDK 21.08. 593 594### event 595 596Added the `disable_signal_handlers` flag to the `spdk_app_opts` struct. 597 598### idxd 599 600Added `spdk_idxd_get_socket` to query the socket that the idxd device 601is on. 602 603### json 604 605Added API to allow for writing and decoding of new types: 606 607- `spdk_json_write_named_uint16` 608- `spdk_json_write_named_uint8` 609- `spdk_json_write_uint16` 610- `spdk_json_write_uint8` 611- `spdk_json_number_to_uint8` 612- `spdk_json_decode_uint8` 613 614### log 615 616Added API `spdk_log_to_syslog_level` to return syslog level based on SPDK's 617log level. 618 619### nvme 620 621Added new functions `spdk_nvme_zns_set_zone_desc_ext` and `spdk_nvme_zns_ext_report_zones` 622to set zone descriptor extension and to get extended zone report respectively. 623 624New API `spdk_nvme_ctrlr_get_memory_domains` has been added, it allows to get SPDK memory domains used by nvme controller. 625 626New API functions `spdk_nvme_ns_cmd_readv_ext` and `spdk_nvme_ns_cmd_writev_ext` 627have been added. These functions accept `spdk_nvme_ns_cmd_ext_io_opts` structure with extended IO request 628options, e.g. DMA memory domain which describes data that may belong to another memory domain and 629can't be accessed directly. 630 631Added a new function `spdk_nvme_ctrlr_get_regs_bpinfo` to get boot partition info of a controller. 632Added new functions `spdk_nvme_ctrlr_write_boot_partition`, 633`spdk_nvme_ctrlr_read_boot_partition_start` and `spdk_nvme_ctrlr_read_boot_partition_poll` 634to write and read the boot partitions of a controller. 635 636Added `spdk_nvme_ctrlr_get_opts` to retrieve the current controller options. 637 638Added `async_mode` to `spdk_nvme_io_qpair_opts` to enable creation of submission and completion 639queues asynchronously. This mode is currently supported at PCIe layer, 640which tracks the qpair creation with state machine and returns to the user immediately. 641Default mode is set to false to create io qpairs synchronously. 642 643Added `spdk_nvme_ctrlr_get_regs_cc` to retrieve NVMe controller CC (Configuration) register. 644 645Added `spdk_nvme_ctrlr_prepare_for_reset` to inform the driver that the application is preparing 646to reset the specified NVMe controller. This function allows the driver to make decisions 647knowing that a reset is about to happen. 648 649### nvmf 650 651Added `oncs` to `struct spdk_nvmf_ctrlr_data` so that the transport layer 652can decide support RESERVATION feature or not. 653 654An `opts_size` element was added in the `spdk_nvmf_ns_opts` structure to solve the 655ABI compatibility issue between different SPDK version. An new option `anagrpid` was 656added in the `spdk_nvmf_ns_opts` structure. 657 658An new parameter `anagrpid` was added to the RPC `nvmf_subsystem_add_ns`. 659 660An new parameter `anagrpid` was added to the RPC `nvmf_subsystem_listener_set_ana_state`. 661 662`spdk_nvmf_subsystem_destroy` is now can be asynchronous, it accepts a callback and callback argument. 663 664A new parameter, `poll_groups_mask` was added to the `nvmf_set_config` RPC that allows specifying 665a subset of cores for the nvmf poll groups. This helps to avoid imbalances when some cores are 666busy with periodic timer tasks that run very frequently. 667 668A new parameter, `commit` was added to `nvmf_bdev_ctrlr_end_zcopy` that allows indicating whether 669the buffers should be committed. 670 671Added `listen_dump_opts` to `spdk_nvmf_transport_ops` so that the transport layer can display 672listen opts. 673 674Added `fabrics_connect_timeout_us` to `bdev_nvme_attach_controller` RPC to allow specifying 675timeout for connect operation. 676 677Added `multipath` to `bdev_nvme_attach_controller` RPC to allow specifying multipath behavior. 678 679### scheduler 680 681New API for implementing schedulers and governors. Please see `include/spdk/scheduler.h` for details. 682 683### scsi 684 685New functions, `spdk_scsi_dev_get_first_lun` and `spdk_scsi_dev_get_next_lun` 686have been added to iterate LUNs of a SCSI device. 687 688Each SCSI device supports 256 LUNs at the maximum now and the macro constant 689`SPDK_SCSI_DEV_MAX_LUN` was removed. 690 691### trace_parser 692 693A new library, lib/trace_parser, has been added. This library provides functions that parse traces 694recorded by an SPDK application. That includes merging traces from multiple cores, 695sorting them by their timestamp and constructing trace entries spanning across multiple buffers. 696 697### util 698 699The `spdk_fd_group_add` API now takes a `name` parameter. 700 701## v21.07 702 703### accel_fw 704 705Added API `spdk_accel_submit_copy_crc32c` to perform a CRC32C while copying data. 706 707Added API `spdk_accel_batch_prep_copy_crc32c` to batch CRC32C + copy commands. 708 709Added API `spdk_accel_submit_copy_crc32cv` to submit chained CRC32C + copy commands. 710 711### bdev 712 713Change `spdk_bdev_read_blocks_with_md` arg offset definition from int64_t to uint64_t. 714 715Red-black tree has been used to organize the bdev names and aliases uniformly 716to provide faster lookup. 717 718Removed ZCOPY emulation: The bdev module can be checked to see if it supports ZCOPY 719and if not supported then use existing READ/WRITE commands. 720 721Added iov to spdk_bdev_zcopy_start. 722 723### dpdk 724 725Updated DPDK submodule to DPDK 21.05. 726 727### idxd 728 729Remove the `probe_cb` parameter in `spdk_idxd_probe` function. And remove the definition 730of `spdk_idxd_probe_cb` function pointer. It should be implemented in idxd_user.c. 731 732Added API `spdk_idxd_submit_copy_crc32c` to perform a CRC32C while copying data. 733 734Added API `spdk_idxd_batch_prep_copy_crc32c` to prepare a batch operation to perform 735a CRC32C while copying data. 736 737We have userspace idxd driver under accel_engine library (module/accel/idxd/accel_engine). 738And kernel idxd driver usage is also added under the accel_engine library. This approach 739implemented a wrapper library to use IDXD device by leveraging the kernel DSA driver in 740SPDK idxd library (lib/idxd). Then users can leverage the RPC later to configure how to 741use the DSA device by user space driver or kernel driver. 742 743### init 744 745Added new `init` library that initializes the SPDK subsystems, which previously was 746internal to application framework. That functionality has been made public and for 747use by applications which don't leverage SPDK's application framework. 748 749### iscsi 750 751New parameters, `pdu_pool_size`, `immediate_data_pool_size`, and `data_out_pool_size`, 752were added to the RPC `iscsi_set_options` to run iSCSI target with varying amount of 753available memory. 754 755### json 756 757Added API `spdk_json_write_named_uint128` and `spdk_json_write_uint128` to perform 758the uint128 related data. 759 760### net 761 762Removed deprecated `net` library. 763 764Added support for ZCOPY. ZCOPY is used in preference to READ and WRITE if the 765bdev module supports ZCOPY and the bdev module has ZCOPY enabled. 766 767### nvme 768 769`spdk_nvme_map_prps` and `spdk_nvme_map_cmd` were moved to nvmf/vfio-user as internal APIs 770as vfio-user is the only user for the above two APIs. 771 772Added a new function `spdk_nvme_ns_cmd_copy` to submit a Simple Copy Command to a Namespace. 773 774Update the `spdk_nvme_generic_command_status_code` structure with new status code 775according to the definition in NVMe 1.4 spec. 776 777`spdk_nvme_ctrlr_get_default_ctrlr_opts` now sets `use_cmb_sqs` to false. This means 778that if a controller has a CMB and supports SQs in the CMB, SPDK will not use 779the CMB for SQs by default - the user must set `use_cmb_sqs` to true in 780the `spdk_nvme_ctrlr_opts` structure prior to controller attach. 781 782Add a new function `spdk_nvme_detach_poll` to simplify a common use case to continue 783polling until all detachments complete. 784 785Added new argument `timeout_admin_us` to `spdk_nvme_ctrlr_register_timeout_callback` so callers 786can specify a different timeout for admin commands vs. io commands. 787 788An existing function `spdk_nvme_detach_async` was updated to add one or more detachments 789to an active context while it is being polled. 790 791Rename a variable in the member `cmic` of the struct `spdk_nvme_ctrlr_data` from 792`multi_host` to `multi_ctrlr`. The variable means that the NVM subsystem may have two or 793more controllers if set to 1. However `multi_host` had indicated a particular use case 794such that the NVM subsystem is used by multiple hosts. 795 796A new option `disable_read_ana_log_page` was added to struct `spdk_nvme_ctrlr_opts` to disable 797reading ANA log page. The upper layer is expected to read ANA log page instead if `true`. 798The default value is `false`. 799 800New APIs, `spdk_nvme_ctrlr_reset_async` and `spdk_nvme_ctrlr_reset_poll_async`, have been added to 801reset a controller asynchronously. 802 803New RPC `bdev_nvme_reset_controller` was added, to reset an NVMe controller. 804 805Added `spdk_nvme_ns_get_nguid` function to get NGUID for the given namespace. 806 807Added `spdk_nvme_ctrlr_is_fabrics` function to indicate whether a ctrlr handle 808is associated with a fabrics controller. 809 810### nvmf 811 812Added `min_cntlid` and `max_cntlid` to `nvmf_create_subsystem` to limit the controller ID range. 813Added `spdk_nvmf_subsystem_get_min_cntlid` and `spdk_nvmf_subsystem_get_max_cntlid` to request those values. 814 815`spdk_nvmf_request_get_buffers_multi` API is removed. 816 817Added the `nvmf_set_crdt` RPC for setting command retry delay times. 818 819Expanded `spdk_nvmf_poll_group_stat` with current qpair count statistics. 820 821Removed following deprecated APIs: 822 823- `spdk_nvmf_poll_group_get_stat` (function in `nvmf.h`), 824- `spdk_nvmf_transport_poll_group_get_stat` (function in `nvmf.h`), 825- `spdk_nvmf_transport_poll_group_free_stat`(function in `nvmf.h`), 826- `spdk_nvmf_rdma_device_stat` (struct in `nvmf.h`), 827- `spdk_nvmf_transport_poll_group_stat` (struct in `nvmf.h`), 828- `poll_group_get_stat` (transport op in `nvmf_transport.h`), 829- `poll_group_free_stat` (transport op in `nvmf_transport.h`). 830 831### rpc 832 833New RPC `bdev_rbd_register_cluster` and `bdev_rbd_unregister_cluster` was added, it allows to create 834and delete the rados object cluster, then users can choose the cluster to create related rbd 835device. 836 837Revised `bdev_rbd_create` parameter, it allows to use an optional parameter `--cluster-name` 838to create a rbd bdev with an already registered Rados Cluster Object. 839 840New RPC `bdev_rbd_get_clusters_info` was added, it allows to get the info of the registered 841Rados Cluster names. 842 843New optional parameter, `timeout_admin_us`, added to the bdev_nvme_set_options RPC. 844 845Revised a parameter `--stripe-size_kb` to `--stripe-size-kb` of `bdev_raid_create` method 846provided in `scripts/rpc.py` for consistency. 847 848An new optional parameter `config_kernel_mode` was added to the RPC `idxd_scan_accel_engine`, 849and this is used to enable using the kernel mode IDXD driver. 850 851Removed deprecated parameter `enable-zerocopy-send` of RPC `sock_impl_set_options`, 852use `enable-zerocopy-send-server` or `enable-zerocopy-send-client` instead. 853Removed deprecated parameter `disable-zerocopy-send` of RPC `sock_impl_set_options`, 854use `disable-zerocopy-send-server` or `disable-zerocopy-send-client` instead. 855 856### rpm 857 858Removed deprecated `pkg/spdk.spec`, please use `rpmbuild/spdk.spec` instead. 859See [RPM documentation](https://spdk.io/doc/rpm.html) for more details. 860 861### thread 862 863Red-black tree has been used for timed pollers to provide faster insertion and deletion 864and for io_devices to provide faster lookup. 865 866Removed `spdk_io_channel` structure from public header and moved it to thread_internal.h. 867 868### trace 869 870Added `spdk_trace_register_description_ext` function to register variable number of tracepoint arguments. 871 872Added ability to chain multiple trace entries together to extend the size of the argument buffer. 873 874### util 875 876Red-black tree macros has been added by using the macros provided by the FreeBSD operating system 877under the same BSD license. 878 879Add an new macro `SPDK_SIZEOF_MEMBER` to get the size of a member of a struct. 880 881`spdk_crc32c_iov_update` function was added to support calculating the crc32c of the iovs. 882 883Added zipf random number generator with power law probability distribution. 884When applied to performance testing of block devices, it will select blocks over 885the full range of LBAs, but will more frequently select lower-numbered LBAs. 886 887## v21.04 888 889### accel 890 891Two new accelerated crc32 functions `spdk_accel_submit_crc32cv` and 892`spdk_accel_batch_prep_crc32cv` are added in order to provide the 893chained accelerated CRC32 computation support. 894 895### bdev 896 897For `bdev_ocssd_create` RPC, the optional parameter `range` was removed. 898Only one OCSSD bdev can be created for one OCSSD namespace. 899 900Removed the `spdk_bdev_open` from bdev library API. 901Removed the `spdk_vbdev_register` and `spdk_bdev_part_base_construct` from bdev module API. 902Removed the `config_text` function for bdev modules to report legacy config. 903 904Added `spdk_bdev_get_max_active_zones` API to display maximum number active zones of a given bdev. 905 906Added `spdk_bdev_get_max_zone_append_size` API to display maximum zone append data transfer size. 907 908### bdev_nvme 909 910Added support for zoned namespaces. 911 912### blobstore 913 914Removed the `spdk_bdev_create_bs_dev_from_desc` and `spdk_bdev_create_bs_dev` API. 915 916### env 917 918Added `spdk_pci_device_allow` API to allow applications to add PCI addresses to 919the allowed list after the application has started. 920 921Removed the `pci_whitelist`, `pci_blacklist` and `master_core` members of struct `spdk_env_opts`. 922 923Added hotplug support based on uevent in `pci_event.c`. Added usage of this functionality in 924nvme, virtio-scsi and virtio_blk libraries. Please see the new API `spdk_pci_event_listen`, 925`spdk_pci_get_event`, `spdk_pci_register_error_handler` and `spdk_pci_unregister_error_handler`. 926 927### event 928 929Removed the `config_file`, `max_delay_us`, `pci_whitelist` 930and `pci_blacklist` members of struct `spdk_app_opts`. 931 932### idxd 933 934A new API `spdk_idxd_device_needs_rebalance` was added so that users of the library 935can know whether they need to rebalance the flow control for the channel 936that was just added/removed. This is based on how the low level library 937shares devices amongst channels. 938 939The API `spdk_idxd_reconfigure_chan` had the `num_channels` removed as this 940is now tracked in the library. The app makes use the new API above to 941determine whether to rebalance or not. This applies to `spdk_idxd_configure_chan` 942as well. 943 944The API `spdk_idxd_put_channel` now returns the rebalance state for the 945underlying device. 946 947### iscsi 948 949A security vulnerability has been identified and fixed in the SPDK iSCSI target. 950A TEXT PDU with no data, but CONTINUE flag set, would result in a NULL pointer dereference 951and crash the SPDK iSCSI target process. All users of the SPDK iSCSI target 952are recommended to update. All SPDK versions <= v21.01 are affected. 953 954### net 955 956The net library is deprecated and will be removed in the 21.07 release. 957 958### nvme 959 960Added a new function `spdk_nvme_ctrlr_get_regs_pmrcap` to get the PMR capabilities. 961 962Added an accelerated table pointer in `spdk_nvme_poll_group` 963which can be used to provide the accelerated functions by users with 964hardware engine, such as crc32c accelerated function. 965 966Added new functions `spdk_nvme_ctrlr_get_pmrsz`, `spdk_nvme_ctrlr_enable_pmr`, 967`spdk_nvme_ctrlr_disable_pmr`, `spdk_nvme_ctrlr_map_pmr` and `spdk_nvme_ctrlr_unmap_pmr`. 968 969Added NVMe transport operations to enable, disable, map and unmap the PMR. 970 971Added `spdk_nvme_qpair_get_optimal_poll_group` function and `qpair_get_optimal_poll_group` 972function pointer in spdk_nvmf_transport_ops structure in order to add the qpair to the most 973suitable polling group. 974 975Added OPTPERF and namespace optimal performance fields to nvme_spec.h. 976 977Added `spdk_nvme_set_hotplug_filter` API to allow applications to choose which 978hot-inserted SSDs should be probed. This is useful for use cases where multiple 979independent SPDK processes are running on one node. The filter function can 980then be implemented in these processes to decide which SSDs to probe based on 981the new SSD's PCI address. 982 983New functions `spdk_nvme_poll_group_get_stats` and `spdk_nvme_poll_group_free_stats` 984were added. These functions allow to get transport statistics per NVME poll group. 985 986Added `spdk_nvme_map_cmd` API to map the NVMe command with SGL cases. 987 988Added support for vector variant of ZNS zone append commands with new API 989`spdk_nvme_zns_zone_appendv` and `spdk_nvme_zns_zone_appendv_with_md`. 990 991Added `spdk_nvme_zns_ns_get_max_open_zones` and `spdk_nvme_zns_ns_get_max_active_zones` API, 992to display maximum number of open and active zones of the given namespace. 993 994Added `spdk_nvme_zns_ns_get_zone_size_sectors` API to provide size of zone in number of 995sectors. 996 997Added `spdk_nvme_qpair_get_id` API to display the ID of the specified qpair. 998 999### nvmf 1000 1001Removed the `spdk_nvmf_tgt_listen` and `spdk_nvmf_subsystem_add_ns` API. 1002 1003Added new APIs: 1004 1005- `spdk_nvmf_poll_group_dump_stat` (function in `nvmf.h`). 1006- `poll_group_dump_stat` (transport op in `nvmf_transport.h`). 1007 1008The following APIs have been deprecated and will be removed in SPDK 21.07: 1009 1010- `spdk_nvmf_poll_group_get_stat` (function in `nvmf.h`), 1011- `spdk_nvmf_transport_poll_group_get_stat` (function in `nvmf.h`), 1012- `spdk_nvmf_transport_poll_group_free_stat`(function in `nvmf.h`), 1013- `spdk_nvmf_rdma_device_stat` (struct in `nvmf.h`), 1014- `spdk_nvmf_transport_poll_group_stat` (struct in `nvmf.h`), 1015- `poll_group_get_stat` (transport op in `nvmf_transport.h`), 1016- `poll_group_free_stat` (transport op in `nvmf_transport.h`). 1017 1018See header files for details. 1019 1020The `trtype` field in JSON returned by `nvmf_get_stats` RPC contains now the name of the transport, 1021which is the same as the type for defined transports and more informative for a custom transport. 1022 1023Added `hdgst` and `ddgst` parameters to `bdev_nvme_attach_controller` RPC in order change 1024state of TCP header and data digest. 1025 1026Added `num_cqe` parameter to `nvmf_create_transport` RPC to set number of completion queues (CQ) 1027for RDMA transport. Useful when CQ resize operation is not supported, for example iWARP. 1028 1029### ocf 1030 1031Updated OCF submodule to v20.12.2 1032 1033Added `bdev_ocf_set_cache_mode` RPC to dynamically switch cache mode of OCF bdev. 1034 1035### opal 1036 1037Removed the `spdk_opal_supported` API. 1038 1039### raid 1040 1041For `bdev_raid_create` RPC, the deprecated parameter `strip_size` was removed. 1042 1043### rpc 1044 1045New RPC `bdev_nvme_get_transport_statistics` was added, it allows to get transport statistics 1046of nvme poll groups. 1047 1048Parameter `enable-zerocopy-send` of RPC `sock_impl_set_options` is deprecated and will be removed in SPDK 21.07, 1049use `enable-zerocopy-send-server` or `enable-zerocopy-send-client` instead. 1050Parameter `disable-zerocopy-send` of RPC `sock_impl_set_options` is deprecated and will be removed in SPDK 21.07, 1051use `disable-zerocopy-send-server` or `disable-zerocopy-send-client` instead. 1052 1053Added cmd_parser.py used to parse the args from argparse. There are 1054two benefit to use command parser: 1055 1056- Simplify the definition of rpc method. It will reduce the rpc method code. 1057- Make the rpc call more versatile. User can add private args into rpc method. 1058 1059### rpm 1060 1061Added support for new RPM spec, rpmbuild/spdk.spec, which can be used for packaging the 1062SPDK. The pkg/spdk.spec is considered to be deprecated and scheduled for removal in SPDK 21.07. 1063See [RPM documentation](https://spdk.io/doc/rpm.html) for more details. 1064 1065### sock 1066 1067The type of `enable_placement_id` in struct `spdk_sock_impl_opts` is changed from 1068bool to int. We can use RPC to configure different value of `enable_placement_id`. 1069Then we can leverage SO_INCOMING_CPU to get placement_id, which aims to utilize 1070CPU cache locality, enabled by setting enable_placement_id=2. 1071 1072A new socket placement mode called PLACEMENT_MARK has been added. Some NICs allow 1073sockets to be marked using the SO_MARK socket option as a hint for which hardware 1074queue they should be associated with. This mode leverages that by setting the same 1075value for all sockets within a poll group. 1076 1077New parameters `enable_zerocopy_send_server` and `enable_zerocopy_send_client` were added 1078to struct spdk_sock_impl_opts, these parameters enable or disable zero copy send for server 1079and client sockets which are created using `spdk_sock_listen` and `spdk_sock_listen_ext` (server); 1080`spdk_sock_connect` and `spdk_sock_connect_ext` (client) functions. Existing parameter 1081`enable_zerocopy_send` enables or disables zero copy send for both server and client sockets. 1082 1083### thread 1084 1085A new API `spdk_io_channel_get_io_device` was added to get the io_device for the specified 1086I/O channel. 1087 1088Added `spdk_thread_set_interrupt_mode` function in order to set present spdk_thread into 1089interrupt mode or back to poll mode. It is valid only when thread interrupt facility is 1090enabled by `spdk_interrupt_mode_enable`. 1091 1092Added `spdk_poller_register_interrupt` function to mark that the poller is capable of 1093entering interrupt mode. Callback function will be called when the poller must transition 1094into or out of interrupt mode. 1095 1096### virtio 1097 1098Added the `bdev_virtio_blk_set_hotplug` RPC for the virtio blk pci device. 1099 1100## v21.01 1101 1102### bdev 1103 1104An `opts_size` element was added in the `spdk_bdev_opts` structure to solve the 1105ABI compatibility issue between different SPDK version. And also add `opts_size` 1106parameter in spdk_bdev_get_opts function. Two fields `small_buf_pool_size` and 1107`large_buf_pool_size` were added into spdk_bdev_opts, which were used to determine 1108the small and large buffer pool size of the whole bdev module. 1109 1110A new API `spdk_bdev_wait_for_examine` was added to allow for checking state of 1111examine process. Along with corresponding `bdev_wait_for_examine` RPC, which 1112is now always called during `spdk_bdev_subsystem_config_json` making sure 1113every bdev is ready to be used. 1114 1115A new API `spdk_bdev_io_get_aio_status` was added for getting the status of 1116bdev_io as Linux AIO errno. Also `spdk_bdev_io_complete_aio_status` function 1117and `SPDK_BDEV_IO_STATUS_AIO_ERROR` were added for bdev module to complete 1118a bdev_io with Linux AIO errno. 1119 1120A new API `spdk_bdev_get_module_name` was added for getting block device module name. 1121Also `spdk_bdev_get_module_ctx` was added to obtain a bdev module context for 1122the block device opened by the specified descriptor. 1123 1124Added `max_segment_size` and `max_num_segments` to `spdk_bdev` structure for 1125bdev modules to specify splitting requirements. 1126 1127### blob 1128 1129An `opts_size` element was added in the `spdk_bs_opts` structure to solve the 1130ABI compatibility issue between different SPDK version. And also add `opts_size` 1131parameter in `spdk_bs_opts_init` function. 1132 1133An `opts_size` element was added in the `spdk_blob_opts` structure to solve the 1134ABI compatibility issue between different SPDK version. And also add `opts_size` 1135parameter in `spdk_blob_opts_init` function. 1136 1137An `opts_size` element was added in the `spdk_blob_open_opts` structure to solve the 1138ABI compatibility issue between different SPDK version. And also add `opts_size` 1139parameter in `spdk_blob_open_opts_init` function. 1140 1141### build 1142 1143SPDK now generates pkg-config files to simplify the process of determining which 1144libraries must be linked into an SPDK application. 1145 1146### dpdk 1147 1148Updated DPDK submodule to DPDK 20.11. 1149 1150Removed `--with-igb-uio-driver` configure option. Since DPDK 20.11 igb_uio driver 1151was moved to separate dpdk-kmods repository. If required, users need to refer to 1152dpdk-kmods repository to build the driver. 1153 1154### env 1155 1156The `pci_whitelist`, `pci_blacklist` and `master_core` members of struct `spdk_env_opts` 1157have been deprecated. The new members are named `pci_allowed`, `pci_blocked` and 1158`main_core` respectively. 1159 1160An `opts_size` element was added in the `spdk_app_opts` structure 1161to solve the ABI compatibility issue between different SPDK version. An `opts_size` 1162parameter is added into `spdk_app_opts_init` function. 1163 1164### event 1165 1166The `pci_whitelist` and `pci_blacklist` members of struct `spdk_app_opts` have been 1167deprecated. The new members are named `pci_allowed` and `pci_blocked` respectively. 1168 1169The `--pci-blacklist` command line option has been deprecated, replaced with 1170`--pci-blocked`. 1171 1172The `--pci-whitelist/-W` command line options have been deprecated, replaced with 1173`--pci-allowed/-A`. 1174 1175Added new experimental `dynamic` scheduler that rebalances idle threads, adjusts CPU frequency 1176using dpdk_governor and turns idle reactor cores to interrupt mode. Please see 1177[scheduler documentation](https://www.spdk.io/doc/scheduler.html) for details. 1178 1179## ioat 1180 1181The PCI BDF whitelist option has been removed from the `ioat_scan_accel_engine` RPC. 1182IOAT PCI functions can still be allowed or denied using SPDK application command 1183line options. 1184 1185### isa-l 1186 1187Updated ISA-L submodule to v2.30.0. 1188 1189### json 1190 1191A new API `spdk_jsonrpc_send_bool_response` was added to allow sending response for 1192writing json bool results into one function. 1193Update API `bdev_nvme_set_options` and add a `keep_alive_timeout_ms` parameter. Now you 1194can specify the `keep_alive_timeout` before creating NVMe bdev. 1195 1196### nbd 1197 1198Change the return type of function `spdk_nbd_stop` from void to int. And update the 1199`spdk_nbd_fini` with two parameters to make its behavior from sync to async. 1200 1201### nvme 1202 1203Directives support was added to the NVMe driver. 1204 1205Two async APIs `spdk_nvme_ctrlr_cmd_directive_receive` and `spdk_nvme_ctrlr_cmd_directive_send` 1206are added for Directive Send and Directive Receive command, respectively. 1207 1208Added a new function `spdk_nvme_ctrlr_reset_subsystem` to perform a NVMe 1209subsystem reset. Note: The NVMe-oF target does not support the subsystem reset yet. 1210 1211Add a new function `spdk_nvme_bytes_to_numd` to transfer bytes to number of 1212dwords. 1213 1214Added a new custom transport `SPDK_NVME_TRANSPORT_VFIOUSER` to enable NVMe 1215driver running with NVMe over vfio-user target. 1216 1217Added the vfio-user custom transport implementation in NVMe driver which can connect 1218to NVMe over vfio-user target via vfio-user transport. 1219 1220Added a new function `spdk_nvme_ctrlr_set_remove_cb` to remove or override 1221`remove_cb` and `remove_ctx` specified when the controller was probed. 1222 1223Added support for ZNS zone append command with new API `spdk_nvme_zns_zone_append` and 1224`spdk_nvme_zns_zone_append_with_md`. 1225 1226### nvmf 1227 1228`nvmf_fc_lld_fini` now takes callback and hence updating FC Broadcom LLD driver 1229to the latest is required. 1230 1231FC transport supported primary tagging and VMID. 1232 1233Broadcom FC LLD driver and SPDK NVMe-oF FC transport consolidated one LLD API, 1234`nvmf_fc_init_poller_queues` into another LLD API `nvmf_fc_init_q`. 1235Hence updating Broadcom FC LLD driver to the latest is required. 1236 1237The functions `destroy` and `qpair_fini` in the transport interface now accept a 1238`cb_fn` and `cb_arg` to call upon completion, and their execution can be asynchronous. 1239 1240The SPDK NVMe-oF target now supports async event notification for discovery log changes. 1241This allows the initiator to create persistent connection to discovery controller and 1242be notified of any discovery log changes. 1243 1244An `opts_size` element was added in the `spdk_nvmf_transport_opts` structure 1245to solve the ABI compatibility issue between different SPDK version. And also add 1246`opts_size` parameter in `spdk_nvmf_transport_opts_init` function. 1247 1248Added a new custom vfio-user transport implementation in NVMe-oF which can provide 1249emulated NVMe devices to QEMU and SPDK NVMe driver. 1250 1251Added new API `spdk_nvmf_tgt_listen_ext` that uses `spdk_nvmf_listen_opts` to allow 1252passing generic options to specific transport. This functionality replaces one provided 1253with `nvmf_subsystem_set_options`. Also removed `nvmf_subsystem_set_options` RPC 1254that allowed transport specific options within subsystem. 1255 1256The `trsvcid` in `nvmf_subsystem_add_listener` RPC is now optional. 1257 1258Pausing a subsystem now only pauses all admin queues. This allows for the 1259management of hosts and listeners, plus the addition of namespaces without a 1260full subsystem pause. Additionally, the target now allows for pausing 1261individual namespaces within a subsystem. To remove a namespace from a 1262subsystem, only the specific namespace must be paused. I/O will continue to 1263other namespaces while these operations execute. 1264 1265### rpc 1266 1267An new optional parameter `wait` was added to the RPC `iscsi_create_portal_group`, 1268and an new RPC `iscsi_start_portal_group` was added. They will be used not to start 1269listening on portals for a portal group until all associated target nodes are created 1270at startup, otherwise some iSCSI initiators may fail to re-login when SPDK iSCSI 1271target application restarts. 1272 1273Two optional parameter `--small-buf-pool-size` and `--large-buf-pool-size` were added 1274into `bdev_set_options` function. 1275 1276### vhost 1277 1278Added optional `packed_ring_recovery` parameter to `vhost_create_blk_controller` RPC 1279enabling packed ring live recovery. 1280 1281## v20.10 1282 1283### accel 1284 1285New APIs `spdk_accel_batch_cancel` and `spdk_idxd_batch_cancel` were added to allow 1286cancelling a batch sequence. 1287 1288### bdev 1289 1290A new `spdk_bdev_part_base_construct_ext` function has been added and the 1291`spdk_bdev_part_base_construct` has been deprecated. The 1292`spdk_bdev_part_base_construct_ext` function takes bdev name as an argument instead 1293of bdev structure to avoid a race condition that can happen when the bdev is being 1294removed between a call to get its structure based on a name and actually opening it. 1295 1296Removed `spdk_bdev_config_text` function for bdev modules to report legacy config. 1297 1298### blobstore 1299 1300A new `spdk_bdev_create_bs_dev_ext` function has been added and `spdk_bdev_create_bs_dev_from_desc` 1301function has been deprecated. 1302 1303### dpdk 1304 1305Updated DPDK submodule to DPDK 20.08. 1306 1307### env 1308 1309A new API `spdk_mem_get_fd_and_offset` was added to get file descriptor and offset for 1310a given virtual address. 1311 1312### event 1313 1314Removed `spdk_subsystem_config` callback for submodules as part of legacy config removal. 1315 1316Removed `spdk_app_get_running_config` function that printed configuration in legacy format, 1317and removed `usr1_handler` from `struct spdk_app_opts` callback that was used to call it. 1318 1319Added SPDK thread scheduler framework which is used to rebalance load among CPU cores. 1320The scheduler implementation is pluggable and two schedulers are added first, `static` scheduler 1321and `gscheduler` scheduler. 1322 1323`static` scheduler is used by default and does not reschedule threads and keeps the initial 1324assignments. 1325 1326CPU frequency governor framework and implementation are also added. 1327`dpdk_governor` is the first implementation and is based on rte_power library of DPDK. 1328`gscheduler` scheduler utilizes the CPU frequency governor. 1329 1330Scheduler and governor frameworks are experimental features. 1331 1332### fio 1333 1334Bdev fio_plugin no longer supports legacy configuration files. Options `spdk_conf` and 1335`spdk_json_conf` have to point to valid JSON configuration. 1336 1337### gpt 1338 1339Removed option to disable GPT probing via legacy configuration. Please use generic 1340bdev functionality to replicate the behavior. See `bdev_set_options --disable-auto-examine` 1341and `bdev_examine` RPC. 1342 1343### intel-ipsec-mb 1344 1345Updated intel-ipsec-mb submodule to v0.54 1346 1347### isa-l 1348 1349Updated ISA-L submodule to v2.29.0. 1350 1351### iscsi 1352 1353The iSCSI target supports iSCSI login redirection feature to achieve fault tolerance 1354and load balancing particularly in scale out iSCSI target system, which runs multiple 1355SPDK iSCSI targets. 1356 1357### json 1358 1359A new API `spdk_json_decode_object_relaxed` was added to allow parsing json values 1360even if decoder for a given key is not found. 1361 1362A new API `spdk_json_free_object` was added to free memory allocated by `spdk_json_decode_object`. 1363 1364### log 1365 1366A log flag structure, `spdk_log_register_flag`, `spdk_log_get_first_flag`, 1367`spdk_log_get_next_flag` and macro `SPDK_LOG_REGISTER_COMPONENT`, have been added which 1368enable registering and interacting with log flags. Added `SPDK_INFOLOG`, `SPDK_DEBUGLOG` 1369and `SPDK_LOGDUMP` macros to display logs for specified log flag. 1370 1371The log flags in the event framework `-L` and several SPDK applications has been enabled on 1372release builds of SPDK. On debug builds this option will additionally set 1373log print level to `SPDK_LOG_DEBUG`. 1374 1375A new API `spdk_log_enable_timestamps` and new RPC `log_enable_timestamps` was added to 1376toggle log timestamps. 1377 1378### miscellaneous 1379 1380The contents of the log_rpc library have been moved to the event library. The log_rpc 1381library now no longer exists. 1382 1383The contents of the app_rpc library have been moved to the event library. The app_rpc 1384library now no longer exists. 1385 1386The contents of the bdev_rpc library have been moved to the bdev library. The app_rpc 1387library now no longer exists. 1388 1389The bdevperf application now disables the zcopy API by default. Prior to this change, 1390bdevperf enabled using the zcopy API by default which caused a performance impact of 1391up to 25% on bdevs that don't natively support zcopy because the API emulates zero-copy 1392by allocating a buffer. The bdevperf `-x` param was renamed to `-Z` and the default 1393value changed to false. For bdevs that support zcopy, use the -Z flag to enable 1394using zcopy API. 1395 1396### net 1397 1398Net framework intended for use by user-space TCP stacks was removed. 1399 1400### nvme 1401 1402Zoned Namespace Command Set support has been added to NVMe driver, see `nvme_zns.h` for details. 1403 1404New APIs, `spdk_nvme_detach_async` and `spdk_nvme_detach_poll_async`, have been added to 1405detach multiple controllers in parallel to mitigate lengthy shutdown notification of 1406a few NVMe SSDs. 1407 1408New APIs, `spdk_nvme_ns_get_ana_group_id` and `spdk_nvme_ns_get_ana_state`, have been 1409added to get ANA group ID and ANA state of the given namespace, respectively. 1410 1411If no specific command set is requested (by setting the command_set member in the 1412`spdk_nvme_ctrlr_opts` structure), SPDK will automatically select the most appropriate 1413command set based on what the controller supports. 1414 1415### nvmf 1416 1417A new API `spdk_nvmf_subsystem_add_ns_ext` has been added and the API `spdk_nvmf_subsystem_add_ns` 1418has been deprecated. 1419 1420The NVMe-oF target now supports Asymmetric Namespace Access (ANA) Reporting to provide 1421multipath to NVMe-oF initiator. 1422 1423Add `no_wr_batching` parameter to `spdk_nvmf_transport_opts` struct to disable 1424Work Requests batching in RDMA transport. 1425 1426NVMf Target transports can now parse any additional JSON params in the nvmf_create_transport RPC 1427through the JSON context provided via spdk_nvmf_target_opts->transport_specific. 1428 1429A new RPC `nvmf_subsystem_set_options` was added to allow transport specific options within subsystem. 1430 1431A new API `spdk_nvmf_transport_stop_listen_async` was added to stop accepting new connections 1432at the provided address. `nvmf_subsystem_remove_listener` RPC now uses this function. 1433 1434A new API `spdk_nvmf_subsystem_disconnect_host` was added to disconnect all connections 1435originating from the provided hostnqn. `nvmf_subsystem_remove_host` RPC now uses this function. 1436 1437### ocf 1438 1439Updated OCF submodule to v20.03.1 1440 1441### rpc 1442 1443New optional parameters, `enable_placement_id` and `enable_quickack` were added to the 1444`sock_impl_set_options` RPC. 1445 1446A new RPC `bdev_examine_bdev` was added to allow users to examine a bdev explicitly. 1447It can be used only if bdev_auto_examine is set to false by the RPC `bdev_set_options`. 1448 1449New optional parameters `no_wr_batching` and `control_msg_num` were added to the RPC 1450'nvmf_create_transport'. 1451 1452New RPCs, `iscsi_target_node_set_redirect` and `iscsi_target_node_request_logout`, have 1453been added, and a new optional parameter `private` has been added to the RPC 1454`iscsi_create_portal_group` to support iSCSI login redirection feature. 1455 1456A new optional parameter `ana_reporting` was added to the RPC `nvmf_create_subsystem`, and 1457a new RPC `nvmf_subsystem_listener_set_ana_state` was added for ANA reporting. 1458 1459New RPCs, `nvmf_subsystem_get_listeners` and `nvmf_subsystem_get_qpairs`, were added to 1460retrieve configuration of the NVMe-oF subsystem. 1461 1462### scsi 1463 1464Two new APIs have been added `spdk_scsi_dev_construct_ext` and 1465`spdk_scsi_dev_add_lun_ext` that allow the upper layer(e.g. vhost-scsi) to 1466receive the notification when the scsi bdev has been resized. 1467 1468The `spdk_scsi_dev_construct` and `spdk_scsi_dev_add_lun` eventually may be 1469deprecated and removed. 1470 1471### sock 1472 1473The `enable_placement_id` field was added in the `struct spdk_sock_impl_opts` to 1474make the placement_id feature configurable by users. The default setting is 1475not enabled. 1476 1477The `enable_quick_ack` field was added in the `struct spdk_sock_impl_opts` to enable 1478or disable quick ack for the POSIX sock module. The default setting is not enabled. 1479 1480### thread 1481 1482`fd_group` is applied to support interrupt mode. 1483New APIs were added to support an experimental interrupt mode. This allows modules or 1484libraries to selectively register file descriptors that the spdk_thread can wait on, 1485as an alternative to polling. In v20.10, this functionality is enabled in a very small 1486subset of SPDK libraries and modules. 1487 1488### util 1489 1490A new utility named `fd_group` was add. It is now 1491implemented by epoll on Linux platform. It can be used by 1492spdk_thread and reactor to implement interrupt mode. 1493 1494Added `spdk_bit_pool` utility, see `bit_pool.h` for details. 1495 1496### vhost 1497 1498SPDK has switched to DPDK's rte_vhost library since 19.07 release, removed the internal 1499rte_vhost library which is used for DPDK older than 19.05, removed the experimental vhost 1500nvme target which depends on the internal rte_vhost library. 1501 1502### vpp 1503 1504Removed socket abstraction for VPP. 1505 1506## v20.07: SPDK CSI driver, new accel_fw commands, I/O abort support 1507 1508spdk_json_free_object() was added to free memory allocated by spdk_json_decode_object(). 1509 1510### accel 1511 1512A new API was added `spdk_accel_get_capabilities` that allows applications to 1513query the capabilities of the currently enabled accel engine back-end. 1514 1515A new capability, CRC-32C, was added via `spdk_accel_submit_crc32c`. 1516 1517The software accel engine implementation has added support for CRC-32C. 1518 1519A new capability, compare, was added via `spdk_accel_submit_compare`. 1520 1521The software accel engine implementation has added support for compare. 1522 1523Several APIs were added to `accel_engine.h` to support batched submission 1524of operations. 1525 1526Several APIs were added to `accel_engine.h` to support dualcast operations. 1527 1528### accel_fw 1529 1530The accel_fw was updated to no longer require the app to allocate an 1531accel_task on its behalf. All public APIs now take a callback arg as 1532the parameter that used to be the accel_task. 1533 1534The accel_fw API `spdk_accel_task_size` was removed as it is no longer 1535required. 1536 1537The accel_fw was updated to support compare, dualcast, crc32c. 1538 1539The accel_fw introduced batching support for all commands in all plug-ins. 1540See docs for detailed information. 1541 1542### bdev 1543 1544A new API `spdk_bdev_abort` has been added to submit abort requests to abort all I/Os 1545whose callback context match to the bdev on the given channel. 1546 1547### build 1548 1549The fio plugins now compile to `build/fio` and are named `spdk_bdev` and `spdk_nvme`. 1550Existing fio configuration files will need to be updated. 1551 1552### dpdk 1553 1554Updated DPDK submodule to DPDK 20.05. 1555 1556### env 1557 1558Several new APIs have been added to provide greater flexibility in registering and 1559accessing polled mode PCI drivers. See `env.h` for more details. 1560 1561### idxd 1562 1563The idxd library and plug-in module for the accel_fw were updated to support 1564all accel_fw commands as well as batching. Batching is supported both 1565through the library and the plug-in module. 1566 1567IDXD engine support for CRC-32C has been added. 1568 1569### ioat 1570 1571A new API `spdk_ioat_get_max_descriptors` was added. 1572 1573### nvme 1574 1575An `opts_size`element was added in the `spdk_nvme_ctrlr_opts` structure 1576to solve the ABI compatibility issue between different SPDK version. 1577 1578A new API `spdk_nvme_ctrlr_cmd_abort_ext` has been added to abort previously submitted 1579commands whose callback argument match. 1580 1581Convenience functions, `spdk_nvme_print_command` and `spdk_nvme-print_completion` were added 1582to the public API. 1583 1584A new function, `spdk_nvmf_cuse_update_namespaces`, updates the cuse representation of an NVMe 1585controller. 1586 1587A new function `qpair_iterate_requests` has been added to the nvme transport interface. ALl 1588implementations of the transport interface will have to implement that function. 1589 1590### nvmf 1591 1592The NVMe-oF target no longer supports connecting scheduling configuration and instead 1593always uses what was previously called "transport" scheduling. 1594 1595`spdk_nvmf_tgt_accept` no longer exists. The accept process now occurs automatically after 1596the creation of an nvmf target and queue pairs are assigned to poll groups by the underlying 1597transport. Further, `spdk_nvmf_transport_ops` has changed such that the accept function 1598pointer no longer takes a function pointer as an argument. Instead, transports should call 1599`spdk_nvmf_tgt_new_qpair` whenever they previously would have called that callback. 1600 1601The NVMe-oF target now supports aborting any submitted NVM or Admin command. Previously, 1602the NVMe-oF target could abort only Asynchronous Event Request commands. 1603 1604### rdma 1605 1606A new `rdma` library has been added. It is an abstraction layer over different RDMA providers. 1607Two providers are available - verbs (used by default when RDMA is enabled or enabled explicitly 1608using --with-rdma=verbs) and mlx5 Direct Verbs aka DV (enabled by --with-rdma=mlx5_dv). 1609Using mlx5_dv requires libmlx5 installed on the system. 1610 1611### rpc 1612 1613Parameter `-p` or `--max-qpairs-per-ctrlr` of `nvmf_create_transport` RPC command accepted by the 1614rpc.py script is deprecated, new parameter `-m` or `--max-io-qpairs-per-ctrlr` was added. 1615 1616Added `sock_impl_get_options` and `sock_impl_set_options` RPC methods. 1617 1618Command line parameters `-r` and `--rpc-socket` will longer accept TCP ports. RPC server 1619must now be started on a Unix domain socket. Exposing RPC on the network, as well as providing 1620proper authentication (if needed) is now a responsibility of the user. 1621 1622The `bdev_set_options` RPC has a new option, `bdev_auto_examine` to control the auto examine function 1623of bdev modules. 1624 1625New RPCs `sock_impl_get_options` and `sock_impl_set_options` been added to expose new socket features. 1626See `sock` section for more details. 1627 1628### sock 1629 1630Added `spdk_sock_impl_get_opts` and `spdk_sock_impl_set_opts` functions to set/get socket layer configuration 1631options. Options can be set independently for each implementation. 1632 1633Added `recv_buf_size` and 'send_buf_size' socket layer options. They are used only in posix implementation. 1634 1635Added `uring` based socket implementation, the code is located in module/sock/uring. This feature is only 1636available in Linux which requires kernel version is greater than 5.4.3. Currently, our CI pool added the uring 1637based socket tests for iSCSI target and also the tests for SPDK NVMe-oF tcp transport. 1638 1639Added `enable_recv_pipe` socket layer option to allow disabling of double buffering on receive. 1640New option is used only in posix implementation. 1641 1642Added `enable_zerocopy_send` socket layer option to allow disabling of zero copy flow on send. 1643New option is used only in posix implementation. 1644 1645### util 1646 1647Some previously exposed CRC32 functions have been removed from the public API - 1648`spdk_crc32_update`, `spdk_crc32_table_init`, and the `spdk_crc32_table` struct. 1649 1650### vhost 1651 1652The function `spdk_vhost_blk_get_dev` has been removed. 1653 1654## v20.04: SPDK Top, IDXD, NVMe qpair groups 1655 1656IDXD engine support for compare has been added. 1657 1658### configuration 1659 1660Legacy INI style configuration for SPDK applications has been deprecated and will be 1661removed in future release. 1662Please switch to JSON-RPC configuration files and/or RPC driven run-time configuration. 1663 1664### copy 1665 1666The copy engine library, modules and public APIs have been renamed. Use of the word `copy` 1667has been replaced with the word `accel` short for accelerator in preparation for adding new 1668capabilities in the future. Additionally, APIs for what was previously called the `memcpy` 1669engine have been renamed to identify the engine as a software accelerator. 1670 1671### crypto 1672 1673Support for AES_XTS was added for the QAT polled mode driver (pmd). The create RPC 1674`bdev_crypto_create` has 2 new optional parameters: cipher and key2. Cipher can be either 1675AES_CBC (default) or AES_XTS. AES_XTS is only valid when using the QAT polled mode driver. 1676The key2 parameter is the second key required for AES_XTS. 1677 1678### event 1679 1680Reactors now accumulate CPU stats and they are retrieved by the RPC `framework_get_reactors`. 1681 1682### idxd 1683 1684IDXD support was added in the form of a low level library that can directly 1685utilize the IDXD offload engine as well as an accel plug-in module allowing for use 1686with the generic accel framework API. IDXD is the first in a family of offload 1687engines that share the same interface, specifically DSA is added here. More info 1688can be found here: https://01.org/blogs/2019/introducing-intel-data-streaming-accelerator 1689 1690Much of the implementation models IOAT, however the low level interfaces are very 1691different. The RPC to enable IDXD requires a configuration number as well. The 1692code includes two pre-defined configurations of IDXD groups/work queues/engines. A future 1693version will provide an interface to allow for setting of individual configuration 1694parameters. 1695 1696IDXD is not yet available so this feature should be considered experimental. It will 1697be built up with additional documentation as an ongoing activity. 1698 1699### iscsi 1700 1701The iSCSI target now creates a lightweight thread per poll group instead of assuming a pool 1702of lightweight threads already exist at start up time. A poll group is a collection of 1703unrelated iSCSI connections. Each poll group is only accessed from the associated 1704lightweight thread. 1705 1706### ftl 1707 1708Several changes have been made to the `spdk_ftl_conf`, `spdk_ftl_dev_init_ops`, and 1709`spdk_ftl_attrs` structs. Please see `include/spdk/ftl.h` for more details. 1710 1711### miscellaneous 1712 1713The `--json-ignore-init-errors` command line parameter has been added to ignore 1714initialization errors on JSON config load. 1715 1716The public header file io_channel.h has been removed. Please use thread.h which has the 1717exact same API. 1718 1719### nvme 1720 1721Exported internal nvme_ctrlr_cmd_security_receive/send() APIs as public APIs with "the spdk_" 1722prefix. 1723 1724Added `priority` field in `spdk_nvme_transport_id`, this field is used to specify the priority 1725of the NVMe-oF connection, and currently it is used for NVMe-oF tcp connection. 1726 1727A new poll group API has been added to allow for pooling of nvme qpairs across a single 1728entity which can be polled for completions. This new API consists of the `spdk_nvme_poll_group` 1729family of functions. As a result of this new API, all NVMe transports are expected to implement 1730several poll group related functions. 1731 1732A new flag, `create_only`, has been added to the `spdk_nvme_io_qpair_opts` structure. This flag 1733allows a user to call `spdk_nvme_ctrlr_get_default_io_qpair` without also connecting the qpair 1734within the context of that call. 1735 1736As a result of the `create_only` flag, two new API functions, `spdk_nvme_ctrlr_connect_io_qpair` 1737and `spdk_nvme_ctrlr_disconnect_io_qpair`, have been added to facilitate connecting newly created 1738qpairs (for example, after they have been added to a poll group) and disconnecting qpairs without 1739destroying them (for example to disconnect a qpair before migrating it to a new poll group and 1740reconnecting it). 1741 1742The functions `spdk_nvme_ctrlr_alloc_cmb_io_buffer` and `spdk_nvme_ctrlr_free_cmb_io_buffer` 1743have been changed to `spdk_nvme_ctrlr_map_cmb` and `spdk_nvme_ctrlr_unmap_cmb` respectively. 1744 1745An additional function, `spdk_nvme_ctrlr_reserve_cmb`, has been added to facilitate reserving 1746the entire size of the controller memory buffer for data transfer. 1747 1748### nvme_cuse 1749 1750`spdk_nvme_cuse_get_ctrlr_name` now takes two additional parameters, `char *name` which 1751stores the pointer to the controller name, and `size_t *size` which stores the length of 1752the name. The return type has also been changed from char * to int. 1753 1754`spdk_nvme_cuse_get_ns_name` now takes two additional parameters, `char *name` which 1755stores the pointer to the namespace name, and `size_t *size` which stores the length of 1756the name. The return type has also been changed from char * to int. 1757 1758### nvme_opal 1759 1760Several public OPAL structure definitions have been changed since the last release. 1761 1762### nvmf 1763 1764`spdk_nvmf_poll_group_destroy()` is now asynchronous and accepts a completion callback. 1765 1766The NVMe-oF target now creates a lightweight thread per poll group instead of assuming a pool 1767of lightweight threads already exist at start up time. A poll group is a collection of 1768unrelated NVMe-oF connections. Each poll group is only accessed from the associated 1769lightweight thread. 1770 1771A new struct, `spdk_nvmf_subsystem_listener`, has been added to encapsulate the subsystem specific 1772nature of a listener object. 1773 1774`spdk_nvmf_tgt_listen` no longer accepts a callback function or argument. It also returns an 1775int to indicate the status of the listen call. 1776 1777The execution of `spdk_nvme_poll_group_destroy` is now asynchronous and the function accepts 1778a cb_fn and cb_arg to call upon completion. 1779 1780The execution of `spdk_nvmf_subsystem_add_listener` is now asynchronous and the function accepts 1781a cb_fn and cb_arg to call upon completion. 1782 1783The `nvmf_transport.h` header has been made public to allow custom NVMe-oF transports to integrate 1784with NVMe-oF libraries without using internal APIs. 1785 1786### ocf 1787 1788Updated the OCF submodule to OCF v20.03 1789 1790New version of OCF comes with API changes and bug fixes 1791 1792### rpc 1793 1794A new RPC `thread_set_cpumask` has been added to set the cpumask of the thread 1795to the specified value. 1796 1797A new RPC `thread_get_pollers` has been added to retrieve pollers from SPDK threads. 1798 1799A new RPC `thread_get_io_channels` has been added to retrieve I/O channels from SPDK threads. 1800 1801A new RPC `bdev_rbd_resize` has been added to resize Ceph RBD bdevs. 1802 1803### sock 1804 1805The `spdk_sock_set_priority` function has been removed since the feature to set the sock priority 1806will be contained in two new functions, i.e., `spdk_sock_listen_ext` and `spdk_sock_connect_ext`. 1807Users may now specify the priority of the socket in the opts that they want to use. 1808 1809### spdk_top 1810 1811A new application, `spdk_top`, has been added which allows users to monitor resource consumption 1812by a running SPDK application. More information on this application can be found in 1813`app/spdk_top/README`. 1814 1815### thread 1816 1817A new function `spdk_thread_lib_init_ext` has been added, and the function 1818`spdk_thread_lib_init` has been deprecated. The user of `spdk_thread_lib_init_ext` is 1819expected to implement both functions `spdk_thread_op_fn` and `spdk_thread_op_supported_fn`. 1820`spdk_thread_op_supported_fn` is called to check whether the SPDK thread operation 1821is supported. `spdk_thread_op_fn` is called to execute the SPDK thread operation. 1822Current SPDK operation types are `SPDK_THREAD_OP_NEW` and `SPDK_THREAD_OP_RESCHED`. 1823The operation `SPDK_THREAD_OP_NEW` is called each time a new thread is created. 1824The operation `SPDK_THREAD_OP_RESCHED` is called when SPDK thread needs to be rescheduled. 1825 1826A unique ID has been added for each created SPDK thread, it is retrieved by a new function 1827`spdk_thread_get_id`, and the SPDK thread which has the specific ID is got by 1828a new function `spdk_thread_get_by_id`. 1829 1830A new function `spdk_thread_cpumask` has been added to set the current thread's cpumask 1831to the specified value. The function requires the operation `SPDK_THREAD_OP_RESCHED` 1832is supported. 1833 1834A new function `spdk_poller_register_named` has been added to set arbitrary name to the 1835created poller. If NULL, the name is set to the pointer of the poller function. 1836 1837The function `spdk_thread_poll` now measures run time per call correctly on multiple SPDK 1838threads configuration, and a new function `spdk_thread_get_last_tsc` has been added to use together. 1839 1840Voluntary termination of SPDK thread has been supported by refining the functions `spdk_thread_exit` 1841and `spdk_thread_poll`. 1842 1843### util 1844 1845New functions `spdk_sn32_lt` and `spdk_sn32_gt` have been added. They compare two sequence 1846numbers based on serial number arithmetic. 1847 1848### vhost 1849 1850Poll groups per session have been replaced by SPDK threads per vhost controller. 1851 1852### vmd 1853 1854A new function, `spdk_vmd_fini`, has been added. It releases all resources acquired by the VMD 1855library through the `spdk_vmd_init` call. 1856 1857## v20.01: Optimized thin provisioning, FTL bdev, VMD hot plug, FUSED support 1858 1859### bdev 1860 1861A new function, `spdk_bdev_set_timeout`, has been added to set per descriptor I/O timeouts. 1862 1863A new class of functions `spdk_bdev_compare*`, have been added to allow native bdev support 1864of block comparisons and compare-and-write. 1865 1866A new class of bdev events, `SPDK_BDEV_EVENT_MEDIA_MANAGEMENT`, has been added to allow bdevs 1867which expose raw media to alert all I/O channels of pending media management events. 1868 1869A new API was added `spdk_bdev_io_get_aux_buf` allowing the caller to request 1870an auxiliary buffer for its own private use. The API is used in the same manner that 1871`spdk_bdev_io_get_buf` is used and the length of the buffer is always the same as the 1872bdev_io primary buffer. 'spdk_bdev_io_put_aux_buf' frees the allocated auxiliary 1873buffer. 1874 1875### blobfs 1876 1877Added boolean return value for function spdk_fs_set_cache_size to indicate its operation result. 1878 1879Added `blobfs_set_cache_size` RPC method to set cache size for blobstore filesystem. 1880 1881### blobstore 1882 1883Added new `use_extent_table` option to `spdk_blob_opts` for creating blobs with Extent Table descriptor. 1884Using this metadata format, dramatically decreases number of writes required to persist each cluster allocation 1885for thin provisioned blobs. Extent Table descriptor is enabled by default. 1886See the [Blobstore Programmer's Guide](https://spdk.io/doc/blob.html#blob_pg_cluster_layout) for more details. 1887 1888### dpdk 1889 1890Updated DPDK submodule to DPDK 19.11. 1891 1892### env_dpdk 1893 1894`spdk_env_dpdk_post_init` now takes a boolean, `legacy_mem`, as an argument. 1895 1896A new function, `spdk_env_dpdk_dump_mem_stats`, prints information about the memory consumed by DPDK to a file specified by 1897the user. A new utility, `scripts/dpdk_mem_info.py`, wraps this function and prints the output in an easy to read way. 1898 1899### event 1900 1901The functions `spdk_reactor_enable_framework_monitor_context_switch()` and 1902`spdk_reactor_framework_monitor_context_switch_enabled()` have been changed to 1903`spdk_framework_enable_context_switch_monitor()` and 1904`spdk_framework_context_switch_monitor_enabled()`, respectively. 1905 1906### ftl 1907 1908All NVMe dependencies were removed from ftl library. 1909From now ftl library is using bdev_zone API. 1910bdev_ftl becomes virtual bdev. 1911 1912`ctrlr` and `trid` fields in `spdk_ftl_dev_init_opts` structure were replaced by 1913`base_bdev_desc`. 1914 1915`bdev_ftl_create` RPC method `trtype` and `traddr` parameters were replaced by `base_bdev` 1916parameter. 1917 1918`spdk_ftl_module_init` and `spdk_ftl_module_fini` functions were removed. 1919 1920`spdk_ftl_punit_range` and `ftl_module_init_opts` structures were removed. 1921 1922### isa-l 1923 1924Updated ISA-L submodule to commit f3993f5c0b6911 which includes implementation and 1925optimization for aarch64. 1926 1927Enabled ISA-L on aarch64 by default in addition to x86. 1928 1929### nvme 1930 1931`delayed_pcie_doorbell` parameter in `spdk_nvme_io_qpair_opts` was renamed to `delay_cmd_submit` 1932to allow reuse in other transports. 1933 1934Added RDMA WR batching to NVMf RDMA initiator. Send and receive WRs are chained together 1935and posted with a single call to ibv_post_send(receive) in the next call to qpair completion 1936processing function. Batching is controlled by 'delay_cmd_submit' qpair option. 1937 1938The NVMe-oF initiator now supports plugging out of tree NVMe-oF transports. In order 1939to facilitate this feature, several small API changes have been made: 1940 1941The `spdk_nvme_transport_id` struct now contains a trstring member used to identify the transport. 1942A new function, `spdk_nvme_transport_available_by_name`, has been added. 1943A function table, `spdk_nvme_transport_ops`, and macro, `SPDK_NVME_TRANSPORT_REGISTER`, have been added which 1944enable registering out of tree transports. 1945 1946A new function, `spdk_nvme_ns_supports_compare`, allows a user to check whether a given namespace supports the compare 1947operation. 1948 1949A new family of functions, `spdk_nvme_ns_compare*`, give the user access to submitting compare commands to NVMe namespaces. 1950 1951A new function, `spdk_nvme_ctrlr_cmd_get_log_page_ext`, gives users more granular control over the command dwords sent in 1952log page requests. 1953 1954### nvmf 1955 1956Add SockPriority option in [Transport] section, this can be used for NVMe-oF target 1957on TCP transport to set sock priority for the incoming TCP connections. 1958 1959The NVMe-oF target now supports plugging out of tree NVMe-oF transports. In order 1960to facilitate this feature, several small API changes have been made: 1961 1962The `spdk_nvme_transport_id` struct now contains a trstring member used to identify the transport. 1963`spdk_nvmf_tgt_get_transport`, `spdk_nvmf_transport_opts_init`, and `spdk_nvmf_transport_create` all have been 1964modified to take a string. 1965A function table, `spdk_nvmf_transport_ops`, and macro, `SPDK_NVMF_TRANSPORT_REGISTER`, have been added which 1966enable registering out of tree transports. 1967 1968Add `spdk_nvmf_tgt_stop_listen()` that can be used to stop listening for 1969incoming connections for specified target and trid. Listener is not stopped 1970implicitly upon destruction of a subsystem any more. 1971 1972A custom NVMe admin command handler has been added which allows the user to use the real drive 1973attributes from one of the target NVMe drives when reporting drive attributes to the initiator. 1974This handler can be enabled via the `nvmf_set_config` RPC. 1975Note: In a future version of SPDK, this handler will be enabled by default. 1976 1977The SPDK target and initiator both now include compare-and-write functionality with one caveat. If using the RDMA transport, 1978the target expects the initiator to send both the compare command and write command either with, or without inline data. The 1979SPDK initiator currently respects this requirement, but this note is included as a flag for other initiators attempting 1980compatibility with this version of SPDK. 1981 1982The `spdk_nvme_ctrlr_opts` struct has been extended with new field `transport_ack_timeout` which allows 1983to configure transport ACK timeout. Applicable for RDMA transport only. 1984 1985### rpc 1986 1987A new RPC, `bdev_zone_block_create`, enables creating an emulated zoned bdev on top of a standard block device. 1988 1989A new RPC, `bdev_ocssd_create`, enables creating an emulated zoned bdev on top of an Open Channel SSD. 1990 1991A new RPC, `blobfs_set_cache_size`, enables managing blobfs cache size. 1992 1993A new RPC, `env_dpdk_get_mem_stats`, has been added to facilitate reading DPDK related memory 1994consumption stats. Please see the env_dpdk section above for more details. 1995 1996A new RPC, `framework_get_reactors`, has been added to retrieve a list of all reactors. 1997 1998`bdev_ftl_create` now takes a `base_bdev` argument in lieu of `trtype`, `traddr`, and `punits`. 1999 2000`bdev_nvme_set_options` now allows users to disable I/O submission batching with the `-d` flag 2001 2002`bdev_nvme_cuse_register` now accepts a `name` parameter. 2003 2004`bdev_uring_create` now takes arguments for `bdev_name` and `block_size` 2005 2006`nvmf_set_config` now takes an argument to enable passthru of identify commands to base NVMe devices. 2007Please see the nvmf section above for more details. 2008 2009### scsi 2010 2011`spdk_scsi_lun_get_dif_ctx` now takes an additional argument of type `spdk_scsi_task`. 2012 2013### sock 2014 2015Added spdk_sock_writev_async for performing asynchronous writes to sockets. This call will 2016never return EAGAIN, instead queueing internally until the data has all been sent. This can 2017simplify many code flows that create pollers to continue attempting to flush writes 2018on sockets. 2019 2020Added `impl_name` parameter in spdk_sock_listen and spdk_sock_connect functions. Users may now 2021specify the sock layer implementation they'd prefer to use. Valid implementations are currently 2022"vpp" and "posix" and NULL, where NULL results in the previous behavior of the functions. 2023 2024### thread 2025 2026`spdk_thread_send_msg` now returns int indicating if the message was successfully 2027sent. 2028 2029A new function `spdk_thread_send_critical_msg`, has been added to support sending a single message from 2030a context that may be interrupted, e.g. a signal handler. 2031 2032Two new functions, `spdk_poller_pause`, and `spdk_poller_resume`, have been added to give greater control 2033of pollers to the application owner. 2034 2035### util 2036 2037`spdk_pipe`, a new utility for buffering data from sockets or files for parsing 2038has been added. The public API is available at `include/spdk/pipe.h`. 2039 2040### bdev 2041 2042Added spdk_bdev_io_get_nvme_fused_status function for translating bdev_io status to NVMe status 2043code for fused compare-and-write operation. 2044 2045Added spdk_bdev_get_acwu function for getting block device atomic compare and write unit size. 2046 2047## v19.10: Zoned bdev API, Opal bdev, NVMe character devices 2048 2049### rpc 2050 2051Many of SPDK's RPCs were renamed to be more consistent and intuitive in this release. 2052The old names will continue to function, but will display a deprecation warning. 2053 2054Added optional parameters '--arbitration-burst' and '--low/medium/high-priority-weight' to 2055'bdev_nvme_set_options' RPC method. 2056 2057Added optional parameter '--md-size' to 'construct_null_bdev' RPC method. 2058 2059Added optional parameters '--dif-type' and '--dif-is-head-of-md' to 'construct_null_bdev' 2060RPC method. 2061 2062Added `blobfs_detect` RPC method to detect whether a blobfs exists on given bdev. 2063 2064Added `blobfs_create` RPC method to build blobfs on given bdev. 2065 2066Added `blobfs_mount` RPC method to mount blobfs on given bdev to a host path by FUSE. 2067Then on the host path, user can directly do some file operations which will be mapped 2068to blobfs. 2069 2070### bdev 2071 2072Added new parameter `cdw0` to `spdk_bdev_io_complete_nvme_status()` and 2073`spdk_bdev_io_get_nvme_status()` that allows setting/getting 2074the NVMe completion queue DW0 entry. This allows vendor specific IO commands 2075to return command specific completion info back to the initiator. 2076 2077Added `spdk_bdev_get_write_unit_size()` function for retrieving required number 2078of logical blocks for write operation. 2079 2080New zone-related fields were added to the result of the `get_bdevs` RPC call: 2081 2082- `zoned`: indicates whether the device is zoned or a regular 2083 block device 2084- `zone_size`: number of blocks in a single zone 2085- `max_open_zones`: maximum number of open zones 2086- `optimal_open_zones`: optimal number of open zones 2087 2088The `zoned` field is a boolean and is always present, while the rest is only available for zoned 2089bdevs. 2090 2091A new `spdk_bdev_open_ext` function has been added and `spdk_bdev_open` function has been deprecated. 2092The new open function introduces requirement to provide callback function that will be called by 2093asynchronous event such as bdev removal. `spdk_bdev_open_ext` function takes bdev name as 2094an argument instead of bdev structure to avoid a race condition that can happen when the bdev 2095is being removed between a call to get its structure based on a name and actually opening it. 2096 2097New 'resize' event has been added to notify about change of block count property of block device. 2098Event is delivered only if block device was opened with `spdk_bdev_open_ext` function. 2099 2100### bdev zone 2101 2102Added new public header for zoned bdev. Zoned bdev is an extension 2103of the bdev interface. 2104 2105`spdk_bdev_get_zone_size()`, `spdk_bdev_get_max_open_zones()`, `spdk_bdev_get_optimal_open_zones()` 2106APIs were added for retrieving zoned device information. 2107`spdk_bdev_get_zone_info()` API was added for retrieving information about zones in zoned 2108device. 2109Added `spdk_bdev_zone_management()` API for changing zone state. 2110`spdk_bdev_zone_append()` and `spdk_bdev_zone_append_with_md()` APIs were added for 2111appending data to a zone. 2112Added `spdk_bdev_io_get_append location()` function for retrieving append location for I/O. 2113Added `spdk_bdev_is_zoned()` function for checking if bdev supports zoned namespace semantics. 2114 2115### bdev opal 2116 2117EXPERIMENTAL: A new opal bdev has been added to support management of 2118NVMe self-encrypting drives through the Opal specification. Users can 2119create opal bdevs from an NVMe namespace bdev, if the controller 2120containing that namespace supports Opal. Currently this is only 2121supported for namespace ID=1. The following RPCs have been added to 2122support Opal: `bdev_nvme_opal_init`, `bdev_nvme_opal_revert`, 2123`bdev_opal_create`, `bdev_opal_delete`, `bdev_opal_get_info`, 2124`bdev_opal_new_user`, `bdev_opal_set_lock_state`. 2125It does not yet support recreating the opal bdevs after application restart. 2126This bdev module should be considered very experimental, and the RPCs may 2127change significantly in future releases. 2128 2129### delay bdev 2130 2131The `bdev_delay_update_latency` has been added to allow users to update 2132a latency value for a given delay bdev. 2133 2134### compress bdev 2135 2136A new RPC `bdev_compress_get_orphans` has been added to list compress bdevs 2137that were not loaded due to a missing pm metadata file. In this state they 2138can only be deleted. 2139 2140### null bdev 2141 2142Metadata support has been added to Null bdev module. 2143 2144Protection information support has been added to Null bdev module. 2145 2146### nvme 2147 2148Added `no_shn_notification` to NVMe controller initialization options, users can enable 2149it for NVMe controllers. When the option is enabled, the controller will not do the 2150shutdown process and just disable the controller, users can start their application 2151later again to initialize the controller to the ready state. 2152 2153A controller flag `SPDK_NVME_CTRLR_WRR_SUPPORTED` was added to indicate the controller 2154can support weighted round robin arbitration feature with submission queue. 2155 2156Added `arbitration_burst` option for arbitration feature, and added three 2157`low/medium/high_priority_weight` options for weighted round robin arbitration. 2158 2159Added `spdk_nvme_ns_cmd_write_uncorrectable`. 2160 2161Added new error handling and reporting functionality. This includes several 2162new API functions to facilitate applications recovering when a qpair or 2163controller fails. 2164 2165`spdk_nvme_ctrlr_reconnect_io_qpair` - Reconnects a failed I/O qpair. 2166`spdk_nvme_ctrlr_set_trid` - Sets the trid of an existing controller. Can be used to 2167change the trid for failover cases. 2168`spdk_nvme_ctrlr_is_failed` - Returns the failed state of a controller. 2169`spdk_nvme_ctrlr_fail` - Forces a controller into a failed state. 2170 2171Modified the return behavior of several API functions to better indicate to 2172applications when a qpair is failed. This list of functions includes: 2173 2174`spdk_nvme_qpair_process_completions` 2175`spdk_nvme_ns_cmd_*` 2176`spdk_nvme_ctrlr_process_admin_completions` 2177`spdk_nvme_ctrlr_cmd_*` 2178 2179These functions now return -ENXIO when the qpair or controller on which they 2180operate is failed. 2181 2182EXPERIMENTAL: Added NVMe character device support to allow to create NVMe device nodes in Linux 2183kernel for controller as well as for namespace and process ioctl requests as usual 2184from linux environment. 2185 2186### nvmf 2187 2188The `spdk_nvmf_tgt_create` function now accepts an object of type `spdk_nvmf_target_opts` 2189as its only parameter. This new structure contains the max_subsystems parameter previously 2190passed into that function. 2191 2192A new public API function `spdk_nvmf_get_tgt` has been added which allows users to 2193retrieve a pointer to an `spdk_nvmf_tgt` object by supplying its name. In the special 2194case where an RPC or application only creates a single target, this function can accept 2195a null name parameter and will return the only available target. 2196 2197The majority of the NVMe-oF RPCs now accept an optional tgt_name parameter. This will 2198allow those RPCs to work with applications that create more than one target. 2199 2200Three new NVMe-oF RPCs have been added `nvmf_create_target`, `nvmf_delete_target`, and 2201`nvmf_get_targets`. These new RPCs provide a basic interface for managing multiple target 2202objects. In SPDK the target object defines a unique discovery service. As of this release, 2203these RPCs are not intended to be used with the in-tree SPDK target applications, spdk_tgt and 2204nvmf_tgt, which use a single, global target structure. As such, they are not included in scripts/rpc.py 2205 2206Three new header functions have also been added to help deal with multiple targets. 2207`spdk_nvmf_tgt_get_name` takes a target pointer as an argument and returns its human readable name. 2208`spdk_nvmf_get_first_target` takes no arguments and returns the first target in the global list. 2209`spdk_nvmf_get_next_tgt` takes a target pointer as an argument and returns the next one in the global list. 2210 2211The `spdk_nvmf_tgt_accept` takes additional argument allowing to pass arbitrary context 2212information to the `new_qpair` callback. This will simplify the code when having multiple 2213nvmf targets or when retrieving the context information from globals is not suitable. 2214 2215### blobstore 2216 2217A new `spdk_bdev_create_bs_dev_from_desc` function has been added and `spdk_bdev_create_bs_dev` 2218function has been deprecated. 2219The new create function can cowork with `spdk_bdev_open_ext` function, which provides callback 2220function that will be called by asynchronous event such as bdev removal. 2221 2222### blobfs_bdev 2223 2224A new blobfs module `bdev` has been added to simplify the operations of blobfs on bdev. 2225 2226Function `spdk_blobfs_bdev_detect` is added to detect whether blobfs exists on the given block device. 2227 2228Function `spdk_blobfs_bdev_create` is added to create a blobfs on the given block device. 2229 2230Function `spdk_blobfs_bdev_mount` is added to mount a blobfs on the given block device to 2231a host path by FUSE. Then, a new thread is created dedicatedly for one mountpoint to handle 2232FUSE request by blobfs API. 2233 2234### build 2235 2236Option to build FUSE components into blobfs_bdev module for mounting a blobfs filesystem. 2237It requires the installation of libfuse3. By default, it is disabled. And it will be 2238enabled if run `./configure` with `--with-fuse` option. 2239 2240### iSCSI 2241 2242Portals may no longer be associated with a cpumask. The scheduling of 2243connections is moving to a more dynamic model. 2244 2245An new RPC `iscsi_portal_group_set_auth` has been added to set CHAP authentication 2246for discovery sessions specific for the existing iSCSI portal group. This RPC overwrites 2247the setting by the global parameters for the iSCSI portal group. 2248 2249### socket 2250 2251Added `spdk_sock_is_connected` to check whether the socket is currently connected. 2252`spdk_sock_group_poll` now returns number of events on success. 2253 2254### env 2255 2256Added `spdk_pci_device_unclaim()` function to cleanup pci claim file. 2257 2258### event 2259 2260`framework_start_init` RPC no longer stops the application on error during 2261initialization. 2262 2263### DPDK 2264 2265Updated DPDK submodule to DPDK 19.08. 2266 2267### ocf 2268 2269Updated OCF submodule to OCF v19.06 2270 2271Along with update, new cache mode 'write only' was added. 2272 2273New cache modes added to use via RPC, wi - write invalidate and wa - write around. 2274 2275New version of OCF provides fully asynchronous management API. 2276 2277## v19.07: NVMe-oF FC Transport, VMD, NVMe-oF Persistent reservations, Bdev I/O with separate metadata 2278 2279### ftl 2280 2281EXPERIMENTAL: Added ability to mirror writes to persistent write buffer cache 2282to allow for recovery from dirty shutdown event. 2283 2284Added handling of Asynchronous Nand Management Events (ANM). 2285 2286### vmd 2287 2288EXPERIMENTAL: Added Intel Volume Management Device (VMD) driver. VMD is an integrated 2289controller inside the CPU PCIe root complex. It enables virtual HBAs for the connected 2290NVMe SSDs. `spdk_vmd_init()` enumerates NVMe SSDs behind VMD device and hook them into 2291SPDK PCI subsystem. `spdk_nvme_probe()` or `spdk_nvme_connect()` can be used to connect 2292NVMe driver to the device located at the given transport ID. 2293 2294To obtain transport ID of NVMe SSD behind VMD `spdk_lspci` can be used. 2295 2296Current implementation does not support hotplug. 2297 2298### blobfs 2299 2300Blobfs file asynchronous operations were added to public APIs. 2301 2302### util 2303 2304A new file API `spdk_posix_file_load` was added to load file content into a data buffer. 2305 2306New APIs `spdk_dif_ctx_set_data_offset`, `spdk_dif_verify_stream`, 2307`spdk_dif_get_range_with_md`, `spdk_dif_get_length_with_md` have been added, 2308and existing APIs `spdk_dif_ctx_init`, `spdk_dif_set_md_interleave_iovs`, and 2309`spdk_dif_generate_stream` have been refined to insert or strip DIF into iovs 2310windows of any alignment. 2311 2312New APIs `spdk_dif_ctx_set_remapped_init_ref_tag`, `spdk_dif_remap_ref_tag`, 2313and `spdk_dix_remap_ref_tag` have been added to remap DIF reference tag. 2314 2315New APIs `spdk_dif_update_crc32c` and `spdk_dif_update_crc32c_stream` have been 2316added to compute CRC-32C checksum for extended LBA payload. 2317 2318Bdevperf and bdevio applications now support starting tests with application specific 2319RPCs. Please see helper Python scripts in their respective directories. 2320This is a move towards simpler RPC-only configuration for all main 2321and auxiliary applications. 2322 2323Legacy INI style configuration for SPDK applications will become deprecated in SPDK 19.10, 2324and removed in SPDK 20.01. Please consider moving to JSON-RPC configuration files and/or 2325RPC driven run-time configuration. 2326 2327### nvmf 2328 2329EXPERIMENTAL: A Fibre Channel transport that supports Broadcom HBAs has been 2330added. This depends on the FC HBA driver at 2331https://github.com/ecdufcdrvr/bcmufctdrvr. See [the documentation](https://spdk.io/doc/nvmf.html#nvmf_fc_transport) 2332for more information. 2333 2334Persistent reservation emulation has been added to the NVMe-oF target. Persistent reservation 2335state is stored in a JSON file on the local filesystem between target restart. To support this, 2336an optional parameter to the RPC method `nvmf_subsystem_add_ns` called `--ptpl-file` was added. 2337This allows the user to specify which file to store the persistent reservation state in. Note 2338that this is done per namespace. 2339 2340The c2h success optimization under which a command capsule response is not sent 2341for reads is turned on by default. A config knob was added to allow disabling 2342the optimization. This will mostly be used for integration testing with 5.0.x kernels 2343while some compatibility fixes make their way down the pipeline for 5.1.x kernels. 2344 2345The sock priority setting of the TCP connection owned by the tcp transport is added. It is 2346used to optimize the TCP connection performance under designated traffic classes. And the 2347priority is used to differentiate the sock priority between SPDK NVMe-oF TCP target application 2348and other TCP based applications. 2349 2350Shared receive queue can now be disabled even for NICs that support it using the 2351`nvmf_create_transport` RPC method parameter `no_srq`. The actual use of a shared 2352receive queue is predicated on hardware support when this flag is not used. 2353 2354spdk_nvmf_get_optimal_poll_group was added, which is used to return the optimal 2355poll group for the qpair. And `ConnectionScheduler` configuration is added into the 2356[Nvmf] section in etc/spdk/nvmf.conf.in to demonstrate how to configure the connection 2357scheduling strategy among different spdk threads. 2358 2359Added infrastructure to retrieve global and per poll group NVMf statistics. 2360 2361DIF strip and insert is now supported for TCP transport. When it is enabled, DIF 2362setting is not exposed to the NVMe-oF initiator, and DIF is attached into data 2363for write I/O and stripped from data for read I/O. 2364 2365Added a field `dif_insert_or_strip` to struct spdk_nvmf_transport_opts, and 2366updated the related rpc function nvmf_create_transport to make this 2367configurable parameter available to users. The `dif_insert_or_strip` is relevant 2368for TCP transport for now and used to configure the DIF strip and insert. 2369 2370Added infrastructure to retrieve NVMf transport statistics. 2371 2372### notify 2373 2374The function `spdk_notify_get_types()` and `spdk_notify_get_events()` were 2375renamed to `spdk_notify_foreach_type()` and `spdk_notify_foreach_event()`, 2376respectively. And update type name of callback accordingly. 2377 2378### bdev 2379 2380The format of the data returned by the get_bdevs_iostat RPC has changed to 2381make it easier to parse. It now returns an object with a "ticks" object 2382and "bdevs" array with the per-bdev statistics. 2383 2384A new bdev module `delay` has been added which simulates a drive latency when placed 2385on top of a Null bdev. This module is intended only for testing and can be created using 2386the new RPC `bdev_delay_create`. That RPC takes the name of the underlying bdev as well 2387as average and p99 latency arguments for both read and write operations. Average latency is 2388defined as a value close to what you would expect a perf tool such as FIO to report back as 2389the mean latency of all I/O submitted to the drive. p99 latency is defined as the value one 2390would expect the drive to see the slowest 1% of I/O report. For underlying drives with already 2391significant latency, the latency values provided to the drive will be additive. This should be 2392taken into account if trying to achieve an artificial latency on top of an nvme drive or aio device. 2393 2394DIF reference tag remapping is now supported for partition type virtual bdev 2395modules. When using partition type virtual bdevs, block address space is 2396remapped during I/O processing and DIF reference tag is remapped accordingly. 2397 2398Added spdk_bdev_*_with_md() functions allowing for IO with metadata being transferred in 2399separate buffer. To check support for separate metadata, use spdk_bdev_is_md_separate(). 2400 2401All bdevs now have a UUID. For devices whose backing hardware does not provide a UUID, 2402one is automatically generated. Across runs of SPDK, bdevs whose UUID is automatically 2403generated may change. 2404 2405A new virtual bdev module `compress` has been added to provide compression services on top of 2406a thinly provisioned logical volume. See documentation for complete details. 2407 2408### nvme 2409 2410Added an optional parameter `--io-queue-requests` to RPC `set_bdev_nvme_options`, which 2411can be used to change the number of requests allocated for one NVMe I/O queue. For 2412very big I/O size, e.g. 128MiB, with this option user will not get an error due to 2413limited requests in NVMe driver layer. 2414 2415Added spdk_nvme_ctrlr_get_transport_id() to get the transport ID from a 2416previously attached controller. 2417 2418Nvme Opal library spdk_opal_cmd deprecated. Adding separate command APIs. 2419NVMe Opal library add support for activating locking SP which will make the transaction 2420from "Manufactured-Inactive" state to "Manufactured" state. Upon successfully invoking 2421of this method, lock and unlock features will be enabled. 2422NVMe Opal library add support for locking/unlocking range and list locking range info. 2423NVMe opal library add support for multiuser. Admin can enable user and add user to specific 2424locking range and the user can lock/unlock his range. 2425 2426Added spdk_nvme_ctrlr_io_cmd_raw_no_payload_build() allowing a caller to pass 2427a completely formed command to an NVMe submission queue (buffer addresses and all). 2428This is supported on the PCIe transport only. 2429 2430Added spdk_nvme_get_ctrlr_registers() to return a pointer to the virtual address 2431of the NVMe controller registers. This is supported on the PCIe transport only. 2432 2433Added additional options to the spdk_nvme_ctrlr_alloc_qpair() option parameter 2434structure to allow caller to override the virtual and optionally physical address 2435of the submission and completion queue pair to be created. This is supported on 2436the PCIe transport only. 2437 2438Added `disable_error_logging` to struct spdk_nvme_ctrlr_opts, which disables 2439logging of failed requests. By default logging is enabled. 2440 2441Added spdk_nvme_qpair_print_command(), spdk_nvme_qpair_print_completion() and 2442spdk_nvme_cpl_get_status_string(). Allowing for easier display of error messages. 2443 2444Added support for NVMe Sanitize command. 2445 2446### env 2447 2448The parameter `free_space` has been added to spdk_ring_enqueue() to wait when 2449the ring is almost full and resume when there is enough space available in 2450the ring. 2451 2452A new API `spdk_mempool_lookup` has been added to lookup the memory pool created 2453by the primary process. 2454 2455Added spdk_pci_get_first_device() and spdk_pci_get_next_device() to allow 2456iterating over PCI devices detected by SPDK. Because of this, all SPDK APIs 2457to attach/detach PCI devices are no longer thread safe. They are now meant to 2458be called from only a single thread only, the same only that called spdk_env_init(). 2459This applies to the newly added APIs as well. 2460 2461### vpp 2462 2463SPDK now supports VPP version 19.04.2, up from VPP 18.01. 2464 2465VPP socket abstraction now uses VPP Session API, instead of VLC (VPP Communications Library). 2466This allows for better control over sessions and queues. 2467Please see VPP documentation for more details: 2468[VPP Host Stack](https://wiki.fd.io/view/VPP/HostStack) 2469 2470### sock 2471 2472Add spdk_sock_get_optimal_sock_group(), which returns the optimal sock group for 2473this socket. When a socket is created, it is often assigned to a sock group using 2474spdk_sock_group_add_sock so that a set of sockets can be polled more efficiently. 2475For some network devices, it is optimal to assign particular sockets to specific 2476sock groups. This API is intended to provide the user with that information. 2477 2478spdk_sock_group_get_ctx() was added to return the context of the spdk_sock_group. 2479spdk_sock_group_create() is updated to allow input the user provided ctx. 2480spdk_sock_set_priority() is added to set the priority of the socket. 2481 2482### rpc 2483 2484Added thread_get_stats RPC method to retrieve existing statistics. 2485 2486Added nvmf_get_stats RPC method to retrieve NVMf subsystem statistics. 2487 2488Response buffers for RPC requests are now always pre-allocated, which implies 2489that all spdk_jsonrpc_begin_result() calls always succeed and return a valid 2490buffer for JSON response. RPC calls no longer need to check if the buffer is 2491non-NULL. 2492 2493Added SPDK_RPC_REGISTER_ALIAS_DEPRECATED to help with deprecation process when 2494renaming existing RPC. First time a deprecated alias is used, it will print 2495a warning message. 2496 2497RPC `get_rpc_methods` was renamed `rpc_get_methods`. The old name is still usable, 2498but is now deprecated. 2499 2500### blobstore 2501 2502A snapshot can now be deleted if there is only a single clone on top of it. 2503 2504### build 2505 2506Preliminary support for cross compilation is now available. Targeting an older 2507CPU on the same architecture using your native compiler can be accomplished by 2508using the `--target-arch` option to `configure` as follows: 2509 2510~~~bash 2511./configure --target-arch=broadwell 2512~~~ 2513 2514Additionally, some support for cross-compiling to other architectures has been 2515added via the `--cross-prefix` argument to `configure`. To cross-compile, set CC 2516and CXX to the cross compilers, then run configure as follows: 2517 2518~~~bash 2519./configure --target-arch=aarm64 --cross-prefix=aarch64-linux-gnu 2520~~~ 2521 2522### vhost 2523 2524A security vulnerability has been identified and fixed in SPDK Vhost-SCSI target. 2525A malicious client (e.g. a virtual machine) could send a carefully prepared, 2526invalid I/O request to crash the entire SPDK process. All users of SPDK Vhost-SCSI 2527target are strongly recommended to update. All SPDK versions < 19.07 are affected. 2528 2529By default, SPDK will now rely on upstream DPDK's rte_vhost instead of its fork 2530located inside SPDK repo. The internal fork is still kept around to support older 2531DPDK versions, but is considered legacy and will be eventually removed. 2532 2533`configure` will now automatically use the upstream rte_vhost if the used DPDK 2534version is >= 19.05. 2535 2536spdk_vhost_init() is now asynchronous and accepts a completion callback. 2537 2538### iscsi target 2539 2540A security vulnerability has been identified and fixed in SPDK iSCSI target. 2541A malicious client (e.g. an iSCSI initiator) could send a carefully prepared, 2542invalid I/O request to crash the entire SPDK process. All users of SPDK iSCSI 2543target are strongly recommended to update. All SPDK versions < 19.07 are affected. 2544 2545### thread 2546 2547Exposed spdk_set_thread() in order for applications to associate 2548with SPDK thread when necessary. 2549 2550Added spdk_thread_destroy() to allow framework polling the thread to 2551release resources associated with that thread. 2552 2553## v19.04: Compression bdev, Notification library, NVMe Opal support 2554 2555### nvme 2556 2557Added asynchronous probe support. New APIs spdk_nvme_probe_async(), 2558spdk_nvme_connect_async() and spdk_nvme_probe_poll_async() were added to enable 2559this feature. spdk_nvme_probe_async() and spdk_nvme_connect_async() return a 2560context associated with the specified controllers. Users then call 2561spdk_nvme_probe_poll_async() until it returns 0, indicating that the operation 2562completed. 2563 2564A new qpair creation option, delay_pcie_doorbell, was added. This can be passed 2565to spdk_nvme_alloc_io_qpair(). This makes the I/O submission functions, such as 2566spdk_nvme_ns_writev(), skip ringing the submission queue doorbell. Instead the 2567doorbell will be rung as necessary inside spdk_nvme_qpair_process_completions(). 2568This can result in significantly fewer MMIO writes to the doorbell register 2569under heavy load, greatly improving performance. 2570 2571spdk_nvme_ctrlr_get_regs_cmbsz() was added to report the size of the controller 2572memory buffer, if available. 2573 2574spdk_nvme_ctrlr_get_flags() was added to return controller feature 2575flags. Two flags are currently tracked: 2576SPDK_NVME_CTRLR_SGL_SUPPORTED 2577SPDK_NVME_CTRLR_SECURITY_SEND_RECV_SUPPORTED 2578 2579The NVMe hotplug poller is now able to detach devices hot-removed from the system 2580via `/sys/bus/pci/devices/<bdf>/remove` and `/sys/bus/pci/devices/<bdf>/driver/unbind`. 2581 2582Opal support was added for scan, take ownership, revert TPer, and dumping device 2583info. The nvme_manage tool can be used to perform these operations. The public 2584API functions are spdk_nvme_ctrlr_security_receive() and 2585spdk_nvme_ctrlr_security_send(). This module should be considered experimental 2586pending additional features and tests. 2587 2588The NVMe-oF initiator is now able to transfer IO whose size is more than 128KiB 2589if the corresponding NVMe-oF target allows. 2590 2591### raid 2592 2593Added new strip_size_kb rpc param on create to replace the more ambiguous 2594strip_size. The strip_size rpc param is deprecated. 2595 2596Changed the raid bdev product_name from "Pooled Device" to "Raid Volume" 2597 2598### thread 2599 2600Added spdk_thread_has_pollers() function to verify if there are any registered 2601pollers to be run on the thread. Added spdk_thread_is_idle() function to check 2602if there are any scheduled operations to be performed on the thread at given 2603time. 2604 2605spdk_thread_create() now takes an optional CPU affinity mask that is passed to 2606the scheduler. 2607 2608spdk_thread_lib_init() now takes an optional context size. For each thread 2609created, an additional region of memory of this size will be allocated. A 2610pointer to this region of memory can be obtained by calling 2611spdk_thread_get_ctx(). The inverse operation is also available via 2612spdk_thread_get_from_ctx(). 2613 2614spdk_thread_poll() now optionally accepts the current time, in ticks. This can 2615be used to avoid some calls to spdk_get_ticks() internally. 2616 2617spdk_app_start() now only accepts a single context argument. 2618 2619### bdev 2620 2621An new API `spdk_bdev_get_data_block_size` has been added to get size of data 2622block except for metadata. 2623 2624spdk_vbdev_register() has been deprecated. spdk_bdev_register() should be used 2625instead. 2626 2627A mechanism for acquiring and releasing data buffers from bdev modules, used 2628to perform zero copy operations, was added. 2629 2630New APIs spdk_bdev_get_md_size(), spdk_bdev_is_md_interleaved(), spdk_bdev_get_dif_type(), 2631spdk_bdev_is_dif_head_of_md(), and spdk_bdev_is_dif_check_enabled() have been 2632added to get metadata and DIF settings. 2633 2634Bdevs claimed by the `examine_config` callback will be now further examined in the 2635`examine_disk` callback. 2636 2637spdk_bdev_io_get_io_channel() was added as a convenient way to get an io_channel 2638from a bdev_io. 2639 2640### ocf 2641 2642Updated OCF submodule to OCF v19.3.2 2643 2644Added support for many-to-one configuration for OCF bdev. 2645Multiple core devices can now be cached on single cache device. 2646 2647Added persistent metadata support, allowing to restore cache state after shutdown. 2648During start of SPDK application, the devices are examined and if OCF metadata 2649is present - appropriate OCF bdevs will be recreated. 2650 2651Added Write-Back mode support. In this mode, data is first written to 2652caching device and periodically synchronized to the core devices. 2653Dirty data is saved as persistent metadata on cache device, 2654allowing for safe restore during application restart. 2655For more details please see OCF documentation: 2656[OpenCAS cache configuration](https://open-cas.github.io/cache_configuration.html) 2657 2658### NVMe-oF Target 2659 2660Support for per-device shared receive queues in the RDMA transport has been added. 2661It is enabled by default for any device that supports it. 2662 2663The size of a shared receive queue is defined by transport configuration file parameter 2664`MaxSRQDepth` and `nvmf_create_transport` RPC method parameter `max_srq_depth`. 2665Default size is 4096. 2666 2667Add model number as parameter to construct_nvmf_subsystem (-d option), 2668rather than using hardcoded define. 2669 2670DIF passthrough feature has been added. DIF setting of the allocated bdevs is 2671exposed to the NVMe-oF initiator and data with DIF from the NVMe-oF initiator is 2672passed through to the allocated bdevs. 2673 2674### env 2675 2676The `phys_addr` parameter in spdk_malloc() and spdk_zmalloc() has been deprecated. 2677For retrieving physical addresses, spdk_vtophys() should be used instead. 2678 2679spdk_realloc() has been added to reallocate DMA/shared memory. 2680 2681spdk_pci_device_is_removed() has been added to let the upper-layer SPDK drivers know 2682that device has a pending external hotremove request. 2683 2684spdk_env_fini() and spdk_env_dpdk_post_fini() were added to release any resources 2685allocated by spdk_env_init() or spdk_env_dpdk_post_init() respectively. It is expected 2686that common usage of those functions is to call them just before terminating the process. 2687 2688Added spdk_iommu_is_enabled() to report if SPDK application is using IOMMU for DMA. 2689 2690### DPDK 2691 2692Dropped support for DPDK 17.07 and earlier, which SPDK won't even compile with right now. 2693 2694Updated DPDK submodule to DPDK 19.02. 2695 2696### rpc 2697 2698New `get_spdk_version` RPC method is introduced to get version info of the running SPDK application. 2699 2700The `start_nbd_disk` RPC method now take nbd_device as an optional parameter. If nbd_device 2701is specified, use that specified nbd device. If it's not specified, pick available one. 2702 2703### iSCSI target 2704 2705DIF strip and insert is now supported. DIF settings are not exposed to the iSCSI initiator. 2706DIF is attached into data for write I/O and stripped from data for read I/O. 2707 2708### vhost 2709 2710Added experimental support for running with the external, upstream rte_vhost library. 2711This can be enabled by configuring SPDK with an `--without-internal-vhost-lib` flag. 2712The minimum supported rte_vhost version (DPDK version) is 19.05-rc1. 2713 2714As a result of fuzz testing, a lot of data races in vhost-scsi LUN hotplug path were identified and 2715fixed. Those data races could have potentially resulted in SPDK crashes, RPC hangs, or memory leaks 2716if Vhost-SCSI LUN hotplug RPCs were executed while connected VMs were in the middle of restarting. 2717 2718The SCSI target id in `add_vhost_scsi_lun` RPC is now optional. If `-1` is passed, the first 2719unoccupied target id will be used. 2720 2721### AIO 2722 2723AIO bdev module can now reap I/O completions directly from userspace, significantly improving 2724the overall performance. 2725 2726### blobfs 2727 2728Synchronous IO operations no longer use spdk_io_channel, but instead use 2729spdk_fs_thread_ctx. The behavior is otherwise identical. 2730 2731### OCF 2732 2733Added support for caching multiple bdevs using a single bdev as a cache. 2734 2735### notify 2736 2737Added the notify library that provides a high performance local event bus 2738between libraries. Example usage was added to bdev module, which reports 2739notifications for added and removed bdevs via RPC. 2740 2741### sock 2742 2743Added new API spdk_sock_readv() to the sock library for performing vectored 2744reads. 2745 2746### event 2747 2748The function spdk_subsystem_init() no longer requires spdk_event as an argument. 2749 2750Changed API of spdk_subsystem_config_json() to no longer be asynchronous. 2751 2752### io_uring 2753 2754A bdev module that issues I/O to kernel block devices using the new io_uring Linux kernel 2755API was added. This module requires liburing. 2756 2757### build 2758 2759Options to easily compile with profile guided optimization have been added to 2760`configure`. To use profile guided optimization with SPDK, run 2761`./configure --with-pgo-capture`, build SPDK, then run a workload of your 2762choosing. Then, simply run `./configure --with-pgo-enable` and recompile to 2763build using the generated profile data. Profile guided optimization can yield 2764very large performance improvements, especially on GCC 8 and clang 7. This may 2765be combined with link time optimization which has been available under the 2766`--enable-lto` configure option for several releases. 2767 2768### compression bdev/reduce library 2769 2770Added "reduce" block compression scheme based on using SSDs for storing 2771compressed blocks of storage and persistent memory for metadata. Please see 2772[compression](https://spdk.io/doc/bdev.html) for more details. 2773 2774## v19.01: NVMe-oF TCP/IP Transport, Open Channel SSD Flash Translation Layer, Caching bdev based on OCF, ISA-L Support, DIF/DIX library 2775 2776### ocf bdev 2777 2778New virtual bdev module based on [Open CAS Framework](https://open-cas.github.io/) has been added. 2779This module allows for the use of one bdev to act as a high performance cache in front of another bdev. 2780Please see [documentation](https://spdk.io/doc/bdev.html#bdev_config_cas) for more details. 2781Only write through mode is currently supported and this feature is considered experimental. 2782 2783### event framework 2784 2785For `spdk_app_parse_args`, add return value to the callback which parses application 2786specific command line parameters to protect SPDK applications from crashing by invalid 2787values from user input. 2788 2789By default, all SPDK applications will now reserve all hugepages at runtime. The pre-reserved 2790memory size can be still set with `-s` or `--mem-size` option, although the default value 2791was reduced down to 0. 2792 2793A custom hugetlbfs directory can now be specified via spdk_app_opts. 2794This can be used to configure hugepages with different sizes, a different size limit, 2795or different access permissions than the system's default hugepage pool. 2796SPDK applications can specify a custom hugetlbfs mount with the `--huge-dir` option. 2797 2798### environment 2799 2800spdk_vtophys() has been refactored to accept length of the translated region as a new 2801parameter. The function will now update that parameter with the largest possible value 2802for which the memory is contiguous in the physical memory address space. 2803 2804The following functions were removed: 2805 2806- spdk_pci_nvme_device_attach() 2807- spdk_pci_nvme_enumerate() 2808- spdk_pci_ioat_device_attach() 2809- spdk_pci_ioat_enumerate() 2810- spdk_pci_virtio_device_attach() 2811- spdk_pci_virtio_enumerate() 2812 2813They were replaced with generic spdk_pci_device_attach() and spdk_pci_enumerate() which 2814require a new spdk_pci_driver object to be provided. It can be one of the following: 2815 2816- spdk_pci_nvme_get_driver() 2817- spdk_pci_ioat_get_driver() 2818- spdk_pci_virtio_get_driver() 2819 2820spdk_pci_hook_device() and spdk_pci_unhook_device() were added. Those allow adding a virtual 2821spdk_pci_device into the SPDK PCI subsystem. A virtual device calls provided callbacks for 2822each BAR mapping request or PCI config access. It's attachable with spdk_pci_device_attach() 2823or spdk_pci_enumerate() like any other device. 2824 2825A new spdk_pause() function was added to pause CPU execution for an implementation specific 2826amount of time. Quoting from DPDK function this is based on: "This call is intended for 2827tight loops which poll a shared resource or wait for an event. A short pause within the loop 2828may reduce the power consumption." 2829 2830A new public header file env_dpdk.h has been introduced, and function spdk_env_dpdk_post_init 2831is added into it. If user is using DPDK, and already called rte_eal_init, then include 2832include/spdk/env_dpdk.h, and call spdk_env_dpdk_post_init() instead of spdk_env_init. 2833 2834ISA-L has been added as an SPDK submodule. ISA-L is enabled by default on x86 architecture 2835to accelerate algorithms such as CRC for iSCSI and NVMe-oF. Users may still disable ISA-L 2836by explicitly passing --without-isal to the configure script. 2837 2838### util 2839 2840A new uuid API `spdk_uuid_copy` was added to make a copy of the source uuid. 2841 2842An new parameter `init_crc` representing the initial CRC value was added to 2843`spdk_crc16_t10dif`. The parameter can be used to calculate a CRC value spanning 2844multiple separate buffers. 2845 2846New DIF APIs were added to generate and verify DIF by byte granularity for both DIF and DIX 2847formats. Among them, DIF with copy APIs will be usable to emulate DIF operations such as DIF 2848insert and strip. 2849 2850Added `spdk_strtol` and `spdk_strtoll` to provide additional error checking around `strtol` 2851and `strtoll`. 2852 2853Added `spdk_sprintf_append_realloc` and `spdk_vsprintf_append_realloc` for appending a string 2854with automatic buffer re-allocation. 2855 2856### nvme 2857 2858Wrapper functions spdk_nvme_ctrlr_security_send() and spdk_nvme_ctrlr_security_receive() are 2859introduced to support further security protocol development. 2860 2861admin_timeout_ms was added to NVMe controller initialization options, users 2862can change the default value when probing a controller. 2863 2864Add two new fields "header_digest" and "data_digest" in struct spdk_nvme_ctrlr_opts, 2865it will be used to enable the digest support for the NVMe/TCP transport. 2866 2867Add a new TCP/IP transport(located in lib/nvme/nvme_tcp.c) in nvme driver. With 2868this new transport, it can be used to connect the NVMe-oF target with the 2869same TCP/IP support. 2870 2871Added API, spdk_nvme_ctrlr_is_discovery(), to indicate whether the ctrlr 2872arg refers to a Discovery Controller or not. 2873 2874Added an API function `spdk_nvme_host_id_parse` and corresponding object `spdk_nvme_host_id` 2875for parsing host address and host service ID arguments on a per connection basis. 2876 2877The RPC `construct_nvme_bdev` now allows a user to specify a source address and service id for the host to 2878use when connecting to the controller backing the NVMe bdev. 2879 2880### NVMe-oF Target 2881 2882The `spdk_nvmf_tgt_opts` struct has been deprecated in favor of `spdk_nvmf_transport_opts`. 2883Users will no longer be able to specify target wide I/O parameters. `spdk_nvmf_tgt_listen` 2884will also no longer implicitly initialize a transport with the default target options (since 2885there are none). Instead, a user must manually instantiate the transport with `spdk_nvmf_transport_create` 2886prior to calling `spdk_nvmf_tgt_listen`. 2887 2888Related to the previous change, the rpc `set_nvmf_target_options` has been renamed to 2889`set_nvmf_target_max_subsystems` to indicate that this is the only target option available for the user to edit. 2890 2891Added fields `num_shared_buffers` and `buf_cache_size` in struct spdk_nvmf_transport_opts, 2892and also updated the related rpc function nvmf_create_transport, to make this 2893configurable parameter available to users. The `num_shared_buffers` is used to 2894configure the shared buffer numbers of the transport used by RDMA or TCP transport. 2895`buf_cache_size` configures number of shared buffers to cache per poll group. 2896 2897### nvmf 2898 2899Add a new TCP/IP transport (located in lib/nvmf/tcp.c). With this transport, 2900the SPDK NVMe-oF target can have a new transport, and can serve the NVMe-oF 2901protocol via TCP/IP from the host. 2902 2903Added optional mechanism to modify the RDMA transport's behavior when creating protection domains and registering memory. 2904By default, the RDMA transport will use the ibverbs library to create protection domains and register memory. 2905Using `spdk_nvme_rdma_init_hooks` will subvert that and use an existing registration. 2906 2907### bdev 2908 2909Added `enable_bdev_histogram` and `get_bdev_histogram` RPC commands to allow gathering latency data for specified bdev. 2910Please see [documentation](https://spdk.io/doc/bdev.html#rpc_bdev_histogram) for more details. 2911 2912Added `required_alignment` field to `spdk_bdev`, that specifies an alignment requirement for data buffers associated with an spdk_bdev_io. 2913Bdev layer will automatically double buffer any spdk_bdev_io that violates this alignment, before the spdk_bdev_io is submitted to the bdev module. 2914 2915On shutdown, bdev unregister now proceeds in top-down fashion, with 2916claimed bdevs skipped (these will be unregistered later, when virtual 2917bdev built on top of the respective base bdev unclaims it). This 2918allows virtual bdevs to be shut down cleanly as opposed to the 2919previous behavior that didn't differentiate between hotremove and 2920planned shutdown. 2921 2922The `delete_bdev` RPC is now deprecated. Users should instead use the specific deletion RPC 2923for the bdev type to be removed (i.e. delete_malloc_bdev). 2924 2925Added support for separate bandwidth rate limits for read and write to QoS in bdev layer. 2926 2927Bdev I/O statistics now track unmap operations. 2928 2929### logical volumes 2930 2931Logical volume bdev can now be marked as read only using `set_read_only_lvol_bdev` RPC. 2932This allows for basing clones on top of lvol_bdev without first creating a snapshot. 2933 2934Added option to change method for data erasure when deleting lvol or resizing down. 2935Default of unmapping clusters can now be changed to writing zeroes or no operation. 2936 2937Added option to change method for erasing data region on lvol store creation. 2938Default of unmapping can now be changed to writing zeroes or no operation. 2939 2940### log 2941 2942"trace flags" are now referred to as "log flags" in the SPDK log API. The 2943set_trace_flag, clear_trace_flag and get_trace_flags RPCs are now deprecated, 2944and set_log_flag, clear_log_flag and get_log_flags RPCs have been added. 2945 2946### trace 2947 2948New `trace_record` application was added. It can be used to poll spdk trace shm file and 2949append any new trace entries into another specified file. This can help retain those entries 2950that would otherwise be overwritten in the shm file. See 2951[Capturing sufficient trace events](https://spdk.io/doc/nvmf_tgt_tracepoints.html#capture_trace_events) 2952for more details. 2953 2954Number of trace entries in circular buffer per lcore can now be assigned by starting SPDK app 2955with argument "--num-trace-entries <NUM>" provided. 2956 2957New `get_tpoint_group_mask` RPC was added to get current tpoint_group_mask, and 2958each tpoint group status. 2959New `enable_tpoint_group` and `disable_tpoint_group` RPC were added to enable or 2960disable a specific tpoint group. 2961 2962### ftl 2963 2964EXPERIMENTAL: Added basic flash translation layer module allowing for using Open Channel SSDs as 2965block devices. The module is split into the library (located in lib/ftl) and bdev_ftl 2966(lib/bdev/ftl). See the [documentation](https://spdk.io/doc/ftl.html) for more details. 2967 2968### vhost 2969 2970A security vulnerability has been identified and fixed in the SPDK vhost target. A malicious 2971vhost client (i.e. virtual machine) could carefully construct a circular descriptor chain which 2972would result in a partial denial of service in the SPDK vhost target. These types of descriptor 2973chains are now properly detected by the vhost target. All SPDK vhost users serving untrusted 2974vhost clients are strongly recommended to upgrade. (Reported by Dima Stepanov and Evgeny 2975Yakovlev.) 2976 2977Vhost SCSI and Vhost Block devices can now accept multiple connections on the same socket file. 2978Each connection (internally called a vhost session) will have access to the same storage, but 2979will use different virtqueues, different features and possibly different memory. 2980 2981### vhost scsi 2982 2983SCSI target hotremove can now be performed even without the VIRTIO_SCSI_F_HOTPLUG feature negotiated. 2984Regardless of VIRTIO_SCSI_F_HOTPLUG support, the hotremoval will be still reported through SCSI sense codes. 2985 2986### DPDK 2987 2988DPDK submodule was updated to DPDK 18.11. Note that SPDK does not fully leverage the new 2989multi-process device hotplug yet and continues to work the same way it always did. 2990 2991Dropped support for DPDK 16.07 and earlier, which SPDK won't even compile with right now. 2992 2993### RPC 2994 2995The following RPC commands deprecated in the previous release are now removed: 2996 2997- construct_virtio_user_scsi_bdev 2998- construct_virtio_pci_scsi_bdev 2999- construct_virtio_user_blk_bdev 3000- construct_virtio_pci_blk_bdev 3001- remove_virtio_scsi_bdev 3002- construct_nvmf_subsystem 3003 3004### Miscellaneous 3005 3006The configure options `--with-raid` and `--without-raid` that were deprecated in the previous 3007release are now removed. 3008 3009### nbd 3010 3011Starting nbd using `spdk_nbd_start` is now performed asynchronously. 3012 3013### net framework 3014 3015Net framework initialization and finish is now done asynchronously. 3016 3017### rpc 3018 3019Added `spdk_rpc_is_method_allowed` function for checking whether method is permitted in a given state. 3020Added `spdk_rpc_get_state` to check current state of RPC server. 3021RPC `wait_subsystem_init` has been added to allow clients to block until all subsystems are initialized. 3022 3023### json rpc 3024 3025JSON RPC client is now running in non-blocking mode. Requests are sent and received during spdk_jsonrpc_client_poll. 3026JSON RPC server can now receive a callback on connection termination or server shutdown using `spdk_jsonrpc_conn_add_close_cb` 3027and `spdk_jsonrpc_conn_del_close_cb`. 3028 3029## v18.10: Dynamic memory allocation, Crypto Virtual Bdev, jsonrpc-client, SPDKCLI iSCSI and NVMe-oF support 3030 3031### nvme 3032 3033spdk_nvme_ctrlr_cmd_security_send() and spdk_nvme_ctrlr_cmd_security_receive() 3034were added to support sending or receiving security protocol data to or from 3035nvme controller. 3036 3037spdk_nvme_ns_get_extended_sector_size() was added. This function includes 3038the metadata size per sector (if any). spdk_nvme_ns_get_sector_size() still 3039returns only the data size per sector, not including metadata. 3040 3041New `send_nvme_cmd` RPC was added to allow sending NVMe commands directly to NVMe controller. 3042See the [send_nvme_cmd](http://spdk.io/doc/jsonrpc.html#rpc_send_nvme_cmd) documentation 3043for more details. 3044 3045### Build System 3046 3047New `configure` options, `--with-shared` and `--without-shared` 3048[default], provide the capability to build, or not, SPDK shared libraries. 3049This includes the single SPDK shared lib encompassing all of the SPDK 3050static libs as well as individual SPDK shared libs corresponding to 3051each of the SPDK static ones. Although the production of the shared 3052libs conforms with conventional version naming practices, such naming 3053does not at this time confer any SPDK ABI compatibility claims. 3054 3055### bdev 3056 3057spdk_bdev_alias_del_all() was added to delete all alias from block device. 3058 3059A new virtual bdev module has been added to perform at rest data encryption using the DPDK CryptoDev 3060Framework. The module initially uses a software AESNI CBC cipher with experimental support for the 3061Intel QAT hardware accelerator also currently implemented with support for CBC cipher. Future work 3062may include additional ciphers as well as consideration for authentication. 3063 3064The RAID virtual bdev module is now always enabled by default. The configure --with-raid and 3065--without-raid options are now ignored and deprecated and will be removed in the next release. 3066 3067Enforcement of bandwidth limits for quality of service (QoS) has been added to the bdev layer. 3068See the new [set_bdev_qos_limit](http://www.spdk.io/doc/jsonrpc.html#rpc_set_bdev_qos_limit) 3069documentation for more details. The previous set_bdev_qos_limit_iops RPC method introduced at 307018.04 release has been deprecated. The new set_bdev_qos_limit RPC method can support both 3071bandwidth and IOPS limits. 3072 3073spdk_bdev_config_json() and corresponding `get_bdevs_config` RPC was removed. 3074 3075### Environment Abstraction Layer and Event Framework 3076 3077The size parameter of spdk_mem_map_translate is now a pointer. This allows the 3078function to report back the actual size of the translation relative to the original 3079request made by the user. 3080 3081A new structure spdk_mem_map_ops has been introduced to hold memory map related 3082callbacks. This structure is now passed as the second argument of spdk_mem_map_alloc 3083in lieu of the notify callback. 3084 3085### DPDK 18.08 3086 3087The DPDK submodule has been updated to the DPDK 18.08 release. SPDK will now automatically 3088utilize DPDK's dynamic memory management with DPDK versions >= 18.05.1. 3089 3090Hugepages can be still reserved with `[-s|--mem-size <size>]` option at application startup, 3091but once we use them all up, instead of failing user allocations with -ENOMEM, we'll try 3092to dynamically reserve even more. This allows starting SPDK with `--mem-size 0` and using 3093only as many hugepages as it is really needed. 3094 3095Due to this change, the memory buffers returned by `spdk_*malloc()` are no longer guaranteed 3096to be physically contiguous. 3097 3098### I/OAT 3099 3100I/OAT driver can now reinitialize I/OAT channels after encountering DMA errors. 3101 3102### iscsi target 3103 3104Parameter names of `set_iscsi_options` and `get_iscsi_global_params` RPC 3105method for CHAP authentication in discovery sessions have been changed to 3106align with `construct_target_node` RPC method. Old names are still usable 3107but will be removed in future release. 3108 3109`set_iscsi_discovery_auth` and `set_iscsi_target_node_auth` RPC methods have 3110been added to set CHAP authentication for discovery sessions and existing 3111target nodes, respectively. 3112 3113The SPDK iSCSI target supports an AuthFile which can be used to load CHAP 3114shared secrets when the iSCSI target starts. SPDK previously provided a 3115default location for this file (`/usr/local/etc/spdk/auth.conf`) if none was 3116specified. This default has been removed. Users must now explicitly specify 3117the location of this file to load CHAP shared secrets from a file, or use 3118the related iSCSI RPC methods to add them at runtime. 3119 3120### iscsi initiator 3121 3122The SPDK iSCSI initiator is no longer considered experimental and becomes 3123a first-class citizen among bdev modules. The basic usage has been briefly 3124described in the bdev user guide: [iSCSI bdev](https://spdk.io/doc/bdev.html#bdev_config_iscsi) 3125 3126### Miscellaneous 3127 3128The SPDK application framework can now parse long name command line parameters. 3129Most single-character parameters have a long name equivalent now. See the 3130[Command Line Parameters](https://spdk.io/doc/app_overview.html) documentation 3131for details or use the `--help` command line parameter to list all available 3132params. 3133 3134bdevperf `-s` param (io size) was renamed to `-o` as `-s` had been already 3135used by existing apps for memory size. 3136 3137bdevio can now accept all SPDK command line parameters. The config now has to 3138be provided with `-c` or `--config` param. 3139 3140The following ioat/perf and nvme/perf parameters were renamed as well: 3141 `-s` (io size) to `-o` 3142 `-d` (mem size) to `-s` 3143 3144The ReactorMask config file parameter has been deprecated. Users should 3145use the -m or --cpumask command line option to specify the CPU core mask 3146for the application. 3147 3148Default config file pathnames have been removed from iscsi_tgt, nvmf_tgt 3149and vhost. Config file pathnames may now only be specified using the 3150-c command line option. 3151 3152Users may no longer set DPDK_DIR in their environment to specify the 3153location of the DPDK installation used to build SPDK. Using DPDK_DIR 3154has not been the documented nor recommended way to specify the DPDK 3155location for several releases, but removing it ensures no unexpected 3156surprises for users who may have DPDK_DIR defined for other reasons. 3157Users should just use the "configure" script to specify the DPDK 3158location before building SPDK. 3159 3160Although we know that many developers still use Python 2 we are officially 3161switching to Python3 with requirement that all new code must be valid also 3162for Python 2 up to the EOL which is year 2020. 3163 3164Invoking interpreter explicitly is forbidden for executable scripts. There 3165is no need to use syntax like "python ./scripts/rpc.py". All executable 3166scripts must contain proper shebang pointing to the right interpreter. 3167Scripts without shebang mustn't be executable. 3168 3169A Python script has been added to enable conversion of old INI config file 3170to new JSON-RPC config file format. This script can be found at 3171scripts/config_converter.py. Example how this script can be used: 3172~~~{.sh} 3173cat old_format.ini | scripts/config_converter.py > new_json_format.json 3174~~~ 3175 3176### Sock 3177 3178Two additional parameters were added to spdk_sock_get_addr() for the server 3179port and client port. These parameters are named "sport" and "cport" 3180respectively. 3181 3182### Virtio 3183 3184The following RPC commands have been deprecated: 3185 3186- construct_virtio_user_scsi_bdev 3187- construct_virtio_pci_scsi_bdev 3188- construct_virtio_user_blk_bdev 3189- construct_virtio_pci_blk_bdev 3190- remove_virtio_scsi_bdev 3191 3192The `construct_virtio_*` ones were replaced with a single `construct_virtio_dev` 3193command that can create any type of Virtio bdev(s). `remove_virtio_scsi_bdev` 3194was replaced with `remove_virtio_bdev` that can delete both Virtio Block and SCSI 3195devices. 3196 3197### Blobfs 3198 3199spdk_file_get_id() returning unique ID for the file was added. 3200 3201### JSON 3202 3203Added jsonrpc-client C library intended for issuing RPC commands from applications. 3204 3205Added API enabling iteration over JSON object: 3206 3207- spdk_json_find() 3208- spdk_json_find_string() 3209- spdk_json_find_array() 3210- spdk_json_object_first() 3211- spdk_json_array_first() 3212- spdk_json_next() 3213 3214### Blobstore 3215 3216Blobstore I/O operations are now based on io_units, instead of blobstore page size. 3217The io_unit size is now the same as the underlying block device's block size. 3218Logical volumes built on a block device with 512B block size can now be used as boot devices 3219in QEMU. 3220 3221### SPDKCLI 3222 3223The SPDKCLI interactive command tool for managing SPDK is no longer considered experimental. 3224Support for the iSCSI and NVMe-oF targets has been added. 3225 3226## v18.07: Raid, Infrastructure Improvements, Bug Fixes 3227 3228### bdev 3229 3230A new public header file bdev_module.h has been introduced to facilitate the 3231development of new bdev modules. This header includes an interface for the 3232spdk_bdev_part and spdk_bdev_part_base objects to enable the creation of 3233multiple virtual bdevs on top of a single base bdev and should act as the 3234primary API for module authors. 3235 3236spdk_bdev_get_opts() and spdk_bdev_set_opts() were added to set bdev-wide 3237options. 3238 3239A mechanism for handling out of memory condition errors (ENOMEM) returned from 3240I/O submission requests at the bdev layer has been added. See 3241spdk_bdev_queue_io_wait(). 3242 3243The spdk_bdev_get_io_stat() function now returns cumulative totals instead of 3244resetting on each call. This allows multiple callers to query I/O statistics 3245without conflicting with each other. Existing users will need to adjust their 3246code to record the previous I/O statistics to calculate the delta between calls. 3247 3248I/O queue depth tracking and samples options have been added. See 3249spdk_bdev_get_qd(), spdk_bdev_get_qd_sampling_period(), and 3250spdk_bdev_set_qd_sampling_period(). 3251 3252### RAID module 3253 3254A new bdev module called "raid" has been added as experimental module which 3255aggregates underlying NVMe bdevs and exposes a single raid bdev. Please note 3256that vhost will not work with this module because it does not yet have support 3257for multi-element io vectors. 3258 3259### Log 3260 3261The debug log component flag available on several SPDK applications has been 3262renamed from `-t` to `-L` to prevent confusion with tracepoints and to allow the 3263option to be added to tools that already use `-t` to mean something else. 3264 3265### Blobstore 3266 3267A new function, spdk_bs_dump(), has been added that dumps all of the contents of 3268a blobstore to a file pointer. This includes the metadata and is very useful for 3269debugging. 3270 3271Two new operations have been added for thin-provisioned blobs. 3272spdk_bs_inflate_blob() will allocate clusters for all thinly provisioned regions 3273of the blob and populate them with the correct data by reading from the backing 3274blob(s). spdk_bs_blob_decouple_parent() works similarly, but will only allocate 3275clusters that correspond to data in the blob's immediate parent. Clusters 3276allocated to grandparents or that aren't allocated at all will remain 3277thin-provisioned. 3278 3279### BlobFS 3280 3281Changed the return type of spdk_file_truncate() from void to int to allow the 3282propagation of `ENOMEM` errors. 3283 3284### NVMe Driver 3285 3286The new API functions spdk_nvme_qpair_add_cmd_error_injection() and 3287spdk_nvme_qpair_remove_cmd_error_injection() have been added for NVMe error 3288emulation. Users can set a specified command to fail with a particular error 3289status. 3290 3291Changed the name `timeout_sec` parameter to `timeout_us` in 3292spdk_nvme_ctrlr_register_timeout_callback(), and also changed the type from 3293uint32_t to uint64_t. This will give users more fine-grained control over the 3294timeout period. 3295 3296Basic support for Open Channel SSDs was added. See nvme_ocssd.h 3297 3298### NVMe Over Fabrics 3299 3300The spdk_nvmf_tgt_destroy() function is now asynchronous and takes a callback 3301as a parameter. 3302 3303spdk_nvmf_qpair_disconnect() was added to allow the user to disconnect qpairs. 3304 3305spdk_nvmf_subsystem_get_max_namespaces() was added to query the maximum allowed 3306number of namespaces for a given subsystem. 3307 3308### Build System 3309 3310The build system now generates a combined shared library (libspdk.so) that may 3311be used in place of the individual static libraries (libspdk_*.a). The combined 3312library includes all components of SPDK and is intended to make linking against 3313SPDK easier. The static libraries are also still provided for users that prefer 3314to link only the minimal set of components required. 3315 3316### git pre-commit and pre-push hooks 3317 3318The pre-commit hook will run `scripts/check_format.sh` and verify there are no 3319formatting errors before allowing `git commit` to run. The pre-push hook runs 3320`make CONFIG_WERROR=y` with and without `CONFIG_DEBUG=y` using both the gcc and 3321clang compiler before allowing `git push` to run. Following each DEBUG build 3322`test/unit/unittest.sh` is run and verified. Results are recorded in the 3323`make.log` file. 3324 3325To enable type: 'git config core.hooksPath .githooks'. To override after 3326configuration use the `git --no-verify` flag. 3327 3328### RPC 3329 3330The `start_nbd_disk` RPC method now returns the path to the kernel NBD device node 3331rather than always returning `true`. 3332 3333### DPDK 18.05 3334 3335The DPDK submodule has been rebased on the DPDK 18.05 release. DPDK 18.05 supports 3336dynamic memory allocation, but due to some issues found after the DPDK 18.05 release, 3337that support is not enabled for SPDK 18.07. Therefore, SPDK 18.07 will continue to use 3338the legacy memory allocation model. The plan is to enable dynamic memory allocation 3339after the DPDK 18.08 release which should fix these issues. 3340 3341### Environment Abstraction Layer and Event Framework 3342 3343The spdk_mem_map_translate() function now takes a size parameter to indicate the size of 3344the memory region. This can be used by environment implementations to validate the 3345requested translation. 3346 3347The I/O Channel implementation has been moved to its own library - lib/thread. The 3348public API that was previously in spdk/io_channel.h is now in spdk/thread.h The 3349file spdk/io_channel.h remains and includes spdk/thread.h. 3350 3351spdk_reactor_get_tsc_stats was added to return interesting statistics for each 3352reactor. 3353 3354### IOAT 3355 3356IOAT for copy engine is disabled by default. It can be enabled by specifying the Enable 3357option with "Yes" in `[Ioat]` section of the configuration file. The Disable option is 3358now deprecated and will be removed in a future release. 3359 3360## v18.04: Logical Volume Snapshot/Clone, iSCSI Initiator, Bdev QoS, VPP Userspace TCP/IP 3361 3362### vhost 3363 3364The SPDK vhost-scsi, vhost-blk and vhost-nvme applications have fixes to address the 3365DPDK rte_vhost vulnerability [CVE-2018-1059](http://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2018-1059). 3366Please see this [security advisory](https://access.redhat.com/security/cve/cve-2018-1059) 3367for additional information on the DPDK vulnerability. 3368 3369Workarounds have been added to ensure vhost compatibility with QEMU 2.12. 3370 3371EXPERIMENTAL: Support for vhost-nvme has been added to the SPDK vhost target. See the 3372[vhost documentation](http://www.spdk.io/doc/vhost.html) for more details. 3373 3374### Unified Target Application 3375 3376A new unified SPDK target application, `spdk_tgt`, has been added. This application combines the 3377functionality of several existing SPDK applications, including the iSCSI target, NVMe-oF target, 3378and vhost target. The new application can be managed through the existing configuration file and 3379[JSON-RPC](http://www.spdk.io/doc/jsonrpc.html) methods. 3380 3381### Env 3382 3383spdk_mempool_get_bulk() has been added to wrap DPDK rte_mempool_get_bulk(). 3384 3385New memory management functions spdk_malloc(), spdk_zmalloc(), and spdk_free() have been added. 3386These new functions have a `flags` parameter that allows the user to specify whether the allocated 3387memory needs to be suitable for DMA and whether it should be shared across processes with the same 3388shm_id. The new functions are intended to replace spdk_dma_malloc() and related functions, which will 3389eventually be deprecated and removed. 3390 3391### Bdev 3392 3393A new optional bdev module interface function, `init_complete`, has been added to notify bdev modules 3394when the bdev subsystem initialization is complete. This may be useful for virtual bdevs that require 3395notification that the set of initialization examine() calls is complete. 3396 3397The bdev layer now allows modules to provide an optional per-bdev UUID, which can be retrieved with 3398the spdk_bdev_get_uuid() function. 3399 3400Enforcement of IOPS limits for quality of service (QoS) has been added to the bdev layer. See the 3401[set_bdev_qos_limit_iops](http://www.spdk.io/doc/jsonrpc.html#rpc_set_bdev_qos_limit_iops) documentation 3402for more details. 3403 3404### RPC 3405 3406The `[Rpc]` configuration file section, which was deprecated in v18.01, has been removed. 3407Users should switch to the `-r` command-line parameter instead. 3408 3409The JSON-RPC server implementation now allows up to 32 megabyte responses, growing as 3410needed; previously, the response was limited to 32 kilobytes. 3411 3412### SPDKCLI 3413 3414EXPERIMENTAL: New SPDKCLI interactive command tool for managing SPDK is available. 3415See the [SPDKCLI](http://www.spdk.io/doc/spdkcli.html) documentation for more details. 3416 3417### NVMe Driver 3418 3419EXPERIMENTAL: Support for WDS and RDS capable CMBs in NVMe controllers has been added. This support is 3420experimental pending a functional allocator to free and reallocate CMB buffers. 3421 3422spdk_nvme_ns_get_uuid() has been added to allow retrieval of per-namespace UUIDs when available. 3423 3424New API functions spdk_nvme_ctrlr_get_first_active_ns() and spdk_nvme_ctrlr_get_next_active_ns() 3425have been added to iterate active namespaces, as well as spdk_nvme_ctrlr_is_active_ns() to check if 3426a namespace ID is active. 3427 3428### NVMe-oF Target 3429 3430Namespaces may now be assigned unique identifiers via new optional `eui64` and `nguid` parameters 3431to the `nvmf_subsystem_add_ns` RPC method. Additionally, the NVMe-oF target automatically exposes 3432the backing bdev's UUID as the namespace UUID when available. 3433 3434spdk_nvmf_subsystem_remove_ns() is now asynchronous and requires a callback to indicate completion. 3435 3436### Blobstore 3437 3438A number of functions have been renamed: 3439 3440- spdk_bs_io_write_blob() => spdk_blob_io_write() 3441- spdk_bs_io_read_blob() => spdk_blob_io_read() 3442- spdk_bs_io_writev_blob() => spdk_blob_io_writev() 3443- spdk_bs_io_readv_blob() => spdk_blob_io_readv() 3444- spdk_bs_io_unmap_blob() => spdk_blob_io_unmap() 3445- spdk_bs_io_write_zeroes_blob() => spdk_blob_io_write_zeroes() 3446 3447The old names still exist but are deprecated. They will be removed in the v18.07 release. 3448 3449spdk_blob_resize() is now an asynchronous operation to enable resizing a blob while I/O 3450are in progress to that blob on other threads. An explicit spdk_blob_sync_md() is still 3451required to sync the updated metadata to disk. 3452 3453### Logical Volumes 3454 3455A new `destroy_lvol_bdev` RPC method to delete logical volumes has been added. 3456 3457Lvols now have their own UUIDs which replace previous LvolStoreUUID_BlobID combination. 3458 3459New Snapshot and Clone functionalities have been added. User may create Snapshots of existing Lvols 3460and Clones of existing Snapshots. 3461See the [lvol snapshots](http://www.spdk.io/doc/logical_volumes.html#lvol_snapshots) documentation 3462for more details. 3463 3464Resizing logical volumes is now supported via the `resize_lvol_bdev` RPC method. 3465 3466### Lib 3467 3468A set of changes were made in the SPDK's lib code altering 3469instances of calls to `exit()` and `abort()` to return a failure instead 3470wherever reasonably possible. 3471 3472spdk_app_start() no longer exit()'s on an internal failure, but 3473instead returns a non-zero error status. 3474 3475spdk_app_parse_args() no longer exit()'s on help, '-h', or an invalid 3476option, but instead returns SPDK_APP_PARSE_ARGS_HELP and 3477SPDK_APP_PARSE_ARGS_FAIL, respectively, and SPDK_APP_PARSE_ARGS_SUCCESS 3478on success. 3479 3480spdk_pci_get_device() has been deprecated and will be removed in SPDK v18.07. 3481 3482### I/O Channels 3483 3484The prototype for spdk_poller_fn() has been modified; it now returns a value indicating 3485whether or not the poller did any work. Existing pollers will need to be updated to 3486return a value. 3487 3488### iSCSI Target 3489 3490The SPDK iSCSI target now supports the fd.io Vector Packet Processing (VPP) framework userspace 3491TCP/IP stack. See the [iSCSI VPP documentation](http://www.spdk.io/doc/iscsi.html#vpp) for more 3492details. 3493 3494### iSCSI initiator 3495 3496An iSCSI initiator bdev module has been added to SPDK. This module should be considered 3497experimental pending additional features and tests. More details can be found in 3498lib/bdev/iscsi/README. 3499 3500### PMDK 3501 3502The persistent memory (PMDK) bdev module is now enabled using `--with-pmdk` instead of 3503`--with-nvml`. This reflects the renaming of the persistent memory library from NVML to 3504PMDK. 3505 3506### Virtio Block driver 3507 3508A userspace driver for Virtio Block devices has been added. It was built on top of the 3509[Virtio](http://www.spdk.io/doc/virtio.html) library and can be managed similarly to 3510the Virtio SCSI driver. See the 3511[Virtio Block](http://www.spdk.io/doc/bdev.html#bdev_config_virtio_blk) reference for 3512more information. 3513 3514### Virtio with 2MB hugepages 3515 3516The previous 1GB hugepage limitation has now been lifted. A new `-g` command-line option 3517enables SPDK Virtio to work with 2MB hugepages. 3518See [2MB hugepages](http://www.spdk.io/doc/virtio.html#virtio_2mb) for details. 3519 3520## v18.01: Blobstore Thin Provisioning 3521 3522### Build System 3523 3524The build system now includes a `make install` rule, including support for the common 3525`DESTDIR` and `prefix` variables as used in other build systems. Additionally, the prefix 3526may be set via the configure `--prefix` option. Example: `make install prefix=/usr`. 3527 3528### RPC 3529 3530A JSON RPC listener is now enabled by default using a UNIX domain socket at /var/tmp/spdk.sock. 3531A -r option command line option has been added to enable an alternative UNIX domain socket location, 3532or a TCP port in the format ip_addr:tcp_port (i.e. 127.0.0.1:5260). The Rpc configuration file 3533section is now deprecated and will be removed in the v18.04 release. 3534 3535### I/O Channels 3536 3537spdk_poller_register() and spdk_poller_unregister() were moved from the event 3538framework (include/spdk/event.h) to the I/O channel library 3539(include/spdk/io_channel.h). This allows code that doesn't depend on the event 3540framework to request registration and unregistration of pollers. 3541 3542spdk_for_each_channel() now allows asynchronous operations during iteration. 3543Instead of immediately continuing the interaction upon returning from the iteration 3544callback, the user must call spdk_for_each_channel_continue() to resume iteration. 3545 3546### Block Device Abstraction Layer (bdev) 3547 3548The poller abstraction was removed from the bdev layer. There is now a general purpose 3549abstraction for pollers available in include/spdk/io_channel.h 3550 3551### Lib 3552 3553A set of changes were made in the SPDK's lib code altering, 3554instances of calls to `exit()` and `abort()` to return a failure instead 3555wherever reasonably possible. This has resulted in return type changes of 3556the API for: 3557 3558- spdk_env_init() from type `void` to `int`. 3559- spdk_mem_map_init() from type `void` to `int`. 3560 3561Applications making use of these APIs should be modified to check for 3562a non-zero return value instead of relying on them to fail without return. 3563 3564### NVMe Driver 3565 3566SPDK now supports hotplug for vfio-attached devices. But there is one thing keep in mind: 3567Only physical removal events are supported; removing devices via the sysfs `remove` file will not work. 3568 3569### NVMe-oF Target 3570 3571Subsystems are no longer tied explicitly to CPU cores. Instead, connections are handed out to the available 3572cores round-robin. The "Core" option in the configuration file has been removed. 3573 3574### Blobstore 3575 3576A number of functions have been renamed: 3577 3578- spdk_bs_md_resize_blob() => spdk_blob_resize() 3579- spdk_bs_md_sync_blob() => spdk_blob_sync_md() 3580- spdk_bs_md_close_blob() => spdk_blob_close() 3581- spdk_bs_md_get_xattr_names() => spdk_blob_get_xattr_names() 3582- spdk_bs_md_get_xattr_value() => spdk_blob_get_xattr_value() 3583- spdk_blob_md_set_xattr() => spdk_blob_set_xattr() 3584- spdk_blob_md_remove_xattr() => spdk_blob_remove_xattr() 3585- spdk_bs_md_create_blob() => spdk_bs_create_blob() 3586- spdk_bs_md_open_blob() => spdk_bs_open_blob() 3587- spdk_bs_md_delete_blob() => spdk_bs_delete_blob() 3588- spdk_bs_md_iter_first() => spdk_bs_iter_first() 3589- spdk_bs_md_iter_next() => spdk_bs_iter_next() 3590 3591The function signature of spdk_blob_close() has changed. It now takes a struct spdk_blob * argument 3592rather than struct spdk_blob **. 3593 3594The function signature of spdk_bs_iter_next() has changed. It now takes a struct spdk_blob * argument 3595rather than struct spdk_blob **. 3596 3597Thin provisioning support has been added to the blobstore. It can be enabled by setting the 3598`thin_provision` flag in struct spdk_blob_opts when calling spdk_bs_create_blob_ext(). 3599 3600### NBD device 3601 3602The NBD application (test/lib/bdev/nbd) has been removed; Same functionality can now be 3603achieved by using the test/app/bdev_svc application and start_nbd_disk RPC method. 3604See the [GPT](http://www.spdk.io/doc/bdev.html#bdev_config_gpt) documentation for more details. 3605 3606### FIO plugin 3607 3608SPDK `fio_plugin` now supports FIO 3.3. The support for previous FIO 2.21 has been dropped, 3609although it still remains to work for now. The new FIO contains huge amount of bugfixes and 3610it's recommended to do an update. 3611 3612### Virtio library 3613 3614Previously a part of the bdev_virtio module, now a separate library. Virtio is now available 3615via `spdk_internal/virtio.h` file. This is an internal interface to be used when implementing 3616new Virtio backends, namely Virtio-BLK. 3617 3618### iSCSI 3619 3620The MinConnectionIdleInterval parameter has been removed, and connections are no longer migrated 3621to an epoll/kqueue descriptor on the master core when idle. 3622 3623## v17.10: Logical Volumes 3624 3625### New dependencies 3626 3627libuuid was added as new dependency for logical volumes. 3628 3629libnuma is now required unconditionally now that the DPDK submodule has been updated to DPDK 17.08. 3630 3631### Block Device Abstraction Layer (bdev) 3632 3633An [fio](http://github.com/axboe/fio) plugin was added that can route 3634I/O to the bdev layer. See the [plugin documentation](https://github.com/spdk/spdk/tree/master/examples/bdev/fio_plugin/) 3635for more information. 3636 3637spdk_bdev_unmap() was modified to take an offset and a length in bytes as 3638arguments instead of requiring the user to provide an array of SCSI 3639unmap descriptors. This limits unmaps to a single contiguous range. 3640 3641spdk_bdev_write_zeroes() was introduced. It ensures that all specified blocks will be zeroed out. 3642If a block device doesn't natively support a write zeroes command, the bdev layer emulates it using 3643write commands. 3644 3645New API functions that accept I/O parameters in units of blocks instead of bytes 3646have been added: 3647 3648- spdk_bdev_read_blocks(), spdk_bdev_readv_blocks() 3649- spdk_bdev_write_blocks(), spdk_bdev_writev_blocks() 3650- spdk_bdev_write_zeroes_blocks() 3651- spdk_bdev_unmap_blocks() 3652 3653The bdev layer now handles temporary out-of-memory I/O failures internally by queueing the I/O to be 3654retried later. 3655 3656### Linux AIO bdev 3657 3658The AIO bdev now allows the user to override the auto-detected block size. 3659 3660### NVMe driver 3661 3662The NVMe driver now recognizes the NVMe 1.3 Namespace Optimal I/O Boundary field. 3663NVMe 1.3 devices may report an optimal I/O boundary, which the driver will take 3664into account when splitting I/O requests. 3665 3666The HotplugEnable option in `[Nvme]` sections of the configuration file is now 3667"No" by default. It was previously "Yes". 3668 3669The NVMe library now includes a spdk_nvme_ns_get_ctrlr() function which returns the 3670NVMe Controller associated with a given namespace. 3671 3672The NVMe library now allows the user to specify a host identifier when attaching 3673to a controller. The host identifier is used as part of the Reservations feature, 3674as well as in the NVMe-oF Connect command. The default host ID is also now a 3675randomly-generated UUID, and the default host NQN uses the host ID to generate 3676a UUID-based NQN. 3677 3678spdk_nvme_connect() was added to allow the user to connect directly to a single 3679NVMe or NVMe-oF controller. 3680 3681### NVMe-oF Target (nvmf_tgt) 3682 3683The NVMe-oF target no longer requires any in-capsule data buffers to run, and 3684the feature is now entirely optional. Previously, at least 4 KiB in-capsule 3685data buffers were required. 3686 3687NVMe-oF subsystems have a new configuration option, AllowAnyHost, to control 3688whether the host NQN whitelist is enforced when accepting new connections. 3689If no Host options have been specified and AllowAnyHost is disabled, the 3690connection will be denied; this is a behavior change from previous releases, 3691which allowed any host NQN to connect if the Host list was empty. 3692AllowAnyHost is disabled by default. 3693 3694NVMe-oF namespaces may now be assigned arbitrary namespace IDs, and the number 3695of namespaces per subsystem is no longer limited. 3696 3697The NVMe-oF target now supports the Write Zeroes command. 3698 3699### Environment Abstraction Layer 3700 3701A new default value, SPDK_MEMPOOL_DEFAULT_CACHE_SIZE, was added to provide 3702additional clarity when constructing spdk_mempools. Previously, -1 could be 3703passed and the library would choose a reasonable default, but this new value 3704makes it explicit that the default is being used. 3705 3706### Blobstore 3707 3708The blobstore super block now contains a bstype field to identify the type of the blobstore. 3709Existing code should be updated to fill out bstype when calling spdk_bs_init() and spdk_bs_load(). 3710 3711spdk_bs_destroy() was added to allow destroying blobstore on device 3712with an initialized blobstore. 3713 3714spdk_bs_io_readv_blob() and spdk_bs_io_writev_blob() were added to enable 3715scattered payloads. 3716 3717A CLI tool for blobstore has been added, allowing basic operations through either command 3718line or shell interface. See the [blobcli](https://github.com/spdk/spdk/tree/master/examples/blob/cli) 3719documentation for more details. 3720 3721### Event Framework 3722 3723The ability to set a thread name, previously only used by the reactor code, is 3724now part of the spdk_thread_allocate() API. Users may specify a thread name 3725which will show up in tools like `gdb`. 3726 3727### Log 3728 3729The spdk_trace_dump() function now takes a new parameter to allow the caller to 3730specify an output file handle (stdout or stderr, for example). 3731 3732### Logical Volumes 3733 3734Logical volumes library built on top of SPDK blobstore has been added. 3735It is possible to create logical volumes on top of other devices using RPC. 3736 3737See the [logical volumes](http://www.spdk.io/doc/logical_volumes.html) documentation for more information. 3738 3739### Persistent Memory 3740 3741A new persistent memory bdev type has been added. 3742The persistent memory block device is built on top of [libpmemblk](http://pmem.io/nvml/libpmemblk/). 3743It is possible to create pmem devices on top of pmem pool files using RPC. 3744 3745See the [Pmem Block Device](http://www.spdk.io/doc/bdev.html#bdev_config_pmem) documentation for more information. 3746 3747### Virtio SCSI driver 3748 3749A userspace driver for Virtio SCSI devices has been added. 3750The driver is capable of creating block devices on top of LUNs exposed by another SPDK vhost-scsi application. 3751 3752See the [Virtio SCSI](http://www.spdk.io/doc/virtio.html) documentation and 3753[Getting Started](http://www.spdk.io/doc/bdev.html#bdev_config_virtio_scsi) guide for more information. 3754 3755### Vhost target 3756 3757The vhost target application now supports live migration between QEMU instances. 3758 3759## v17.07: Build system improvements, userspace vhost-blk target, and GPT bdev 3760 3761### Build System 3762 3763A `configure` script has been added to simplify the build configuration process. 3764The existing CONFIG file and `make CONFIG_...` options are also still supported. 3765Run `./configure --help` for information about available configuration options. 3766 3767A DPDK submodule has been added to make building SPDK easier. If no `--with-dpdk` 3768option is specified to configure, the SPDK build system will automatically build a 3769known-good configuration of DPDK with the minimal options enabled. See the Building 3770section of README.md for more information. 3771 3772A [Vagrant](https://www.vagrantup.com/) setup has been added to make it easier to 3773develop and use SPDK on systems without suitable NVMe hardware. See the Vagrant 3774section of README.md for more information. 3775 3776### Userspace vhost-blk target 3777 3778The vhost library and example app have been updated to support the vhost-blk 3779protocol in addition to the existing vhost-scsi protocol. 3780See the [vhost documentation](http://www.spdk.io/doc/vhost.html) for more details. 3781 3782### Block device abstraction layer (bdev) 3783 3784A GPT virtual block device has been added, which automatically exposes GPT partitions 3785with a special SPDK-specific partition type as bdevs. 3786See the [GPT bdev documentation](http://www.spdk.io/doc/bdev.md#bdev_config_gpt) for 3787more information. 3788 3789### NVMe driver 3790 3791The NVMe driver has been updated to support recent Intel SSDs, including the Intel® 3792Optane™ SSD DC P4800X series. 3793 3794A workaround has been added for devices that failed to recognize register writes 3795during controller reset. 3796 3797The NVMe driver now allocates request tracking objects on a per-queue basis. The 3798number of requests allowed on an I/O queue may be set during `spdk_nvme_probe()` by 3799modifying `io_queue_requests` in the opts structure. 3800 3801The SPDK NVMe `fio_plugin` has been updated to support multiple threads (`numjobs`). 3802 3803spdk_nvme_ctrlr_alloc_io_qpair() has been modified to allow the user to override 3804controller-level options for each individual I/O queue pair. 3805Existing callers with qprio == 0 can be updated to: 3806~~~ 3807... = spdk_nvme_ctrlr_alloc_io_qpair(ctrlr, NULL, 0); 3808~~~ 3809Callers that need to specify a non-default qprio should be updated to: 3810~~~ 3811struct spdk_nvme_io_qpair_opts opts; 3812spdk_nvme_ctrlr_get_default_io_qpair_opts(ctrlr, &opts, sizeof(opts)); 3813opts.qprio = SPDK_NVME_QPRIO_...; 3814... = spdk_nvme_ctrlr_alloc_io_qpair(ctrlr, &opts, sizeof(opts)); 3815~~~ 3816 3817### Environment Abstraction Layer 3818 3819The environment abstraction layer has been updated to include several new functions 3820in order to wrap additional DPDK functionality. See `include/spdk/env.h` for the 3821current set of functions. 3822 3823### SPDK Performance Analysis with Intel® VTune™ Amplifier 3824 3825Support for SPDK performance analysis has been added to Intel® VTune™ Amplifier 2018. 3826 3827This analysis provides: 3828 3829- I/O performance monitoring (calculating standard I/O metrics like IOPS, throughput, etc.) 3830- Tuning insights on the interplay of I/O and compute devices by estimating how many cores 3831 would be reasonable to provide for SPDK to keep up with a current storage workload. 3832 3833See the VTune Amplifier documentation for more information. 3834 3835## v17.03: Blobstore and userspace vhost-scsi target 3836 3837### Blobstore and BlobFS 3838 3839The blobstore is a persistent, power-fail safe block allocator designed to be 3840used as the local storage system backing a higher-level storage service. 3841See the [blobstore documentation](http://www.spdk.io/doc/blob.html) for more details. 3842 3843BlobFS adds basic filesystem functionality like filenames on top of the blobstore. 3844This release also includes a RocksDB Env implementation using BlobFS in place of the 3845kernel filesystem. 3846See the [BlobFS documentation](http://www.spdk.io/doc/blobfs.html) for more details. 3847 3848### Userspace vhost-scsi target 3849 3850A userspace implementation of the QEMU vhost-scsi protocol has been added. 3851The vhost target is capable of exporting SPDK bdevs to QEMU-based VMs as virtio devices. 3852See the [vhost documentation](http://www.spdk.io/doc/vhost.html) for more details. 3853 3854### Event framework 3855 3856The overhead of the main reactor event loop was reduced by optimizing the number of 3857calls to spdk_get_ticks() per iteration. 3858 3859### NVMe library 3860 3861The NVMe library will now automatically split readv/writev requests with scatter-gather 3862lists that do not map to valid PRP lists when the NVMe controller does not natively 3863support SGLs. 3864 3865The `identify` and `perf` NVMe examples were modified to add a consistent format for 3866specifying remote NVMe over Fabrics devices via the `-r` option. 3867This is implemented using the new `spdk_nvme_transport_id_parse()` function. 3868 3869### iSCSI Target 3870 3871The [Nvme] section of the configuration file was modified to remove the `BDF` directive 3872and replace it with a `TransportID` directive. Both local (PCIe) and remote (NVMe-oF) 3873devices can now be specified as the backing block device. A script to generate an 3874entire [Nvme] section based on the local NVMe devices attached was added at 3875`scripts/gen_nvme.sh`. 3876 3877### NVMe-oF Target 3878 3879The [Nvme] section of the configuration file was modified to remove the `BDF` directive 3880and replace it with a `TransportID` directive. Both local (PCIe) and remote (NVMe-oF) 3881devices can now be specified as the backing block device. A script to generate an 3882entire [Nvme] section based on the local NVMe devices attached was added at 3883`scripts/gen_nvme.sh`. 3884 3885## v16.12: NVMe over Fabrics host, hotplug, and multi-process 3886 3887### NVMe library 3888 3889The NVMe library has been changed to create its own request memory pool rather than 3890requiring the user to initialize the global `request_mempool` variable. Apps can be 3891updated by simply removing the initialization of `request_mempool`. Since the NVMe 3892library user no longer needs to know the size of the internal NVMe request 3893structure to create the pool, the `spdk_nvme_request_size()` function was also removed. 3894 3895The `spdk_nvme_ns_cmd_deallocate()` function was renamed and extended to become 3896`spdk_nvme_ns_cmd_dataset_management()`, which allows access to all of the NVMe 3897Dataset Management command's parameters. Existing callers can be updated to use 3898`spdk_nvme_ns_cmd_dataset_management()` with `SPDK_NVME_DSM_ATTR_DEALLOCATE` as the 3899`type` parameter. 3900 3901The NVMe library SGL callback prototype has been changed to return virtual addresses 3902rather than physical addresses. Callers of `spdk_nvme_ns_cmd_readv()` and 3903`spdk_nvme_ns_cmd_writev()` must update their `next_sge_fn` callbacks to match. 3904 3905The NVMe library now supports NVMe over Fabrics devices in addition to the existing 3906support for local PCIe-attached NVMe devices. For an example of how to enable 3907NVMe over Fabrics support in an application, see `examples/nvme/identify` and 3908`examples/nvme/perf`. 3909 3910Hot insert/remove support for NVMe devices has been added. To enable NVMe hotplug 3911support, an application should call the `spdk_nvme_probe()` function on a regular 3912basis to probe for new devices (reported via the existing `probe_cb` callback) and 3913removed devices (reported via a new `remove_cb` callback). Hotplug is currently 3914only supported on Linux with the `uio_pci_generic` driver, and newly-added NVMe 3915devices must be bound to `uio_pci_generic` by an external script or tool. 3916 3917Multiple processes may now coordinate and use a single NVMe device simultaneously 3918using [DPDK Multi-process Support](http://dpdk.org/doc/guides/prog_guide/multi_proc_support.html). 3919 3920### NVMe over Fabrics target (`nvmf_tgt`) 3921 3922The `nvmf_tgt` configuration file format has been updated significantly to enable 3923new features. See the example configuration file `etc/spdk/nvmf.conf.in` for 3924more details on the new and changed options. 3925 3926The NVMe over Fabrics target now supports virtual mode subsystems, which allow the 3927user to export devices from the SPDK block device abstraction layer as NVMe over 3928Fabrics subsystems. Direct mode (raw NVMe device access) is also still supported, 3929and a single `nvmf_tgt` may export both types of subsystems simultaneously. 3930 3931### Block device abstraction layer (bdev) 3932 3933The bdev layer now supports scatter/gather read and write I/O APIs, and the NVMe 3934blockdev driver has been updated to support scatter/gather. Apps can use the 3935new scatter/gather support via the `spdk_bdev_readv()` and `spdk_bdev_writev()` 3936functions. 3937 3938The bdev status returned from each I/O has been extended to pass through NVMe 3939or SCSI status codes directly in cases where the underlying device can provide 3940a more specific status code. 3941 3942A Ceph RBD (RADOS Block Device) blockdev driver has been added. This allows the 3943`iscsi_tgt` and `nvmf_tgt` apps to export Ceph RBD volumes as iSCSI LUNs or 3944NVMe namespaces. 3945 3946### General changes 3947 3948`libpciaccess` has been removed as a dependency and DPDK PCI enumeration is 3949used instead. Prior to DPDK 16.07 enumeration by class code was not supported, 3950so for earlier DPDK versions, only Intel SSD DC P3x00 devices will be discovered 3951by the NVMe library. 3952 3953The `env` environment abstraction library has been introduced, and a default 3954DPDK-based implementation is provided as part of SPDK. The goal of the `env` 3955layer is to enable use of alternate user-mode memory allocation and PCI access 3956libraries. See `doc/porting.md` for more details. 3957 3958The build process has been modified to produce all of the library files in the 3959`build/lib` directory. This is intended to simplify the use of SPDK from external 3960projects, which can now link to SPDK libraries by adding the `build/lib` directory 3961to the library path via `-L` and linking the SPDK libraries by name (for example, 3962`-lspdk_nvme -lspdk_log -lspdk_util`). 3963 3964`nvmf_tgt` and `iscsi_tgt` now have a JSON-RPC interface, which allows the user 3965to query and modify the configuration at runtime. The RPC service is disabled by 3966default, since it currently does not provide any authentication or security 3967mechanisms; it should only be enabled on systems with controlled user access 3968behind a firewall. An example RPC client implemented in Python is provided in 3969`scripts/rpc.py`. 3970 3971## v16.08: iSCSI target, NVMe over Fabrics maturity 3972 3973This release adds a userspace iSCSI target. The iSCSI target is capable of exporting 3974NVMe devices over a network using the iSCSI protocol. The application is located 3975in app/iscsi_tgt and a documented configuration file can be found at etc/spdk/spdk.conf.in. 3976 3977This release also significantly improves the existing NVMe over Fabrics target. 3978 3979- The configuration file format was changed, which will require updates to 3980 any existing nvmf.conf files (see `etc/spdk/nvmf.conf.in`): 3981 - `SubsystemGroup` was renamed to `Subsystem`. 3982 - `AuthFile` was removed (it was unimplemented). 3983 - `nvmf_tgt` was updated to correctly recognize NQN (NVMe Qualified Names) 3984 when naming subsystems. The default node name was changed to reflect this; 3985 it is now "nqn.2016-06.io.spdk". 3986 - `Port` and `Host` sections were merged into the `Subsystem` section 3987 - Global options to control max queue depth, number of queues, max I/O 3988 size, and max in-capsule data size were added. 3989 - The Nvme section was removed. Now a list of devices is specified by 3990 bus/device/function directly in the Subsystem section. 3991 - Subsystems now have a Mode, which can be Direct or Virtual. This is an attempt 3992 to future-proof the interface, so the only mode supported by this release 3993 is "Direct". 3994- Many bug fixes and cleanups were applied to the `nvmf_tgt` app and library. 3995- The target now supports discovery. 3996 3997This release also adds one new feature and provides some better examples and tools 3998for the NVMe driver. 3999 4000- The Weighted Round Robin arbitration method is now supported. This allows 4001 the user to specify different priorities on a per-I/O-queue basis. To 4002 enable WRR, set the `arb_mechanism` field during `spdk_nvme_probe()`. 4003- A simplified "Hello World" example was added to show the proper way to use 4004 the NVMe library API; see `examples/nvme/hello_world/hello_world.c`. 4005- A test for measuring software overhead was added. See `test/lib/nvme/overhead`. 4006 4007## v16.06: NVMf userspace target 4008 4009This release adds a userspace NVMf (NVMe over Fabrics) target, conforming to the 4010newly-released NVMf 1.0/NVMe 1.2.1 specification. The NVMf target exports NVMe 4011devices from a host machine over the network via RDMA. Currently, the target is 4012limited to directly exporting physical NVMe devices, and the discovery subsystem 4013is not supported. 4014 4015This release includes a general API cleanup, including renaming all declarations 4016in public headers to include a `spdk` prefix to prevent namespace clashes with 4017user code. 4018 4019- NVMe 4020 - The `nvme_attach()` API was reworked into a new probe/attach model, which 4021 moves device detection into the NVMe library. The new API also allows 4022 parallel initialization of NVMe controllers, providing a major reduction in 4023 startup time when using multiple controllers. 4024 - I/O queue allocation was changed to be explicit in the API. Each function 4025 that generates I/O requests now takes a queue pair (`spdk_nvme_qpair *`) 4026 argument, and I/O queues may be allocated using 4027 `spdk_nvme_ctrlr_alloc_io_qpair()`. This allows more flexible assignment of 4028 queue pairs than the previous model, which only allowed a single queue 4029 per thread and limited the total number of I/O queues to the lowest number 4030 supported on any attached controller. 4031 - Added support for the Write Zeroes command. 4032 - `examples/nvme/perf` can now report I/O command latency from the 4033 the controller's viewpoint using the Intel vendor-specific read/write latency 4034 log page. 4035 - Added namespace reservation command support, which can be used to coordinate 4036 sharing of a namespace between multiple hosts. 4037 - Added hardware SGL support, which enables use of scattered buffers that 4038 don't conform to the PRP list alignment and length requirements on supported 4039 NVMe controllers. 4040 - Added end-to-end data protection support, including the ability to write and 4041 read metadata in extended LBA (metadata appended to each block of data in the 4042 buffer) and separate metadata buffer modes. 4043 See `spdk_nvme_ns_cmd_write_with_md()` and `spdk_nvme_ns_cmd_read_with_md()` 4044 for details. 4045- IOAT 4046 - The DMA block fill feature is now exposed via the `ioat_submit_fill()` 4047 function. This is functionally similar to `memset()`, except the memory is 4048 filled with an 8-byte repeating pattern instead of a single byte like memset. 4049- PCI 4050 - Added support for using DPDK for PCI device mapping in addition to the 4051 existing libpciaccess option. Using the DPDK PCI support also allows use of 4052 the Linux VFIO driver model, which means that SPDK userspace drivers will work 4053 with the IOMMU enabled. Additionally, SPDK applications may be run as an 4054 unprivileged user with access restricted to a specific set of PCIe devices. 4055 - The PCI library API was made more generic to abstract away differences 4056 between the underlying PCI access implementations. 4057 4058## v1.2.0: IOAT user-space driver 4059 4060This release adds a user-space driver with support for the Intel I/O Acceleration Technology 4061(I/OAT, also known as "Crystal Beach") DMA offload engine. 4062 4063- IOAT 4064 - New user-space driver supporting DMA memory copy offload 4065 - Example programs `ioat/perf` and `ioat/verify` 4066 - Kernel-mode DMA engine test driver `kperf` for performance comparison 4067- NVMe 4068 - Per-I/O flags for Force Unit Access (FUA) and Limited Retry 4069 - Public API for retrieving log pages 4070 - Reservation register/acquire/release/report command support 4071 - Scattered payload support - an alternate API to provide I/O buffers via a sequence of callbacks 4072 - Declarations and `nvme/identify` support for Intel SSD DC P3700 series vendor-specific log pages and features 4073- Updated to support DPDK 2.2.0 4074 4075## v1.0.0: NVMe user-space driver 4076 4077This is the initial open source release of the Storage Performance Development Kit (SPDK). 4078 4079Features: 4080 4081- NVMe user-space driver 4082- NVMe example programs 4083 - `examples/nvme/perf` tests performance (IOPS) using the NVMe user-space driver 4084 - `examples/nvme/identify` displays NVMe controller information in a human-readable format 4085- Linux and FreeBSD support 4086