1# Changelog 2 3## v23.01: (Upcoming Release) 4 5### bdev 6 7Both of interleaved and separated metadata are now supported by the malloc bdev module. 8 9Protection information is now supported by the malloc bdev module. 10 11A new API `spkd_bdev_part_submit_request_ext` was added to specify a custom completion callback. 12 13### scheduler 14 15Changing scheduler from dynamic back to static is no longer possible, 16as there is no way of saving original SPDK thread distribution on reactors. 17 18### bdev 19 20New APIs `spdk_bdev_for_each_channel` and `spdk_bdev_for_each_channel_continue` and 21associated function pointers were added to iterate each channel of the required bdev. 22 23The RPC `bdev_get_iostat` now allows a user to query the per channel IO statistics for 24required bdev. 25 26New `spdk_bdev_copy_blocks` and `spdk_bdev_get_max_copy` APIs to support copy commands. 27 28A new API `spdk_bdev_io_get_submit_tsc` was added to get the submit_tsc of the bdev I/O. 29 30Bdevs will no longer have UUIDs generated based on timestamp and are responsible for 31setting this field themselves. Generation of UUIDs for NVMe bdevs may be enabled by 32running `bdev_nvme_set_options` RPC with `--generate-uuids` option. These identifiers 33are based on serial number and namespace ID and will always be the same for a given 34device. 35 36A new API `spdk_bdev_channel_get_histogram` was added to get the histogram of a specified 37channel for a bdev. 38 39Converted internal use of `pthread_mutex_t` to `struct spdk_spinlock`. Consumers of bdev 40API functions must be on an SPDK thread or the program will abort. It is now enforced 41that no internal bdev locks can be held when a poller or message goes off CPU. 42 43### event 44 45Added core lock file mechanism to prevent the same CPU cores from being used by multiple 46SPDK application instances. 47 48Added a command line switch to disable CPU locks on SPDK startup. 49 50Added RPCs framework_enable_cpumask_locks and framework_disable_cpumask_locks to enable 51and disable CPU core locks in runtime. 52 53Added --rpcs-allowed command line option. Users can specify a comma-separated list of RPC 54names with this option to restrict allowed RPCs to only that list. 55 56### rpc 57 58Added spdk_rpc_set_allowlist to restrict allowed RPCs to the specified list. 59 60Changed `bdev_raid_get_bdevs` RPC output format to include raid_bdev details. 61 62### bdevperf 63 64Promoted the application to example to match similar programs: fio_plugin and perf. 65It can now be found inside `examples/bdev/bdevperf`. 66 67### trace 68 69New `trace_get_info` RPC was added to get name of shared memory file, list of the 70available trace point groups and mask of the available trace points for each group. 71 72### util 73 74New API `spdk_fd_group_get_epoll_event` that returns the epoll(7) event that 75caused a function callback in file descriptor group to execute. 76 77### json 78 79Added API `spdk_json_write_double` and `spdk_json_write_named_double` to allow 80for writing and decoding of the the double data type. 81 82### thread 83 84Added `spdk_thread_get_app_thread` which returns the first thread that was created using 85`spdk_thread_create`. 86 87Added `spdk_thread_is_running`. This returns `true` for a running thread, or `false` if 88its exit process has started using `spdk_thread_exit`. 89 90Added API `spdk_spin_init`, `spdk_spin_destroy`, `spdk_spin_lock`, `spdk_spin_unlock`, and 91`spdk_spin_held` to support spinlocks that are aware of the SPDK concurrency model. 92 93## v22.09 94 95### accel 96 97Many names were changed in the accel framework to make them consistent both with themselves and 98the rest of SPDK. The primary public header file is now named `include/spdk/accel.h`. 99 100Added a new runtime RPC `accel_get_opc_assignments` to get a list of current opcode to engine 101assignments. 102 103Added a new startup RPC `accel_assign_opc` to assign/override a specific opcode to 104an engine. 105 106### bdev 107 108New RPCs `bdev_xnvme_create` and `bdev_xnvme_delete` were added to support the xNVMe bdev. 109 110A new API `spdk_bdev_for_each_bdev_io` was added to execute the function on the appropriate 111thread for each bdev_io submitted to the bdev. 112 113A new API `spdk_bdev_get_current_qd` was added to measure and return the queue depth from a 114bdev. This API is available even when queue depth sampling is disabled. 115 116Added `spdk_bdev_seek_data`, `spdk_bdev_seek_hole` and `spdk_bdev_io_get_seek_offset` functions 117that start from a given offset and seek for next data or for next hole. At this time only 118supported by lvol bdev module. 119 120A new parameter `io_drain_timeout` has been added to `spdk_bdev` structure. It controls how long 121a bdev reset must wait for IO to complete prior to issuing a reset to the underlying device. 122If there is no outstanding IO at the end of that period, the reset is skipped. 123Best defined for bdevs that share an underlying bdev, such as multiple lvol bdevs sharing 124an nvme device, to avoid unnecessarily resetting the underlying bdev and affecting other 125bdevs that are sharing it. 126Modifying this field is optional. Setting the value to 0 keeps the original behavior 127to always send resets immediately, even if there is no I/O outstanding. 128 129### blobstore 130 131Reserve space for `used_clusters` bitmap. The reserved space could be used for blobstore growing 132in the future. 133 134Added `is_zeroes` operation to `spdk_bs_dev`. It allows to detect if logical blocks are backed 135by zeroes device and do a shortcut in copy-on-write flow by excluding copy part from zeroes device. 136 137Added `spdk_blob_get_next_allocated_io_unit` and `spdk_blob_get_next_unallocated_io_unit` functions 138that start from a given offset and seek for first io_unit belonging to an allocated cluster 139or first io_unit belonging to an unallocated cluster. 140 141Added `spdk_bs_grow` function to grow blobstore size if the underlying bdev size is increased. 142This is used by lvol bdev via `bdev_lvol_grow_lvstore` RPC. 143 144### daos 145 146Added new DAOS bdev module, that creates SPDK block device on top of DAOS DFS. 147Please see [documentation](https://spdk.io/doc/bdev.html#bdev_config_daos) for more details. 148 149### env 150 151Added `spdk_pci_register_device_provider` and matching `SPDK_PCI_REGISTER_DEVICE_PROVIDER` 152functions for registering PCI device providers. That allow the VMD driver to be notified 153when users want to attach a device under a given BDF and when a device is detached. 154 155### init 156 157`spdk_subsystem_init_from_json_config` now fails if the JSON configuration file is not 158an object with an array named "subsystems". 159 160### iscsi 161 162Added `bdev_iscsi_set_options` RPC to modify options of the iSCSI bdev module. 163 164### json 165 166Added `spdk_json_find` API return errcode: EPROTOTYPE - json not enclosed in {}. 167`spdk_json_find` now returns -EPROTOTYPE instead of -ENOENT if the object parameter 168does not point to a JSON object (i.e. is not enclosed with {}). 169 170### lvol 171 172Add `num_md_pages_per_cluster_ratio` parameter to the `bdev_lvol_create_lvstore` RPC. 173Calculate `num_md_pages` from `num_md_pages_per_cluster_ratio`, and pass it to `spdk_bs_opts`. 174 175### nvme 176 177Added SPDK_NVME_TRANSPORT_CUSTOM_FABRICS to enum `spdk_nvme_transport_type` to support custom 178fabric transport. SPDK_NVME_TRANSPORT_CUSTOM was intended to be non-fabric custom transport. 179 180Added a new function `spdk_nvme_ns_cmd_verify` to submit a Verify Command to a Namespace. 181 182Added `spdk_nvme_ctrlr_disable_read_changed_ns_list_log_page` to allow an application to 183tell the driver to not read the CHANGED_NS_LIST log page in response to a NS_ATTR_CHANGED 184AEN. When called the application is required to read this log page instead to clear the 185AEN. 186 187Added `psk` field to `spdk_nvme_ctrlr_opts` struct in order to enable SSL socket implementation 188of TCP connection and set the PSK. Applicable for TCP transport only. 189 190### raid 191 192Renamed the `raid5` module to `raid5f` to reflect that it is not a traditional 193RAID5 implementation - only full stripe writes are supported, partial stripe 194writes (read-modify-write) are not. 195 196### rpc 197 198Added `psk` parameter to `bdev_nvme_attach_controller` RPC in order to enable SSL socket implementation 199of TCP connection and set the PSK. Applicable for TCP transport only. 200 201New options `enable_ktls` and `tls_version` were added to the `sock_impl_set_options` structure. 202New options `psk_key` and `psk_identity` were added to the `sock_impl_set_options` structure. 203 204Added warning message for `bdev_rbd_create`, if it is used without -c. 205`bdev_rbd_create()` API without specifying -c is deprecated and will be removed in future release. 206 207Renamed `enable_vmd` RPC to `vmd_enable` to make it consistent with our naming scheme of 208`<subsystem>_<action>`. For now, the old name is still available, but is marked as deprecated. 209 210New function `spdk_rpc_get_method_state_mask` was added to RPC library. 211 212### sma 213 214Extended `VolumeParameters` with crypto parameters allowing the user to configure crypto when 215attaching a volume to a device. This interface is now supported by all upstream device types 216(nvmf-tcp, virtio-blk, vfio-user) using `bdev_crypto`. Users must specify the crypto engine to use 217under `crypto` section in config. It is also possible to register out-of-tree crypto engines by 218inheriting from the `CryptoEngine` class. 219 220Added two new methods: `SetQos` and `GetQosCapabilities` allowing the user to configure QoS on a 221per-device or per-volume level. Not all QoS settings have to be supported by each device, so users 222can use `GetQosCapabilities` to query them for that. All upstream device types support QoS on a 223per-volume level using bdev layer's QoS mechanism. 224 225### sock 226 227Added new `ssl` based socket implementation, the code is located in `module/sock/posix`. 228For now we are using hard-coded PSK and only support TLS 1.3. 229 230### util 231 232Added new functions: `spdk_hexlify` and `spdk_unhexlify`. 233 234A new API `spdk_xor_gen` was added to generate XOR from multiple source buffers. It is going to be 235used by `raid5f` for calculating parity. 236 237### vfu_tgt 238 239Added vfu_tgt library abstraction based on libvfio-user for PCI device emulation. 240Besides the NVMe device emulation, its now possible to emulate virtio-blk and virtio-scsi 241devices. 242 243### virtio 244 245virtio-vhost-user no longer tries to support dynamic memory allocation. The vhost target does 246not support the high rate of SET_MEM_TABLE vhost messages that result from dynamic memory 247allocation, so a virtio-vhost-user device will now present an ERRLOG, assert, and skip the 248SET_MEM_TABLE vhost message if a memory notification is received outside of the normal device 249start/stop. Applications using the virtio library in vhost-user mode should now pre-allocate 250the application's memory using the -s/--mem-size option and use single shared memory file 251segments using the -g/--single-file-segments option. 252 253Added `vfio-user` transport type for virtio-blk and virtio-scsi devices. 254 255### vmd 256 257Fixed hotplug when a device is inserted in a slot in which a disk was already enumerated previously. 258Added two new RPCs: `vmd_remove_device` simulating a hotremove, and `vmd_rescan`, which rescans all 259buses managed by the VMD driver and hotplugs all newfound devices. 260 261## v22.05 262 263### sock 264 265A new option `ack_timeout` was added to the `spdk_sock_opts` structure. 266 267### util 268 269A new parameter `bounce_iovcnt` was added to `spdk_dif_generate_copy` and `spdk_dif_verify_copy`. 270The `bounce_iovcnt` is used to specify the number of bounce_iov to support multiple block-aligned 271fragment copies. 272 273A new API `spdk_copy_iovs_to_buf` and `spdk_copy_buf_to_iovs` were added to copy iovs to buf or 274copy buf to iovs. There're many cases need to use these two APIs. 275 276### bdev 277 278Removed deprecated spdk_bdev_module_finish_done(). Use spdk_bdev_module_fini_done() instead. 279 280A new API `spdk_bdev_unregister_by_name` was added to handle race conditions correctly. 281 282New APIs, `spdk_for_each_bdev` and `spdk_for_each_bdev_leaf`, were added to provide iteration 283safe for race conditions. 284 285A new RPC `bdev_nvme_get_io_paths` was added to get all active I/O paths. 286 287A new RPC `bdev_nvme_set_preferred_path` was added to set preferred I/O path for an NVMe bdev 288when in multipath mode. This RPC does not support NVMe bdevs in failover mode. 289 290A new RPC `bdev_nvme_set_multipath_policy` was added to set multipath policy of a NVMe bdev 291in multipath mode. 292 293A new option `disable_auto_failback` was added to the `bdev_nvme_set_options` RPC to disable 294automatic failback. 295 296### idxd / dsa 297 298A new parameter `flags` was added to all low level submission and preparation 299APIs to enable the caller to pass operation flags per the DSA specification. 300 301A new flag 'SPDK_IDXD_FLAG_PERSISTENT' was added to let DSA know that 302the destination is persistent. 303 304The RPC `idxd_scan_accel_engine` has been renamed to `dsa_scan_accel_engine` 305 306The RPC `iaa_scan_accel_engine` has been added. 307 308Many HW related structs/functions with the name `idxd` have been renamed `dsa` 309to more accurately represent the HW they are associated with. 310 311Two new functions were added to the library `spdk_idxd_submit_compress` and 312`spdk_idxd_submit_decompress` 313 314### accel_fw 315 316A new parameter `flags` was added to accel API. 317The APIs include: 318`spdk_accel_submit_copy` 319`spdk_accel_submit_dualcast` 320`spdk_accel_submit_fill` 321`spdk_accel_submit_copy_crc32c` 322`spdk_accel_submit_copy_crc32cv` 323`spdk_accel_submit_compress` 324`spdk_accel_submit_decompress` 325 326A new flag `ACCEL_FLAG_PERSISTENT` was added to indicate the target memory is PMEM. 327 328The API `spdk_accel_get_capabilities` has been removed. 329 330### crypto 331 332Support for AES_XTS was added for MLX5 polled mode driver (pmd). 333 334bdev_crypto_create RPC now requires hexlified 'key' and 'key2' params for all pmd drivers. 335Unhexlifying is performed during RPC command processing and the vbdev crypto module runs on 336binary keys as before. 337 338### bdev 339 340Added a timeout option to the `bdev_get_bdevs` RPC. It allows the user to specify the amount of 341time to wait until a bdev with a given name appears in the system. 342 343### bdev_nvme 344 345Added `bdev_nvme_add_error_injection` and `bdev_nvme_remove_error_injection` RPCs to add and 346remove NVMe error injections. 347 348New parameters, `ctrlr_loss_timeout_sec`, `reconnect_delay_sec`, and `fast_io_fail_timeout_sec`, are 349added to the RPC `bdev_nvme_set_options`. They can be overridden if they are given by the RPC 350`bdev_nvme_attach_controller`. 351 352### blobstore 353 354New functions `spdk_blob_io_writev_ext` and `spdk_blob_io_readv_ext` are added. The new functions accept 355`spdk_blob_ext_io_opts` structure with extended IO request options. 356 357### event 358 359Added `msg_mempool_size` parameter to `spdk_reactors_init` and `spdk_thread_lib_init_ext`. 360The size of `g_spdk_msg_mempool` can now be controlled through the same-named 361user option of `spdk_app_opts` structure. 362 363### nvme 364 365The API `spdk_nvme_ctrlr_prepare_for_reset()` was deprecated. The functionality provided by the 366`spdk_nvme_ctrlr_prepare_for_reset()` was merged into the API `spdk_nvme_ctrlr_disconnect()`. 367 368### nvmf 369 370Removed deprecated max_qpairs_per_ctrlr parameter from nvmf_create_transport RPC. Use 371max_io_qpairs_per_ctrlr instead. 372 373Deprecated the ability for hosts to connect to the discovery subsystem automatically on any 374existing listener. Users should now explicitly add listeners for the discovery subsystem. 375Host can still connect to the discovery subsystem as before, but a warning message will be 376emitted if no listener was configured for the transport ID of the incoming connection. 377 378Added adaptive interrupt feature for vfio-user transport. New parameter `disable_adaptive_irq` 379is added to the RPC `nvmf_create_transport`. 380 381### thread 382 383Added `spdk_thread_exec_msg()` API. 384 385### scheduler 386 387`framework_set_scheduler` can now be called after application initialization. 388Added callbacks to set custom parameters specific for each scheduler implementation 389and `framework_get_scheduler` to retrieve them. 390 391Added `dynamic` scheduler options: load_limit, core_limit, core_busy. Their descriptions 392are available in JSON-RPC document, in section 393[framework_set_scheduler](jsonrpc.html#rpc_framework_set_scheduler). 394 395### raid 396 397Add concat as a special raid module. The concat module could create a virtual bdev. The 398virtual bdev combines multiple underlying bdevs together. The layout of the underlying 399bdevs is one after another. The concat bdev is extendable. When the free space of the 400concat bdev is not enough, the user can deconstruct the concat bdev, then reconstruct it 401with an additional underlying bdev. 402 403### sock 404 405Allow MSG_ZEROCOPY flag to be set or not according to data size, which can be enabled and 406set by setting "zerocopy_threshold". zerocopy_threshold = 0 means disable this function; 407zerocopy_threshold > 0 means enable it and use this value as the threshold. 408 409### rpc 410 411Introduced `zerocopy_threshold` to enable zerocopy on send for server sockets according to 412data size to be flushed. 413 414## v22.01 415 416### accel 417 418The batching capability was removed. Batching is now considered an implementation 419detail of the low level drivers. 420 421### bdev 422 423The NVMe bdev module supports multipath and improved I/O error resiliency. 424 425The parameter `retry_count` of the RPC `bdev_nvme_set_options` was deprecated and will be 426removed in SPDK 22.04, and the parameter `transport_retry_count` is added and used instead. 427 428An new parameter `bdev_retry_count` is added to the RPC `bdev_nvme_set_options`. 429 430New parameters, `ctrlr_loss_timeout_sec`, `reconnect_delay_sec`, and `fast_io_fail_timeout_sec`, are 431added to the RPC `bdev_nvme_attach_controller`. 432 433An new parameter `num_io_queues` is added to `bdev_nvme_attach_controller` RPC to allow specifying amount 434of requested IO queues. 435 436Added `key_file` parameter to the `rbd_register_cluster` RPC. It is an optional parameter to 437specify a keyring file to connect to a RADOS cluster. 438 439Added `spdk_bdev_get_zone_id` API to retrieve zone_id for a certain LBA. 440 441### bdev_aio 442 443Added `bdev_aio_rescan` RPC to allow rescanning the size of aio bdev. 444 445### bdev_nvme 446 447Added discovery service to bdev_nvme module, which now can connect to a discovery controller. 448bdev_nvme will connect to all current and future subsystems in the discovery controller. 449Added `bdev_nvme_start_discovery` and `bdev_nvme_stop_discovery` RPC. 450 451### dpdk 452 453Updated DPDK submodule to DPDK 21.11. 454 455### env 456 457Added `spdk_pci_for_each_device`. 458 459Removed `spdk_pci_get_first_device` and `spdk_pci_get_next_device`. These APIs were unsafe, because 460they did not account for PCI devices being inserted or removed while the caller was using handles 461returned from these APIs. Existing users of these APIs should switch to `spdk_pci_for_each_device` 462instead. 463 464Added 3 experimental APIs to handle PCI device interrupts (`spdk_pci_device_enable_interrupt`, 465`spdk_pci_device_disable_interrupt`, `spdk_pci_device_get_interrupt_efd`). 466 467Added `framework_get_pci_devices` RPC to list PCIe devices attached to an SPDK application. 468 469### idxd 470 471Many APIs are now vectored rather than scalar, meaning they take iovecs instead of individual pointers. 472 473### json 474 475Added `spdk_json_write_bytearray` API to serialize a buffer as a hex string. 476 477### nvme 478 479API `spdk_nvme_trtype_is_fabrics` was added to return existing transport type 480is fabric or not. 481 482API `spdk_nvme_poll_group_remove` was limited to be available only for a 483disconnected qpair in the group. 484 485New APIs, `spdk_nvme_ctrlr_disconnect`, `spdk_nvme_ctrlr_reconnect_async`, and 486`spdk_nvme_ctrlr_reconnect_poll_async`, have been added to improve error recovery, and 487the existing APIs,`spdk_nvme_ctrlr_reset_async` and `spdk_nvme_ctrlr_reset_poll_async` 488were deprecated. 489 490Added `spdk_nvme_ctrlr_get_discovery_log_page` API for getting the full discovery log page 491from a discovery controller. 492 493### nvmf 494 495Added support for zero-copy operations in the NVMe-oF TCP target. It can be enabled via 496the `zcopy` parameter when creating a transport. The zero-copy operations are only used 497for requests not using in-capsule data. 498 499Added a `subsystem` parameter to `spdk_nvmf_transport_stop_listen_async`. When not NULL, 500it will only disconnect qpairs for controllers associated with the specified subsystem. 501 502Removed accept poller from transport layer. Each transport can have its own policy of 503handling new connections. To notify transport layer `spdk_nvmf_poll_group_add` and 504`spdk_nvmf_tgt_new_qpair` can be used. 505 506### scsi 507 508Structure `spdk_scsi_lun` has been extended with new member `resizing` so that SCSI layer now reports 509unit attention for disk resize. 510 511### trace 512 513Added `spdk_trace_create_tpoint_group_mask` to return tracepoint group mask from 514a tracepoint group name. 515 516Added `trace_set_tpoint_mask` and `trace_clear_tpoint_mask` RPC to allow enabling 517individual traces. 518 519### util 520 521Added `spdk_ioviter_first` and `spdk_ioviter_next` to iterate over two iovecs and 522yield pointers to common length segments. 523 524### sock 525 526A new parameter `hint` is added to `spdk_sock_get_optimal_sock_group`. It allows to suggest 527a poll group when no optimal poll group is found. 528 529## v21.10 530 531Structure `spdk_nvmf_target_opts` has been extended with new member `discovery_filter` which allows to specify 532filtering rules applied during discovery log generation. Refer to `enum spdk_nvmf_tgt_discovery_filter` for more info. 533 534### bdev 535 536New API `spdk_bdev_get_memory_domains` has been added, it allows to get SPDK memory domains used by bdev. 537 538New API functions `spdk_bdev_readv_blocks_ext` and `spdk_bdev_writev_blocks_ext` have been added. 539These functions accept `spdk_bdev_ext_io_opts` structure with extended IO request 540options, e.g. DMA memory domain which describes data that may belong to another memory domain and 541can't be accessed directly. 542 543Added `async_fini_start` to allow bdev modules to complete the `fini_start` asynchronously, 544with new `spdk_bdev_module_fini_start_done` API. 545 546Deprecated `spdk_bdev_module_finish_done()` API, which will be removed in SPDK 22.01. 547Bdev modules should use `spdk_bdev_module_fini_done()` instead. 548 549The `ocssd` bdev has been removed. The Open Channel specification has been largely superseded by 550zoned namespaces, few if any devices have been brought to market, and there is little reason to 551continue to support this. OCSSD support in the nvme driver will remain for now. 552 553### dma 554 555A new library, lib/dma, has been added. This library provides the necessary infrastructure for 556handling systems and devices with multiple memory domains. For example, a PCIe add-in card with an 557SoC may be running SPDK on the SoC. That SoC has its own local memory, but SPDK may be controlling 558devices that can also access the host system memory. This library provides infrastructure to enumerate 559the memory domains and request hardware perform DMA transfers between them. 560 561### dpdk 562 563Updated DPDK submodule to DPDK 21.08. 564 565### event 566 567Added the `disable_signal_handlers` flag to the `spdk_app_opts` struct. 568 569### idxd 570 571Added `spdk_idxd_get_socket` to query the socket that the idxd device 572is on. 573 574### json 575 576Added API to allow for writing and decoding of new types: 577 578- `spdk_json_write_named_uint16` 579- `spdk_json_write_named_uint8` 580- `spdk_json_write_uint16` 581- `spdk_json_write_uint8` 582- `spdk_json_number_to_uint8` 583- `spdk_json_decode_uint8` 584 585### log 586 587Added API `spdk_log_to_syslog_level` to return syslog level based on SPDK's 588log level. 589 590### nvme 591 592Added new functions `spdk_nvme_zns_set_zone_desc_ext` and `spdk_nvme_zns_ext_report_zones` 593to set zone descriptor extension and to get extended zone report respectively. 594 595New API `spdk_nvme_ctrlr_get_memory_domains` has been added, it allows to get SPDK memory domains used by nvme controller. 596 597New API functions `spdk_nvme_ns_cmd_readv_ext` and `spdk_nvme_ns_cmd_writev_ext` 598have been added. These functions accept `spdk_nvme_ns_cmd_ext_io_opts` structure with extended IO request 599options, e.g. DMA memory domain which describes data that may belong to another memory domain and 600can't be accessed directly. 601 602Added a new function `spdk_nvme_ctrlr_get_regs_bpinfo` to get boot partition info of a controller. 603Added new functions `spdk_nvme_ctrlr_write_boot_partition`, 604`spdk_nvme_ctrlr_read_boot_partition_start` and `spdk_nvme_ctrlr_read_boot_partition_poll` 605to write and read the boot partitions of a controller. 606 607Added `spdk_nvme_ctrlr_get_opts` to retrieve the current controller options. 608 609Added `async_mode` to `spdk_nvme_io_qpair_opts` to enable creation of submission and completion 610queues asynchronously. This mode is currently supported at PCIe layer, 611which tracks the qpair creation with state machine and returns to the user immediately. 612Default mode is set to false to create io qpairs synchronously. 613 614Added `spdk_nvme_ctrlr_get_regs_cc` to retrieve NVMe controller CC (Configuration) register. 615 616Added `spdk_nvme_ctrlr_prepare_for_reset` to inform the driver that the application is preparing 617to reset the specified NVMe controller. This function allows the driver to make decisions 618knowing that a reset is about to happen. 619 620### nvmf 621 622Added `oncs` to `struct spdk_nvmf_ctrlr_data` so that the transport layer 623can decide support RESERVATION feature or not. 624 625An `opts_size` element was added in the `spdk_nvmf_ns_opts` structure to solve the 626ABI compatibility issue between different SPDK version. An new option `anagrpid` was 627added in the `spdk_nvmf_ns_opts` structure. 628 629An new parameter `anagrpid` was added to the RPC `nvmf_subsystem_add_ns`. 630 631An new parameter `anagrpid` was added to the RPC `nvmf_subsystem_listener_set_ana_state`. 632 633`spdk_nvmf_subsystem_destroy` is now can be asynchronous, it accepts a callback and callback argument. 634 635A new parameter, `poll_groups_mask` was added to the `nvmf_set_config` RPC that allows specifying 636a subset of cores for the nvmf poll groups. This helps to avoid imbalances when some cores are 637busy with periodic timer tasks that run very frequently. 638 639A new parameter, `commit` was added to `nvmf_bdev_ctrlr_end_zcopy` that allows indicating whether 640the buffers should be committed. 641 642Added `listen_dump_opts` to `spdk_nvmf_transport_ops` so that the transport layer can display 643listen opts. 644 645Added `fabrics_connect_timeout_us` to `bdev_nvme_attach_controller` RPC to allow specifying 646timeout for connect operation. 647 648Added `multipath` to `bdev_nvme_attach_controller` RPC to allow specifying multipath behavior. 649 650### scheduler 651 652New API for implementing schedulers and governors. Please see `include/spdk/scheduler.h` for details. 653 654### scsi 655 656New functions, `spdk_scsi_dev_get_first_lun` and `spdk_scsi_dev_get_next_lun` 657have been added to iterate LUNs of a SCSI device. 658 659Each SCSI device supports 256 LUNs at the maximum now and the macro constant 660`SPDK_SCSI_DEV_MAX_LUN` was removed. 661 662### trace_parser 663 664A new library, lib/trace_parser, has been added. This library provides functions that parse traces 665recorded by an SPDK application. That includes merging traces from multiple cores, 666sorting them by their timestamp and constructing trace entries spanning across multiple buffers. 667 668### util 669 670The `spdk_fd_group_add` API now takes a `name` parameter. 671 672## v21.07 673 674### accel_fw 675 676Added API `spdk_accel_submit_copy_crc32c` to perform a CRC32C while copying data. 677 678Added API `spdk_accel_batch_prep_copy_crc32c` to batch CRC32C + copy commands. 679 680Added API `spdk_accel_submit_copy_crc32cv` to submit chained CRC32C + copy commands. 681 682### bdev 683 684Change `spdk_bdev_read_blocks_with_md` arg offset definition from int64_t to uint64_t. 685 686Red-black tree has been used to organize the bdev names and aliases uniformly 687to provide faster lookup. 688 689Removed ZCOPY emulation: The bdev module can be checked to see if it supports ZCOPY 690and if not supported then use existing READ/WRITE commands. 691 692Added iov to spdk_bdev_zcopy_start. 693 694### dpdk 695 696Updated DPDK submodule to DPDK 21.05. 697 698### idxd 699 700Remove the `probe_cb` parameter in `spdk_idxd_probe` function. And remove the definition 701of `spdk_idxd_probe_cb` function pointer. It should be implemented in idxd_user.c. 702 703Added API `spdk_idxd_submit_copy_crc32c` to perform a CRC32C while copying data. 704 705Added API `spdk_idxd_batch_prep_copy_crc32c` to prepare a batch operation to perform 706a CRC32C while copying data. 707 708We have userspace idxd driver under accel_engine library (module/accel/idxd/accel_engine). 709And kernel idxd driver usage is also added under the accel_engine library. This approach 710implemented a wrapper library to use IDXD device by leveraging the kernel DSA driver in 711SPDK idxd library (lib/idxd). Then users can leverage the RPC later to configure how to 712use the DSA device by user space driver or kernel driver. 713 714### init 715 716Added new `init` library that initializes the SPDK subsystems, which previously was 717internal to application framework. That functionality has been made public and for 718use by applications which don't leverage SPDK's application framework. 719 720### iscsi 721 722New parameters, `pdu_pool_size`, `immediate_data_pool_size`, and `data_out_pool_size`, 723were added to the RPC `iscsi_set_options` to run iSCSI target with varying amount of 724available memory. 725 726### json 727 728Added API `spdk_json_write_named_uint128` and `spdk_json_write_uint128` to perform 729the uint128 related data. 730 731### net 732 733Removed deprecated `net` library. 734 735Added support for ZCOPY. ZCOPY is used in preference to READ and WRITE if the 736bdev module supports ZCOPY and the bdev module has ZCOPY enabled. 737 738### nvme 739 740`spdk_nvme_map_prps` and `spdk_nvme_map_cmd` were moved to nvmf/vfio-user as internal APIs 741as vfio-user is the only user for the above two APIs. 742 743Added a new function `spdk_nvme_ns_cmd_copy` to submit a Simple Copy Command to a Namespace. 744 745Update the `spdk_nvme_generic_command_status_code` structure with new status code 746according to the definition in NVMe 1.4 spec. 747 748`spdk_nvme_ctrlr_get_default_ctrlr_opts` now sets `use_cmb_sqs` to false. This means 749that if a controller has a CMB and supports SQs in the CMB, SPDK will not use 750the CMB for SQs by default - the user must set `use_cmb_sqs` to true in 751the `spdk_nvme_ctrlr_opts` structure prior to controller attach. 752 753Add a new function `spdk_nvme_detach_poll` to simplify a common use case to continue 754polling until all detachments complete. 755 756Added new argument `timeout_admin_us` to `spdk_nvme_ctrlr_register_timeout_callback` so callers 757can specify a different timeout for admin commands vs. io commands. 758 759An existing function `spdk_nvme_detach_async` was updated to add one or more detachments 760to an active context while it is being polled. 761 762Rename a variable in the member `cmic` of the struct `spdk_nvme_ctrlr_data` from 763`multi_host` to `multi_ctrlr`. The variable means that the NVM subsystem may have two or 764more controllers if set to 1. However `multi_host` had indicated a particular use case 765such that the NVM subsystem is used by multiple hosts. 766 767A new option `disable_read_ana_log_page` was added to struct `spdk_nvme_ctrlr_opts` to disable 768reading ANA log page. The upper layer is expected to read ANA log page instead if `true`. 769The default value is `false`. 770 771New APIs, `spdk_nvme_ctrlr_reset_async` and `spdk_nvme_ctrlr_reset_poll_async`, have been added to 772reset a controller asynchronously. 773 774New RPC `bdev_nvme_reset_controller` was added, to reset an NVMe controller. 775 776Added `spdk_nvme_ns_get_nguid` function to get NGUID for the given namespace. 777 778Added `spdk_nvme_ctrlr_is_fabrics` function to indicate whether a ctrlr handle 779is associated with a fabrics controller. 780 781### nvmf 782 783Added `min_cntlid` and `max_cntlid` to `nvmf_create_subsystem` to limit the controller ID range. 784Added `spdk_nvmf_subsystem_get_min_cntlid` and `spdk_nvmf_subsystem_get_max_cntlid` to request those values. 785 786`spdk_nvmf_request_get_buffers_multi` API is removed. 787 788Added the `nvmf_set_crdt` RPC for setting command retry delay times. 789 790Expanded `spdk_nvmf_poll_group_stat` with current qpair count statistics. 791 792Removed following deprecated APIs: 793 794- `spdk_nvmf_poll_group_get_stat` (function in `nvmf.h`), 795- `spdk_nvmf_transport_poll_group_get_stat` (function in `nvmf.h`), 796- `spdk_nvmf_transport_poll_group_free_stat`(function in `nvmf.h`), 797- `spdk_nvmf_rdma_device_stat` (struct in `nvmf.h`), 798- `spdk_nvmf_transport_poll_group_stat` (struct in `nvmf.h`), 799- `poll_group_get_stat` (transport op in `nvmf_transport.h`), 800- `poll_group_free_stat` (transport op in `nvmf_transport.h`). 801 802### rpc 803 804New RPC `bdev_rbd_register_cluster` and `bdev_rbd_unregister_cluster` was added, it allows to create 805and delete the rados object cluster, then users can choose the cluster to create related rbd 806device. 807 808Revised `bdev_rbd_create` parameter, it allows to use an optional parameter `--cluster-name` 809to create a rbd bdev with an already registered Rados Cluster Object. 810 811New RPC `bdev_rbd_get_clusters_info` was added, it allows to get the info of the registered 812Rados Cluster names. 813 814New optional parameter, `timeout_admin_us`, added to the bdev_nvme_set_options RPC. 815 816Revised a parameter `--stripe-size_kb` to `--stripe-size-kb` of `bdev_raid_create` method 817provided in `scripts/rpc.py` for consistency. 818 819An new optional parameter `config_kernel_mode` was added to the RPC `idxd_scan_accel_engine`, 820and this is used to enable using the kernel mode IDXD driver. 821 822Removed deprecated parameter `enable-zerocopy-send` of RPC `sock_impl_set_options`, 823use `enable-zerocopy-send-server` or `enable-zerocopy-send-client` instead. 824Removed deprecated parameter `disable-zerocopy-send` of RPC `sock_impl_set_options`, 825use `disable-zerocopy-send-server` or `disable-zerocopy-send-client` instead. 826 827### rpm 828 829Removed deprecated `pkg/spdk.spec`, please use `rpmbuild/spdk.spec` instead. 830See [RPM documentation](https://spdk.io/doc/rpm.html) for more details. 831 832### thread 833 834Red-black tree has been used for timed pollers to provide faster insertion and deletion 835and for io_devices to provide faster lookup. 836 837Removed `spdk_io_channel` structure from public header and moved it to thread_internal.h. 838 839### trace 840 841Added `spdk_trace_register_description_ext` function to register variable number of tracepoint arguments. 842 843Added ability to chain multiple trace entries together to extend the size of the argument buffer. 844 845### util 846 847Red-black tree macros has been added by using the macros provided by the FreeBSD operating system 848under the same BSD license. 849 850Add an new macro `SPDK_SIZEOF_MEMBER` to get the size of a member of a struct. 851 852`spdk_crc32c_iov_update` function was added to support calculating the crc32c of the iovs. 853 854Added zipf random number generator with power law probability distribution. 855When applied to performance testing of block devices, it will select blocks over 856the full range of LBAs, but will more frequently select lower-numbered LBAs. 857 858## v21.04 859 860### accel 861 862Two new accelerated crc32 functions `spdk_accel_submit_crc32cv` and 863`spdk_accel_batch_prep_crc32cv` are added in order to provide the 864chained accelerated CRC32 computation support. 865 866### bdev 867 868For `bdev_ocssd_create` RPC, the optional parameter `range` was removed. 869Only one OCSSD bdev can be created for one OCSSD namespace. 870 871Removed the `spdk_bdev_open` from bdev library API. 872Removed the `spdk_vbdev_register` and `spdk_bdev_part_base_construct` from bdev module API. 873Removed the `config_text` function for bdev modules to report legacy config. 874 875Added `spdk_bdev_get_max_active_zones` API to display maximum number active zones of a given bdev. 876 877Added `spdk_bdev_get_max_zone_append_size` API to display maximum zone append data transfer size. 878 879### bdev_nvme 880 881Added support for zoned namespaces. 882 883### blobstore 884 885Removed the `spdk_bdev_create_bs_dev_from_desc` and `spdk_bdev_create_bs_dev` API. 886 887### env 888 889Added `spdk_pci_device_allow` API to allow applications to add PCI addresses to 890the allowed list after the application has started. 891 892Removed the `pci_whitelist`, `pci_blacklist` and `master_core` members of struct `spdk_env_opts`. 893 894Added hotplug support based on uevent in `pci_event.c`. Added usage of this functionality in 895nvme, virtio-scsi and virtio_blk libraries. Please see the new API `spdk_pci_event_listen`, 896`spdk_pci_get_event`, `spdk_pci_register_error_handler` and `spdk_pci_unregister_error_handler`. 897 898### event 899 900Removed the `config_file`, `max_delay_us`, `pci_whitelist` 901and `pci_blacklist` members of struct `spdk_app_opts`. 902 903### idxd 904 905A new API `spdk_idxd_device_needs_rebalance` was added so that users of the library 906can know whether they need to rebalance the flow control for the channel 907that was just added/removed. This is based on how the low level library 908shares devices amongst channels. 909 910The API `spdk_idxd_reconfigure_chan` had the `num_channels` removed as this 911is now tracked in the library. The app makes use the new API above to 912determine whether to rebalance or not. This applies to `spdk_idxd_configure_chan` 913as well. 914 915The API `spdk_idxd_put_channel` now returns the rebalance state for the 916underlying device. 917 918### iscsi 919 920A security vulnerability has been identified and fixed in the SPDK iSCSI target. 921A TEXT PDU with no data, but CONTINUE flag set, would result in a NULL pointer dereference 922and crash the SPDK iSCSI target process. All users of the SPDK iSCSI target 923are recommended to update. All SPDK versions <= v21.01 are affected. 924 925### net 926 927The net library is deprecated and will be removed in the 21.07 release. 928 929### nvme 930 931Added a new function `spdk_nvme_ctrlr_get_regs_pmrcap` to get the PMR capabilities. 932 933Added an accelerated table pointer in `spdk_nvme_poll_group` 934which can be used to provide the accelerated functions by users with 935hardware engine, such as crc32c accelerated function. 936 937Added new functions `spdk_nvme_ctrlr_get_pmrsz`, `spdk_nvme_ctrlr_enable_pmr`, 938`spdk_nvme_ctrlr_disable_pmr`, `spdk_nvme_ctrlr_map_pmr` and `spdk_nvme_ctrlr_unmap_pmr`. 939 940Added NVMe transport operations to enable, disable, map and unmap the PMR. 941 942Added `spdk_nvme_qpair_get_optimal_poll_group` function and `qpair_get_optimal_poll_group` 943function pointer in spdk_nvmf_transport_ops structure in order to add the qpair to the most 944suitable polling group. 945 946Added OPTPERF and namespace optimal performance fields to nvme_spec.h. 947 948Added `spdk_nvme_set_hotplug_filter` API to allow applications to choose which 949hot-inserted SSDs should be probed. This is useful for use cases where multiple 950independent SPDK processes are running on one node. The filter function can 951then be implemented in these processes to decide which SSDs to probe based on 952the new SSD's PCI address. 953 954New functions `spdk_nvme_poll_group_get_stats` and `spdk_nvme_poll_group_free_stats` 955were added. These functions allow to get transport statistics per NVME poll group. 956 957Added `spdk_nvme_map_cmd` API to map the NVMe command with SGL cases. 958 959Added support for vector variant of ZNS zone append commands with new API 960`spdk_nvme_zns_zone_appendv` and `spdk_nvme_zns_zone_appendv_with_md`. 961 962Added `spdk_nvme_zns_ns_get_max_open_zones` and `spdk_nvme_zns_ns_get_max_active_zones` API, 963to display maximum number of open and active zones of the given namespace. 964 965Added `spdk_nvme_zns_ns_get_zone_size_sectors` API to provide size of zone in number of 966sectors. 967 968Added `spdk_nvme_qpair_get_id` API to display the ID of the specified qpair. 969 970### nvmf 971 972Removed the `spdk_nvmf_tgt_listen` and `spdk_nvmf_subsystem_add_ns` API. 973 974Added new APIs: 975 976- `spdk_nvmf_poll_group_dump_stat` (function in `nvmf.h`). 977- `poll_group_dump_stat` (transport op in `nvmf_transport.h`). 978 979The following APIs have been deprecated and will be removed in SPDK 21.07: 980 981- `spdk_nvmf_poll_group_get_stat` (function in `nvmf.h`), 982- `spdk_nvmf_transport_poll_group_get_stat` (function in `nvmf.h`), 983- `spdk_nvmf_transport_poll_group_free_stat`(function in `nvmf.h`), 984- `spdk_nvmf_rdma_device_stat` (struct in `nvmf.h`), 985- `spdk_nvmf_transport_poll_group_stat` (struct in `nvmf.h`), 986- `poll_group_get_stat` (transport op in `nvmf_transport.h`), 987- `poll_group_free_stat` (transport op in `nvmf_transport.h`). 988 989See header files for details. 990 991The `trtype` field in JSON returned by `nvmf_get_stats` RPC contains now the name of the transport, 992which is the same as the type for defined transports and more informative for a custom transport. 993 994Added `hdgst` and `ddgst` parameters to `bdev_nvme_attach_controller` RPC in order change 995state of TCP header and data digest. 996 997Added `num_cqe` parameter to `nvmf_create_transport` RPC to set number of completion queues (CQ) 998for RDMA transport. Useful when CQ resize operation is not supported, for example iWARP. 999 1000### ocf 1001 1002Updated OCF submodule to v20.12.2 1003 1004Added `bdev_ocf_set_cache_mode` RPC to dynamically switch cache mode of OCF bdev. 1005 1006### opal 1007 1008Removed the `spdk_opal_supported` API. 1009 1010### raid 1011 1012For `bdev_raid_create` RPC, the deprecated parameter `strip_size` was removed. 1013 1014### rpc 1015 1016New RPC `bdev_nvme_get_transport_statistics` was added, it allows to get transport statistics 1017of nvme poll groups. 1018 1019Parameter `enable-zerocopy-send` of RPC `sock_impl_set_options` is deprecated and will be removed in SPDK 21.07, 1020use `enable-zerocopy-send-server` or `enable-zerocopy-send-client` instead. 1021Parameter `disable-zerocopy-send` of RPC `sock_impl_set_options` is deprecated and will be removed in SPDK 21.07, 1022use `disable-zerocopy-send-server` or `disable-zerocopy-send-client` instead. 1023 1024Added cmd_parser.py used to parse the args from argparse. There are 1025two benefit to use command parser: 1026 1027- Simplify the definition of rpc method. It will reduce the rpc method code. 1028- Make the rpc call more versatile. User can add private args into rpc method. 1029 1030### rpm 1031 1032Added support for new RPM spec, rpmbuild/spdk.spec, which can be used for packaging the 1033SPDK. The pkg/spdk.spec is considered to be deprecated and scheduled for removal in SPDK 21.07. 1034See [RPM documentation](https://spdk.io/doc/rpm.html) for more details. 1035 1036### sock 1037 1038The type of `enable_placement_id` in struct `spdk_sock_impl_opts` is changed from 1039bool to int. We can use RPC to configure different value of `enable_placement_id`. 1040Then we can leverage SO_INCOMING_CPU to get placement_id, which aims to utilize 1041CPU cache locality, enabled by setting enable_placement_id=2. 1042 1043A new socket placement mode called PLACEMENT_MARK has been added. Some NICs allow 1044sockets to be marked using the SO_MARK socket option as a hint for which hardware 1045queue they should be associated with. This mode leverages that by setting the same 1046value for all sockets within a poll group. 1047 1048New parameters `enable_zerocopy_send_server` and `enable_zerocopy_send_client` were added 1049to struct spdk_sock_impl_opts, these parameters enable or disable zero copy send for server 1050and client sockets which are created using `spdk_sock_listen` and `spdk_sock_listen_ext` (server); 1051`spdk_sock_connect` and `spdk_sock_connect_ext` (client) functions. Existing parameter 1052`enable_zerocopy_send` enables or disables zero copy send for both server and client sockets. 1053 1054### thread 1055 1056A new API `spdk_io_channel_get_io_device` was added to get the io_device for the specified 1057I/O channel. 1058 1059Added `spdk_thread_set_interrupt_mode` function in order to set present spdk_thread into 1060interrupt mode or back to poll mode. It is valid only when thread interrupt facility is 1061enabled by `spdk_interrupt_mode_enable`. 1062 1063Added `spdk_poller_register_interrupt` function to mark that the poller is capable of 1064entering interrupt mode. Callback function will be called when the poller must transition 1065into or out of interrupt mode. 1066 1067### virtio 1068 1069Added the `bdev_virtio_blk_set_hotplug` RPC for the virtio blk pci device. 1070 1071## v21.01 1072 1073### bdev 1074 1075An `opts_size` element was added in the `spdk_bdev_opts` structure to solve the 1076ABI compatibility issue between different SPDK version. And also add `opts_size` 1077parameter in spdk_bdev_get_opts function. Two fields `small_buf_pool_size` and 1078`large_buf_pool_size` were added into spdk_bdev_opts, which were used to determine 1079the small and large buffer pool size of the whole bdev module. 1080 1081A new API `spdk_bdev_wait_for_examine` was added to allow for checking state of 1082examine process. Along with corresponding `bdev_wait_for_examine` RPC, which 1083is now always called during `spdk_bdev_subsystem_config_json` making sure 1084every bdev is ready to be used. 1085 1086A new API `spdk_bdev_io_get_aio_status` was added for getting the status of 1087bdev_io as Linux AIO errno. Also `spdk_bdev_io_complete_aio_status` function 1088and `SPDK_BDEV_IO_STATUS_AIO_ERROR` were added for bdev module to complete 1089a bdev_io with Linux AIO errno. 1090 1091A new API `spdk_bdev_get_module_name` was added for getting block device module name. 1092Also `spdk_bdev_get_module_ctx` was added to obtain a bdev module context for 1093the block device opened by the specified descriptor. 1094 1095Added `max_segment_size` and `max_num_segments` to `spdk_bdev` structure for 1096bdev modules to specify splitting requirements. 1097 1098### blob 1099 1100An `opts_size` element was added in the `spdk_bs_opts` structure to solve the 1101ABI compatibility issue between different SPDK version. And also add `opts_size` 1102parameter in `spdk_bs_opts_init` function. 1103 1104An `opts_size` element was added in the `spdk_blob_opts` structure to solve the 1105ABI compatibility issue between different SPDK version. And also add `opts_size` 1106parameter in `spdk_blob_opts_init` function. 1107 1108An `opts_size` element was added in the `spdk_blob_open_opts` structure to solve the 1109ABI compatibility issue between different SPDK version. And also add `opts_size` 1110parameter in `spdk_blob_open_opts_init` function. 1111 1112### build 1113 1114SPDK now generates pkg-config files to simplify the process of determining which 1115libraries must be linked into an SPDK application. 1116 1117### dpdk 1118 1119Updated DPDK submodule to DPDK 20.11. 1120 1121Removed `--with-igb-uio-driver` configure option. Since DPDK 20.11 igb_uio driver 1122was moved to separate dpdk-kmods repository. If required, users need to refer to 1123dpdk-kmods repository to build the driver. 1124 1125### env 1126 1127The `pci_whitelist`, `pci_blacklist` and `master_core` members of struct `spdk_env_opts` 1128have been deprecated. The new members are named `pci_allowed`, `pci_blocked` and 1129`main_core` respectively. 1130 1131An `opts_size` element was added in the `spdk_app_opts` structure 1132to solve the ABI compatibility issue between different SPDK version. An `opts_size` 1133parameter is added into `spdk_app_opts_init` function. 1134 1135### event 1136 1137The `pci_whitelist` and `pci_blacklist` members of struct `spdk_app_opts` have been 1138deprecated. The new members are named `pci_allowed` and `pci_blocked` respectively. 1139 1140The `--pci-blacklist` command line option has been deprecated, replaced with 1141`--pci-blocked`. 1142 1143The `--pci-whitelist/-W` command line options have been deprecated, replaced with 1144`--pci-allowed/-A`. 1145 1146Added new experimental `dynamic` scheduler that rebalances idle threads, adjusts CPU frequency 1147using dpdk_governor and turns idle reactor cores to interrupt mode. Please see 1148[scheduler documentation](https://www.spdk.io/doc/scheduler.html) for details. 1149 1150## ioat 1151 1152The PCI BDF whitelist option has been removed from the `ioat_scan_accel_engine` RPC. 1153IOAT PCI functions can still be allowed or denied using SPDK application command 1154line options. 1155 1156### isa-l 1157 1158Updated ISA-L submodule to v2.30.0. 1159 1160### json 1161 1162A new API `spdk_jsonrpc_send_bool_response` was added to allow sending response for 1163writing json bool results into one function. 1164Update API `bdev_nvme_set_options` and add a `keep_alive_timeout_ms` parameter. Now you 1165can specify the `keep_alive_timeout` before creating NVMe bdev. 1166 1167### nbd 1168 1169Change the return type of function `spdk_nbd_stop` from void to int. And update the 1170`spdk_nbd_fini` with two parameters to make its behavior from sync to async. 1171 1172### nvme 1173 1174Directives support was added to the NVMe driver. 1175 1176Two async APIs `spdk_nvme_ctrlr_cmd_directive_receive` and `spdk_nvme_ctrlr_cmd_directive_send` 1177are added for Directive Send and Directive Receive command, respectively. 1178 1179Added a new function `spdk_nvme_ctrlr_reset_subsystem` to perform a NVMe 1180subsystem reset. Note: The NVMe-oF target does not support the subsystem reset yet. 1181 1182Add a new function `spdk_nvme_bytes_to_numd` to transfer bytes to number of 1183dwords. 1184 1185Added a new custom transport `SPDK_NVME_TRANSPORT_VFIOUSER` to enable NVMe 1186driver running with NVMe over vfio-user target. 1187 1188Added the vfio-user custom transport implementation in NVMe driver which can connect 1189to NVMe over vfio-user target via vfio-user transport. 1190 1191Added a new function `spdk_nvme_ctrlr_set_remove_cb` to remove or override 1192`remove_cb` and `remove_ctx` specified when the controller was probed. 1193 1194Added support for ZNS zone append command with new API `spdk_nvme_zns_zone_append` and 1195`spdk_nvme_zns_zone_append_with_md`. 1196 1197### nvmf 1198 1199`nvmf_fc_lld_fini` now takes callback and hence updating FC Broadcom LLD driver 1200to the latest is required. 1201 1202FC transport supported primary tagging and VMID. 1203 1204Broadcom FC LLD driver and SPDK NVMe-oF FC transport consolidated one LLD API, 1205`nvmf_fc_init_poller_queues` into another LLD API `nvmf_fc_init_q`. 1206Hence updating Broadcom FC LLD driver to the latest is required. 1207 1208The functions `destroy` and `qpair_fini` in the transport interface now accept a 1209`cb_fn` and `cb_arg` to call upon completion, and their execution can be asynchronous. 1210 1211The SPDK NVMe-oF target now supports async event notification for discovery log changes. 1212This allows the initiator to create persistent connection to discovery controller and 1213be notified of any discovery log changes. 1214 1215An `opts_size` element was added in the `spdk_nvmf_transport_opts` structure 1216to solve the ABI compatibility issue between different SPDK version. And also add 1217`opts_size` parameter in `spdk_nvmf_transport_opts_init` function. 1218 1219Added a new custom vfio-user transport implementation in NVMe-oF which can provide 1220emulated NVMe devices to QEMU and SPDK NVMe driver. 1221 1222Added new API `spdk_nvmf_tgt_listen_ext` that uses `spdk_nvmf_listen_opts` to allow 1223passing generic options to specific transport. This functionality replaces one provided 1224with `nvmf_subsystem_set_options`. Also removed `nvmf_subsystem_set_options` RPC 1225that allowed transport specific options within subsystem. 1226 1227The `trsvcid` in `nvmf_subsystem_add_listener` RPC is now optional. 1228 1229Pausing a subsystem now only pauses all admin queues. This allows for the 1230management of hosts and listeners, plus the addition of namespaces without a 1231full subsystem pause. Additionally, the target now allows for pausing 1232individual namespaces within a subsystem. To remove a namespace from a 1233subsystem, only the specific namespace must be paused. I/O will continue to 1234other namespaces while these operations execute. 1235 1236### rpc 1237 1238An new optional parameter `wait` was added to the RPC `iscsi_create_portal_group`, 1239and an new RPC `iscsi_start_portal_group` was added. They will be used not to start 1240listening on portals for a portal group until all associated target nodes are created 1241at startup, otherwise some iSCSI initiators may fail to re-login when SPDK iSCSI 1242target application restarts. 1243 1244Two optional parameter `--small-buf-pool-size` and `--large-buf-pool-size` were added 1245into `bdev_set_options` function. 1246 1247### vhost 1248 1249Added optional `packed_ring_recovery` parameter to `vhost_create_blk_controller` RPC 1250enabling packed ring live recovery. 1251 1252## v20.10 1253 1254### accel 1255 1256New APIs `spdk_accel_batch_cancel` and `spdk_idxd_batch_cancel` were added to allow 1257cancelling a batch sequence. 1258 1259### bdev 1260 1261A new `spdk_bdev_part_base_construct_ext` function has been added and the 1262`spdk_bdev_part_base_construct` has been deprecated. The 1263`spdk_bdev_part_base_construct_ext` function takes bdev name as an argument instead 1264of bdev structure to avoid a race condition that can happen when the bdev is being 1265removed between a call to get its structure based on a name and actually opening it. 1266 1267Removed `spdk_bdev_config_text` function for bdev modules to report legacy config. 1268 1269### blobstore 1270 1271A new `spdk_bdev_create_bs_dev_ext` function has been added and `spdk_bdev_create_bs_dev_from_desc` 1272function has been deprecated. 1273 1274### dpdk 1275 1276Updated DPDK submodule to DPDK 20.08. 1277 1278### env 1279 1280A new API `spdk_mem_get_fd_and_offset` was added to get file descriptor and offset for 1281a given virtual address. 1282 1283### event 1284 1285Removed `spdk_subsystem_config` callback for submodules as part of legacy config removal. 1286 1287Removed `spdk_app_get_running_config` function that printed configuration in legacy format, 1288and removed `usr1_handler` from `struct spdk_app_opts` callback that was used to call it. 1289 1290Added SPDK thread scheduler framework which is used to rebalance load among CPU cores. 1291The scheduler implementation is pluggable and two schedulers are added first, `static` scheduler 1292and `gscheduler` scheduler. 1293 1294`static` scheduler is used by default and does not reschedule threads and keeps the initial 1295assignments. 1296 1297CPU frequency governor framework and implementation are also added. 1298`dpdk_governor` is the first implementation and is based on rte_power library of DPDK. 1299`gscheduler` scheduler utilizes the CPU frequency governor. 1300 1301Scheduler and governor frameworks are experimental features. 1302 1303### fio 1304 1305Bdev fio_plugin no longer supports legacy configuration files. Options `spdk_conf` and 1306`spdk_json_conf` have to point to valid JSON configuration. 1307 1308### gpt 1309 1310Removed option to disable GPT probing via legacy configuration. Please use generic 1311bdev functionality to replicate the behavior. See `bdev_set_options --disable-auto-examine` 1312and `bdev_examine` RPC. 1313 1314### intel-ipsec-mb 1315 1316Updated intel-ipsec-mb submodule to v0.54 1317 1318### isa-l 1319 1320Updated ISA-L submodule to v2.29.0. 1321 1322### iscsi 1323 1324The iSCSI target supports iSCSI login redirection feature to achieve fault tolerance 1325and load balancing particularly in scale out iSCSI target system, which runs multiple 1326SPDK iSCSI targets. 1327 1328### json 1329 1330A new API `spdk_json_decode_object_relaxed` was added to allow parsing json values 1331even if decoder for a given key is not found. 1332 1333A new API `spdk_json_free_object` was added to free memory allocated by `spdk_json_decode_object`. 1334 1335### log 1336 1337A log flag structure, `spdk_log_register_flag`, `spdk_log_get_first_flag`, 1338`spdk_log_get_next_flag` and macro `SPDK_LOG_REGISTER_COMPONENT`, have been added which 1339enable registering and interacting with log flags. Added `SPDK_INFOLOG`, `SPDK_DEBUGLOG` 1340and `SPDK_LOGDUMP` macros to display logs for specified log flag. 1341 1342The log flags in the event framework `-L` and several SPDK applications has been enabled on 1343release builds of SPDK. On debug builds this option will additionally set 1344log print level to `SPDK_LOG_DEBUG`. 1345 1346A new API `spdk_log_enable_timestamps` and new RPC `log_enable_timestamps` was added to 1347toggle log timestamps. 1348 1349### miscellaneous 1350 1351The contents of the log_rpc library have been moved to the event library. The log_rpc 1352library now no longer exists. 1353 1354The contents of the app_rpc library have been moved to the event library. The app_rpc 1355library now no longer exists. 1356 1357The contents of the bdev_rpc library have been moved to the bdev library. The app_rpc 1358library now no longer exists. 1359 1360The bdevperf application now disables the zcopy API by default. Prior to this change, 1361bdevperf enabled using the zcopy API by default which caused a performance impact of 1362up to 25% on bdevs that don't natively support zcopy because the API emulates zero-copy 1363by allocating a buffer. The bdevperf `-x` param was renamed to `-Z` and the default 1364value changed to false. For bdevs that support zcopy, use the -Z flag to enable 1365using zcopy API. 1366 1367### net 1368 1369Net framework intended for use by user-space TCP stacks was removed. 1370 1371### nvme 1372 1373Zoned Namespace Command Set support has been added to NVMe driver, see `nvme_zns.h` for details. 1374 1375New APIs, `spdk_nvme_detach_async` and `spdk_nvme_detach_poll_async`, have been added to 1376detach multiple controllers in parallel to mitigate lengthy shutdown notification of 1377a few NVMe SSDs. 1378 1379New APIs, `spdk_nvme_ns_get_ana_group_id` and `spdk_nvme_ns_get_ana_state`, have been 1380added to get ANA group ID and ANA state of the given namespace, respectively. 1381 1382If no specific command set is requested (by setting the command_set member in the 1383`spdk_nvme_ctrlr_opts` structure), SPDK will automatically select the most appropriate 1384command set based on what the controller supports. 1385 1386### nvmf 1387 1388A new API `spdk_nvmf_subsystem_add_ns_ext` has been added and the API `spdk_nvmf_subsystem_add_ns` 1389has been deprecated. 1390 1391The NVMe-oF target now supports Asymmetric Namespace Access (ANA) Reporting to provide 1392multipath to NVMe-oF initiator. 1393 1394Add `no_wr_batching` parameter to `spdk_nvmf_transport_opts` struct to disable 1395Work Requests batching in RDMA transport. 1396 1397NVMf Target transports can now parse any additional JSON params in the nvmf_create_transport RPC 1398through the JSON context provided via spdk_nvmf_target_opts->transport_specific. 1399 1400A new RPC `nvmf_subsystem_set_options` was added to allow transport specific options within subsystem. 1401 1402A new API `spdk_nvmf_transport_stop_listen_async` was added to stop accepting new connections 1403at the provided address. `nvmf_subsystem_remove_listener` RPC now uses this function. 1404 1405A new API `spdk_nvmf_subsystem_disconnect_host` was added to disconnect all connections 1406originating from the provided hostnqn. `nvmf_subsystem_remove_host` RPC now uses this function. 1407 1408### ocf 1409 1410Updated OCF submodule to v20.03.1 1411 1412### rpc 1413 1414New optional parameters, `enable_placement_id` and `enable_quickack` were added to the 1415`sock_impl_set_options` RPC. 1416 1417A new RPC `bdev_examine_bdev` was added to allow users to examine a bdev explicitly. 1418It can be used only if bdev_auto_examine is set to false by the RPC `bdev_set_options`. 1419 1420New optional parameters `no_wr_batching` and `control_msg_num` were added to the RPC 1421'nvmf_create_transport'. 1422 1423New RPCs, `iscsi_target_node_set_redirect` and `iscsi_target_node_request_logout`, have 1424been added, and a new optional parameter `private` has been added to the RPC 1425`iscsi_create_portal_group` to support iSCSI login redirection feature. 1426 1427A new optional parameter `ana_reporting` was added to the RPC `nvmf_create_subsystem`, and 1428a new RPC `nvmf_subsystem_listener_set_ana_state` was added for ANA reporting. 1429 1430New RPCs, `nvmf_subsystem_get_listeners` and `nvmf_subsystem_get_qpairs`, were added to 1431retrieve configuration of the NVMe-oF subsystem. 1432 1433### scsi 1434 1435Two new APIs have been added `spdk_scsi_dev_construct_ext` and 1436`spdk_scsi_dev_add_lun_ext` that allow the upper layer(e.g. vhost-scsi) to 1437receive the notification when the scsi bdev has been resized. 1438 1439The `spdk_scsi_dev_construct` and `spdk_scsi_dev_add_lun` eventually may be 1440deprecated and removed. 1441 1442### sock 1443 1444The `enable_placement_id` field was added in the `struct spdk_sock_impl_opts` to 1445make the placement_id feature configurable by users. The default setting is 1446not enabled. 1447 1448The `enable_quick_ack` field was added in the `struct spdk_sock_impl_opts` to enable 1449or disable quick ack for the POSIX sock module. The default setting is not enabled. 1450 1451### thread 1452 1453`fd_group` is applied to support interrupt mode. 1454New APIs were added to support an experimental interrupt mode. This allows modules or 1455libraries to selectively register file descriptors that the spdk_thread can wait on, 1456as an alternative to polling. In v20.10, this functionality is enabled in a very small 1457subset of SPDK libraries and modules. 1458 1459### util 1460 1461A new utility named `fd_group` was add. It is now 1462implemented by epoll on Linux platform. It can be used by 1463spdk_thread and reactor to implement interrupt mode. 1464 1465Added `spdk_bit_pool` utility, see `bit_pool.h` for details. 1466 1467### vhost 1468 1469SPDK has switched to DPDK's rte_vhost library since 19.07 release, removed the internal 1470rte_vhost library which is used for DPDK older than 19.05, removed the experimental vhost 1471nvme target which depends on the internal rte_vhost library. 1472 1473### vpp 1474 1475Removed socket abstraction for VPP. 1476 1477## v20.07: SPDK CSI driver, new accel_fw commands, I/O abort support 1478 1479spdk_json_free_object() was added to free memory allocated by spdk_json_decode_object(). 1480 1481### accel 1482 1483A new API was added `spdk_accel_get_capabilities` that allows applications to 1484query the capabilities of the currently enabled accel engine back-end. 1485 1486A new capability, CRC-32C, was added via `spdk_accel_submit_crc32c`. 1487 1488The software accel engine implementation has added support for CRC-32C. 1489 1490A new capability, compare, was added via `spdk_accel_submit_compare`. 1491 1492The software accel engine implementation has added support for compare. 1493 1494Several APIs were added to `accel_engine.h` to support batched submission 1495of operations. 1496 1497Several APIs were added to `accel_engine.h` to support dualcast operations. 1498 1499### accel_fw 1500 1501The accel_fw was updated to no longer require the app to allocate an 1502accel_task on its behalf. All public APIs now take a callback arg as 1503the parameter that used to be the accel_task. 1504 1505The accel_fw API `spdk_accel_task_size` was removed as it is no longer 1506required. 1507 1508The accel_fw was updated to support compare, dualcast, crc32c. 1509 1510The accel_fw introduced batching support for all commands in all plug-ins. 1511See docs for detailed information. 1512 1513### bdev 1514 1515A new API `spdk_bdev_abort` has been added to submit abort requests to abort all I/Os 1516whose callback context match to the bdev on the given channel. 1517 1518### build 1519 1520The fio plugins now compile to `build/fio` and are named `spdk_bdev` and `spdk_nvme`. 1521Existing fio configuration files will need to be updated. 1522 1523### dpdk 1524 1525Updated DPDK submodule to DPDK 20.05. 1526 1527### env 1528 1529Several new APIs have been added to provide greater flexibility in registering and 1530accessing polled mode PCI drivers. See `env.h` for more details. 1531 1532### idxd 1533 1534The idxd library and plug-in module for the accel_fw were updated to support 1535all accel_fw commands as well as batching. Batching is supported both 1536through the library and the plug-in module. 1537 1538IDXD engine support for CRC-32C has been added. 1539 1540### ioat 1541 1542A new API `spdk_ioat_get_max_descriptors` was added. 1543 1544### nvme 1545 1546An `opts_size`element was added in the `spdk_nvme_ctrlr_opts` structure 1547to solve the ABI compatibility issue between different SPDK version. 1548 1549A new API `spdk_nvme_ctrlr_cmd_abort_ext` has been added to abort previously submitted 1550commands whose callback argument match. 1551 1552Convenience functions, `spdk_nvme_print_command` and `spdk_nvme-print_completion` were added 1553to the public API. 1554 1555A new function, `spdk_nvmf_cuse_update_namespaces`, updates the cuse representation of an NVMe 1556controller. 1557 1558A new function `qpair_iterate_requests` has been added to the nvme transport interface. ALl 1559implementations of the transport interface will have to implement that function. 1560 1561### nvmf 1562 1563The NVMe-oF target no longer supports connecting scheduling configuration and instead 1564always uses what was previously called "transport" scheduling. 1565 1566`spdk_nvmf_tgt_accept` no longer exists. The accept process now occurs automatically after 1567the creation of an nvmf target and queue pairs are assigned to poll groups by the underlying 1568transport. Further, `spdk_nvmf_transport_ops` has changed such that the accept function 1569pointer no longer takes a function pointer as an argument. Instead, transports should call 1570`spdk_nvmf_tgt_new_qpair` whenever they previously would have called that callback. 1571 1572The NVMe-oF target now supports aborting any submitted NVM or Admin command. Previously, 1573the NVMe-oF target could abort only Asynchronous Event Request commands. 1574 1575### rdma 1576 1577A new `rdma` library has been added. It is an abstraction layer over different RDMA providers. 1578Two providers are available - verbs (used by default when RDMA is enabled or enabled explicitly 1579using --with-rdma=verbs) and mlx5 Direct Verbs aka DV (enabled by --with-rdma=mlx5_dv). 1580Using mlx5_dv requires libmlx5 installed on the system. 1581 1582### rpc 1583 1584Parameter `-p` or `--max-qpairs-per-ctrlr` of `nvmf_create_transport` RPC command accepted by the 1585rpc.py script is deprecated, new parameter `-m` or `--max-io-qpairs-per-ctrlr` was added. 1586 1587Added `sock_impl_get_options` and `sock_impl_set_options` RPC methods. 1588 1589Command line parameters `-r` and `--rpc-socket` will longer accept TCP ports. RPC server 1590must now be started on a Unix domain socket. Exposing RPC on the network, as well as providing 1591proper authentication (if needed) is now a responsibility of the user. 1592 1593The `bdev_set_options` RPC has a new option, `bdev_auto_examine` to control the auto examine function 1594of bdev modules. 1595 1596New RPCs `sock_impl_get_options` and `sock_impl_set_options` been added to expose new socket features. 1597See `sock` section for more details. 1598 1599### sock 1600 1601Added `spdk_sock_impl_get_opts` and `spdk_sock_impl_set_opts` functions to set/get socket layer configuration 1602options. Options can be set independently for each implementation. 1603 1604Added `recv_buf_size` and 'send_buf_size' socket layer options. They are used only in posix implementation. 1605 1606Added `uring` based socket implementation, the code is located in module/sock/uring. This feature is only 1607available in Linux which requires kernel version is greater than 5.4.3. Currently, our CI pool added the uring 1608based socket tests for iSCSI target and also the tests for SPDK NVMe-oF tcp transport. 1609 1610Added `enable_recv_pipe` socket layer option to allow disabling of double buffering on receive. 1611New option is used only in posix implementation. 1612 1613Added `enable_zerocopy_send` socket layer option to allow disabling of zero copy flow on send. 1614New option is used only in posix implementation. 1615 1616### util 1617 1618Some previously exposed CRC32 functions have been removed from the public API - 1619`spdk_crc32_update`, `spdk_crc32_table_init`, and the `spdk_crc32_table` struct. 1620 1621### vhost 1622 1623The function `spdk_vhost_blk_get_dev` has been removed. 1624 1625## v20.04: SPDK Top, IDXD, NVMe qpair groups 1626 1627IDXD engine support for compare has been added. 1628 1629### configuration 1630 1631Legacy INI style configuration for SPDK applications has been deprecated and will be 1632removed in future release. 1633Please switch to JSON-RPC configuration files and/or RPC driven run-time configuration. 1634 1635### copy 1636 1637The copy engine library, modules and public APIs have been renamed. Use of the word `copy` 1638has been replaced with the word `accel` short for accelerator in preparation for adding new 1639capabilities in the future. Additionally, APIs for what was previously called the `memcpy` 1640engine have been renamed to identify the engine as a software accelerator. 1641 1642### crypto 1643 1644Support for AES_XTS was added for the QAT polled mode driver (pmd). The create RPC 1645`bdev_crypto_create` has 2 new optional parameters: cipher and key2. Cipher can be either 1646AES_CBC (default) or AES_XTS. AES_XTS is only valid when using the QAT polled mode driver. 1647The key2 parameter is the second key required for AES_XTS. 1648 1649### event 1650 1651Reactors now accumulate CPU stats and they are retrieved by the RPC `framework_get_reactors`. 1652 1653### idxd 1654 1655IDXD support was added in the form of a low level library that can directly 1656utilize the IDXD offload engine as well as an accel plug-in module allowing for use 1657with the generic accel framework API. IDXD is the first in a family of offload 1658engines that share the same interface, specifically DSA is added here. More info 1659can be found here: https://01.org/blogs/2019/introducing-intel-data-streaming-accelerator 1660 1661Much of the implementation models IOAT, however the low level interfaces are very 1662different. The RPC to enable IDXD requires a configuration number as well. The 1663code includes two pre-defined configurations of IDXD groups/work queues/engines. A future 1664version will provide an interface to allow for setting of individual configuration 1665parameters. 1666 1667IDXD is not yet available so this feature should be considered experimental. It will 1668be built up with additional documentation as an ongoing activity. 1669 1670### iscsi 1671 1672The iSCSI target now creates a lightweight thread per poll group instead of assuming a pool 1673of lightweight threads already exist at start up time. A poll group is a collection of 1674unrelated iSCSI connections. Each poll group is only accessed from the associated 1675lightweight thread. 1676 1677### ftl 1678 1679Several changes have been made to the `spdk_ftl_conf`, `spdk_ftl_dev_init_ops`, and 1680`spdk_ftl_attrs` structs. Please see `include/spdk/ftl.h` for more details. 1681 1682### miscellaneous 1683 1684The `--json-ignore-init-errors` command line parameter has been added to ignore 1685initialization errors on JSON config load. 1686 1687The public header file io_channel.h has been removed. Please use thread.h which has the 1688exact same API. 1689 1690### nvme 1691 1692Exported internal nvme_ctrlr_cmd_security_receive/send() APIs as public APIs with "the spdk_" 1693prefix. 1694 1695Added `priority` field in `spdk_nvme_transport_id`, this field is used to specify the priority 1696of the NVMe-oF connection, and currently it is used for NVMe-oF tcp connection. 1697 1698A new poll group API has been added to allow for pooling of nvme qpairs across a single 1699entity which can be polled for completions. This new API consists of the `spdk_nvme_poll_group` 1700family of functions. As a result of this new API, all NVMe transports are expected to implement 1701several poll group related functions. 1702 1703A new flag, `create_only`, has been added to the `spdk_nvme_io_qpair_opts` structure. This flag 1704allows a user to call `spdk_nvme_ctrlr_get_default_io_qpair` without also connecting the qpair 1705within the context of that call. 1706 1707As a result of the `create_only` flag, two new API functions, `spdk_nvme_ctrlr_connect_io_qpair` 1708and `spdk_nvme_ctrlr_disconnect_io_qpair`, have been added to facilitate connecting newly created 1709qpairs (for example, after they have been added to a poll group) and disconnecting qpairs without 1710destroying them (for example to disconnect a qpair before migrating it to a new poll group and 1711reconnecting it). 1712 1713The functions `spdk_nvme_ctrlr_alloc_cmb_io_buffer` and `spdk_nvme_ctrlr_free_cmb_io_buffer` 1714have been changed to `spdk_nvme_ctrlr_map_cmb` and `spdk_nvme_ctrlr_unmap_cmb` respectively. 1715 1716An additional function, `spdk_nvme_ctrlr_reserve_cmb`, has been added to facilitate reserving 1717the entire size of the controller memory buffer for data transfer. 1718 1719### nvme_cuse 1720 1721`spdk_nvme_cuse_get_ctrlr_name` now takes two additional parameters, `char *name` which 1722stores the pointer to the controller name, and `size_t *size` which stores the length of 1723the name. The return type has also been changed from char * to int. 1724 1725`spdk_nvme_cuse_get_ns_name` now takes two additional parameters, `char *name` which 1726stores the pointer to the namespace name, and `size_t *size` which stores the length of 1727the name. The return type has also been changed from char * to int. 1728 1729### nvme_opal 1730 1731Several public OPAL structure definitions have been changed since the last release. 1732 1733### nvmf 1734 1735`spdk_nvmf_poll_group_destroy()` is now asynchronous and accepts a completion callback. 1736 1737The NVMe-oF target now creates a lightweight thread per poll group instead of assuming a pool 1738of lightweight threads already exist at start up time. A poll group is a collection of 1739unrelated NVMe-oF connections. Each poll group is only accessed from the associated 1740lightweight thread. 1741 1742A new struct, `spdk_nvmf_subsystem_listener`, has been added to encapsulate the subsystem specific 1743nature of a listener object. 1744 1745`spdk_nvmf_tgt_listen` no longer accepts a callback function or argument. It also returns an 1746int to indicate the status of the listen call. 1747 1748The execution of `spdk_nvme_poll_group_destroy` is now asynchronous and the function accepts 1749a cb_fn and cb_arg to call upon completion. 1750 1751The execution of `spdk_nvmf_subsystem_add_listener` is now asynchronous and the function accepts 1752a cb_fn and cb_arg to call upon completion. 1753 1754The `nvmf_transport.h` header has been made public to allow custom NVMe-oF transports to integrate 1755with NVMe-oF libraries without using internal APIs. 1756 1757### ocf 1758 1759Updated the OCF submodule to OCF v20.03 1760 1761New version of OCF comes with API changes and bug fixes 1762 1763### rpc 1764 1765A new RPC `thread_set_cpumask` has been added to set the cpumask of the thread 1766to the specified value. 1767 1768A new RPC `thread_get_pollers` has been added to retrieve pollers from SPDK threads. 1769 1770A new RPC `thread_get_io_channels` has been added to retrieve I/O channels from SPDK threads. 1771 1772A new RPC `bdev_rbd_resize` has been added to resize Ceph RBD bdevs. 1773 1774### sock 1775 1776The `spdk_sock_set_priority` function has been removed since the feature to set the sock priority 1777will be contained in two new functions, i.e., `spdk_sock_listen_ext` and `spdk_sock_connect_ext`. 1778Users may now specify the priority of the socket in the opts that they want to use. 1779 1780### spdk_top 1781 1782A new application, `spdk_top`, has been added which allows users to monitor resource consumption 1783by a running SPDK application. More information on this application can be found in 1784`app/spdk_top/README`. 1785 1786### thread 1787 1788A new function `spdk_thread_lib_init_ext` has been added, and the function 1789`spdk_thread_lib_init` has been deprecated. The user of `spdk_thread_lib_init_ext` is 1790expected to implement both functions `spdk_thread_op_fn` and `spdk_thread_op_supported_fn`. 1791`spdk_thread_op_supported_fn` is called to check whether the SPDK thread operation 1792is supported. `spdk_thread_op_fn` is called to execute the SPDK thread operation. 1793Current SPDK operation types are `SPDK_THREAD_OP_NEW` and `SPDK_THREAD_OP_RESCHED`. 1794The operation `SPDK_THREAD_OP_NEW` is called each time a new thread is created. 1795The operation `SPDK_THREAD_OP_RESCHED` is called when SPDK thread needs to be rescheduled. 1796 1797A unique ID has been added for each created SPDK thread, it is retrieved by a new function 1798`spdk_thread_get_id`, and the SPDK thread which has the specific ID is got by 1799a new function `spdk_thread_get_by_id`. 1800 1801A new function `spdk_thread_cpumask` has been added to set the current thread's cpumask 1802to the specified value. The function requires the operation `SPDK_THREAD_OP_RESCHED` 1803is supported. 1804 1805A new function `spdk_poller_register_named` has been added to set arbitrary name to the 1806created poller. If NULL, the name is set to the pointer of the poller function. 1807 1808The function `spdk_thread_poll` now measures run time per call correctly on multiple SPDK 1809threads configuration, and a new function `spdk_thread_get_last_tsc` has been added to use together. 1810 1811Voluntary termination of SPDK thread has been supported by refining the functions `spdk_thread_exit` 1812and `spdk_thread_poll`. 1813 1814### util 1815 1816New functions `spdk_sn32_lt` and `spdk_sn32_gt` have been added. They compare two sequence 1817numbers based on serial number arithmetic. 1818 1819### vhost 1820 1821Poll groups per session have been replaced by SPDK threads per vhost controller. 1822 1823### vmd 1824 1825A new function, `spdk_vmd_fini`, has been added. It releases all resources acquired by the VMD 1826library through the `spdk_vmd_init` call. 1827 1828## v20.01: Optimized thin provisioning, FTL bdev, VMD hot plug, FUSED support 1829 1830### bdev 1831 1832A new function, `spdk_bdev_set_timeout`, has been added to set per descriptor I/O timeouts. 1833 1834A new class of functions `spdk_bdev_compare*`, have been added to allow native bdev support 1835of block comparisons and compare-and-write. 1836 1837A new class of bdev events, `SPDK_BDEV_EVENT_MEDIA_MANAGEMENT`, has been added to allow bdevs 1838which expose raw media to alert all I/O channels of pending media management events. 1839 1840A new API was added `spdk_bdev_io_get_aux_buf` allowing the caller to request 1841an auxiliary buffer for its own private use. The API is used in the same manner that 1842`spdk_bdev_io_get_buf` is used and the length of the buffer is always the same as the 1843bdev_io primary buffer. 'spdk_bdev_io_put_aux_buf' frees the allocated auxiliary 1844buffer. 1845 1846### blobfs 1847 1848Added boolean return value for function spdk_fs_set_cache_size to indicate its operation result. 1849 1850Added `blobfs_set_cache_size` RPC method to set cache size for blobstore filesystem. 1851 1852### blobstore 1853 1854Added new `use_extent_table` option to `spdk_blob_opts` for creating blobs with Extent Table descriptor. 1855Using this metadata format, dramatically decreases number of writes required to persist each cluster allocation 1856for thin provisioned blobs. Extent Table descriptor is enabled by default. 1857See the [Blobstore Programmer's Guide](https://spdk.io/doc/blob.html#blob_pg_cluster_layout) for more details. 1858 1859### dpdk 1860 1861Updated DPDK submodule to DPDK 19.11. 1862 1863### env_dpdk 1864 1865`spdk_env_dpdk_post_init` now takes a boolean, `legacy_mem`, as an argument. 1866 1867A new function, `spdk_env_dpdk_dump_mem_stats`, prints information about the memory consumed by DPDK to a file specified by 1868the user. A new utility, `scripts/dpdk_mem_info.py`, wraps this function and prints the output in an easy to read way. 1869 1870### event 1871 1872The functions `spdk_reactor_enable_framework_monitor_context_switch()` and 1873`spdk_reactor_framework_monitor_context_switch_enabled()` have been changed to 1874`spdk_framework_enable_context_switch_monitor()` and 1875`spdk_framework_context_switch_monitor_enabled()`, respectively. 1876 1877### ftl 1878 1879All NVMe dependencies were removed from ftl library. 1880From now ftl library is using bdev_zone API. 1881bdev_ftl becomes virtual bdev. 1882 1883`ctrlr` and `trid` fields in `spdk_ftl_dev_init_opts` structure were replaced by 1884`base_bdev_desc`. 1885 1886`bdev_ftl_create` RPC method `trtype` and `traddr` parameters were replaced by `base_bdev` 1887parameter. 1888 1889`spdk_ftl_module_init` and `spdk_ftl_module_fini` functions were removed. 1890 1891`spdk_ftl_punit_range` and `ftl_module_init_opts` structures were removed. 1892 1893### isa-l 1894 1895Updated ISA-L submodule to commit f3993f5c0b6911 which includes implementation and 1896optimization for aarch64. 1897 1898Enabled ISA-L on aarch64 by default in addition to x86. 1899 1900### nvme 1901 1902`delayed_pcie_doorbell` parameter in `spdk_nvme_io_qpair_opts` was renamed to `delay_cmd_submit` 1903to allow reuse in other transports. 1904 1905Added RDMA WR batching to NVMf RDMA initiator. Send and receive WRs are chained together 1906and posted with a single call to ibv_post_send(receive) in the next call to qpair completion 1907processing function. Batching is controlled by 'delay_cmd_submit' qpair option. 1908 1909The NVMe-oF initiator now supports plugging out of tree NVMe-oF transports. In order 1910to facilitate this feature, several small API changes have been made: 1911 1912The `spdk_nvme_transport_id` struct now contains a trstring member used to identify the transport. 1913A new function, `spdk_nvme_transport_available_by_name`, has been added. 1914A function table, `spdk_nvme_transport_ops`, and macro, `SPDK_NVME_TRANSPORT_REGISTER`, have been added which 1915enable registering out of tree transports. 1916 1917A new function, `spdk_nvme_ns_supports_compare`, allows a user to check whether a given namespace supports the compare 1918operation. 1919 1920A new family of functions, `spdk_nvme_ns_compare*`, give the user access to submitting compare commands to NVMe namespaces. 1921 1922A new function, `spdk_nvme_ctrlr_cmd_get_log_page_ext`, gives users more granular control over the command dwords sent in 1923log page requests. 1924 1925### nvmf 1926 1927Add SockPriority option in [Transport] section, this can be used for NVMe-oF target 1928on TCP transport to set sock priority for the incoming TCP connections. 1929 1930The NVMe-oF target now supports plugging out of tree NVMe-oF transports. In order 1931to facilitate this feature, several small API changes have been made: 1932 1933The `spdk_nvme_transport_id` struct now contains a trstring member used to identify the transport. 1934`spdk_nvmf_tgt_get_transport`, `spdk_nvmf_transport_opts_init`, and `spdk_nvmf_transport_create` all have been 1935modified to take a string. 1936A function table, `spdk_nvmf_transport_ops`, and macro, `SPDK_NVMF_TRANSPORT_REGISTER`, have been added which 1937enable registering out of tree transports. 1938 1939Add `spdk_nvmf_tgt_stop_listen()` that can be used to stop listening for 1940incoming connections for specified target and trid. Listener is not stopped 1941implicitly upon destruction of a subsystem any more. 1942 1943A custom NVMe admin command handler has been added which allows the user to use the real drive 1944attributes from one of the target NVMe drives when reporting drive attributes to the initiator. 1945This handler can be enabled via the `nvmf_set_config` RPC. 1946Note: In a future version of SPDK, this handler will be enabled by default. 1947 1948The SPDK target and initiator both now include compare-and-write functionality with one caveat. If using the RDMA transport, 1949the target expects the initiator to send both the compare command and write command either with, or without inline data. The 1950SPDK initiator currently respects this requirement, but this note is included as a flag for other initiators attempting 1951compatibility with this version of SPDK. 1952 1953The `spdk_nvme_ctrlr_opts` struct has been extended with new field `transport_ack_timeout` which allows 1954to configure transport ACK timeout. Applicable for RDMA transport only. 1955 1956### rpc 1957 1958A new RPC, `bdev_zone_block_create`, enables creating an emulated zoned bdev on top of a standard block device. 1959 1960A new RPC, `bdev_ocssd_create`, enables creating an emulated zoned bdev on top of an Open Channel SSD. 1961 1962A new RPC, `blobfs_set_cache_size`, enables managing blobfs cache size. 1963 1964A new RPC, `env_dpdk_get_mem_stats`, has been added to facilitate reading DPDK related memory 1965consumption stats. Please see the env_dpdk section above for more details. 1966 1967A new RPC, `framework_get_reactors`, has been added to retrieve a list of all reactors. 1968 1969`bdev_ftl_create` now takes a `base_bdev` argument in lieu of `trtype`, `traddr`, and `punits`. 1970 1971`bdev_nvme_set_options` now allows users to disable I/O submission batching with the `-d` flag 1972 1973`bdev_nvme_cuse_register` now accepts a `name` parameter. 1974 1975`bdev_uring_create` now takes arguments for `bdev_name` and `block_size` 1976 1977`nvmf_set_config` now takes an argument to enable passthru of identify commands to base NVMe devices. 1978Please see the nvmf section above for more details. 1979 1980### scsi 1981 1982`spdk_scsi_lun_get_dif_ctx` now takes an additional argument of type `spdk_scsi_task`. 1983 1984### sock 1985 1986Added spdk_sock_writev_async for performing asynchronous writes to sockets. This call will 1987never return EAGAIN, instead queueing internally until the data has all been sent. This can 1988simplify many code flows that create pollers to continue attempting to flush writes 1989on sockets. 1990 1991Added `impl_name` parameter in spdk_sock_listen and spdk_sock_connect functions. Users may now 1992specify the sock layer implementation they'd prefer to use. Valid implementations are currently 1993"vpp" and "posix" and NULL, where NULL results in the previous behavior of the functions. 1994 1995### thread 1996 1997`spdk_thread_send_msg` now returns int indicating if the message was successfully 1998sent. 1999 2000A new function `spdk_thread_send_critical_msg`, has been added to support sending a single message from 2001a context that may be interrupted, e.g. a signal handler. 2002 2003Two new functions, `spdk_poller_pause`, and `spdk_poller_resume`, have been added to give greater control 2004of pollers to the application owner. 2005 2006### util 2007 2008`spdk_pipe`, a new utility for buffering data from sockets or files for parsing 2009has been added. The public API is available at `include/spdk/pipe.h`. 2010 2011### bdev 2012 2013Added spdk_bdev_io_get_nvme_fused_status function for translating bdev_io status to NVMe status 2014code for fused compare-and-write operation. 2015 2016Added spdk_bdev_get_acwu function for getting block device atomic compare and write unit size. 2017 2018## v19.10: Zoned bdev API, Opal bdev, NVMe character devices 2019 2020### rpc 2021 2022Many of SPDK's RPCs were renamed to be more consistent and intuitive in this release. 2023The old names will continue to function, but will display a deprecation warning. 2024 2025Added optional parameters '--arbitration-burst' and '--low/medium/high-priority-weight' to 2026'bdev_nvme_set_options' RPC method. 2027 2028Added optional parameter '--md-size' to 'construct_null_bdev' RPC method. 2029 2030Added optional parameters '--dif-type' and '--dif-is-head-of-md' to 'construct_null_bdev' 2031RPC method. 2032 2033Added `blobfs_detect` RPC method to detect whether a blobfs exists on given bdev. 2034 2035Added `blobfs_create` RPC method to build blobfs on given bdev. 2036 2037Added `blobfs_mount` RPC method to mount blobfs on given bdev to a host path by FUSE. 2038Then on the host path, user can directly do some file operations which will be mapped 2039to blobfs. 2040 2041### bdev 2042 2043Added new parameter `cdw0` to `spdk_bdev_io_complete_nvme_status()` and 2044`spdk_bdev_io_get_nvme_status()` that allows setting/getting 2045the NVMe completion queue DW0 entry. This allows vendor specific IO commands 2046to return command specific completion info back to the initiator. 2047 2048Added `spdk_bdev_get_write_unit_size()` function for retrieving required number 2049of logical blocks for write operation. 2050 2051New zone-related fields were added to the result of the `get_bdevs` RPC call: 2052 2053- `zoned`: indicates whether the device is zoned or a regular 2054 block device 2055- `zone_size`: number of blocks in a single zone 2056- `max_open_zones`: maximum number of open zones 2057- `optimal_open_zones`: optimal number of open zones 2058 2059The `zoned` field is a boolean and is always present, while the rest is only available for zoned 2060bdevs. 2061 2062A new `spdk_bdev_open_ext` function has been added and `spdk_bdev_open` function has been deprecated. 2063The new open function introduces requirement to provide callback function that will be called by 2064asynchronous event such as bdev removal. `spdk_bdev_open_ext` function takes bdev name as 2065an argument instead of bdev structure to avoid a race condition that can happen when the bdev 2066is being removed between a call to get its structure based on a name and actually openning it. 2067 2068New 'resize' event has been added to notify about change of block count property of block device. 2069Event is delivered only if block device was opened with `spdk_bdev_open_ext` function. 2070 2071### bdev zone 2072 2073Added new public header for zoned bdev. Zoned bdev is an extension 2074of the bdev interface. 2075 2076`spdk_bdev_get_zone_size()`, `spdk_bdev_get_max_open_zones()`, `spdk_bdev_get_optimal_open_zones()` 2077APIs were added for retrieving zoned device information. 2078`spdk_bdev_get_zone_info()` API was added for retrieving information about zones in zoned 2079device. 2080Added `spdk_bdev_zone_management()` API for changing zone state. 2081`spdk_bdev_zone_append()` and `spdk_bdev_zone_append_with_md()` APIs were added for 2082appending data to a zone. 2083Added `spdk_bdev_io_get_append location()` function for retrieving append location for I/O. 2084Added `spdk_bdev_is_zoned()` function for checking if bdev supports zoned namespace semantics. 2085 2086### bdev opal 2087 2088EXPERIMENTAL: A new opal bdev has been added to support management of 2089NVMe self-encrypting drives through the Opal specification. Users can 2090create opal bdevs from an NVMe namespace bdev, if the controller 2091containing that namespace supports Opal. Currently this is only 2092supported for namespace ID=1. The following RPCs have been added to 2093support Opal: `bdev_nvme_opal_init`, `bdev_nvme_opal_revert`, 2094`bdev_opal_create`, `bdev_opal_delete`, `bdev_opal_get_info`, 2095`bdev_opal_new_user`, `bdev_opal_set_lock_state`. 2096It does not yet support recreating the opal bdevs after application restart. 2097This bdev module should be considered very experimental, and the RPCs may 2098change significantly in future releases. 2099 2100### delay bdev 2101 2102The `bdev_delay_update_latency` has been added to allow users to update 2103a latency value for a given delay bdev. 2104 2105### compress bdev 2106 2107A new RPC `bdev_compress_get_orphans` has been added to list compress bdevs 2108that were not loaded due to a missing pm metadata file. In this state they 2109can only be deleted. 2110 2111### null bdev 2112 2113Metadata support has been added to Null bdev module. 2114 2115Protection information support has been added to Null bdev module. 2116 2117### nvme 2118 2119Added `no_shn_notification` to NVMe controller initialization options, users can enable 2120it for NVMe controllers. When the option is enabled, the controller will not do the 2121shutdown process and just disable the controller, users can start their application 2122later again to initialize the controller to the ready state. 2123 2124A controller flag `SPDK_NVME_CTRLR_WRR_SUPPORTED` was added to indicate the controller 2125can support weighted round robin arbitration feature with submission queue. 2126 2127Added `arbitration_burst` option for arbitration feature, and added three 2128`low/medium/high_priority_weight` options for weighted round robin arbitration. 2129 2130Added `spdk_nvme_ns_cmd_write_uncorrectable`. 2131 2132Added new error handling and reporting functionality. This includes several 2133new API functions to facilitate applications recovering when a qpair or 2134controller fails. 2135 2136`spdk_nvme_ctrlr_reconnect_io_qpair` - Reconnects a failed I/O qpair. 2137`spdk_nvme_ctrlr_set_trid` - Sets the trid of an existing controller. Can be used to 2138change the trid for failover cases. 2139`spdk_nvme_ctrlr_is_failed` - Returns the failed state of a controller. 2140`spdk_nvme_ctrlr_fail` - Forces a controller into a failed state. 2141 2142Modified the return behavior of several API functions to better indicate to 2143applications when a qpair is failed. This list of functions includes: 2144 2145`spdk_nvme_qpair_process_completions` 2146`spdk_nvme_ns_cmd_*` 2147`spdk_nvme_ctrlr_process_admin_completions` 2148`spdk_nvme_ctrlr_cmd_*` 2149 2150These functions now return -ENXIO when the qpair or controller on which they 2151operate is failed. 2152 2153EXPERIMENTAL: Added NVMe character device support to allow to create NVMe device nodes in Linux 2154kernel for controller as well as for namespace and process ioctl requests as usual 2155from linux environment. 2156 2157### nvmf 2158 2159The `spdk_nvmf_tgt_create` function now accepts an object of type `spdk_nvmf_target_opts` 2160as its only parameter. This new structure contains the max_subsystems parameter previously 2161passed into that function. 2162 2163A new public API function `spdk_nvmf_get_tgt` has been added which allows users to 2164retrieve a pointer to an `spdk_nvmf_tgt` object by supplying its name. In the special 2165case where an RPC or application only creates a single target, this function can accept 2166a null name parameter and will return the only available target. 2167 2168The majority of the NVMe-oF RPCs now accept an optional tgt_name parameter. This will 2169allow those RPCs to work with applications that create more than one target. 2170 2171Three new NVMe-oF RPCs have been added `nvmf_create_target`, `nvmf_delete_target`, and 2172`nvmf_get_targets`. These new RPCs provide a basic interface for managing multiple target 2173objects. In SPDK the target object defines a unique discovery service. As of this release, 2174these RPCs are not intended to be used with the in-tree SPDK target applications, spdk_tgt and 2175nvmf_tgt, which use a single, global target structure. As such, they are not included in scripts/rpc.py 2176 2177Three new header functions have also been added to help deal with multiple targets. 2178`spdk_nvmf_tgt_get_name` takes a target pointer as an argument and returns its human readable name. 2179`spdk_nvmf_get_first_target` takes no arguments and returns the first target in the global list. 2180`spdk_nvmf_get_next_tgt` takes a target pointer as an argument and returns the next one in the global list. 2181 2182The `spdk_nvmf_tgt_accept` takes additional argument allowing to pass arbitrary context 2183information to the `new_qpair` callback. This will simplify the code when having multiple 2184nvmf targets or when retrieving the context information from globals is not suitable. 2185 2186### blobstore 2187 2188A new `spdk_bdev_create_bs_dev_from_desc` function has been added and `spdk_bdev_create_bs_dev` 2189function has been deprecated. 2190The new create function can cowork with `spdk_bdev_open_ext` function, which provides callback 2191function that will be called by asynchronous event such as bdev removal. 2192 2193### blobfs_bdev 2194 2195A new blobfs module `bdev` has been added to simplify the operations of blobfs on bdev. 2196 2197Function `spdk_blobfs_bdev_detect` is added to detect whether blobfs exists on the given block device. 2198 2199Function `spdk_blobfs_bdev_create` is added to create a blobfs on the given block device. 2200 2201Function `spdk_blobfs_bdev_mount` is added to mount a blobfs on the given block device to 2202a host path by FUSE. Then, a new thread is created dedicatedly for one mountpoint to handle 2203FUSE request by blobfs API. 2204 2205### build 2206 2207Option to build FUSE components into blobfs_bdev module for mounting a blobfs filesystem. 2208It requires the installation of libfuse3. By default, it is disabled. And it will be 2209enabled if run `./configure` with `--with-fuse` option. 2210 2211### iSCSI 2212 2213Portals may no longer be associated with a cpumask. The scheduling of 2214connections is moving to a more dynamic model. 2215 2216An new RPC `iscsi_portal_group_set_auth` has been added to set CHAP authentication 2217for discovery sessions specific for the existing iSCSI portal group. This RPC overwrites 2218the setting by the global parameters for the iSCSI portal group. 2219 2220### socket 2221 2222Added `spdk_sock_is_connected` to check whether the socket is currently connected. 2223`spdk_sock_group_poll` now returns number of events on success. 2224 2225### env 2226 2227Added `spdk_pci_device_unclaim()` function to cleanup pci claim file. 2228 2229### event 2230 2231`framework_start_init` RPC no longer stops the application on error during 2232initialization. 2233 2234### DPDK 2235 2236Updated DPDK submodule to DPDK 19.08. 2237 2238### ocf 2239 2240Updated OCF submodule to OCF v19.06 2241 2242Along with update, new cache mode 'write only' was added. 2243 2244New cache modes added to use via RPC, wi - write invalidate and wa - write around. 2245 2246New version of OCF provides fully asynchronous management API. 2247 2248## v19.07: NVMe-oF FC Transport, VMD, NVMe-oF Persistent reservations, Bdev I/O with separate metadata 2249 2250### ftl 2251 2252EXPERIMENTAL: Added ability to mirror writes to persistent write buffer cache 2253to allow for recovery from dirty shutdown event. 2254 2255Added handling of Asynchronous Nand Management Events (ANM). 2256 2257### vmd 2258 2259EXPERIMENTAL: Added Intel Volume Management Device (VMD) driver. VMD is an integrated 2260controller inside the CPU PCIe root complex. It enables virtual HBAs for the connected 2261NVMe SSDs. `spdk_vmd_init()` enumerates NVMe SSDs behind VMD device and hook them into 2262SPDK PCI subsystem. `spdk_nvme_probe()` or `spdk_nvme_connect()` can be used to connect 2263NVMe driver to the device located at the given transport ID. 2264 2265To obtain transport ID of NVMe SSD behind VMD `spdk_lspci` can be used. 2266 2267Current implementation does not support hotplug. 2268 2269### blobfs 2270 2271Blobfs file asynchronous operations were added to public APIs. 2272 2273### util 2274 2275A new file API `spdk_posix_file_load` was added to load file content into a data buffer. 2276 2277New APIs `spdk_dif_ctx_set_data_offset`, `spdk_dif_verify_stream`, 2278`spdk_dif_get_range_with_md`, `spdk_dif_get_length_with_md` have been added, 2279and existing APIs `spdk_dif_ctx_init`, `spdk_dif_set_md_interleave_iovs`, and 2280`spdk_dif_generate_stream` have been refined to insert or strip DIF into iovs 2281windows of any alignment. 2282 2283New APIs `spdk_dif_ctx_set_remapped_init_ref_tag`, `spdk_dif_remap_ref_tag`, 2284and `spdk_dix_remap_ref_tag` have been added to remap DIF reference tag. 2285 2286New APIs `spdk_dif_update_crc32c` and `spdk_dif_update_crc32c_stream` have been 2287added to compute CRC-32C checksum for extended LBA payload. 2288 2289Bdevperf and bdevio applications now support starting tests with application specific 2290RPCs. Please see helper Python scripts in their respective directories. 2291This is a move towards simpler RPC-only configuration for all main 2292and auxiliary applications. 2293 2294Legacy INI style configuration for SPDK applications will become deprecated in SPDK 19.10, 2295and removed in SPDK 20.01. Please consider moving to JSON-RPC configuration files and/or 2296RPC driven run-time configuration. 2297 2298### nvmf 2299 2300EXPERIMENTAL: A Fibre Channel transport that supports Broadcom HBAs has been 2301added. This depends on the FC HBA driver at 2302https://github.com/ecdufcdrvr/bcmufctdrvr. See [the documentation](https://spdk.io/doc/nvmf.html#nvmf_fc_transport) 2303for more information. 2304 2305Persistent reservation emulation has been added to the NVMe-oF target. Persistent reservation 2306state is stored in a JSON file on the local filesystem between target restart. To support this, 2307an optional parameter to the RPC method `nvmf_subsystem_add_ns` called `--ptpl-file` was added. 2308This allows the user to specify which file to store the persistent reservation state in. Note 2309that this is done per namespace. 2310 2311The c2h success optimization under which a command capsule response is not sent 2312for reads is turned on by default. A config knob was added to allow disabling 2313the optimization. This will mostly be used for integration testing with 5.0.x kernels 2314while some compatibility fixes make their way down the pipeline for 5.1.x kernels. 2315 2316The sock priority setting of the TCP connection owned by the tcp transport is added. It is 2317used to optimize the TCP connection performance under designated traffic classes. And the 2318priority is used to differentiate the sock priority between SPDK NVMe-oF TCP target application 2319and other TCP based applications. 2320 2321Shared receive queue can now be disabled even for NICs that support it using the 2322`nvmf_create_transport` RPC method parameter `no_srq`. The actual use of a shared 2323receive queue is predicated on hardware support when this flag is not used. 2324 2325spdk_nvmf_get_optimal_poll_group was added, which is used to return the optimal 2326poll group for the qpair. And `ConnectionScheduler` configuration is added into the 2327[Nvmf] section in etc/spdk/nvmf.conf.in to demonstrate how to configure the connection 2328scheduling strategy among different spdk threads. 2329 2330Added infrastructure to retrieve global and per poll group NVMf statistics. 2331 2332DIF strip and insert is now supported for TCP transport. When it is enabled, DIF 2333setting is not exposed to the NVMe-oF initiator, and DIF is attached into data 2334for write I/O and stripped from data for read I/O. 2335 2336Added a field `dif_insert_or_strip` to struct spdk_nvmf_transport_opts, and 2337updated the related rpc function nvmf_create_transport to make this 2338configurable parameter available to users. The `dif_insert_or_strip` is relevant 2339for TCP transport for now and used to configure the DIF strip and insert. 2340 2341Added infrastructure to retrieve NVMf transport statistics. 2342 2343### notify 2344 2345The function `spdk_notify_get_types()` and `spdk_notify_get_events()` were 2346renamed to `spdk_notify_foreach_type()` and `spdk_notify_foreach_event()`, 2347respectively. And update type name of callback accordingly. 2348 2349### bdev 2350 2351The format of the data returned by the get_bdevs_iostat RPC has changed to 2352make it easier to parse. It now returns an object with a "ticks" object 2353and "bdevs" array with the per-bdev statistics. 2354 2355A new bdev module `delay` has been added which simulates a drive latency when placed 2356on top of a Null bdev. This module is intended only for testing and can be created using 2357the new RPC `bdev_delay_create`. That RPC takes the name of the underlying bdev as well 2358as average and p99 latency arguments for both read and write operations. Average latency is 2359defined as a value close to what you would expect a perf tool such as FIO to report back as 2360the mean latency of all I/O submitted to the drive. p99 latency is defined as the value one 2361would expect the drive to see the slowest 1% of I/O report. For underlying drives with already 2362significant latency, the latency values provided to the drive will be additive. This should be 2363taken into account if trying to achieve an artificial latency on top of an nvme drive or aio device. 2364 2365DIF reference tag remapping is now supported for partition type virtual bdev 2366modules. When using partition type virtual bdevs, block address space is 2367remapped during I/O processing and DIF reference tag is remapped accordingly. 2368 2369Added spdk_bdev_*_with_md() functions allowing for IO with metadata being transferred in 2370separate buffer. To check support for separate metadata, use spdk_bdev_is_md_separate(). 2371 2372All bdevs now have a UUID. For devices whose backing hardware does not provide a UUID, 2373one is automatically generated. Across runs of SPDK, bdevs whose UUID is automatically 2374generated may change. 2375 2376A new virtual bdev module `compress` has been added to provide compression services on top of 2377a thinly provisioned logical volume. See documentation for complete details. 2378 2379### nvme 2380 2381Added an optional parameter `--io-queue-requests` to RPC `set_bdev_nvme_options`, which 2382can be used to change the number of requests allocated for one NVMe I/O queue. For 2383very big I/O size, e.g. 128MiB, with this option user will not get an error due to 2384limited requests in NVMe driver layer. 2385 2386Added spdk_nvme_ctrlr_get_transport_id() to get the transport ID from a 2387previously attached controller. 2388 2389Nvme Opal library spdk_opal_cmd deprecated. Adding separate command APIs. 2390NVMe Opal library add support for activating locking SP which will make the transaction 2391from "Manufactured-Inactive" state to "Manufactured" state. Upon successfully invoking 2392of this method, lock and unlock features will be enabled. 2393NVMe Opal library add support for locking/unlocking range and list locking range info. 2394NVMe opal library add support for multiuser. Admin can enable user and add user to specific 2395locking range and the user can lock/unlock his range. 2396 2397Added spdk_nvme_ctrlr_io_cmd_raw_no_payload_build() allowing a caller to pass 2398a completely formed command to an NVMe submission queue (buffer addresses and all). 2399This is supported on the PCIe transport only. 2400 2401Added spdk_nvme_get_ctrlr_registers() to return a pointer to the virtual address 2402of the NVMe controller registers. This is supported on the PCIe transport only. 2403 2404Added additional options to the spdk_nvme_ctrlr_alloc_qpair() option parameter 2405structure to allow caller to override the virtual and optionally physical address 2406of the submission and completion queue pair to be created. This is supported on 2407the PCIe transport only. 2408 2409Added `disable_error_logging` to struct spdk_nvme_ctrlr_opts, which disables 2410logging of failed requests. By default logging is enabled. 2411 2412Added spdk_nvme_qpair_print_command(), spdk_nvme_qpair_print_completion() and 2413spdk_nvme_cpl_get_status_string(). Allowing for easier display of error messages. 2414 2415Added support for NVMe Sanitize command. 2416 2417### env 2418 2419The parameter `free_space` has been added to spdk_ring_enqueue() to wait when 2420the ring is almost full and resume when there is enough space available in 2421the ring. 2422 2423A new API `spdk_mempool_lookup` has been added to lookup the memory pool created 2424by the primary process. 2425 2426Added spdk_pci_get_first_device() and spdk_pci_get_next_device() to allow 2427iterating over PCI devices detected by SPDK. Because of this, all SPDK APIs 2428to attach/detach PCI devices are no longer thread safe. They are now meant to 2429be called from only a single thread only, the same only that called spdk_env_init(). 2430This applies to the newly added APIs as well. 2431 2432### vpp 2433 2434SPDK now supports VPP version 19.04.2, up from VPP 18.01. 2435 2436VPP socket abstraction now uses VPP Session API, instead of VLC (VPP Communications Library). 2437This allows for better control over sessions and queues. 2438Please see VPP documentation for more details: 2439[VPP Host Stack](https://wiki.fd.io/view/VPP/HostStack) 2440 2441### sock 2442 2443Add spdk_sock_get_optimal_sock_group(), which returns the optimal sock group for 2444this socket. When a socket is created, it is often assigned to a sock group using 2445spdk_sock_group_add_sock so that a set of sockets can be polled more efficiently. 2446For some network devices, it is optimal to assign particular sockets to specific 2447sock groups. This API is intended to provide the user with that information. 2448 2449spdk_sock_group_get_ctx() was added to return the context of the spdk_sock_group. 2450spdk_sock_group_create() is updated to allow input the user provided ctx. 2451spdk_sock_set_priority() is added to set the priority of the socket. 2452 2453### rpc 2454 2455Added thread_get_stats RPC method to retrieve existing statistics. 2456 2457Added nvmf_get_stats RPC method to retrieve NVMf subsystem statistics. 2458 2459Response buffers for RPC requests are now always pre-allocated, which implies 2460that all spdk_jsonrpc_begin_result() calls always succeed and return a valid 2461buffer for JSON response. RPC calls no longer need to check if the buffer is 2462non-NULL. 2463 2464Added SPDK_RPC_REGISTER_ALIAS_DEPRECATED to help with deprecation process when 2465renaming existing RPC. First time a deprecated alias is used, it will print 2466a warning message. 2467 2468RPC `get_rpc_methods` was renamed `rpc_get_methods`. The old name is still usable, 2469but is now deprecated. 2470 2471### blobstore 2472 2473A snapshot can now be deleted if there is only a single clone on top of it. 2474 2475### build 2476 2477Preliminary support for cross compilation is now available. Targeting an older 2478CPU on the same architecture using your native compiler can be accomplished by 2479using the `--target-arch` option to `configure` as follows: 2480 2481~~~bash 2482./configure --target-arch=broadwell 2483~~~ 2484 2485Additionally, some support for cross-compiling to other architectures has been 2486added via the `--cross-prefix` argument to `configure`. To cross-compile, set CC 2487and CXX to the cross compilers, then run configure as follows: 2488 2489~~~bash 2490./configure --target-arch=aarm64 --cross-prefix=aarch64-linux-gnu 2491~~~ 2492 2493### vhost 2494 2495A security vulnerability has been identified and fixed in SPDK Vhost-SCSI target. 2496A malicious client (e.g. a virtual machine) could send a carefully prepared, 2497invalid I/O request to crash the entire SPDK process. All users of SPDK Vhost-SCSI 2498target are strongly recommended to update. All SPDK versions < 19.07 are affected. 2499 2500By default, SPDK will now rely on upstream DPDK's rte_vhost instead of its fork 2501located inside SPDK repo. The internal fork is still kept around to support older 2502DPDK versions, but is considered legacy and will be eventually removed. 2503 2504`configure` will now automatically use the upstream rte_vhost if the used DPDK 2505version is >= 19.05. 2506 2507spdk_vhost_init() is now asynchronous and accepts a completion callback. 2508 2509### iscsi target 2510 2511A security vulnerability has been identified and fixed in SPDK iSCSI target. 2512A malicious client (e.g. an iSCSI initiator) could send a carefully prepared, 2513invalid I/O request to crash the entire SPDK process. All users of SPDK iSCSI 2514target are strongly recommended to update. All SPDK versions < 19.07 are affected. 2515 2516### thread 2517 2518Exposed spdk_set_thread() in order for applications to associate 2519with SPDK thread when necessary. 2520 2521Added spdk_thread_destroy() to allow framework polling the thread to 2522release resources associated with that thread. 2523 2524## v19.04: Compression bdev, Notification library, NVMe Opal support 2525 2526### nvme 2527 2528Added asynchronous probe support. New APIs spdk_nvme_probe_async(), 2529spdk_nvme_connect_async() and spdk_nvme_probe_poll_async() were added to enable 2530this feature. spdk_nvme_probe_async() and spdk_nvme_connect_async() return a 2531context associated with the specified controllers. Users then call 2532spdk_nvme_probe_poll_async() until it returns 0, indicating that the operation 2533completed. 2534 2535A new qpair creation option, delay_pcie_doorbell, was added. This can be passed 2536to spdk_nvme_alloc_io_qpair(). This makes the I/O submission functions, such as 2537spdk_nvme_ns_writev(), skip ringing the submission queue doorbell. Instead the 2538doorbell will be rung as necessary inside spdk_nvme_qpair_process_completions(). 2539This can result in significantly fewer MMIO writes to the doorbell register 2540under heavy load, greatly improving performance. 2541 2542spdk_nvme_ctrlr_get_regs_cmbsz() was added to report the size of the controller 2543memory buffer, if available. 2544 2545spdk_nvme_ctrlr_get_flags() was added to return controller feature 2546flags. Two flags are currently tracked: 2547SPDK_NVME_CTRLR_SGL_SUPPORTED 2548SPDK_NVME_CTRLR_SECURITY_SEND_RECV_SUPPORTED 2549 2550The NVMe hotplug poller is now able to detach devices hot-removed from the system 2551via `/sys/bus/pci/devices/<bdf>/remove` and `/sys/bus/pci/devices/<bdf>/driver/unbind`. 2552 2553Opal support was added for scan, take ownership, revert TPer, and dumping device 2554info. The nvme_manage tool can be used to perform these operations. The public 2555API functions are spdk_nvme_ctrlr_security_receive() and 2556spdk_nvme_ctrlr_security_send(). This module should be considered experimental 2557pending additional features and tests. 2558 2559The NVMe-oF initiator is now able to transfer IO whose size is more than 128KiB 2560if the corresponding NVMe-oF target allows. 2561 2562### raid 2563 2564Added new strip_size_kb rpc param on create to replace the more ambiguous 2565strip_size. The strip_size rpc param is deprecated. 2566 2567Changed the raid bdev product_name from "Pooled Device" to "Raid Volume" 2568 2569### thread 2570 2571Added spdk_thread_has_pollers() function to verify if there are any registered 2572pollers to be run on the thread. Added spdk_thread_is_idle() function to check 2573if there are any scheduled operations to be performed on the thread at given 2574time. 2575 2576spdk_thread_create() now takes an optional CPU affinity mask that is passed to 2577the scheduler. 2578 2579spdk_thread_lib_init() now takes an optional context size. For each thread 2580created, an additional region of memory of this size will be allocated. A 2581pointer to this region of memory can be obtained by calling 2582spdk_thread_get_ctx(). The inverse operation is also available via 2583spdk_thread_get_from_ctx(). 2584 2585spdk_thread_poll() now optionally accepts the current time, in ticks. This can 2586be used to avoid some calls to spdk_get_ticks() internally. 2587 2588spdk_app_start() now only accepts a single context argument. 2589 2590### bdev 2591 2592An new API `spdk_bdev_get_data_block_size` has been added to get size of data 2593block except for metadata. 2594 2595spdk_vbdev_register() has been deprecated. spdk_bdev_register() should be used 2596instead. 2597 2598A mechanism for acquiring and releasing data buffers from bdev modules, used 2599to perform zero copy operations, was added. 2600 2601New APIs spdk_bdev_get_md_size(), spdk_bdev_is_md_interleaved(), spdk_bdev_get_dif_type(), 2602spdk_bdev_is_dif_head_of_md(), and spdk_bdev_is_dif_check_enabled() have been 2603added to get metadata and DIF settings. 2604 2605Bdevs claimed by the `examine_config` callback will be now further examined in the 2606`examine_disk` callback. 2607 2608spdk_bdev_io_get_io_channel() was added as a convenient way to get an io_channel 2609from a bdev_io. 2610 2611### ocf 2612 2613Updated OCF submodule to OCF v19.3.2 2614 2615Added support for many-to-one configuration for OCF bdev. 2616Multiple core devices can now be cached on single cache device. 2617 2618Added persistent metadata support, allowing to restore cache state after shutdown. 2619During start of SPDK application, the devices are examined and if OCF metadata 2620is present - appropriate OCF bdevs will be recreated. 2621 2622Added Write-Back mode support. In this mode, data is first written to 2623caching device and periodically synchronized to the core devices. 2624Dirty data is saved as persistent metadata on cache device, 2625allowing for safe restore during application restart. 2626For more details please see OCF documentation: 2627[OpenCAS cache configuration](https://open-cas.github.io/cache_configuration.html) 2628 2629### NVMe-oF Target 2630 2631Support for per-device shared receive queues in the RDMA transport has been added. 2632It is enabled by default for any device that supports it. 2633 2634The size of a shared receive queue is defined by transport configuration file parameter 2635`MaxSRQDepth` and `nvmf_create_transport` RPC method parameter `max_srq_depth`. 2636Default size is 4096. 2637 2638Add model number as parameter to construct_nvmf_subsystem (-d option), 2639rather than using hardcoded define. 2640 2641DIF passthrough feature has been added. DIF setting of the allocated bdevs is 2642exposed to the NVMe-oF initiator and data with DIF from the NVMe-oF initiator is 2643passed through to the allocated bdevs. 2644 2645### env 2646 2647The `phys_addr` parameter in spdk_malloc() and spdk_zmalloc() has been deprecated. 2648For retrieving physical addresses, spdk_vtophys() should be used instead. 2649 2650spdk_realloc() has been added to reallocate DMA/shared memory. 2651 2652spdk_pci_device_is_removed() has been added to let the upper-layer SPDK drivers know 2653that device has a pending external hotremove request. 2654 2655spdk_env_fini() and spdk_env_dpdk_post_fini() were added to release any resources 2656allocated by spdk_env_init() or spdk_env_dpdk_post_init() respectively. It is expected 2657that common usage of those functions is to call them just before terminating the process. 2658 2659Added spdk_iommu_is_enabled() to report if SPDK application is using IOMMU for DMA. 2660 2661### DPDK 2662 2663Dropped support for DPDK 17.07 and earlier, which SPDK won't even compile with right now. 2664 2665Updated DPDK submodule to DPDK 19.02. 2666 2667### rpc 2668 2669New `get_spdk_version` RPC method is introduced to get version info of the running SPDK application. 2670 2671The `start_nbd_disk` RPC method now take nbd_device as an optional parameter. If nbd_device 2672is specified, use that specified nbd device. If it's not specified, pick available one. 2673 2674### iSCSI target 2675 2676DIF strip and insert is now supported. DIF settings are not exposed to the iSCSI initiator. 2677DIF is attached into data for write I/O and stripped from data for read I/O. 2678 2679### vhost 2680 2681Added experimental support for running with the external, upstream rte_vhost library. 2682This can be enabled by configuring SPDK with an `--without-internal-vhost-lib` flag. 2683The minimum supported rte_vhost version (DPDK version) is 19.05-rc1. 2684 2685As a result of fuzz testing, a lot of data races in vhost-scsi LUN hotplug path were identified and 2686fixed. Those data races could have potentially resulted in SPDK crashes, RPC hangs, or memory leaks 2687if Vhost-SCSI LUN hotplug RPCs were executed while connected VMs were in the middle of restarting. 2688 2689The SCSI target id in `add_vhost_scsi_lun` RPC is now optional. If `-1` is passed, the first 2690unoccupied target id will be used. 2691 2692### AIO 2693 2694AIO bdev module can now reap I/O completions directly from userspace, significantly improving 2695the overall performance. 2696 2697### blobfs 2698 2699Synchronous IO operations no longer use spdk_io_channel, but instead use 2700spdk_fs_thread_ctx. The behavior is otherwise identical. 2701 2702### OCF 2703 2704Added support for caching multiple bdevs using a single bdev as a cache. 2705 2706### notify 2707 2708Added the notify library that provides a high performance local event bus 2709between libraries. Example usage was added to bdev module, which reports 2710notifications for added and removed bdevs via RPC. 2711 2712### sock 2713 2714Added new API spdk_sock_readv() to the sock library for performing vectored 2715reads. 2716 2717### event 2718 2719The function spdk_subsystem_init() no longer requires spdk_event as an argument. 2720 2721Changed API of spdk_subsystem_config_json() to no longer be asynchronous. 2722 2723### io_uring 2724 2725A bdev module that issues I/O to kernel block devices using the new io_uring Linux kernel 2726API was added. This module requires liburing. 2727 2728### build 2729 2730Options to easily compile with profile guided optimization have been added to 2731`configure`. To use profile guided optimization with SPDK, run 2732`./configure --with-pgo-capture`, build SPDK, then run a workload of your 2733choosing. Then, simply run `./configure --with-pgo-enable` and recompile to 2734build using the generated profile data. Profile guided optimization can yield 2735very large performance improvements, especially on GCC 8 and clang 7. This may 2736be combined with link time optimization which has been available under the 2737`--enable-lto` configure option for several releases. 2738 2739### compression bdev/reduce library 2740 2741Added "reduce" block compression scheme based on using SSDs for storing 2742compressed blocks of storage and persistent memory for metadata. Please see 2743[compression](https://spdk.io/doc/bdev.html) for more details. 2744 2745## v19.01: NVMe-oF TCP/IP Transport, Open Channel SSD Flash Translation Layer, Caching bdev based on OCF, ISA-L Support, DIF/DIX library 2746 2747### ocf bdev 2748 2749New virtual bdev module based on [Open CAS Framework](https://open-cas.github.io/) has been added. 2750This module allows for the use of one bdev to act as a high performance cache in front of another bdev. 2751Please see [documentation](https://spdk.io/doc/bdev.html#bdev_config_cas) for more details. 2752Only write through mode is currently supported and this feature is considered experimental. 2753 2754### event framework 2755 2756For `spdk_app_parse_args`, add return value to the callback which parses application 2757specific command line parameters to protect SPDK applications from crashing by invalid 2758values from user input. 2759 2760By default, all SPDK applications will now reserve all hugepages at runtime. The pre-reserved 2761memory size can be still set with `-s` or `--mem-size` option, although the default value 2762was reduced down to 0. 2763 2764A custom hugetlbfs directory can now be specified via spdk_app_opts. 2765This can be used to configure hugepages with different sizes, a different size limit, 2766or different access permissions than the system's default hugepage pool. 2767SPDK applications can specify a custom hugetlbfs mount with the `--huge-dir` option. 2768 2769### environment 2770 2771spdk_vtophys() has been refactored to accept length of the translated region as a new 2772parameter. The function will now update that parameter with the largest possible value 2773for which the memory is contiguous in the physical memory address space. 2774 2775The following functions were removed: 2776 2777- spdk_pci_nvme_device_attach() 2778- spdk_pci_nvme_enumerate() 2779- spdk_pci_ioat_device_attach() 2780- spdk_pci_ioat_enumerate() 2781- spdk_pci_virtio_device_attach() 2782- spdk_pci_virtio_enumerate() 2783 2784They were replaced with generic spdk_pci_device_attach() and spdk_pci_enumerate() which 2785require a new spdk_pci_driver object to be provided. It can be one of the following: 2786 2787- spdk_pci_nvme_get_driver() 2788- spdk_pci_ioat_get_driver() 2789- spdk_pci_virtio_get_driver() 2790 2791spdk_pci_hook_device() and spdk_pci_unhook_device() were added. Those allow adding a virtual 2792spdk_pci_device into the SPDK PCI subsystem. A virtual device calls provided callbacks for 2793each BAR mapping request or PCI config access. It's attachable with spdk_pci_device_attach() 2794or spdk_pci_enumerate() like any other device. 2795 2796A new spdk_pause() function was added to pause CPU execution for an implementation specific 2797amount of time. Quoting from DPDK function this is based on: "This call is intended for 2798tight loops which poll a shared resource or wait for an event. A short pause within the loop 2799may reduce the power consumption." 2800 2801A new public header file env_dpdk.h has been introduced, and function spdk_env_dpdk_post_init 2802is added into it. If user is using DPDK, and already called rte_eal_init, then include 2803include/spdk/env_dpdk.h, and call spdk_env_dpdk_post_init() instead of spdk_env_init. 2804 2805ISA-L has been added as an SPDK submodule. ISA-L is enabled by default on x86 architecture 2806to accelerate algorithms such as CRC for iSCSI and NVMe-oF. Users may still disable ISA-L 2807by explicitly passing --without-isal to the configure script. 2808 2809### util 2810 2811A new uuid API `spdk_uuid_copy` was added to make a copy of the source uuid. 2812 2813An new parameter `init_crc` representing the initial CRC value was added to 2814`spdk_crc16_t10dif`. The parameter can be used to calculate a CRC value spanning 2815multiple separate buffers. 2816 2817New DIF APIs were added to generate and verify DIF by byte granularity for both DIF and DIX 2818formats. Among them, DIF with copy APIs will be usable to emulate DIF operations such as DIF 2819insert and strip. 2820 2821Added `spdk_strtol` and `spdk_strtoll` to provide additional error checking around `strtol` 2822and `strtoll`. 2823 2824Added `spdk_sprintf_append_realloc` and `spdk_vsprintf_append_realloc` for appending a string 2825with automatic buffer re-allocation. 2826 2827### nvme 2828 2829Wrapper functions spdk_nvme_ctrlr_security_send() and spdk_nvme_ctrlr_security_receive() are 2830introduced to support further security protocol development. 2831 2832admin_timeout_ms was added to NVMe controller initialization options, users 2833can change the default value when probing a controller. 2834 2835Add two new fields "header_digest" and "data_digest" in struct spdk_nvme_ctrlr_opts, 2836it will be used to enable the digest support for the NVMe/TCP transport. 2837 2838Add a new TCP/IP transport(located in lib/nvme/nvme_tcp.c) in nvme driver. With 2839this new transport, it can be used to connect the NVMe-oF target with the 2840same TCP/IP support. 2841 2842Added API, spdk_nvme_ctrlr_is_discovery(), to indicate whether the ctrlr 2843arg refers to a Discovery Controller or not. 2844 2845Added an API function `spdk_nvme_host_id_parse` and corresponding object `spdk_nvme_host_id` 2846for parsing host address and host service ID arguments on a per connection basis. 2847 2848The RPC `construct_nvme_bdev` now allows a user to specify a source address and service id for the host to 2849use when connecting to the controller backing the NVMe bdev. 2850 2851### NVMe-oF Target 2852 2853The `spdk_nvmf_tgt_opts` struct has been deprecated in favor of `spdk_nvmf_transport_opts`. 2854Users will no longer be able to specify target wide I/O parameters. `spdk_nvmf_tgt_listen` 2855will also no longer implicitly initialize a transport with the default target options (since 2856there are none). Instead, a user must manually instantiate the transport with `spdk_nvmf_transport_create` 2857prior to calling `spdk_nvmf_tgt_listen`. 2858 2859Related to the previous change, the rpc `set_nvmf_target_options` has been renamed to 2860`set_nvmf_target_max_subsystems` to indicate that this is the only target option available for the user to edit. 2861 2862Added fields `num_shared_buffers` and `buf_cache_size` in struct spdk_nvmf_transport_opts, 2863and also updated the related rpc function nvmf_create_transport, to make this 2864configurable parameter available to users. The `num_shared_buffers` is used to 2865configure the shared buffer numbers of the transport used by RDMA or TCP transport. 2866`buf_cache_size` configures number of shared buffers to cache per poll group. 2867 2868### nvmf 2869 2870Add a new TCP/IP transport (located in lib/nvmf/tcp.c). With this transport, 2871the SPDK NVMe-oF target can have a new transport, and can serve the NVMe-oF 2872protocol via TCP/IP from the host. 2873 2874Added optional mechanism to modify the RDMA transport's behavior when creating protection domains and registering memory. 2875By default, the RDMA transport will use the ibverbs library to create protection domains and register memory. 2876Using `spdk_nvme_rdma_init_hooks` will subvert that and use an existing registration. 2877 2878### bdev 2879 2880Added `enable_bdev_histogram` and `get_bdev_histogram` RPC commands to allow gathering latency data for specified bdev. 2881Please see [documentation](https://spdk.io/doc/bdev.html#rpc_bdev_histogram) for more details. 2882 2883Added `required_alignment` field to `spdk_bdev`, that specifies an alignment requirement for data buffers associated with an spdk_bdev_io. 2884Bdev layer will automatically double buffer any spdk_bdev_io that violates this alignment, before the spdk_bdev_io is submitted to the bdev module. 2885 2886On shutdown, bdev unregister now proceeds in top-down fashion, with 2887claimed bdevs skipped (these will be unregistered later, when virtual 2888bdev built on top of the respective base bdev unclaims it). This 2889allows virtual bdevs to be shut down cleanly as opposed to the 2890previous behavior that didn't differentiate between hotremove and 2891planned shutdown. 2892 2893The `delete_bdev` RPC is now deprecated. Users should instead use the specific deletion RPC 2894for the bdev type to be removed (i.e. delete_malloc_bdev). 2895 2896Added support for separate bandwidth rate limits for read and write to QoS in bdev layer. 2897 2898Bdev I/O statistics now track unmap operations. 2899 2900### logical volumes 2901 2902Logical volume bdev can now be marked as read only using `set_read_only_lvol_bdev` RPC. 2903This allows for basing clones on top of lvol_bdev without first creating a snapshot. 2904 2905Added option to change method for data erasure when deleting lvol or resizing down. 2906Default of unmapping clusters can now be changed to writing zeroes or no operation. 2907 2908Added option to change method for erasing data region on lvol store creation. 2909Default of unmapping can now be changed to writing zeroes or no operation. 2910 2911### log 2912 2913"trace flags" are now referred to as "log flags" in the SPDK log API. The 2914set_trace_flag, clear_trace_flag and get_trace_flags RPCs are now deprecated, 2915and set_log_flag, clear_log_flag and get_log_flags RPCs have been added. 2916 2917### trace 2918 2919New `trace_record` application was added. It can be used to poll spdk trace shm file and 2920append any new trace entries into another specified file. This can help retain those entries 2921that would otherwise be overwritten in the shm file. See 2922[Capturing sufficient trace events](https://spdk.io/doc/nvmf_tgt_tracepoints.html#capture_trace_events) 2923for more details. 2924 2925Number of trace entries in circular buffer per lcore can now be assigned by starting SPDK app 2926with argument "--num-trace-entries <NUM>" provided. 2927 2928New `get_tpoint_group_mask` RPC was added to get current tpoint_group_mask, and 2929each tpoint group status. 2930New `enable_tpoint_group` and `disable_tpoint_group` RPC were added to enable or 2931disable a specific tpoint group. 2932 2933### ftl 2934 2935EXPERIMENTAL: Added basic flash translation layer module allowing for using Open Channel SSDs as 2936block devices. The module is split into the library (located in lib/ftl) and bdev_ftl 2937(lib/bdev/ftl). See the [documentation](https://spdk.io/doc/ftl.html) for more details. 2938 2939### vhost 2940 2941A security vulnerability has been identified and fixed in the SPDK vhost target. A malicious 2942vhost client (i.e. virtual machine) could carefully construct a circular descriptor chain which 2943would result in a partial denial of service in the SPDK vhost target. These types of descriptor 2944chains are now properly detected by the vhost target. All SPDK vhost users serving untrusted 2945vhost clients are strongly recommended to upgrade. (Reported by Dima Stepanov and Evgeny 2946Yakovlev.) 2947 2948Vhost SCSI and Vhost Block devices can now accept multiple connections on the same socket file. 2949Each connection (internally called a vhost session) will have access to the same storage, but 2950will use different virtqueues, different features and possibly different memory. 2951 2952### vhost scsi 2953 2954SCSI target hotremove can now be performed even without the VIRTIO_SCSI_F_HOTPLUG feature negotiated. 2955Regardless of VIRTIO_SCSI_F_HOTPLUG support, the hotremoval will be still reported through SCSI sense codes. 2956 2957### DPDK 2958 2959DPDK submodule was updated to DPDK 18.11. Note that SPDK does not fully leverage the new 2960multi-process device hotplug yet and continues to work the same way it always did. 2961 2962Dropped support for DPDK 16.07 and earlier, which SPDK won't even compile with right now. 2963 2964### RPC 2965 2966The following RPC commands deprecated in the previous release are now removed: 2967 2968- construct_virtio_user_scsi_bdev 2969- construct_virtio_pci_scsi_bdev 2970- construct_virtio_user_blk_bdev 2971- construct_virtio_pci_blk_bdev 2972- remove_virtio_scsi_bdev 2973- construct_nvmf_subsystem 2974 2975### Miscellaneous 2976 2977The configure options `--with-raid` and `--without-raid` that were deprecated in the previous 2978release are now removed. 2979 2980### nbd 2981 2982Starting nbd using `spdk_nbd_start` is now performed asynchronously. 2983 2984### net framework 2985 2986Net framework initialization and finish is now done asynchronously. 2987 2988### rpc 2989 2990Added `spdk_rpc_is_method_allowed` function for checking whether method is permitted in a given state. 2991Added `spdk_rpc_get_state` to check current state of RPC server. 2992RPC `wait_subsystem_init` has been added to allow clients to block untill all subsystems are initialized. 2993 2994### json rpc 2995 2996JSON RPC client is now running in non-blocking mode. Requests are sent and received during spdk_jsonrpc_client_poll. 2997JSON RPC server can now receive a callback on connection termination or server shutdown using `spdk_jsonrpc_conn_add_close_cb` 2998and `spdk_jsonrpc_conn_del_close_cb`. 2999 3000## v18.10: Dynamic memory allocation, Crypto Virtual Bdev, jsonrpc-client, SPDKCLI iSCSI and NVMe-oF support 3001 3002### nvme 3003 3004spdk_nvme_ctrlr_cmd_security_send() and spdk_nvme_ctrlr_cmd_security_receive() 3005were added to support sending or receiving security protocol data to or from 3006nvme controller. 3007 3008spdk_nvme_ns_get_extended_sector_size() was added. This function includes 3009the metadata size per sector (if any). spdk_nvme_ns_get_sector_size() still 3010returns only the data size per sector, not including metadata. 3011 3012New `send_nvme_cmd` RPC was added to allow sending NVMe commands directly to NVMe controller. 3013See the [send_nvme_cmd](http://spdk.io/doc/jsonrpc.html#rpc_send_nvme_cmd) documentation 3014for more details. 3015 3016### Build System 3017 3018New `configure` options, `--with-shared` and `--without-shared` 3019[default], provide the capability to build, or not, SPDK shared libraries. 3020This includes the single SPDK shared lib encompassing all of the SPDK 3021static libs as well as individual SPDK shared libs corresponding to 3022each of the SPDK static ones. Although the production of the shared 3023libs conforms with conventional version naming practices, such naming 3024does not at this time confer any SPDK ABI compatibility claims. 3025 3026### bdev 3027 3028spdk_bdev_alias_del_all() was added to delete all alias from block device. 3029 3030A new virtual bdev module has been added to perform at rest data encryption using the DPDK CryptoDev 3031Framework. The module initially uses a software AESNI CBC cipher with experimental support for the 3032Intel QAT hardware accelerator also currently implemented with support for CBC cipher. Future work 3033may include additional ciphers as well as consideration for authentication. 3034 3035The RAID virtual bdev module is now always enabled by default. The configure --with-raid and 3036--without-raid options are now ignored and deprecated and will be removed in the next release. 3037 3038Enforcement of bandwidth limits for quality of service (QoS) has been added to the bdev layer. 3039See the new [set_bdev_qos_limit](http://www.spdk.io/doc/jsonrpc.html#rpc_set_bdev_qos_limit) 3040documentation for more details. The previous set_bdev_qos_limit_iops RPC method introduced at 304118.04 release has been deprecated. The new set_bdev_qos_limit RPC method can support both 3042bandwidth and IOPS limits. 3043 3044spdk_bdev_config_json() and corresponding `get_bdevs_config` RPC was removed. 3045 3046### Environment Abstraction Layer and Event Framework 3047 3048The size parameter of spdk_mem_map_translate is now a pointer. This allows the 3049function to report back the actual size of the translation relative to the original 3050request made by the user. 3051 3052A new structure spdk_mem_map_ops has been introduced to hold memory map related 3053callbacks. This structure is now passed as the second argument of spdk_mem_map_alloc 3054in lieu of the notify callback. 3055 3056### DPDK 18.08 3057 3058The DPDK submodule has been updated to the DPDK 18.08 release. SPDK will now automatically 3059utilize DPDK's dynamic memory management with DPDK versions >= 18.05.1. 3060 3061Hugepages can be still reserved with `[-s|--mem-size <size>]` option at application startup, 3062but once we use them all up, instead of failing user allocations with -ENOMEM, we'll try 3063to dynamically reserve even more. This allows starting SPDK with `--mem-size 0` and using 3064only as many hugepages as it is really needed. 3065 3066Due to this change, the memory buffers returned by `spdk_*malloc()` are no longer guaranteed 3067to be physically contiguous. 3068 3069### I/OAT 3070 3071I/OAT driver can now reinitialize I/OAT channels after encountering DMA errors. 3072 3073### iscsi target 3074 3075Parameter names of `set_iscsi_options` and `get_iscsi_global_params` RPC 3076method for CHAP authentication in discovery sessions have been changed to 3077align with `construct_target_node` RPC method. Old names are still usable 3078but will be removed in future release. 3079 3080`set_iscsi_discovery_auth` and `set_iscsi_target_node_auth` RPC methods have 3081been added to set CHAP authentication for discovery sessions and existing 3082target nodes, respectively. 3083 3084The SPDK iSCSI target supports an AuthFile which can be used to load CHAP 3085shared secrets when the iSCSI target starts. SPDK previously provided a 3086default location for this file (`/usr/local/etc/spdk/auth.conf`) if none was 3087specified. This default has been removed. Users must now explicitly specify 3088the location of this file to load CHAP shared secrets from a file, or use 3089the related iSCSI RPC methods to add them at runtime. 3090 3091### iscsi initiator 3092 3093The SPDK iSCSI initiator is no longer considered experimental and becomes 3094a first-class citizen among bdev modules. The basic usage has been briefly 3095described in the bdev user guide: [iSCSI bdev](https://spdk.io/doc/bdev.html#bdev_config_iscsi) 3096 3097### Miscellaneous 3098 3099The SPDK application framework can now parse long name command line parameters. 3100Most single-character parameters have a long name equivalent now. See the 3101[Command Line Parameters](https://spdk.io/doc/app_overview.html) documentation 3102for details or use the `--help` command line parameter to list all available 3103params. 3104 3105bdevperf `-s` param (io size) was renamed to `-o` as `-s` had been already 3106used by existing apps for memory size. 3107 3108bdevio can now accept all SPDK command line parameters. The config now has to 3109be provided with `-c` or `--config` param. 3110 3111The following ioat/perf and nvme/perf parameters were renamed as well: 3112 `-s` (io size) to `-o` 3113 `-d` (mem size) to `-s` 3114 3115The ReactorMask config file parameter has been deprecated. Users should 3116use the -m or --cpumask command line option to specify the CPU core mask 3117for the application. 3118 3119Default config file pathnames have been removed from iscsi_tgt, nvmf_tgt 3120and vhost. Config file pathnames may now only be specified using the 3121-c command line option. 3122 3123Users may no longer set DPDK_DIR in their environment to specify the 3124location of the DPDK installation used to build SPDK. Using DPDK_DIR 3125has not been the documented nor recommended way to specify the DPDK 3126location for several releases, but removing it ensures no unexpected 3127surprises for users who may have DPDK_DIR defined for other reasons. 3128Users should just use the "configure" script to specify the DPDK 3129location before building SPDK. 3130 3131Although we know that many developers still use Python 2 we are officially 3132switching to Python3 with requirement that all new code must be valid also 3133for Python 2 up to the EOL which is year 2020. 3134 3135Invoking interpreter explicitly is forbidden for executable scripts. There 3136is no need to use syntax like "python ./scripts/rpc.py". All executable 3137scripts must contain proper shebang pointing to the right interpreter. 3138Scripts without shebang mustn't be executable. 3139 3140A Python script has been added to enable conversion of old INI config file 3141to new JSON-RPC config file format. This script can be found at 3142scripts/config_converter.py. Example how this script can be used: 3143~~~{.sh} 3144cat old_format.ini | scripts/config_converter.py > new_json_format.json 3145~~~ 3146 3147### Sock 3148 3149Two additional parameters were added to spdk_sock_get_addr() for the server 3150port and client port. These parameters are named "sport" and "cport" 3151respectively. 3152 3153### Virtio 3154 3155The following RPC commands have been deprecated: 3156 3157- construct_virtio_user_scsi_bdev 3158- construct_virtio_pci_scsi_bdev 3159- construct_virtio_user_blk_bdev 3160- construct_virtio_pci_blk_bdev 3161- remove_virtio_scsi_bdev 3162 3163The `construct_virtio_*` ones were replaced with a single `construct_virtio_dev` 3164command that can create any type of Virtio bdev(s). `remove_virtio_scsi_bdev` 3165was replaced with `remove_virtio_bdev` that can delete both Virtio Block and SCSI 3166devices. 3167 3168### Blobfs 3169 3170spdk_file_get_id() returning unique ID for the file was added. 3171 3172### JSON 3173 3174Added jsonrpc-client C library intended for issuing RPC commands from applications. 3175 3176Added API enabling iteration over JSON object: 3177 3178- spdk_json_find() 3179- spdk_json_find_string() 3180- spdk_json_find_array() 3181- spdk_json_object_first() 3182- spdk_json_array_first() 3183- spdk_json_next() 3184 3185### Blobstore 3186 3187Blobstore I/O operations are now based on io_units, instead of blobstore page size. 3188The io_unit size is now the same as the underlying block device's block size. 3189Logical volumes built on a block device with 512B block size can now be used as boot devices 3190in QEMU. 3191 3192### SPDKCLI 3193 3194The SPDKCLI interactive command tool for managing SPDK is no longer considered experimental. 3195Support for the iSCSI and NVMe-oF targets has been added. 3196 3197## v18.07: Raid, Infrastructure Improvements, Bug Fixes 3198 3199### bdev 3200 3201A new public header file bdev_module.h has been introduced to facilitate the 3202development of new bdev modules. This header includes an interface for the 3203spdk_bdev_part and spdk_bdev_part_base objects to enable the creation of 3204multiple virtual bdevs on top of a single base bdev and should act as the 3205primary API for module authors. 3206 3207spdk_bdev_get_opts() and spdk_bdev_set_opts() were added to set bdev-wide 3208options. 3209 3210A mechanism for handling out of memory condition errors (ENOMEM) returned from 3211I/O submission requests at the bdev layer has been added. See 3212spdk_bdev_queue_io_wait(). 3213 3214The spdk_bdev_get_io_stat() function now returns cumulative totals instead of 3215resetting on each call. This allows multiple callers to query I/O statistics 3216without conflicting with each other. Existing users will need to adjust their 3217code to record the previous I/O statistics to calculate the delta between calls. 3218 3219I/O queue depth tracking and samples options have been added. See 3220spdk_bdev_get_qd(), spdk_bdev_get_qd_sampling_period(), and 3221spdk_bdev_set_qd_sampling_period(). 3222 3223### RAID module 3224 3225A new bdev module called "raid" has been added as experimental module which 3226aggregates underlying NVMe bdevs and exposes a single raid bdev. Please note 3227that vhost will not work with this module because it does not yet have support 3228for multi-element io vectors. 3229 3230### Log 3231 3232The debug log component flag available on several SPDK applications has been 3233renamed from `-t` to `-L` to prevent confusion with tracepoints and to allow the 3234option to be added to tools that already use `-t` to mean something else. 3235 3236### Blobstore 3237 3238A new function, spdk_bs_dump(), has been added that dumps all of the contents of 3239a blobstore to a file pointer. This includes the metadata and is very useful for 3240debugging. 3241 3242Two new operations have been added for thin-provisioned blobs. 3243spdk_bs_inflate_blob() will allocate clusters for all thinly provisioned regions 3244of the blob and populate them with the correct data by reading from the backing 3245blob(s). spdk_bs_blob_decouple_parent() works similarly, but will only allocate 3246clusters that correspond to data in the blob's immediate parent. Clusters 3247allocated to grandparents or that aren't allocated at all will remain 3248thin-provisioned. 3249 3250### BlobFS 3251 3252Changed the return type of spdk_file_truncate() from void to int to allow the 3253propagation of `ENOMEM` errors. 3254 3255### NVMe Driver 3256 3257The new API functions spdk_nvme_qpair_add_cmd_error_injection() and 3258spdk_nvme_qpair_remove_cmd_error_injection() have been added for NVMe error 3259emulation. Users can set a specified command to fail with a particular error 3260status. 3261 3262Changed the name `timeout_sec` parameter to `timeout_us` in 3263spdk_nvme_ctrlr_register_timeout_callback(), and also changed the type from 3264uint32_t to uint64_t. This will give users more fine-grained control over the 3265timeout period. 3266 3267Basic support for Open Channel SSDs was added. See nvme_ocssd.h 3268 3269### NVMe Over Fabrics 3270 3271The spdk_nvmf_tgt_destroy() function is now asynchronous and takes a callback 3272as a parameter. 3273 3274spdk_nvmf_qpair_disconnect() was added to allow the user to disconnect qpairs. 3275 3276spdk_nvmf_subsystem_get_max_namespaces() was added to query the maximum allowed 3277number of namespaces for a given subsystem. 3278 3279### Build System 3280 3281The build system now generates a combined shared library (libspdk.so) that may 3282be used in place of the individual static libraries (libspdk_*.a). The combined 3283library includes all components of SPDK and is intended to make linking against 3284SPDK easier. The static libraries are also still provided for users that prefer 3285to link only the minimal set of components required. 3286 3287### git pre-commit and pre-push hooks 3288 3289The pre-commit hook will run `scripts/check_format.sh` and verify there are no 3290formating errors before allowing `git commit` to run. The pre-push hook runs 3291`make CONFIG_WERROR=y` with and without `CONFIG_DEBUG=y` using both the gcc and 3292clang compiler before allowing `git push` to run. Following each DEBUG build 3293`test/unit/unittest.sh` is run and verified. Results are recorded in the 3294`make.log` file. 3295 3296To enable type: 'git config core.hooksPath .githooks'. To override after 3297configuration use the `git --no-verify` flag. 3298 3299### RPC 3300 3301The `start_nbd_disk` RPC method now returns the path to the kernel NBD device node 3302rather than always returning `true`. 3303 3304### DPDK 18.05 3305 3306The DPDK submodule has been rebased on the DPDK 18.05 release. DPDK 18.05 supports 3307dynamic memory allocation, but due to some issues found after the DPDK 18.05 release, 3308that support is not enabled for SPDK 18.07. Therefore, SPDK 18.07 will continue to use 3309the legacy memory allocation model. The plan is to enable dynamic memory allocation 3310after the DPDK 18.08 release which should fix these issues. 3311 3312### Environment Abstraction Layer and Event Framework 3313 3314The spdk_mem_map_translate() function now takes a size parameter to indicate the size of 3315the memory region. This can be used by environment implementations to validate the 3316requested translation. 3317 3318The I/O Channel implementation has been moved to its own library - lib/thread. The 3319public API that was previously in spdk/io_channel.h is now in spdk/thread.h The 3320file spdk/io_channel.h remains and includes spdk/thread.h. 3321 3322spdk_reactor_get_tsc_stats was added to return interesting statistics for each 3323reactor. 3324 3325### IOAT 3326 3327IOAT for copy engine is disabled by default. It can be enabled by specifying the Enable 3328option with "Yes" in `[Ioat]` section of the configuration file. The Disable option is 3329now deprecated and will be removed in a future release. 3330 3331## v18.04: Logical Volume Snapshot/Clone, iSCSI Initiator, Bdev QoS, VPP Userspace TCP/IP 3332 3333### vhost 3334 3335The SPDK vhost-scsi, vhost-blk and vhost-nvme applications have fixes to address the 3336DPDK rte_vhost vulnerability [CVE-2018-1059](http://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2018-1059). 3337Please see this [security advisory](https://access.redhat.com/security/cve/cve-2018-1059) 3338for additional information on the DPDK vulnerability. 3339 3340Workarounds have been added to ensure vhost compatibility with QEMU 2.12. 3341 3342EXPERIMENTAL: Support for vhost-nvme has been added to the SPDK vhost target. See the 3343[vhost documentation](http://www.spdk.io/doc/vhost.html) for more details. 3344 3345### Unified Target Application 3346 3347A new unified SPDK target application, `spdk_tgt`, has been added. This application combines the 3348functionality of several existing SPDK applications, including the iSCSI target, NVMe-oF target, 3349and vhost target. The new application can be managed through the existing configuration file and 3350[JSON-RPC](http://www.spdk.io/doc/jsonrpc.html) methods. 3351 3352### Env 3353 3354spdk_mempool_get_bulk() has been added to wrap DPDK rte_mempool_get_bulk(). 3355 3356New memory management functions spdk_malloc(), spdk_zmalloc(), and spdk_free() have been added. 3357These new functions have a `flags` parameter that allows the user to specify whether the allocated 3358memory needs to be suitable for DMA and whether it should be shared across processes with the same 3359shm_id. The new functions are intended to replace spdk_dma_malloc() and related functions, which will 3360eventually be deprecated and removed. 3361 3362### Bdev 3363 3364A new optional bdev module interface function, `init_complete`, has been added to notify bdev modules 3365when the bdev subsystem initialization is complete. This may be useful for virtual bdevs that require 3366notification that the set of initialization examine() calls is complete. 3367 3368The bdev layer now allows modules to provide an optional per-bdev UUID, which can be retrieved with 3369the spdk_bdev_get_uuid() function. 3370 3371Enforcement of IOPS limits for quality of service (QoS) has been added to the bdev layer. See the 3372[set_bdev_qos_limit_iops](http://www.spdk.io/doc/jsonrpc.html#rpc_set_bdev_qos_limit_iops) documentation 3373for more details. 3374 3375### RPC 3376 3377The `[Rpc]` configuration file section, which was deprecated in v18.01, has been removed. 3378Users should switch to the `-r` command-line parameter instead. 3379 3380The JSON-RPC server implementation now allows up to 32 megabyte responses, growing as 3381needed; previously, the response was limited to 32 kilobytes. 3382 3383### SPDKCLI 3384 3385EXPERIMENTAL: New SPDKCLI interactive command tool for managing SPDK is available. 3386See the [SPDKCLI](http://www.spdk.io/doc/spdkcli.html) documentation for more details. 3387 3388### NVMe Driver 3389 3390EXPERIMENTAL: Support for WDS and RDS capable CMBs in NVMe controllers has been added. This support is 3391experimental pending a functional allocator to free and reallocate CMB buffers. 3392 3393spdk_nvme_ns_get_uuid() has been added to allow retrieval of per-namespace UUIDs when available. 3394 3395New API functions spdk_nvme_ctrlr_get_first_active_ns() and spdk_nvme_ctrlr_get_next_active_ns() 3396have been added to iterate active namespaces, as well as spdk_nvme_ctrlr_is_active_ns() to check if 3397a namespace ID is active. 3398 3399### NVMe-oF Target 3400 3401Namespaces may now be assigned unique identifiers via new optional `eui64` and `nguid` parameters 3402to the `nvmf_subsystem_add_ns` RPC method. Additionally, the NVMe-oF target automatically exposes 3403the backing bdev's UUID as the namespace UUID when available. 3404 3405spdk_nvmf_subsystem_remove_ns() is now asynchronous and requires a callback to indicate completion. 3406 3407### Blobstore 3408 3409A number of functions have been renamed: 3410 3411- spdk_bs_io_write_blob() => spdk_blob_io_write() 3412- spdk_bs_io_read_blob() => spdk_blob_io_read() 3413- spdk_bs_io_writev_blob() => spdk_blob_io_writev() 3414- spdk_bs_io_readv_blob() => spdk_blob_io_readv() 3415- spdk_bs_io_unmap_blob() => spdk_blob_io_unmap() 3416- spdk_bs_io_write_zeroes_blob() => spdk_blob_io_write_zeroes() 3417 3418The old names still exist but are deprecated. They will be removed in the v18.07 release. 3419 3420spdk_blob_resize() is now an asynchronous operation to enable resizing a blob while I/O 3421are in progress to that blob on other threads. An explicit spdk_blob_sync_md() is still 3422required to sync the updated metadata to disk. 3423 3424### Logical Volumes 3425 3426A new `destroy_lvol_bdev` RPC method to delete logical volumes has been added. 3427 3428Lvols now have their own UUIDs which replace previous LvolStoreUUID_BlobID combination. 3429 3430New Snapshot and Clone functionalities have been added. User may create Snapshots of existing Lvols 3431and Clones of existing Snapshots. 3432See the [lvol snapshots](http://www.spdk.io/doc/logical_volumes.html#lvol_snapshots) documentation 3433for more details. 3434 3435Resizing logical volumes is now supported via the `resize_lvol_bdev` RPC method. 3436 3437### Lib 3438 3439A set of changes were made in the SPDK's lib code altering 3440instances of calls to `exit()` and `abort()` to return a failure instead 3441wherever reasonably possible. 3442 3443spdk_app_start() no longer exit()'s on an internal failure, but 3444instead returns a non-zero error status. 3445 3446spdk_app_parse_args() no longer exit()'s on help, '-h', or an invalid 3447option, but instead returns SPDK_APP_PARSE_ARGS_HELP and 3448SPDK_APP_PARSE_ARGS_FAIL, respectively, and SPDK_APP_PARSE_ARGS_SUCCESS 3449on success. 3450 3451spdk_pci_get_device() has been deprecated and will be removed in SPDK v18.07. 3452 3453### I/O Channels 3454 3455The prototype for spdk_poller_fn() has been modified; it now returns a value indicating 3456whether or not the poller did any work. Existing pollers will need to be updated to 3457return a value. 3458 3459### iSCSI Target 3460 3461The SPDK iSCSI target now supports the fd.io Vector Packet Processing (VPP) framework userspace 3462TCP/IP stack. See the [iSCSI VPP documentation](http://www.spdk.io/doc/iscsi.html#vpp) for more 3463details. 3464 3465### iSCSI initiator 3466 3467An iSCSI initiator bdev module has been added to SPDK. This module should be considered 3468experimental pending additional features and tests. More details can be found in 3469lib/bdev/iscsi/README. 3470 3471### PMDK 3472 3473The persistent memory (PMDK) bdev module is now enabled using `--with-pmdk` instead of 3474`--with-nvml`. This reflects the renaming of the persistent memory library from NVML to 3475PMDK. 3476 3477### Virtio Block driver 3478 3479A userspace driver for Virtio Block devices has been added. It was built on top of the 3480[Virtio](http://www.spdk.io/doc/virtio.html) library and can be managed similarly to 3481the Virtio SCSI driver. See the 3482[Virtio Block](http://www.spdk.io/doc/bdev.html#bdev_config_virtio_blk) reference for 3483more information. 3484 3485### Virtio with 2MB hugepages 3486 3487The previous 1GB hugepage limitation has now been lifted. A new `-g` command-line option 3488enables SPDK Virtio to work with 2MB hugepages. 3489See [2MB hugepages](http://www.spdk.io/doc/virtio.html#virtio_2mb) for details. 3490 3491## v18.01: Blobstore Thin Provisioning 3492 3493### Build System 3494 3495The build system now includes a `make install` rule, including support for the common 3496`DESTDIR` and `prefix` variables as used in other build systems. Additionally, the prefix 3497may be set via the configure `--prefix` option. Example: `make install prefix=/usr`. 3498 3499### RPC 3500 3501A JSON RPC listener is now enabled by default using a UNIX domain socket at /var/tmp/spdk.sock. 3502A -r option command line option has been added to enable an alternative UNIX domain socket location, 3503or a TCP port in the format ip_addr:tcp_port (i.e. 127.0.0.1:5260). The Rpc configuration file 3504section is now deprecated and will be removed in the v18.04 release. 3505 3506### I/O Channels 3507 3508spdk_poller_register() and spdk_poller_unregister() were moved from the event 3509framework (include/spdk/event.h) to the I/O channel library 3510(include/spdk/io_channel.h). This allows code that doesn't depend on the event 3511framework to request registration and unregistration of pollers. 3512 3513spdk_for_each_channel() now allows asynchronous operations during iteration. 3514Instead of immediately continuing the interaction upon returning from the iteration 3515callback, the user must call spdk_for_each_channel_continue() to resume iteration. 3516 3517### Block Device Abstraction Layer (bdev) 3518 3519The poller abstraction was removed from the bdev layer. There is now a general purpose 3520abstraction for pollers available in include/spdk/io_channel.h 3521 3522### Lib 3523 3524A set of changes were made in the SPDK's lib code altering, 3525instances of calls to `exit()` and `abort()` to return a failure instead 3526wherever reasonably possible. This has resulted in return type changes of 3527the API for: 3528 3529- spdk_env_init() from type `void` to `int`. 3530- spdk_mem_map_init() from type `void` to `int`. 3531 3532Applications making use of these APIs should be modified to check for 3533a non-zero return value instead of relying on them to fail without return. 3534 3535### NVMe Driver 3536 3537SPDK now supports hotplug for vfio-attached devices. But there is one thing keep in mind: 3538Only physical removal events are supported; removing devices via the sysfs `remove` file will not work. 3539 3540### NVMe-oF Target 3541 3542Subsystems are no longer tied explicitly to CPU cores. Instead, connections are handed out to the available 3543cores round-robin. The "Core" option in the configuration file has been removed. 3544 3545### Blobstore 3546 3547A number of functions have been renamed: 3548 3549- spdk_bs_md_resize_blob() => spdk_blob_resize() 3550- spdk_bs_md_sync_blob() => spdk_blob_sync_md() 3551- spdk_bs_md_close_blob() => spdk_blob_close() 3552- spdk_bs_md_get_xattr_names() => spdk_blob_get_xattr_names() 3553- spdk_bs_md_get_xattr_value() => spdk_blob_get_xattr_value() 3554- spdk_blob_md_set_xattr() => spdk_blob_set_xattr() 3555- spdk_blob_md_remove_xattr() => spdk_blob_remove_xattr() 3556- spdk_bs_md_create_blob() => spdk_bs_create_blob() 3557- spdk_bs_md_open_blob() => spdk_bs_open_blob() 3558- spdk_bs_md_delete_blob() => spdk_bs_delete_blob() 3559- spdk_bs_md_iter_first() => spdk_bs_iter_first() 3560- spdk_bs_md_iter_next() => spdk_bs_iter_next() 3561 3562The function signature of spdk_blob_close() has changed. It now takes a struct spdk_blob * argument 3563rather than struct spdk_blob **. 3564 3565The function signature of spdk_bs_iter_next() has changed. It now takes a struct spdk_blob * argument 3566rather than struct spdk_blob **. 3567 3568Thin provisioning support has been added to the blobstore. It can be enabled by setting the 3569`thin_provision` flag in struct spdk_blob_opts when calling spdk_bs_create_blob_ext(). 3570 3571### NBD device 3572 3573The NBD application (test/lib/bdev/nbd) has been removed; Same functionality can now be 3574achieved by using the test/app/bdev_svc application and start_nbd_disk RPC method. 3575See the [GPT](http://www.spdk.io/doc/bdev.html#bdev_config_gpt) documentation for more details. 3576 3577### FIO plugin 3578 3579SPDK `fio_plugin` now supports FIO 3.3. The support for previous FIO 2.21 has been dropped, 3580although it still remains to work for now. The new FIO contains huge amount of bugfixes and 3581it's recommended to do an update. 3582 3583### Virtio library 3584 3585Previously a part of the bdev_virtio module, now a separate library. Virtio is now available 3586via `spdk_internal/virtio.h` file. This is an internal interface to be used when implementing 3587new Virtio backends, namely Virtio-BLK. 3588 3589### iSCSI 3590 3591The MinConnectionIdleInterval parameter has been removed, and connections are no longer migrated 3592to an epoll/kqueue descriptor on the master core when idle. 3593 3594## v17.10: Logical Volumes 3595 3596### New dependencies 3597 3598libuuid was added as new dependency for logical volumes. 3599 3600libnuma is now required unconditionally now that the DPDK submodule has been updated to DPDK 17.08. 3601 3602### Block Device Abstraction Layer (bdev) 3603 3604An [fio](http://github.com/axboe/fio) plugin was added that can route 3605I/O to the bdev layer. See the [plugin documentation](https://github.com/spdk/spdk/tree/master/examples/bdev/fio_plugin/) 3606for more information. 3607 3608spdk_bdev_unmap() was modified to take an offset and a length in bytes as 3609arguments instead of requiring the user to provide an array of SCSI 3610unmap descriptors. This limits unmaps to a single contiguous range. 3611 3612spdk_bdev_write_zeroes() was introduced. It ensures that all specified blocks will be zeroed out. 3613If a block device doesn't natively support a write zeroes command, the bdev layer emulates it using 3614write commands. 3615 3616New API functions that accept I/O parameters in units of blocks instead of bytes 3617have been added: 3618 3619- spdk_bdev_read_blocks(), spdk_bdev_readv_blocks() 3620- spdk_bdev_write_blocks(), spdk_bdev_writev_blocks() 3621- spdk_bdev_write_zeroes_blocks() 3622- spdk_bdev_unmap_blocks() 3623 3624The bdev layer now handles temporary out-of-memory I/O failures internally by queueing the I/O to be 3625retried later. 3626 3627### Linux AIO bdev 3628 3629The AIO bdev now allows the user to override the auto-detected block size. 3630 3631### NVMe driver 3632 3633The NVMe driver now recognizes the NVMe 1.3 Namespace Optimal I/O Boundary field. 3634NVMe 1.3 devices may report an optimal I/O boundary, which the driver will take 3635into account when splitting I/O requests. 3636 3637The HotplugEnable option in `[Nvme]` sections of the configuration file is now 3638"No" by default. It was previously "Yes". 3639 3640The NVMe library now includes a spdk_nvme_ns_get_ctrlr() function which returns the 3641NVMe Controller associated with a given namespace. 3642 3643The NVMe library now allows the user to specify a host identifier when attaching 3644to a controller. The host identifier is used as part of the Reservations feature, 3645as well as in the NVMe-oF Connect command. The default host ID is also now a 3646randomly-generated UUID, and the default host NQN uses the host ID to generate 3647a UUID-based NQN. 3648 3649spdk_nvme_connect() was added to allow the user to connect directly to a single 3650NVMe or NVMe-oF controller. 3651 3652### NVMe-oF Target (nvmf_tgt) 3653 3654The NVMe-oF target no longer requires any in-capsule data buffers to run, and 3655the feature is now entirely optional. Previously, at least 4 KiB in-capsule 3656data buffers were required. 3657 3658NVMe-oF subsystems have a new configuration option, AllowAnyHost, to control 3659whether the host NQN whitelist is enforced when accepting new connections. 3660If no Host options have been specified and AllowAnyHost is disabled, the 3661connection will be denied; this is a behavior change from previous releases, 3662which allowed any host NQN to connect if the Host list was empty. 3663AllowAnyHost is disabled by default. 3664 3665NVMe-oF namespaces may now be assigned arbitrary namespace IDs, and the number 3666of namespaces per subsystem is no longer limited. 3667 3668The NVMe-oF target now supports the Write Zeroes command. 3669 3670### Environment Abstraction Layer 3671 3672A new default value, SPDK_MEMPOOL_DEFAULT_CACHE_SIZE, was added to provide 3673additional clarity when constructing spdk_mempools. Previously, -1 could be 3674passed and the library would choose a reasonable default, but this new value 3675makes it explicit that the default is being used. 3676 3677### Blobstore 3678 3679The blobstore super block now contains a bstype field to identify the type of the blobstore. 3680Existing code should be updated to fill out bstype when calling spdk_bs_init() and spdk_bs_load(). 3681 3682spdk_bs_destroy() was added to allow destroying blobstore on device 3683with an initialized blobstore. 3684 3685spdk_bs_io_readv_blob() and spdk_bs_io_writev_blob() were added to enable 3686scattered payloads. 3687 3688A CLI tool for blobstore has been added, allowing basic operations through either command 3689line or shell interface. See the [blobcli](https://github.com/spdk/spdk/tree/master/examples/blob/cli) 3690documentation for more details. 3691 3692### Event Framework 3693 3694The ability to set a thread name, previously only used by the reactor code, is 3695now part of the spdk_thread_allocate() API. Users may specify a thread name 3696which will show up in tools like `gdb`. 3697 3698### Log 3699 3700The spdk_trace_dump() function now takes a new parameter to allow the caller to 3701specify an output file handle (stdout or stderr, for example). 3702 3703### Logical Volumes 3704 3705Logical volumes library built on top of SPDK blobstore has been added. 3706It is possible to create logical volumes on top of other devices using RPC. 3707 3708See the [logical volumes](http://www.spdk.io/doc/logical_volumes.html) documentation for more information. 3709 3710### Persistent Memory 3711 3712A new persistent memory bdev type has been added. 3713The persistent memory block device is built on top of [libpmemblk](http://pmem.io/nvml/libpmemblk/). 3714It is possible to create pmem devices on top of pmem pool files using RPC. 3715 3716See the [Pmem Block Device](http://www.spdk.io/doc/bdev.html#bdev_config_pmem) documentation for more information. 3717 3718### Virtio SCSI driver 3719 3720A userspace driver for Virtio SCSI devices has been added. 3721The driver is capable of creating block devices on top of LUNs exposed by another SPDK vhost-scsi application. 3722 3723See the [Virtio SCSI](http://www.spdk.io/doc/virtio.html) documentation and 3724[Getting Started](http://www.spdk.io/doc/bdev.html#bdev_config_virtio_scsi) guide for more information. 3725 3726### Vhost target 3727 3728The vhost target application now supports live migration between QEMU instances. 3729 3730## v17.07: Build system improvements, userspace vhost-blk target, and GPT bdev 3731 3732### Build System 3733 3734A `configure` script has been added to simplify the build configuration process. 3735The existing CONFIG file and `make CONFIG_...` options are also still supported. 3736Run `./configure --help` for information about available configuration options. 3737 3738A DPDK submodule has been added to make building SPDK easier. If no `--with-dpdk` 3739option is specified to configure, the SPDK build system will automatically build a 3740known-good configuration of DPDK with the minimal options enabled. See the Building 3741section of README.md for more information. 3742 3743A [Vagrant](https://www.vagrantup.com/) setup has been added to make it easier to 3744develop and use SPDK on systems without suitable NVMe hardware. See the Vagrant 3745section of README.md for more information. 3746 3747### Userspace vhost-blk target 3748 3749The vhost library and example app have been updated to support the vhost-blk 3750protocol in addition to the existing vhost-scsi protocol. 3751See the [vhost documentation](http://www.spdk.io/doc/vhost.html) for more details. 3752 3753### Block device abstraction layer (bdev) 3754 3755A GPT virtual block device has been added, which automatically exposes GPT partitions 3756with a special SPDK-specific partition type as bdevs. 3757See the [GPT bdev documentation](http://www.spdk.io/doc/bdev.md#bdev_config_gpt) for 3758more information. 3759 3760### NVMe driver 3761 3762The NVMe driver has been updated to support recent Intel SSDs, including the Intel® 3763Optane™ SSD DC P4800X series. 3764 3765A workaround has been added for devices that failed to recognize register writes 3766during controller reset. 3767 3768The NVMe driver now allocates request tracking objects on a per-queue basis. The 3769number of requests allowed on an I/O queue may be set during `spdk_nvme_probe()` by 3770modifying `io_queue_requests` in the opts structure. 3771 3772The SPDK NVMe `fio_plugin` has been updated to support multiple threads (`numjobs`). 3773 3774spdk_nvme_ctrlr_alloc_io_qpair() has been modified to allow the user to override 3775controller-level options for each individual I/O queue pair. 3776Existing callers with qprio == 0 can be updated to: 3777~~~ 3778... = spdk_nvme_ctrlr_alloc_io_qpair(ctrlr, NULL, 0); 3779~~~ 3780Callers that need to specify a non-default qprio should be updated to: 3781~~~ 3782struct spdk_nvme_io_qpair_opts opts; 3783spdk_nvme_ctrlr_get_default_io_qpair_opts(ctrlr, &opts, sizeof(opts)); 3784opts.qprio = SPDK_NVME_QPRIO_...; 3785... = spdk_nvme_ctrlr_alloc_io_qpair(ctrlr, &opts, sizeof(opts)); 3786~~~ 3787 3788### Environment Abstraction Layer 3789 3790The environment abstraction layer has been updated to include several new functions 3791in order to wrap additional DPDK functionality. See `include/spdk/env.h` for the 3792current set of functions. 3793 3794### SPDK Performance Analysis with Intel® VTune™ Amplifier 3795 3796Support for SPDK performance analysis has been added to Intel® VTune™ Amplifier 2018. 3797 3798This analysis provides: 3799 3800- I/O performance monitoring (calculating standard I/O metrics like IOPS, throughput, etc.) 3801- Tuning insights on the interplay of I/O and compute devices by estimating how many cores 3802 would be reasonable to provide for SPDK to keep up with a current storage workload. 3803 3804See the VTune Amplifier documentation for more information. 3805 3806## v17.03: Blobstore and userspace vhost-scsi target 3807 3808### Blobstore and BlobFS 3809 3810The blobstore is a persistent, power-fail safe block allocator designed to be 3811used as the local storage system backing a higher-level storage service. 3812See the [blobstore documentation](http://www.spdk.io/doc/blob.html) for more details. 3813 3814BlobFS adds basic filesystem functionality like filenames on top of the blobstore. 3815This release also includes a RocksDB Env implementation using BlobFS in place of the 3816kernel filesystem. 3817See the [BlobFS documentation](http://www.spdk.io/doc/blobfs.html) for more details. 3818 3819### Userspace vhost-scsi target 3820 3821A userspace implementation of the QEMU vhost-scsi protocol has been added. 3822The vhost target is capable of exporting SPDK bdevs to QEMU-based VMs as virtio devices. 3823See the [vhost documentation](http://www.spdk.io/doc/vhost.html) for more details. 3824 3825### Event framework 3826 3827The overhead of the main reactor event loop was reduced by optimizing the number of 3828calls to spdk_get_ticks() per iteration. 3829 3830### NVMe library 3831 3832The NVMe library will now automatically split readv/writev requests with scatter-gather 3833lists that do not map to valid PRP lists when the NVMe controller does not natively 3834support SGLs. 3835 3836The `identify` and `perf` NVMe examples were modified to add a consistent format for 3837specifying remote NVMe over Fabrics devices via the `-r` option. 3838This is implemented using the new `spdk_nvme_transport_id_parse()` function. 3839 3840### iSCSI Target 3841 3842The [Nvme] section of the configuration file was modified to remove the `BDF` directive 3843and replace it with a `TransportID` directive. Both local (PCIe) and remote (NVMe-oF) 3844devices can now be specified as the backing block device. A script to generate an 3845entire [Nvme] section based on the local NVMe devices attached was added at 3846`scripts/gen_nvme.sh`. 3847 3848### NVMe-oF Target 3849 3850The [Nvme] section of the configuration file was modified to remove the `BDF` directive 3851and replace it with a `TransportID` directive. Both local (PCIe) and remote (NVMe-oF) 3852devices can now be specified as the backing block device. A script to generate an 3853entire [Nvme] section based on the local NVMe devices attached was added at 3854`scripts/gen_nvme.sh`. 3855 3856## v16.12: NVMe over Fabrics host, hotplug, and multi-process 3857 3858### NVMe library 3859 3860The NVMe library has been changed to create its own request memory pool rather than 3861requiring the user to initialize the global `request_mempool` variable. Apps can be 3862updated by simply removing the initialization of `request_mempool`. Since the NVMe 3863library user no longer needs to know the size of the internal NVMe request 3864structure to create the pool, the `spdk_nvme_request_size()` function was also removed. 3865 3866The `spdk_nvme_ns_cmd_deallocate()` function was renamed and extended to become 3867`spdk_nvme_ns_cmd_dataset_management()`, which allows access to all of the NVMe 3868Dataset Management command's parameters. Existing callers can be updated to use 3869`spdk_nvme_ns_cmd_dataset_management()` with `SPDK_NVME_DSM_ATTR_DEALLOCATE` as the 3870`type` parameter. 3871 3872The NVMe library SGL callback prototype has been changed to return virtual addresses 3873rather than physical addresses. Callers of `spdk_nvme_ns_cmd_readv()` and 3874`spdk_nvme_ns_cmd_writev()` must update their `next_sge_fn` callbacks to match. 3875 3876The NVMe library now supports NVMe over Fabrics devices in addition to the existing 3877support for local PCIe-attached NVMe devices. For an example of how to enable 3878NVMe over Fabrics support in an application, see `examples/nvme/identify` and 3879`examples/nvme/perf`. 3880 3881Hot insert/remove support for NVMe devices has been added. To enable NVMe hotplug 3882support, an application should call the `spdk_nvme_probe()` function on a regular 3883basis to probe for new devices (reported via the existing `probe_cb` callback) and 3884removed devices (reported via a new `remove_cb` callback). Hotplug is currently 3885only supported on Linux with the `uio_pci_generic` driver, and newly-added NVMe 3886devices must be bound to `uio_pci_generic` by an external script or tool. 3887 3888Multiple processes may now coordinate and use a single NVMe device simultaneously 3889using [DPDK Multi-process Support](http://dpdk.org/doc/guides/prog_guide/multi_proc_support.html). 3890 3891### NVMe over Fabrics target (`nvmf_tgt`) 3892 3893The `nvmf_tgt` configuration file format has been updated significantly to enable 3894new features. See the example configuration file `etc/spdk/nvmf.conf.in` for 3895more details on the new and changed options. 3896 3897The NVMe over Fabrics target now supports virtual mode subsystems, which allow the 3898user to export devices from the SPDK block device abstraction layer as NVMe over 3899Fabrics subsystems. Direct mode (raw NVMe device access) is also still supported, 3900and a single `nvmf_tgt` may export both types of subsystems simultaneously. 3901 3902### Block device abstraction layer (bdev) 3903 3904The bdev layer now supports scatter/gather read and write I/O APIs, and the NVMe 3905blockdev driver has been updated to support scatter/gather. Apps can use the 3906new scatter/gather support via the `spdk_bdev_readv()` and `spdk_bdev_writev()` 3907functions. 3908 3909The bdev status returned from each I/O has been extended to pass through NVMe 3910or SCSI status codes directly in cases where the underlying device can provide 3911a more specific status code. 3912 3913A Ceph RBD (RADOS Block Device) blockdev driver has been added. This allows the 3914`iscsi_tgt` and `nvmf_tgt` apps to export Ceph RBD volumes as iSCSI LUNs or 3915NVMe namespaces. 3916 3917### General changes 3918 3919`libpciaccess` has been removed as a dependency and DPDK PCI enumeration is 3920used instead. Prior to DPDK 16.07 enumeration by class code was not supported, 3921so for earlier DPDK versions, only Intel SSD DC P3x00 devices will be discovered 3922by the NVMe library. 3923 3924The `env` environment abstraction library has been introduced, and a default 3925DPDK-based implementation is provided as part of SPDK. The goal of the `env` 3926layer is to enable use of alternate user-mode memory allocation and PCI access 3927libraries. See `doc/porting.md` for more details. 3928 3929The build process has been modified to produce all of the library files in the 3930`build/lib` directory. This is intended to simplify the use of SPDK from external 3931projects, which can now link to SPDK libraries by adding the `build/lib` directory 3932to the library path via `-L` and linking the SPDK libraries by name (for example, 3933`-lspdk_nvme -lspdk_log -lspdk_util`). 3934 3935`nvmf_tgt` and `iscsi_tgt` now have a JSON-RPC interface, which allows the user 3936to query and modify the configuration at runtime. The RPC service is disabled by 3937default, since it currently does not provide any authentication or security 3938mechanisms; it should only be enabled on systems with controlled user access 3939behind a firewall. An example RPC client implemented in Python is provided in 3940`scripts/rpc.py`. 3941 3942## v16.08: iSCSI target, NVMe over Fabrics maturity 3943 3944This release adds a userspace iSCSI target. The iSCSI target is capable of exporting 3945NVMe devices over a network using the iSCSI protocol. The application is located 3946in app/iscsi_tgt and a documented configuration file can be found at etc/spdk/spdk.conf.in. 3947 3948This release also significantly improves the existing NVMe over Fabrics target. 3949 3950- The configuration file format was changed, which will require updates to 3951 any existing nvmf.conf files (see `etc/spdk/nvmf.conf.in`): 3952 - `SubsystemGroup` was renamed to `Subsystem`. 3953 - `AuthFile` was removed (it was unimplemented). 3954 - `nvmf_tgt` was updated to correctly recognize NQN (NVMe Qualified Names) 3955 when naming subsystems. The default node name was changed to reflect this; 3956 it is now "nqn.2016-06.io.spdk". 3957 - `Port` and `Host` sections were merged into the `Subsystem` section 3958 - Global options to control max queue depth, number of queues, max I/O 3959 size, and max in-capsule data size were added. 3960 - The Nvme section was removed. Now a list of devices is specified by 3961 bus/device/function directly in the Subsystem section. 3962 - Subsystems now have a Mode, which can be Direct or Virtual. This is an attempt 3963 to future-proof the interface, so the only mode supported by this release 3964 is "Direct". 3965- Many bug fixes and cleanups were applied to the `nvmf_tgt` app and library. 3966- The target now supports discovery. 3967 3968This release also adds one new feature and provides some better examples and tools 3969for the NVMe driver. 3970 3971- The Weighted Round Robin arbitration method is now supported. This allows 3972 the user to specify different priorities on a per-I/O-queue basis. To 3973 enable WRR, set the `arb_mechanism` field during `spdk_nvme_probe()`. 3974- A simplified "Hello World" example was added to show the proper way to use 3975 the NVMe library API; see `examples/nvme/hello_world/hello_world.c`. 3976- A test for measuring software overhead was added. See `test/lib/nvme/overhead`. 3977 3978## v16.06: NVMf userspace target 3979 3980This release adds a userspace NVMf (NVMe over Fabrics) target, conforming to the 3981newly-released NVMf 1.0/NVMe 1.2.1 specification. The NVMf target exports NVMe 3982devices from a host machine over the network via RDMA. Currently, the target is 3983limited to directly exporting physical NVMe devices, and the discovery subsystem 3984is not supported. 3985 3986This release includes a general API cleanup, including renaming all declarations 3987in public headers to include a `spdk` prefix to prevent namespace clashes with 3988user code. 3989 3990- NVMe 3991 - The `nvme_attach()` API was reworked into a new probe/attach model, which 3992 moves device detection into the NVMe library. The new API also allows 3993 parallel initialization of NVMe controllers, providing a major reduction in 3994 startup time when using multiple controllers. 3995 - I/O queue allocation was changed to be explicit in the API. Each function 3996 that generates I/O requests now takes a queue pair (`spdk_nvme_qpair *`) 3997 argument, and I/O queues may be allocated using 3998 `spdk_nvme_ctrlr_alloc_io_qpair()`. This allows more flexible assignment of 3999 queue pairs than the previous model, which only allowed a single queue 4000 per thread and limited the total number of I/O queues to the lowest number 4001 supported on any attached controller. 4002 - Added support for the Write Zeroes command. 4003 - `examples/nvme/perf` can now report I/O command latency from the 4004 the controller's viewpoint using the Intel vendor-specific read/write latency 4005 log page. 4006 - Added namespace reservation command support, which can be used to coordinate 4007 sharing of a namespace between multiple hosts. 4008 - Added hardware SGL support, which enables use of scattered buffers that 4009 don't conform to the PRP list alignment and length requirements on supported 4010 NVMe controllers. 4011 - Added end-to-end data protection support, including the ability to write and 4012 read metadata in extended LBA (metadata appended to each block of data in the 4013 buffer) and separate metadata buffer modes. 4014 See `spdk_nvme_ns_cmd_write_with_md()` and `spdk_nvme_ns_cmd_read_with_md()` 4015 for details. 4016- IOAT 4017 - The DMA block fill feature is now exposed via the `ioat_submit_fill()` 4018 function. This is functionally similar to `memset()`, except the memory is 4019 filled with an 8-byte repeating pattern instead of a single byte like memset. 4020- PCI 4021 - Added support for using DPDK for PCI device mapping in addition to the 4022 existing libpciaccess option. Using the DPDK PCI support also allows use of 4023 the Linux VFIO driver model, which means that SPDK userspace drivers will work 4024 with the IOMMU enabled. Additionally, SPDK applications may be run as an 4025 unprivileged user with access restricted to a specific set of PCIe devices. 4026 - The PCI library API was made more generic to abstract away differences 4027 between the underlying PCI access implementations. 4028 4029## v1.2.0: IOAT user-space driver 4030 4031This release adds a user-space driver with support for the Intel I/O Acceleration Technology 4032(I/OAT, also known as "Crystal Beach") DMA offload engine. 4033 4034- IOAT 4035 - New user-space driver supporting DMA memory copy offload 4036 - Example programs `ioat/perf` and `ioat/verify` 4037 - Kernel-mode DMA engine test driver `kperf` for performance comparison 4038- NVMe 4039 - Per-I/O flags for Force Unit Access (FUA) and Limited Retry 4040 - Public API for retrieving log pages 4041 - Reservation register/acquire/release/report command support 4042 - Scattered payload support - an alternate API to provide I/O buffers via a sequence of callbacks 4043 - Declarations and `nvme/identify` support for Intel SSD DC P3700 series vendor-specific log pages and features 4044- Updated to support DPDK 2.2.0 4045 4046## v1.0.0: NVMe user-space driver 4047 4048This is the initial open source release of the Storage Performance Development Kit (SPDK). 4049 4050Features: 4051 4052- NVMe user-space driver 4053- NVMe example programs 4054 - `examples/nvme/perf` tests performance (IOPS) using the NVMe user-space driver 4055 - `examples/nvme/identify` displays NVMe controller information in a human-readable format 4056- Linux and FreeBSD support 4057