1# Changelog 2 3## v23.05: (Upcoming Release) 4 5### accel 6 7Added API `spdk_accel_submit_xor` to perform XOR. 8 9### bdev 10 11A new API `spdk_bdev_module_claim_bdev_desc` was added. Unlike `spdk_bdev_module_claim_bdev`, this 12function requires a bdev descriptor to be passed and the claim is automatically released when the 13descriptor is closed. It allows bdev modules to claim bdevs as a single writer, multiple writers, or 14multiple readers. 15 16### env 17 18New function `spdk_env_get_main_core` was added. 19 20### nvmf 21 22New `spdk_nvmf_request_copy_to/from_buf()` APIs have been added, which support 23iovecs, unlike the deprecated `spdk_nvmf_request_get_data()`. 24 25### nvme 26 27New API `spdk_nvme_ns_get_format_index` was added to calculate the exact format index, that 28was used to format the namespace. 29 30Added two new APIs `spdk_nvme_ns_cmd_io_mgmt_recv` and `spdk_nvme_ns_cmd_io_mgmt_send` to 31receive and send the I/O management commands. 32 33New `spdk_nvmf_transport_create_async` was added, it accepts a callback and callback argument. 34`spdk_nvmf_transport_create` is marked deprecated. 35 36## v23.01 37 38### accel 39 40New library isa-l-crypto has been added, it is used by accel library in crypto operations. 41 42New functions `spdk_accel_submit_encrypt` and `spdk_accel_submit_decrypt` were added. 43 44New accel module `dpdk_cryptodev` has been added. It uses DPDK crypto PMD and support encrypt and 45decrypt operations. New RPC `dpdk_cryptodev_scan_accel_module` has been added to enable this accel module. 46 47New accel module `mlx5` was added. It implements crypto operations, enabled when SPDK is configured with 48RDMA provider is mlx5_dv and crypto support. 49 50Introduced the concept of chaining multiple accel operations together and executing them all in a 51single call. The operation can be chained via one of the `spdk_accel_append_*` functions and then 52executed using `spdk_accel_sequence_finish`. Currently, copy, decompress, decrypt, encrypt, and 53fill operations support this mode. This feature is considered experimental. 54 55### bdev 56 57A new API `spkd_bdev_part_submit_request_ext` was added to specify a custom completion callback. 58 59New APIs `spdk_bdev_for_each_channel` and `spdk_bdev_for_each_channel_continue` and 60associated function pointers were added to iterate each channel of the required bdev. 61 62The RPC `bdev_get_iostat` now allows a user to query the per channel IO statistics for 63required bdev, and displays maximum and minimum latencies and I/O error counts. 64 65New `spdk_bdev_copy_blocks` and `spdk_bdev_get_max_copy` APIs to support copy commands. 66 67A new API `spdk_bdev_io_get_submit_tsc` was added to get the submit_tsc of the bdev I/O. 68 69Bdevs will no longer have UUIDs generated based on timestamp and are responsible for 70setting this field themselves. Generation of UUIDs for NVMe bdevs may be enabled by 71running `bdev_nvme_set_options` RPC with `--generate-uuids` option. These identifiers 72are based on serial number and namespace ID and will always be the same for a given 73device. 74 75A new API `spdk_bdev_channel_get_histogram` was added to get the histogram of a specified 76channel for a bdev. 77 78Converted internal use of `pthread_mutex_t` to `struct spdk_spinlock`. Consumers of bdev 79API functions must be on an SPDK thread or the program will abort. It is now enforced 80that no internal bdev locks can be held when a poller or message goes off CPU. 81 82A new RPC `bdev_reset_iostat` was added to reset I/O statistics of bdevs. Note that if one 83consumer reset I/O statistics, it affects all other consumers. 84 85Add function pointers, `dump_device_stat_json` and `reset_device_stat` to the bdev module 86function table to display and reset I/O statistics specific for the module specific bdev 87context. 88 89New APIs `spdk_bdev_reset_io_stat`, `spdk_bdev_add_io_stat` and `spdk_bdev_dump_io_stat_json` 90were added to process I/O statistics outside the generic bdev layer, especially in bdev modules. 91 92### bdev_crypto 93 94vbdev_crypto is updated to use accel framework instead of DPDK PMDs. 95 96### bdev_daos 97 98New RPC `bdev_daos_resize` was added to allow resizing the daos bdev. 99 100### bdev_malloc 101 102Both of interleaved and separated metadata are now supported by the malloc bdev module. 103 104Protection information is now supported by the malloc bdev module. 105 106### bdev_nvme 107 108Updated `bdev_nvme_set_options` RPC (and rpc.py) to support the new `transport_tos` parameter. 109 110For the active-active policy of the multipath mode, in addition to the default round-robin path 111selector, the minimum queue depth path selector was added. The minimum queue depth path selector 112selects an I/O path according to the number of outstanding requests of each nvme qpair. 113 114Added RPCs `bdev_nvme_start_mdns_discovery`, `bdev_nvme_get_mdns_discovery_info` and 115`bdev_nvme_stop_mdns_discovery` to perform Avahi based mDNS discovery service, 116as per NVMe TP 8009 - Automated Discovery of Ethernet Discovery Controllers. 117 118A new option `nvme_error_stat` was added to the `bdev_nvme_set_options` RPC to enable 119collecting NVMe error counts. 120 121Added I/O statistics per I/O path to the NVMe bdev module for NVMe bdev multipath. It can be 122enabled by a new option `io_path_stat` of RPC `bdev_nvme_set_options`. 123 124Added RPC `bdev_nvme_get_path_iostat` to get I/O statistics for IO paths of the NVMe bdev. 125 126Added `selector` parameter to `bdev_nvme_set_multipath_policy` RPC to set path selector for multipath. 127Option `round_robin` and `queue_depth` are available. 128 129Added `rr_min_io` option to RPC `bdev_nvme_set_multipath_policy`. It switches I/O to 130another path after rr_min_io I/Os are routed to current io path for the round-robin 131path selector. 132 133Added option `--io-path-stat` for RPC `bdev_nvme_set_option` to enable collecting io path stat. 134 135### bdev_raid 136 137Changed `bdev_raid_get_bdevs` RPC output format to include raid_bdev details. 138 139### bdevperf 140 141Promoted the application to example to match similar programs: fio_plugin and perf. 142It can now be found inside `examples/bdev/bdevperf`. 143 144### env 145 146Added `spdk_mempool_mem_iter` that allows to get start address and length of each memory 147chunk in order to create app-specific resources. 148 149### event 150 151Added core lock file mechanism to prevent the same CPU cores from being used by multiple 152SPDK application instances. 153 154Added `--disable-cpumask-locks` command line switch to disable CPU locks on SPDK startup. 155 156Added RPCs `framework_enable_cpumask_locks` and `framework_disable_cpumask_locks` to enable 157and disable CPU core locks in runtime. 158 159Added `--rpcs-allowed` command line option. Users can specify a comma-separated list of RPC 160names with this option to restrict allowed RPCs to only that list. 161 162Added `--vfio-vf-token` command line option to specify a VF token (UUID) 163shared between SR-IOV PF and VFs for vfio_pci driver. 164 165The `spdk_iov_xfer_*()` APIs were added for handling incremental copying between 166an iovec and a buffer. 167 168### json 169 170Added API `spdk_json_write_double` and `spdk_json_write_named_double` to allow 171for writing and decoding of the the double data type. 172 173### log 174 175Added consistent tracking of use of deprecated behavior. Deprecations continue to be 176listed in deprecation.md and now are in subsections with headers matching a tag 177used in `SPDK_LOG_DEPRECATED()` calls. When deprecated behavior is used, these 178tags will appear in SPDK's log at the warn level. As the SPDK application exits, it 179will log a summary of how many times `SPDK_LOG_DEPRECATED()` was called for each 180tag that was logged at least once. 181 182### nvme 183 184NVMe transport options were newly introduced. The NVMe transport options are defined via 185the `spdk_nvme_transport_opts` structure and configured via `spdk_nvme_transport_get_opts` 186and `spdk_nvme_transport_get_opts` functions. 187 188Shared receive queue was supported by the RDMA transport. It can be configured by 189a new NVMe transport option `rdma_srq_size`. 190 191Added `transport_tos` to `spdk_nvme_ctrlr_opts` to support setting of the "type of service" 192value in the IPv4 header. Only RDMA is supported at this time. 193 194Added API `spdk_nvme_qpair_get_num_outstanding_reqs` to get the number of outstanding reqs of 195a specified qpair. 196 197### nvmf 198 199Added API `spdk_nvmf_tgt_pause_polling` and `spdk_nvmf_tgt_resume_polling` to allow 200pausing polling on poll group of a given target. 201 202### rpc 203 204Added `spdk_rpc_set_allowlist` to restrict allowed RPCs to the specified list. 205 206### scheduler 207 208Changing scheduler from dynamic back to static is no longer possible, 209as there is no way of saving original SPDK thread distribution on reactors. 210 211### thread 212 213Added `spdk_thread_get_app_thread` which returns the first thread that was created using 214`spdk_thread_create`. 215 216Added `spdk_thread_is_running`. This returns `true` for a running thread, or `false` if 217its exit process has started using `spdk_thread_exit`. 218 219Added API `spdk_spin_init`, `spdk_spin_destroy`, `spdk_spin_lock`, `spdk_spin_unlock`, and 220`spdk_spin_held` to support spinlocks that are aware of the SPDK concurrency model. 221 222Added iobuf buffer pool, set of API calls for allocating data buffers across libraries. 223Please see new API `spdk_iobuf_*`. 224 225### trace 226 227New `trace_get_info` RPC was added to get name of shared memory file, list of the 228available trace point groups and mask of the available trace points for each group. 229 230### ublk device 231 232The ublk application supports the ublk kernel driver. It's implemented as a ublk backend 233in spdk_tgt and could be started with specifying configuration. See the 234[ublk](https://www.kernel.org/doc/html/latest/block/ublk.html) documentation for more details. 235 236ublk bdev could export a block device via Linux ublk. It will move this backend device into userspace 237as `/dev/ublkb*`. Before to adding ublk device, need to create ublk target by RPC method. 238 239### util 240 241New API `spdk_fd_group_get_epoll_event` that returns the epoll(7) event that 242caused a function callback in file descriptor group to execute. 243 244Added API `spdk_strarray_from_string`, `spdk_strarray_dup` and `spdk_strarray_free` 245for handling arrays of strings. 246 247A new API `spdk_strcpy_replace` was added to replace all occurrences of the search string 248with the replacement string. 249 250New API `spdk_iov_memset()` was added to memset an iovec. 251 252New API `spdk_iov_one()` was added to initialize an iovec for a single buffer. 253 254## v22.09 255 256### accel 257 258Many names were changed in the accel framework to make them consistent both with themselves and 259the rest of SPDK. The primary public header file is now named `include/spdk/accel.h`. 260 261Added a new runtime RPC `accel_get_opc_assignments` to get a list of current opcode to engine 262assignments. 263 264Added a new startup RPC `accel_assign_opc` to assign/override a specific opcode to 265an engine. 266 267### bdev 268 269New RPCs `bdev_xnvme_create` and `bdev_xnvme_delete` were added to support the xNVMe bdev. 270 271A new API `spdk_bdev_for_each_bdev_io` was added to execute the function on the appropriate 272thread for each bdev_io submitted to the bdev. 273 274A new API `spdk_bdev_get_current_qd` was added to measure and return the queue depth from a 275bdev. This API is available even when queue depth sampling is disabled. 276 277Added `spdk_bdev_seek_data`, `spdk_bdev_seek_hole` and `spdk_bdev_io_get_seek_offset` functions 278that start from a given offset and seek for next data or for next hole. At this time only 279supported by lvol bdev module. 280 281A new parameter `io_drain_timeout` has been added to `spdk_bdev` structure. It controls how long 282a bdev reset must wait for IO to complete prior to issuing a reset to the underlying device. 283If there is no outstanding IO at the end of that period, the reset is skipped. 284Best defined for bdevs that share an underlying bdev, such as multiple lvol bdevs sharing 285an nvme device, to avoid unnecessarily resetting the underlying bdev and affecting other 286bdevs that are sharing it. 287Modifying this field is optional. Setting the value to 0 keeps the original behavior 288to always send resets immediately, even if there is no I/O outstanding. 289 290### blobstore 291 292Reserve space for `used_clusters` bitmap. The reserved space could be used for blobstore growing 293in the future. 294 295Added `is_zeroes` operation to `spdk_bs_dev`. It allows to detect if logical blocks are backed 296by zeroes device and do a shortcut in copy-on-write flow by excluding copy part from zeroes device. 297 298Added `spdk_blob_get_next_allocated_io_unit` and `spdk_blob_get_next_unallocated_io_unit` functions 299that start from a given offset and seek for first io_unit belonging to an allocated cluster 300or first io_unit belonging to an unallocated cluster. 301 302Added `spdk_bs_grow` function to grow blobstore size if the underlying bdev size is increased. 303This is used by lvol bdev via `bdev_lvol_grow_lvstore` RPC. 304 305### daos 306 307Added new DAOS bdev module, that creates SPDK block device on top of DAOS DFS. 308Please see [documentation](https://spdk.io/doc/bdev.html#bdev_config_daos) for more details. 309 310### env 311 312Added `spdk_pci_register_device_provider` and matching `SPDK_PCI_REGISTER_DEVICE_PROVIDER` 313functions for registering PCI device providers. That allow the VMD driver to be notified 314when users want to attach a device under a given BDF and when a device is detached. 315 316### init 317 318`spdk_subsystem_init_from_json_config` now fails if the JSON configuration file is not 319an object with an array named "subsystems". 320 321### iscsi 322 323Added `bdev_iscsi_set_options` RPC to modify options of the iSCSI bdev module. 324 325### json 326 327Added `spdk_json_find` API return errcode: EPROTOTYPE - json not enclosed in {}. 328`spdk_json_find` now returns -EPROTOTYPE instead of -ENOENT if the object parameter 329does not point to a JSON object (i.e. is not enclosed with {}). 330 331### lvol 332 333Add `num_md_pages_per_cluster_ratio` parameter to the `bdev_lvol_create_lvstore` RPC. 334Calculate `num_md_pages` from `num_md_pages_per_cluster_ratio`, and pass it to `spdk_bs_opts`. 335 336### nvme 337 338Added SPDK_NVME_TRANSPORT_CUSTOM_FABRICS to enum `spdk_nvme_transport_type` to support custom 339fabric transport. SPDK_NVME_TRANSPORT_CUSTOM was intended to be non-fabric custom transport. 340 341Added a new function `spdk_nvme_ns_cmd_verify` to submit a Verify Command to a Namespace. 342 343Added `spdk_nvme_ctrlr_disable_read_changed_ns_list_log_page` to allow an application to 344tell the driver to not read the CHANGED_NS_LIST log page in response to a NS_ATTR_CHANGED 345AEN. When called the application is required to read this log page instead to clear the 346AEN. 347 348Added `psk` field to `spdk_nvme_ctrlr_opts` struct in order to enable SSL socket implementation 349of TCP connection and set the PSK. Applicable for TCP transport only. 350 351### raid 352 353Renamed the `raid5` module to `raid5f` to reflect that it is not a traditional 354RAID5 implementation - only full stripe writes are supported, partial stripe 355writes (read-modify-write) are not. 356 357### rpc 358 359Added `psk` parameter to `bdev_nvme_attach_controller` RPC in order to enable SSL socket implementation 360of TCP connection and set the PSK. Applicable for TCP transport only. 361 362New options `enable_ktls` and `tls_version` were added to the `sock_impl_set_options` structure. 363New options `psk_key` and `psk_identity` were added to the `sock_impl_set_options` structure. 364 365Added warning message for `bdev_rbd_create`, if it is used without -c. 366`bdev_rbd_create()` API without specifying -c is deprecated and will be removed in future release. 367 368Renamed `enable_vmd` RPC to `vmd_enable` to make it consistent with our naming scheme of 369`<subsystem>_<action>`. For now, the old name is still available, but is marked as deprecated. 370 371New function `spdk_rpc_get_method_state_mask` was added to RPC library. 372 373### sma 374 375Extended `VolumeParameters` with crypto parameters allowing the user to configure crypto when 376attaching a volume to a device. This interface is now supported by all upstream device types 377(nvmf-tcp, virtio-blk, vfio-user) using `bdev_crypto`. Users must specify the crypto engine to use 378under `crypto` section in config. It is also possible to register out-of-tree crypto engines by 379inheriting from the `CryptoEngine` class. 380 381Added two new methods: `SetQos` and `GetQosCapabilities` allowing the user to configure QoS on a 382per-device or per-volume level. Not all QoS settings have to be supported by each device, so users 383can use `GetQosCapabilities` to query them for that. All upstream device types support QoS on a 384per-volume level using bdev layer's QoS mechanism. 385 386### sock 387 388Added new `ssl` based socket implementation, the code is located in `module/sock/posix`. 389For now we are using hard-coded PSK and only support TLS 1.3. 390 391### util 392 393Added new functions: `spdk_hexlify` and `spdk_unhexlify`. 394 395A new API `spdk_xor_gen` was added to generate XOR from multiple source buffers. It is going to be 396used by `raid5f` for calculating parity. 397 398### vfu_tgt 399 400Added vfu_tgt library abstraction based on libvfio-user for PCI device emulation. 401Besides the NVMe device emulation, its now possible to emulate virtio-blk and virtio-scsi 402devices. 403 404### virtio 405 406virtio-vhost-user no longer tries to support dynamic memory allocation. The vhost target does 407not support the high rate of SET_MEM_TABLE vhost messages that result from dynamic memory 408allocation, so a virtio-vhost-user device will now present an ERRLOG, assert, and skip the 409SET_MEM_TABLE vhost message if a memory notification is received outside of the normal device 410start/stop. Applications using the virtio library in vhost-user mode should now pre-allocate 411the application's memory using the -s/--mem-size option and use single shared memory file 412segments using the -g/--single-file-segments option. 413 414Added `vfio-user` transport type for virtio-blk and virtio-scsi devices. 415 416### vmd 417 418Fixed hotplug when a device is inserted in a slot in which a disk was already enumerated previously. 419Added two new RPCs: `vmd_remove_device` simulating a hotremove, and `vmd_rescan`, which rescans all 420buses managed by the VMD driver and hotplugs all newfound devices. 421 422## v22.05 423 424### sock 425 426A new option `ack_timeout` was added to the `spdk_sock_opts` structure. 427 428### util 429 430A new parameter `bounce_iovcnt` was added to `spdk_dif_generate_copy` and `spdk_dif_verify_copy`. 431The `bounce_iovcnt` is used to specify the number of bounce_iov to support multiple block-aligned 432fragment copies. 433 434A new API `spdk_copy_iovs_to_buf` and `spdk_copy_buf_to_iovs` were added to copy iovs to buf or 435copy buf to iovs. There're many cases need to use these two APIs. 436 437### bdev 438 439Removed deprecated spdk_bdev_module_finish_done(). Use spdk_bdev_module_fini_done() instead. 440 441A new API `spdk_bdev_unregister_by_name` was added to handle race conditions correctly. 442 443New APIs, `spdk_for_each_bdev` and `spdk_for_each_bdev_leaf`, were added to provide iteration 444safe for race conditions. 445 446A new RPC `bdev_nvme_get_io_paths` was added to get all active I/O paths. 447 448A new RPC `bdev_nvme_set_preferred_path` was added to set preferred I/O path for an NVMe bdev 449when in multipath mode. This RPC does not support NVMe bdevs in failover mode. 450 451A new RPC `bdev_nvme_set_multipath_policy` was added to set multipath policy of a NVMe bdev 452in multipath mode. 453 454A new option `disable_auto_failback` was added to the `bdev_nvme_set_options` RPC to disable 455automatic failback. 456 457### idxd / dsa 458 459A new parameter `flags` was added to all low level submission and preparation 460APIs to enable the caller to pass operation flags per the DSA specification. 461 462A new flag 'SPDK_IDXD_FLAG_PERSISTENT' was added to let DSA know that 463the destination is persistent. 464 465The RPC `idxd_scan_accel_engine` has been renamed to `dsa_scan_accel_engine` 466 467The RPC `iaa_scan_accel_engine` has been added. 468 469Many HW related structs/functions with the name `idxd` have been renamed `dsa` 470to more accurately represent the HW they are associated with. 471 472Two new functions were added to the library `spdk_idxd_submit_compress` and 473`spdk_idxd_submit_decompress` 474 475### accel_fw 476 477A new parameter `flags` was added to accel API. 478The APIs include: 479`spdk_accel_submit_copy` 480`spdk_accel_submit_dualcast` 481`spdk_accel_submit_fill` 482`spdk_accel_submit_copy_crc32c` 483`spdk_accel_submit_copy_crc32cv` 484`spdk_accel_submit_compress` 485`spdk_accel_submit_decompress` 486 487A new flag `ACCEL_FLAG_PERSISTENT` was added to indicate the target memory is PMEM. 488 489The API `spdk_accel_get_capabilities` has been removed. 490 491### crypto 492 493Support for AES_XTS was added for MLX5 polled mode driver (pmd). 494 495bdev_crypto_create RPC now requires hexlified 'key' and 'key2' params for all pmd drivers. 496Unhexlifying is performed during RPC command processing and the vbdev crypto module runs on 497binary keys as before. 498 499### bdev 500 501Added a timeout option to the `bdev_get_bdevs` RPC. It allows the user to specify the amount of 502time to wait until a bdev with a given name appears in the system. 503 504### bdev_nvme 505 506Added `bdev_nvme_add_error_injection` and `bdev_nvme_remove_error_injection` RPCs to add and 507remove NVMe error injections. 508 509New parameters, `ctrlr_loss_timeout_sec`, `reconnect_delay_sec`, and `fast_io_fail_timeout_sec`, are 510added to the RPC `bdev_nvme_set_options`. They can be overridden if they are given by the RPC 511`bdev_nvme_attach_controller`. 512 513### blobstore 514 515New functions `spdk_blob_io_writev_ext` and `spdk_blob_io_readv_ext` are added. The new functions accept 516`spdk_blob_ext_io_opts` structure with extended IO request options. 517 518### event 519 520Added `msg_mempool_size` parameter to `spdk_reactors_init` and `spdk_thread_lib_init_ext`. 521The size of `g_spdk_msg_mempool` can now be controlled through the same-named 522user option of `spdk_app_opts` structure. 523 524### nvme 525 526The API `spdk_nvme_ctrlr_prepare_for_reset()` was deprecated. The functionality provided by the 527`spdk_nvme_ctrlr_prepare_for_reset()` was merged into the API `spdk_nvme_ctrlr_disconnect()`. 528 529### nvmf 530 531Removed deprecated max_qpairs_per_ctrlr parameter from nvmf_create_transport RPC. Use 532max_io_qpairs_per_ctrlr instead. 533 534Deprecated the ability for hosts to connect to the discovery subsystem automatically on any 535existing listener. Users should now explicitly add listeners for the discovery subsystem. 536Host can still connect to the discovery subsystem as before, but a warning message will be 537emitted if no listener was configured for the transport ID of the incoming connection. 538 539Added adaptive interrupt feature for vfio-user transport. New parameter `disable_adaptive_irq` 540is added to the RPC `nvmf_create_transport`. 541 542### thread 543 544Added `spdk_thread_exec_msg()` API. 545 546### scheduler 547 548`framework_set_scheduler` can now be called after application initialization. 549Added callbacks to set custom parameters specific for each scheduler implementation 550and `framework_get_scheduler` to retrieve them. 551 552Added `dynamic` scheduler options: load_limit, core_limit, core_busy. Their descriptions 553are available in JSON-RPC document, in section 554[framework_set_scheduler](jsonrpc.html#rpc_framework_set_scheduler). 555 556### raid 557 558Add concat as a special raid module. The concat module could create a virtual bdev. The 559virtual bdev combines multiple underlying bdevs together. The layout of the underlying 560bdevs is one after another. The concat bdev is extendable. When the free space of the 561concat bdev is not enough, the user can deconstruct the concat bdev, then reconstruct it 562with an additional underlying bdev. 563 564### sock 565 566Allow MSG_ZEROCOPY flag to be set or not according to data size, which can be enabled and 567set by setting "zerocopy_threshold". zerocopy_threshold = 0 means disable this function; 568zerocopy_threshold > 0 means enable it and use this value as the threshold. 569 570### rpc 571 572Introduced `zerocopy_threshold` to enable zerocopy on send for server sockets according to 573data size to be flushed. 574 575## v22.01 576 577### accel 578 579The batching capability was removed. Batching is now considered an implementation 580detail of the low level drivers. 581 582### bdev 583 584The NVMe bdev module supports multipath and improved I/O error resiliency. 585 586The parameter `retry_count` of the RPC `bdev_nvme_set_options` was deprecated and will be 587removed in SPDK 22.04, and the parameter `transport_retry_count` is added and used instead. 588 589An new parameter `bdev_retry_count` is added to the RPC `bdev_nvme_set_options`. 590 591New parameters, `ctrlr_loss_timeout_sec`, `reconnect_delay_sec`, and `fast_io_fail_timeout_sec`, are 592added to the RPC `bdev_nvme_attach_controller`. 593 594An new parameter `num_io_queues` is added to `bdev_nvme_attach_controller` RPC to allow specifying amount 595of requested IO queues. 596 597Added `key_file` parameter to the `rbd_register_cluster` RPC. It is an optional parameter to 598specify a keyring file to connect to a RADOS cluster. 599 600Added `spdk_bdev_get_zone_id` API to retrieve zone_id for a certain LBA. 601 602### bdev_aio 603 604Added `bdev_aio_rescan` RPC to allow rescanning the size of aio bdev. 605 606### bdev_nvme 607 608Added discovery service to bdev_nvme module, which now can connect to a discovery controller. 609bdev_nvme will connect to all current and future subsystems in the discovery controller. 610Added `bdev_nvme_start_discovery` and `bdev_nvme_stop_discovery` RPC. 611 612### dpdk 613 614Updated DPDK submodule to DPDK 21.11. 615 616### env 617 618Added `spdk_pci_for_each_device`. 619 620Removed `spdk_pci_get_first_device` and `spdk_pci_get_next_device`. These APIs were unsafe, because 621they did not account for PCI devices being inserted or removed while the caller was using handles 622returned from these APIs. Existing users of these APIs should switch to `spdk_pci_for_each_device` 623instead. 624 625Added 3 experimental APIs to handle PCI device interrupts (`spdk_pci_device_enable_interrupt`, 626`spdk_pci_device_disable_interrupt`, `spdk_pci_device_get_interrupt_efd`). 627 628Added `framework_get_pci_devices` RPC to list PCIe devices attached to an SPDK application. 629 630### idxd 631 632Many APIs are now vectored rather than scalar, meaning they take iovecs instead of individual pointers. 633 634### json 635 636Added `spdk_json_write_bytearray` API to serialize a buffer as a hex string. 637 638### nvme 639 640API `spdk_nvme_trtype_is_fabrics` was added to return existing transport type 641is fabric or not. 642 643API `spdk_nvme_poll_group_remove` was limited to be available only for a 644disconnected qpair in the group. 645 646New APIs, `spdk_nvme_ctrlr_disconnect`, `spdk_nvme_ctrlr_reconnect_async`, and 647`spdk_nvme_ctrlr_reconnect_poll_async`, have been added to improve error recovery, and 648the existing APIs,`spdk_nvme_ctrlr_reset_async` and `spdk_nvme_ctrlr_reset_poll_async` 649were deprecated. 650 651Added `spdk_nvme_ctrlr_get_discovery_log_page` API for getting the full discovery log page 652from a discovery controller. 653 654### nvmf 655 656Added support for zero-copy operations in the NVMe-oF TCP target. It can be enabled via 657the `zcopy` parameter when creating a transport. The zero-copy operations are only used 658for requests not using in-capsule data. 659 660Added a `subsystem` parameter to `spdk_nvmf_transport_stop_listen_async`. When not NULL, 661it will only disconnect qpairs for controllers associated with the specified subsystem. 662 663Removed accept poller from transport layer. Each transport can have its own policy of 664handling new connections. To notify transport layer `spdk_nvmf_poll_group_add` and 665`spdk_nvmf_tgt_new_qpair` can be used. 666 667### scsi 668 669Structure `spdk_scsi_lun` has been extended with new member `resizing` so that SCSI layer now reports 670unit attention for disk resize. 671 672### trace 673 674Added `spdk_trace_create_tpoint_group_mask` to return tracepoint group mask from 675a tracepoint group name. 676 677Added `trace_set_tpoint_mask` and `trace_clear_tpoint_mask` RPC to allow enabling 678individual traces. 679 680### util 681 682Added `spdk_ioviter_first` and `spdk_ioviter_next` to iterate over two iovecs and 683yield pointers to common length segments. 684 685### sock 686 687A new parameter `hint` is added to `spdk_sock_get_optimal_sock_group`. It allows to suggest 688a poll group when no optimal poll group is found. 689 690## v21.10 691 692Structure `spdk_nvmf_target_opts` has been extended with new member `discovery_filter` which allows to specify 693filtering rules applied during discovery log generation. Refer to `enum spdk_nvmf_tgt_discovery_filter` for more info. 694 695### bdev 696 697New API `spdk_bdev_get_memory_domains` has been added, it allows to get SPDK memory domains used by bdev. 698 699New API functions `spdk_bdev_readv_blocks_ext` and `spdk_bdev_writev_blocks_ext` have been added. 700These functions accept `spdk_bdev_ext_io_opts` structure with extended IO request 701options, e.g. DMA memory domain which describes data that may belong to another memory domain and 702can't be accessed directly. 703 704Added `async_fini_start` to allow bdev modules to complete the `fini_start` asynchronously, 705with new `spdk_bdev_module_fini_start_done` API. 706 707Deprecated `spdk_bdev_module_finish_done()` API, which will be removed in SPDK 22.01. 708Bdev modules should use `spdk_bdev_module_fini_done()` instead. 709 710The `ocssd` bdev has been removed. The Open Channel specification has been largely superseded by 711zoned namespaces, few if any devices have been brought to market, and there is little reason to 712continue to support this. OCSSD support in the nvme driver will remain for now. 713 714### dma 715 716A new library, lib/dma, has been added. This library provides the necessary infrastructure for 717handling systems and devices with multiple memory domains. For example, a PCIe add-in card with an 718SoC may be running SPDK on the SoC. That SoC has its own local memory, but SPDK may be controlling 719devices that can also access the host system memory. This library provides infrastructure to enumerate 720the memory domains and request hardware perform DMA transfers between them. 721 722### dpdk 723 724Updated DPDK submodule to DPDK 21.08. 725 726### event 727 728Added the `disable_signal_handlers` flag to the `spdk_app_opts` struct. 729 730### idxd 731 732Added `spdk_idxd_get_socket` to query the socket that the idxd device 733is on. 734 735### json 736 737Added API to allow for writing and decoding of new types: 738 739- `spdk_json_write_named_uint16` 740- `spdk_json_write_named_uint8` 741- `spdk_json_write_uint16` 742- `spdk_json_write_uint8` 743- `spdk_json_number_to_uint8` 744- `spdk_json_decode_uint8` 745 746### log 747 748Added API `spdk_log_to_syslog_level` to return syslog level based on SPDK's 749log level. 750 751### nvme 752 753Added new functions `spdk_nvme_zns_set_zone_desc_ext` and `spdk_nvme_zns_ext_report_zones` 754to set zone descriptor extension and to get extended zone report respectively. 755 756New API `spdk_nvme_ctrlr_get_memory_domains` has been added, it allows to get SPDK memory domains used by nvme controller. 757 758New API functions `spdk_nvme_ns_cmd_readv_ext` and `spdk_nvme_ns_cmd_writev_ext` 759have been added. These functions accept `spdk_nvme_ns_cmd_ext_io_opts` structure with extended IO request 760options, e.g. DMA memory domain which describes data that may belong to another memory domain and 761can't be accessed directly. 762 763Added a new function `spdk_nvme_ctrlr_get_regs_bpinfo` to get boot partition info of a controller. 764Added new functions `spdk_nvme_ctrlr_write_boot_partition`, 765`spdk_nvme_ctrlr_read_boot_partition_start` and `spdk_nvme_ctrlr_read_boot_partition_poll` 766to write and read the boot partitions of a controller. 767 768Added `spdk_nvme_ctrlr_get_opts` to retrieve the current controller options. 769 770Added `async_mode` to `spdk_nvme_io_qpair_opts` to enable creation of submission and completion 771queues asynchronously. This mode is currently supported at PCIe layer, 772which tracks the qpair creation with state machine and returns to the user immediately. 773Default mode is set to false to create io qpairs synchronously. 774 775Added `spdk_nvme_ctrlr_get_regs_cc` to retrieve NVMe controller CC (Configuration) register. 776 777Added `spdk_nvme_ctrlr_prepare_for_reset` to inform the driver that the application is preparing 778to reset the specified NVMe controller. This function allows the driver to make decisions 779knowing that a reset is about to happen. 780 781### nvmf 782 783Added `oncs` to `struct spdk_nvmf_ctrlr_data` so that the transport layer 784can decide support RESERVATION feature or not. 785 786An `opts_size` element was added in the `spdk_nvmf_ns_opts` structure to solve the 787ABI compatibility issue between different SPDK version. An new option `anagrpid` was 788added in the `spdk_nvmf_ns_opts` structure. 789 790An new parameter `anagrpid` was added to the RPC `nvmf_subsystem_add_ns`. 791 792An new parameter `anagrpid` was added to the RPC `nvmf_subsystem_listener_set_ana_state`. 793 794`spdk_nvmf_subsystem_destroy` is now can be asynchronous, it accepts a callback and callback argument. 795 796A new parameter, `poll_groups_mask` was added to the `nvmf_set_config` RPC that allows specifying 797a subset of cores for the nvmf poll groups. This helps to avoid imbalances when some cores are 798busy with periodic timer tasks that run very frequently. 799 800A new parameter, `commit` was added to `nvmf_bdev_ctrlr_end_zcopy` that allows indicating whether 801the buffers should be committed. 802 803Added `listen_dump_opts` to `spdk_nvmf_transport_ops` so that the transport layer can display 804listen opts. 805 806Added `fabrics_connect_timeout_us` to `bdev_nvme_attach_controller` RPC to allow specifying 807timeout for connect operation. 808 809Added `multipath` to `bdev_nvme_attach_controller` RPC to allow specifying multipath behavior. 810 811### scheduler 812 813New API for implementing schedulers and governors. Please see `include/spdk/scheduler.h` for details. 814 815### scsi 816 817New functions, `spdk_scsi_dev_get_first_lun` and `spdk_scsi_dev_get_next_lun` 818have been added to iterate LUNs of a SCSI device. 819 820Each SCSI device supports 256 LUNs at the maximum now and the macro constant 821`SPDK_SCSI_DEV_MAX_LUN` was removed. 822 823### trace_parser 824 825A new library, lib/trace_parser, has been added. This library provides functions that parse traces 826recorded by an SPDK application. That includes merging traces from multiple cores, 827sorting them by their timestamp and constructing trace entries spanning across multiple buffers. 828 829### util 830 831The `spdk_fd_group_add` API now takes a `name` parameter. 832 833## v21.07 834 835### accel_fw 836 837Added API `spdk_accel_submit_copy_crc32c` to perform a CRC32C while copying data. 838 839Added API `spdk_accel_batch_prep_copy_crc32c` to batch CRC32C + copy commands. 840 841Added API `spdk_accel_submit_copy_crc32cv` to submit chained CRC32C + copy commands. 842 843### bdev 844 845Change `spdk_bdev_read_blocks_with_md` arg offset definition from int64_t to uint64_t. 846 847Red-black tree has been used to organize the bdev names and aliases uniformly 848to provide faster lookup. 849 850Removed ZCOPY emulation: The bdev module can be checked to see if it supports ZCOPY 851and if not supported then use existing READ/WRITE commands. 852 853Added iov to spdk_bdev_zcopy_start. 854 855### dpdk 856 857Updated DPDK submodule to DPDK 21.05. 858 859### idxd 860 861Remove the `probe_cb` parameter in `spdk_idxd_probe` function. And remove the definition 862of `spdk_idxd_probe_cb` function pointer. It should be implemented in idxd_user.c. 863 864Added API `spdk_idxd_submit_copy_crc32c` to perform a CRC32C while copying data. 865 866Added API `spdk_idxd_batch_prep_copy_crc32c` to prepare a batch operation to perform 867a CRC32C while copying data. 868 869We have userspace idxd driver under accel_engine library (module/accel/idxd/accel_engine). 870And kernel idxd driver usage is also added under the accel_engine library. This approach 871implemented a wrapper library to use IDXD device by leveraging the kernel DSA driver in 872SPDK idxd library (lib/idxd). Then users can leverage the RPC later to configure how to 873use the DSA device by user space driver or kernel driver. 874 875### init 876 877Added new `init` library that initializes the SPDK subsystems, which previously was 878internal to application framework. That functionality has been made public and for 879use by applications which don't leverage SPDK's application framework. 880 881### iscsi 882 883New parameters, `pdu_pool_size`, `immediate_data_pool_size`, and `data_out_pool_size`, 884were added to the RPC `iscsi_set_options` to run iSCSI target with varying amount of 885available memory. 886 887### json 888 889Added API `spdk_json_write_named_uint128` and `spdk_json_write_uint128` to perform 890the uint128 related data. 891 892### net 893 894Removed deprecated `net` library. 895 896Added support for ZCOPY. ZCOPY is used in preference to READ and WRITE if the 897bdev module supports ZCOPY and the bdev module has ZCOPY enabled. 898 899### nvme 900 901`spdk_nvme_map_prps` and `spdk_nvme_map_cmd` were moved to nvmf/vfio-user as internal APIs 902as vfio-user is the only user for the above two APIs. 903 904Added a new function `spdk_nvme_ns_cmd_copy` to submit a Simple Copy Command to a Namespace. 905 906Update the `spdk_nvme_generic_command_status_code` structure with new status code 907according to the definition in NVMe 1.4 spec. 908 909`spdk_nvme_ctrlr_get_default_ctrlr_opts` now sets `use_cmb_sqs` to false. This means 910that if a controller has a CMB and supports SQs in the CMB, SPDK will not use 911the CMB for SQs by default - the user must set `use_cmb_sqs` to true in 912the `spdk_nvme_ctrlr_opts` structure prior to controller attach. 913 914Add a new function `spdk_nvme_detach_poll` to simplify a common use case to continue 915polling until all detachments complete. 916 917Added new argument `timeout_admin_us` to `spdk_nvme_ctrlr_register_timeout_callback` so callers 918can specify a different timeout for admin commands vs. io commands. 919 920An existing function `spdk_nvme_detach_async` was updated to add one or more detachments 921to an active context while it is being polled. 922 923Rename a variable in the member `cmic` of the struct `spdk_nvme_ctrlr_data` from 924`multi_host` to `multi_ctrlr`. The variable means that the NVM subsystem may have two or 925more controllers if set to 1. However `multi_host` had indicated a particular use case 926such that the NVM subsystem is used by multiple hosts. 927 928A new option `disable_read_ana_log_page` was added to struct `spdk_nvme_ctrlr_opts` to disable 929reading ANA log page. The upper layer is expected to read ANA log page instead if `true`. 930The default value is `false`. 931 932New APIs, `spdk_nvme_ctrlr_reset_async` and `spdk_nvme_ctrlr_reset_poll_async`, have been added to 933reset a controller asynchronously. 934 935New RPC `bdev_nvme_reset_controller` was added, to reset an NVMe controller. 936 937Added `spdk_nvme_ns_get_nguid` function to get NGUID for the given namespace. 938 939Added `spdk_nvme_ctrlr_is_fabrics` function to indicate whether a ctrlr handle 940is associated with a fabrics controller. 941 942### nvmf 943 944Added `min_cntlid` and `max_cntlid` to `nvmf_create_subsystem` to limit the controller ID range. 945Added `spdk_nvmf_subsystem_get_min_cntlid` and `spdk_nvmf_subsystem_get_max_cntlid` to request those values. 946 947`spdk_nvmf_request_get_buffers_multi` API is removed. 948 949Added the `nvmf_set_crdt` RPC for setting command retry delay times. 950 951Expanded `spdk_nvmf_poll_group_stat` with current qpair count statistics. 952 953Removed following deprecated APIs: 954 955- `spdk_nvmf_poll_group_get_stat` (function in `nvmf.h`), 956- `spdk_nvmf_transport_poll_group_get_stat` (function in `nvmf.h`), 957- `spdk_nvmf_transport_poll_group_free_stat`(function in `nvmf.h`), 958- `spdk_nvmf_rdma_device_stat` (struct in `nvmf.h`), 959- `spdk_nvmf_transport_poll_group_stat` (struct in `nvmf.h`), 960- `poll_group_get_stat` (transport op in `nvmf_transport.h`), 961- `poll_group_free_stat` (transport op in `nvmf_transport.h`). 962 963### rpc 964 965New RPC `bdev_rbd_register_cluster` and `bdev_rbd_unregister_cluster` was added, it allows to create 966and delete the rados object cluster, then users can choose the cluster to create related rbd 967device. 968 969Revised `bdev_rbd_create` parameter, it allows to use an optional parameter `--cluster-name` 970to create a rbd bdev with an already registered Rados Cluster Object. 971 972New RPC `bdev_rbd_get_clusters_info` was added, it allows to get the info of the registered 973Rados Cluster names. 974 975New optional parameter, `timeout_admin_us`, added to the bdev_nvme_set_options RPC. 976 977Revised a parameter `--stripe-size_kb` to `--stripe-size-kb` of `bdev_raid_create` method 978provided in `scripts/rpc.py` for consistency. 979 980An new optional parameter `config_kernel_mode` was added to the RPC `idxd_scan_accel_engine`, 981and this is used to enable using the kernel mode IDXD driver. 982 983Removed deprecated parameter `enable-zerocopy-send` of RPC `sock_impl_set_options`, 984use `enable-zerocopy-send-server` or `enable-zerocopy-send-client` instead. 985Removed deprecated parameter `disable-zerocopy-send` of RPC `sock_impl_set_options`, 986use `disable-zerocopy-send-server` or `disable-zerocopy-send-client` instead. 987 988### rpm 989 990Removed deprecated `pkg/spdk.spec`, please use `rpmbuild/spdk.spec` instead. 991See [RPM documentation](https://spdk.io/doc/rpm.html) for more details. 992 993### thread 994 995Red-black tree has been used for timed pollers to provide faster insertion and deletion 996and for io_devices to provide faster lookup. 997 998Removed `spdk_io_channel` structure from public header and moved it to thread_internal.h. 999 1000### trace 1001 1002Added `spdk_trace_register_description_ext` function to register variable number of tracepoint arguments. 1003 1004Added ability to chain multiple trace entries together to extend the size of the argument buffer. 1005 1006### util 1007 1008Red-black tree macros has been added by using the macros provided by the FreeBSD operating system 1009under the same BSD license. 1010 1011Add an new macro `SPDK_SIZEOF_MEMBER` to get the size of a member of a struct. 1012 1013`spdk_crc32c_iov_update` function was added to support calculating the crc32c of the iovs. 1014 1015Added zipf random number generator with power law probability distribution. 1016When applied to performance testing of block devices, it will select blocks over 1017the full range of LBAs, but will more frequently select lower-numbered LBAs. 1018 1019## v21.04 1020 1021### accel 1022 1023Two new accelerated crc32 functions `spdk_accel_submit_crc32cv` and 1024`spdk_accel_batch_prep_crc32cv` are added in order to provide the 1025chained accelerated CRC32 computation support. 1026 1027### bdev 1028 1029For `bdev_ocssd_create` RPC, the optional parameter `range` was removed. 1030Only one OCSSD bdev can be created for one OCSSD namespace. 1031 1032Removed the `spdk_bdev_open` from bdev library API. 1033Removed the `spdk_vbdev_register` and `spdk_bdev_part_base_construct` from bdev module API. 1034Removed the `config_text` function for bdev modules to report legacy config. 1035 1036Added `spdk_bdev_get_max_active_zones` API to display maximum number active zones of a given bdev. 1037 1038Added `spdk_bdev_get_max_zone_append_size` API to display maximum zone append data transfer size. 1039 1040### bdev_nvme 1041 1042Added support for zoned namespaces. 1043 1044### blobstore 1045 1046Removed the `spdk_bdev_create_bs_dev_from_desc` and `spdk_bdev_create_bs_dev` API. 1047 1048### env 1049 1050Added `spdk_pci_device_allow` API to allow applications to add PCI addresses to 1051the allowed list after the application has started. 1052 1053Removed the `pci_whitelist`, `pci_blacklist` and `master_core` members of struct `spdk_env_opts`. 1054 1055Added hotplug support based on uevent in `pci_event.c`. Added usage of this functionality in 1056nvme, virtio-scsi and virtio_blk libraries. Please see the new API `spdk_pci_event_listen`, 1057`spdk_pci_get_event`, `spdk_pci_register_error_handler` and `spdk_pci_unregister_error_handler`. 1058 1059### event 1060 1061Removed the `config_file`, `max_delay_us`, `pci_whitelist` 1062and `pci_blacklist` members of struct `spdk_app_opts`. 1063 1064### idxd 1065 1066A new API `spdk_idxd_device_needs_rebalance` was added so that users of the library 1067can know whether they need to rebalance the flow control for the channel 1068that was just added/removed. This is based on how the low level library 1069shares devices amongst channels. 1070 1071The API `spdk_idxd_reconfigure_chan` had the `num_channels` removed as this 1072is now tracked in the library. The app makes use the new API above to 1073determine whether to rebalance or not. This applies to `spdk_idxd_configure_chan` 1074as well. 1075 1076The API `spdk_idxd_put_channel` now returns the rebalance state for the 1077underlying device. 1078 1079### iscsi 1080 1081A security vulnerability has been identified and fixed in the SPDK iSCSI target. 1082A TEXT PDU with no data, but CONTINUE flag set, would result in a NULL pointer dereference 1083and crash the SPDK iSCSI target process. All users of the SPDK iSCSI target 1084are recommended to update. All SPDK versions <= v21.01 are affected. 1085 1086### net 1087 1088The net library is deprecated and will be removed in the 21.07 release. 1089 1090### nvme 1091 1092Added a new function `spdk_nvme_ctrlr_get_regs_pmrcap` to get the PMR capabilities. 1093 1094Added an accelerated table pointer in `spdk_nvme_poll_group` 1095which can be used to provide the accelerated functions by users with 1096hardware engine, such as crc32c accelerated function. 1097 1098Added new functions `spdk_nvme_ctrlr_get_pmrsz`, `spdk_nvme_ctrlr_enable_pmr`, 1099`spdk_nvme_ctrlr_disable_pmr`, `spdk_nvme_ctrlr_map_pmr` and `spdk_nvme_ctrlr_unmap_pmr`. 1100 1101Added NVMe transport operations to enable, disable, map and unmap the PMR. 1102 1103Added `spdk_nvme_qpair_get_optimal_poll_group` function and `qpair_get_optimal_poll_group` 1104function pointer in spdk_nvmf_transport_ops structure in order to add the qpair to the most 1105suitable polling group. 1106 1107Added OPTPERF and namespace optimal performance fields to nvme_spec.h. 1108 1109Added `spdk_nvme_set_hotplug_filter` API to allow applications to choose which 1110hot-inserted SSDs should be probed. This is useful for use cases where multiple 1111independent SPDK processes are running on one node. The filter function can 1112then be implemented in these processes to decide which SSDs to probe based on 1113the new SSD's PCI address. 1114 1115New functions `spdk_nvme_poll_group_get_stats` and `spdk_nvme_poll_group_free_stats` 1116were added. These functions allow to get transport statistics per NVME poll group. 1117 1118Added `spdk_nvme_map_cmd` API to map the NVMe command with SGL cases. 1119 1120Added support for vector variant of ZNS zone append commands with new API 1121`spdk_nvme_zns_zone_appendv` and `spdk_nvme_zns_zone_appendv_with_md`. 1122 1123Added `spdk_nvme_zns_ns_get_max_open_zones` and `spdk_nvme_zns_ns_get_max_active_zones` API, 1124to display maximum number of open and active zones of the given namespace. 1125 1126Added `spdk_nvme_zns_ns_get_zone_size_sectors` API to provide size of zone in number of 1127sectors. 1128 1129Added `spdk_nvme_qpair_get_id` API to display the ID of the specified qpair. 1130 1131### nvmf 1132 1133Removed the `spdk_nvmf_tgt_listen` and `spdk_nvmf_subsystem_add_ns` API. 1134 1135Added new APIs: 1136 1137- `spdk_nvmf_poll_group_dump_stat` (function in `nvmf.h`). 1138- `poll_group_dump_stat` (transport op in `nvmf_transport.h`). 1139 1140The following APIs have been deprecated and will be removed in SPDK 21.07: 1141 1142- `spdk_nvmf_poll_group_get_stat` (function in `nvmf.h`), 1143- `spdk_nvmf_transport_poll_group_get_stat` (function in `nvmf.h`), 1144- `spdk_nvmf_transport_poll_group_free_stat`(function in `nvmf.h`), 1145- `spdk_nvmf_rdma_device_stat` (struct in `nvmf.h`), 1146- `spdk_nvmf_transport_poll_group_stat` (struct in `nvmf.h`), 1147- `poll_group_get_stat` (transport op in `nvmf_transport.h`), 1148- `poll_group_free_stat` (transport op in `nvmf_transport.h`). 1149 1150See header files for details. 1151 1152The `trtype` field in JSON returned by `nvmf_get_stats` RPC contains now the name of the transport, 1153which is the same as the type for defined transports and more informative for a custom transport. 1154 1155Added `hdgst` and `ddgst` parameters to `bdev_nvme_attach_controller` RPC in order change 1156state of TCP header and data digest. 1157 1158Added `num_cqe` parameter to `nvmf_create_transport` RPC to set number of completion queues (CQ) 1159for RDMA transport. Useful when CQ resize operation is not supported, for example iWARP. 1160 1161### ocf 1162 1163Updated OCF submodule to v20.12.2 1164 1165Added `bdev_ocf_set_cache_mode` RPC to dynamically switch cache mode of OCF bdev. 1166 1167### opal 1168 1169Removed the `spdk_opal_supported` API. 1170 1171### raid 1172 1173For `bdev_raid_create` RPC, the deprecated parameter `strip_size` was removed. 1174 1175### rpc 1176 1177New RPC `bdev_nvme_get_transport_statistics` was added, it allows to get transport statistics 1178of nvme poll groups. 1179 1180Parameter `enable-zerocopy-send` of RPC `sock_impl_set_options` is deprecated and will be removed in SPDK 21.07, 1181use `enable-zerocopy-send-server` or `enable-zerocopy-send-client` instead. 1182Parameter `disable-zerocopy-send` of RPC `sock_impl_set_options` is deprecated and will be removed in SPDK 21.07, 1183use `disable-zerocopy-send-server` or `disable-zerocopy-send-client` instead. 1184 1185Added cmd_parser.py used to parse the args from argparse. There are 1186two benefit to use command parser: 1187 1188- Simplify the definition of rpc method. It will reduce the rpc method code. 1189- Make the rpc call more versatile. User can add private args into rpc method. 1190 1191### rpm 1192 1193Added support for new RPM spec, rpmbuild/spdk.spec, which can be used for packaging the 1194SPDK. The pkg/spdk.spec is considered to be deprecated and scheduled for removal in SPDK 21.07. 1195See [RPM documentation](https://spdk.io/doc/rpm.html) for more details. 1196 1197### sock 1198 1199The type of `enable_placement_id` in struct `spdk_sock_impl_opts` is changed from 1200bool to int. We can use RPC to configure different value of `enable_placement_id`. 1201Then we can leverage SO_INCOMING_CPU to get placement_id, which aims to utilize 1202CPU cache locality, enabled by setting enable_placement_id=2. 1203 1204A new socket placement mode called PLACEMENT_MARK has been added. Some NICs allow 1205sockets to be marked using the SO_MARK socket option as a hint for which hardware 1206queue they should be associated with. This mode leverages that by setting the same 1207value for all sockets within a poll group. 1208 1209New parameters `enable_zerocopy_send_server` and `enable_zerocopy_send_client` were added 1210to struct spdk_sock_impl_opts, these parameters enable or disable zero copy send for server 1211and client sockets which are created using `spdk_sock_listen` and `spdk_sock_listen_ext` (server); 1212`spdk_sock_connect` and `spdk_sock_connect_ext` (client) functions. Existing parameter 1213`enable_zerocopy_send` enables or disables zero copy send for both server and client sockets. 1214 1215### thread 1216 1217A new API `spdk_io_channel_get_io_device` was added to get the io_device for the specified 1218I/O channel. 1219 1220Added `spdk_thread_set_interrupt_mode` function in order to set present spdk_thread into 1221interrupt mode or back to poll mode. It is valid only when thread interrupt facility is 1222enabled by `spdk_interrupt_mode_enable`. 1223 1224Added `spdk_poller_register_interrupt` function to mark that the poller is capable of 1225entering interrupt mode. Callback function will be called when the poller must transition 1226into or out of interrupt mode. 1227 1228### virtio 1229 1230Added the `bdev_virtio_blk_set_hotplug` RPC for the virtio blk pci device. 1231 1232## v21.01 1233 1234### bdev 1235 1236An `opts_size` element was added in the `spdk_bdev_opts` structure to solve the 1237ABI compatibility issue between different SPDK version. And also add `opts_size` 1238parameter in spdk_bdev_get_opts function. Two fields `small_buf_pool_size` and 1239`large_buf_pool_size` were added into spdk_bdev_opts, which were used to determine 1240the small and large buffer pool size of the whole bdev module. 1241 1242A new API `spdk_bdev_wait_for_examine` was added to allow for checking state of 1243examine process. Along with corresponding `bdev_wait_for_examine` RPC, which 1244is now always called during `spdk_bdev_subsystem_config_json` making sure 1245every bdev is ready to be used. 1246 1247A new API `spdk_bdev_io_get_aio_status` was added for getting the status of 1248bdev_io as Linux AIO errno. Also `spdk_bdev_io_complete_aio_status` function 1249and `SPDK_BDEV_IO_STATUS_AIO_ERROR` were added for bdev module to complete 1250a bdev_io with Linux AIO errno. 1251 1252A new API `spdk_bdev_get_module_name` was added for getting block device module name. 1253Also `spdk_bdev_get_module_ctx` was added to obtain a bdev module context for 1254the block device opened by the specified descriptor. 1255 1256Added `max_segment_size` and `max_num_segments` to `spdk_bdev` structure for 1257bdev modules to specify splitting requirements. 1258 1259### blob 1260 1261An `opts_size` element was added in the `spdk_bs_opts` structure to solve the 1262ABI compatibility issue between different SPDK version. And also add `opts_size` 1263parameter in `spdk_bs_opts_init` function. 1264 1265An `opts_size` element was added in the `spdk_blob_opts` structure to solve the 1266ABI compatibility issue between different SPDK version. And also add `opts_size` 1267parameter in `spdk_blob_opts_init` function. 1268 1269An `opts_size` element was added in the `spdk_blob_open_opts` structure to solve the 1270ABI compatibility issue between different SPDK version. And also add `opts_size` 1271parameter in `spdk_blob_open_opts_init` function. 1272 1273### build 1274 1275SPDK now generates pkg-config files to simplify the process of determining which 1276libraries must be linked into an SPDK application. 1277 1278### dpdk 1279 1280Updated DPDK submodule to DPDK 20.11. 1281 1282Removed `--with-igb-uio-driver` configure option. Since DPDK 20.11 igb_uio driver 1283was moved to separate dpdk-kmods repository. If required, users need to refer to 1284dpdk-kmods repository to build the driver. 1285 1286### env 1287 1288The `pci_whitelist`, `pci_blacklist` and `master_core` members of struct `spdk_env_opts` 1289have been deprecated. The new members are named `pci_allowed`, `pci_blocked` and 1290`main_core` respectively. 1291 1292An `opts_size` element was added in the `spdk_app_opts` structure 1293to solve the ABI compatibility issue between different SPDK version. An `opts_size` 1294parameter is added into `spdk_app_opts_init` function. 1295 1296### event 1297 1298The `pci_whitelist` and `pci_blacklist` members of struct `spdk_app_opts` have been 1299deprecated. The new members are named `pci_allowed` and `pci_blocked` respectively. 1300 1301The `--pci-blacklist` command line option has been deprecated, replaced with 1302`--pci-blocked`. 1303 1304The `--pci-whitelist/-W` command line options have been deprecated, replaced with 1305`--pci-allowed/-A`. 1306 1307Added new experimental `dynamic` scheduler that rebalances idle threads, adjusts CPU frequency 1308using dpdk_governor and turns idle reactor cores to interrupt mode. Please see 1309[scheduler documentation](https://www.spdk.io/doc/scheduler.html) for details. 1310 1311## ioat 1312 1313The PCI BDF whitelist option has been removed from the `ioat_scan_accel_engine` RPC. 1314IOAT PCI functions can still be allowed or denied using SPDK application command 1315line options. 1316 1317### isa-l 1318 1319Updated ISA-L submodule to v2.30.0. 1320 1321### json 1322 1323A new API `spdk_jsonrpc_send_bool_response` was added to allow sending response for 1324writing json bool results into one function. 1325Update API `bdev_nvme_set_options` and add a `keep_alive_timeout_ms` parameter. Now you 1326can specify the `keep_alive_timeout` before creating NVMe bdev. 1327 1328### nbd 1329 1330Change the return type of function `spdk_nbd_stop` from void to int. And update the 1331`spdk_nbd_fini` with two parameters to make its behavior from sync to async. 1332 1333### nvme 1334 1335Directives support was added to the NVMe driver. 1336 1337Two async APIs `spdk_nvme_ctrlr_cmd_directive_receive` and `spdk_nvme_ctrlr_cmd_directive_send` 1338are added for Directive Send and Directive Receive command, respectively. 1339 1340Added a new function `spdk_nvme_ctrlr_reset_subsystem` to perform a NVMe 1341subsystem reset. Note: The NVMe-oF target does not support the subsystem reset yet. 1342 1343Add a new function `spdk_nvme_bytes_to_numd` to transfer bytes to number of 1344dwords. 1345 1346Added a new custom transport `SPDK_NVME_TRANSPORT_VFIOUSER` to enable NVMe 1347driver running with NVMe over vfio-user target. 1348 1349Added the vfio-user custom transport implementation in NVMe driver which can connect 1350to NVMe over vfio-user target via vfio-user transport. 1351 1352Added a new function `spdk_nvme_ctrlr_set_remove_cb` to remove or override 1353`remove_cb` and `remove_ctx` specified when the controller was probed. 1354 1355Added support for ZNS zone append command with new API `spdk_nvme_zns_zone_append` and 1356`spdk_nvme_zns_zone_append_with_md`. 1357 1358### nvmf 1359 1360`nvmf_fc_lld_fini` now takes callback and hence updating FC Broadcom LLD driver 1361to the latest is required. 1362 1363FC transport supported primary tagging and VMID. 1364 1365Broadcom FC LLD driver and SPDK NVMe-oF FC transport consolidated one LLD API, 1366`nvmf_fc_init_poller_queues` into another LLD API `nvmf_fc_init_q`. 1367Hence updating Broadcom FC LLD driver to the latest is required. 1368 1369The functions `destroy` and `qpair_fini` in the transport interface now accept a 1370`cb_fn` and `cb_arg` to call upon completion, and their execution can be asynchronous. 1371 1372The SPDK NVMe-oF target now supports async event notification for discovery log changes. 1373This allows the initiator to create persistent connection to discovery controller and 1374be notified of any discovery log changes. 1375 1376An `opts_size` element was added in the `spdk_nvmf_transport_opts` structure 1377to solve the ABI compatibility issue between different SPDK version. And also add 1378`opts_size` parameter in `spdk_nvmf_transport_opts_init` function. 1379 1380Added a new custom vfio-user transport implementation in NVMe-oF which can provide 1381emulated NVMe devices to QEMU and SPDK NVMe driver. 1382 1383Added new API `spdk_nvmf_tgt_listen_ext` that uses `spdk_nvmf_listen_opts` to allow 1384passing generic options to specific transport. This functionality replaces one provided 1385with `nvmf_subsystem_set_options`. Also removed `nvmf_subsystem_set_options` RPC 1386that allowed transport specific options within subsystem. 1387 1388The `trsvcid` in `nvmf_subsystem_add_listener` RPC is now optional. 1389 1390Pausing a subsystem now only pauses all admin queues. This allows for the 1391management of hosts and listeners, plus the addition of namespaces without a 1392full subsystem pause. Additionally, the target now allows for pausing 1393individual namespaces within a subsystem. To remove a namespace from a 1394subsystem, only the specific namespace must be paused. I/O will continue to 1395other namespaces while these operations execute. 1396 1397### rpc 1398 1399An new optional parameter `wait` was added to the RPC `iscsi_create_portal_group`, 1400and an new RPC `iscsi_start_portal_group` was added. They will be used not to start 1401listening on portals for a portal group until all associated target nodes are created 1402at startup, otherwise some iSCSI initiators may fail to re-login when SPDK iSCSI 1403target application restarts. 1404 1405Two optional parameter `--small-buf-pool-size` and `--large-buf-pool-size` were added 1406into `bdev_set_options` function. 1407 1408### vhost 1409 1410Added optional `packed_ring_recovery` parameter to `vhost_create_blk_controller` RPC 1411enabling packed ring live recovery. 1412 1413## v20.10 1414 1415### accel 1416 1417New APIs `spdk_accel_batch_cancel` and `spdk_idxd_batch_cancel` were added to allow 1418cancelling a batch sequence. 1419 1420### bdev 1421 1422A new `spdk_bdev_part_base_construct_ext` function has been added and the 1423`spdk_bdev_part_base_construct` has been deprecated. The 1424`spdk_bdev_part_base_construct_ext` function takes bdev name as an argument instead 1425of bdev structure to avoid a race condition that can happen when the bdev is being 1426removed between a call to get its structure based on a name and actually opening it. 1427 1428Removed `spdk_bdev_config_text` function for bdev modules to report legacy config. 1429 1430### blobstore 1431 1432A new `spdk_bdev_create_bs_dev_ext` function has been added and `spdk_bdev_create_bs_dev_from_desc` 1433function has been deprecated. 1434 1435### dpdk 1436 1437Updated DPDK submodule to DPDK 20.08. 1438 1439### env 1440 1441A new API `spdk_mem_get_fd_and_offset` was added to get file descriptor and offset for 1442a given virtual address. 1443 1444### event 1445 1446Removed `spdk_subsystem_config` callback for submodules as part of legacy config removal. 1447 1448Removed `spdk_app_get_running_config` function that printed configuration in legacy format, 1449and removed `usr1_handler` from `struct spdk_app_opts` callback that was used to call it. 1450 1451Added SPDK thread scheduler framework which is used to rebalance load among CPU cores. 1452The scheduler implementation is pluggable and two schedulers are added first, `static` scheduler 1453and `gscheduler` scheduler. 1454 1455`static` scheduler is used by default and does not reschedule threads and keeps the initial 1456assignments. 1457 1458CPU frequency governor framework and implementation are also added. 1459`dpdk_governor` is the first implementation and is based on rte_power library of DPDK. 1460`gscheduler` scheduler utilizes the CPU frequency governor. 1461 1462Scheduler and governor frameworks are experimental features. 1463 1464### fio 1465 1466Bdev fio_plugin no longer supports legacy configuration files. Options `spdk_conf` and 1467`spdk_json_conf` have to point to valid JSON configuration. 1468 1469### gpt 1470 1471Removed option to disable GPT probing via legacy configuration. Please use generic 1472bdev functionality to replicate the behavior. See `bdev_set_options --disable-auto-examine` 1473and `bdev_examine` RPC. 1474 1475### intel-ipsec-mb 1476 1477Updated intel-ipsec-mb submodule to v0.54 1478 1479### isa-l 1480 1481Updated ISA-L submodule to v2.29.0. 1482 1483### iscsi 1484 1485The iSCSI target supports iSCSI login redirection feature to achieve fault tolerance 1486and load balancing particularly in scale out iSCSI target system, which runs multiple 1487SPDK iSCSI targets. 1488 1489### json 1490 1491A new API `spdk_json_decode_object_relaxed` was added to allow parsing json values 1492even if decoder for a given key is not found. 1493 1494A new API `spdk_json_free_object` was added to free memory allocated by `spdk_json_decode_object`. 1495 1496### log 1497 1498A log flag structure, `spdk_log_register_flag`, `spdk_log_get_first_flag`, 1499`spdk_log_get_next_flag` and macro `SPDK_LOG_REGISTER_COMPONENT`, have been added which 1500enable registering and interacting with log flags. Added `SPDK_INFOLOG`, `SPDK_DEBUGLOG` 1501and `SPDK_LOGDUMP` macros to display logs for specified log flag. 1502 1503The log flags in the event framework `-L` and several SPDK applications has been enabled on 1504release builds of SPDK. On debug builds this option will additionally set 1505log print level to `SPDK_LOG_DEBUG`. 1506 1507A new API `spdk_log_enable_timestamps` and new RPC `log_enable_timestamps` was added to 1508toggle log timestamps. 1509 1510### miscellaneous 1511 1512The contents of the log_rpc library have been moved to the event library. The log_rpc 1513library now no longer exists. 1514 1515The contents of the app_rpc library have been moved to the event library. The app_rpc 1516library now no longer exists. 1517 1518The contents of the bdev_rpc library have been moved to the bdev library. The app_rpc 1519library now no longer exists. 1520 1521The bdevperf application now disables the zcopy API by default. Prior to this change, 1522bdevperf enabled using the zcopy API by default which caused a performance impact of 1523up to 25% on bdevs that don't natively support zcopy because the API emulates zero-copy 1524by allocating a buffer. The bdevperf `-x` param was renamed to `-Z` and the default 1525value changed to false. For bdevs that support zcopy, use the -Z flag to enable 1526using zcopy API. 1527 1528### net 1529 1530Net framework intended for use by user-space TCP stacks was removed. 1531 1532### nvme 1533 1534Zoned Namespace Command Set support has been added to NVMe driver, see `nvme_zns.h` for details. 1535 1536New APIs, `spdk_nvme_detach_async` and `spdk_nvme_detach_poll_async`, have been added to 1537detach multiple controllers in parallel to mitigate lengthy shutdown notification of 1538a few NVMe SSDs. 1539 1540New APIs, `spdk_nvme_ns_get_ana_group_id` and `spdk_nvme_ns_get_ana_state`, have been 1541added to get ANA group ID and ANA state of the given namespace, respectively. 1542 1543If no specific command set is requested (by setting the command_set member in the 1544`spdk_nvme_ctrlr_opts` structure), SPDK will automatically select the most appropriate 1545command set based on what the controller supports. 1546 1547### nvmf 1548 1549A new API `spdk_nvmf_subsystem_add_ns_ext` has been added and the API `spdk_nvmf_subsystem_add_ns` 1550has been deprecated. 1551 1552The NVMe-oF target now supports Asymmetric Namespace Access (ANA) Reporting to provide 1553multipath to NVMe-oF initiator. 1554 1555Add `no_wr_batching` parameter to `spdk_nvmf_transport_opts` struct to disable 1556Work Requests batching in RDMA transport. 1557 1558NVMf Target transports can now parse any additional JSON params in the nvmf_create_transport RPC 1559through the JSON context provided via spdk_nvmf_target_opts->transport_specific. 1560 1561A new RPC `nvmf_subsystem_set_options` was added to allow transport specific options within subsystem. 1562 1563A new API `spdk_nvmf_transport_stop_listen_async` was added to stop accepting new connections 1564at the provided address. `nvmf_subsystem_remove_listener` RPC now uses this function. 1565 1566A new API `spdk_nvmf_subsystem_disconnect_host` was added to disconnect all connections 1567originating from the provided hostnqn. `nvmf_subsystem_remove_host` RPC now uses this function. 1568 1569### ocf 1570 1571Updated OCF submodule to v20.03.1 1572 1573### rpc 1574 1575New optional parameters, `enable_placement_id` and `enable_quickack` were added to the 1576`sock_impl_set_options` RPC. 1577 1578A new RPC `bdev_examine_bdev` was added to allow users to examine a bdev explicitly. 1579It can be used only if bdev_auto_examine is set to false by the RPC `bdev_set_options`. 1580 1581New optional parameters `no_wr_batching` and `control_msg_num` were added to the RPC 1582'nvmf_create_transport'. 1583 1584New RPCs, `iscsi_target_node_set_redirect` and `iscsi_target_node_request_logout`, have 1585been added, and a new optional parameter `private` has been added to the RPC 1586`iscsi_create_portal_group` to support iSCSI login redirection feature. 1587 1588A new optional parameter `ana_reporting` was added to the RPC `nvmf_create_subsystem`, and 1589a new RPC `nvmf_subsystem_listener_set_ana_state` was added for ANA reporting. 1590 1591New RPCs, `nvmf_subsystem_get_listeners` and `nvmf_subsystem_get_qpairs`, were added to 1592retrieve configuration of the NVMe-oF subsystem. 1593 1594### scsi 1595 1596Two new APIs have been added `spdk_scsi_dev_construct_ext` and 1597`spdk_scsi_dev_add_lun_ext` that allow the upper layer(e.g. vhost-scsi) to 1598receive the notification when the scsi bdev has been resized. 1599 1600The `spdk_scsi_dev_construct` and `spdk_scsi_dev_add_lun` eventually may be 1601deprecated and removed. 1602 1603### sock 1604 1605The `enable_placement_id` field was added in the `struct spdk_sock_impl_opts` to 1606make the placement_id feature configurable by users. The default setting is 1607not enabled. 1608 1609The `enable_quick_ack` field was added in the `struct spdk_sock_impl_opts` to enable 1610or disable quick ack for the POSIX sock module. The default setting is not enabled. 1611 1612### thread 1613 1614`fd_group` is applied to support interrupt mode. 1615New APIs were added to support an experimental interrupt mode. This allows modules or 1616libraries to selectively register file descriptors that the spdk_thread can wait on, 1617as an alternative to polling. In v20.10, this functionality is enabled in a very small 1618subset of SPDK libraries and modules. 1619 1620### util 1621 1622A new utility named `fd_group` was add. It is now 1623implemented by epoll on Linux platform. It can be used by 1624spdk_thread and reactor to implement interrupt mode. 1625 1626Added `spdk_bit_pool` utility, see `bit_pool.h` for details. 1627 1628### vhost 1629 1630SPDK has switched to DPDK's rte_vhost library since 19.07 release, removed the internal 1631rte_vhost library which is used for DPDK older than 19.05, removed the experimental vhost 1632nvme target which depends on the internal rte_vhost library. 1633 1634### vpp 1635 1636Removed socket abstraction for VPP. 1637 1638## v20.07: SPDK CSI driver, new accel_fw commands, I/O abort support 1639 1640spdk_json_free_object() was added to free memory allocated by spdk_json_decode_object(). 1641 1642### accel 1643 1644A new API was added `spdk_accel_get_capabilities` that allows applications to 1645query the capabilities of the currently enabled accel engine back-end. 1646 1647A new capability, CRC-32C, was added via `spdk_accel_submit_crc32c`. 1648 1649The software accel engine implementation has added support for CRC-32C. 1650 1651A new capability, compare, was added via `spdk_accel_submit_compare`. 1652 1653The software accel engine implementation has added support for compare. 1654 1655Several APIs were added to `accel_engine.h` to support batched submission 1656of operations. 1657 1658Several APIs were added to `accel_engine.h` to support dualcast operations. 1659 1660### accel_fw 1661 1662The accel_fw was updated to no longer require the app to allocate an 1663accel_task on its behalf. All public APIs now take a callback arg as 1664the parameter that used to be the accel_task. 1665 1666The accel_fw API `spdk_accel_task_size` was removed as it is no longer 1667required. 1668 1669The accel_fw was updated to support compare, dualcast, crc32c. 1670 1671The accel_fw introduced batching support for all commands in all plug-ins. 1672See docs for detailed information. 1673 1674### bdev 1675 1676A new API `spdk_bdev_abort` has been added to submit abort requests to abort all I/Os 1677whose callback context match to the bdev on the given channel. 1678 1679### build 1680 1681The fio plugins now compile to `build/fio` and are named `spdk_bdev` and `spdk_nvme`. 1682Existing fio configuration files will need to be updated. 1683 1684### dpdk 1685 1686Updated DPDK submodule to DPDK 20.05. 1687 1688### env 1689 1690Several new APIs have been added to provide greater flexibility in registering and 1691accessing polled mode PCI drivers. See `env.h` for more details. 1692 1693### idxd 1694 1695The idxd library and plug-in module for the accel_fw were updated to support 1696all accel_fw commands as well as batching. Batching is supported both 1697through the library and the plug-in module. 1698 1699IDXD engine support for CRC-32C has been added. 1700 1701### ioat 1702 1703A new API `spdk_ioat_get_max_descriptors` was added. 1704 1705### nvme 1706 1707An `opts_size`element was added in the `spdk_nvme_ctrlr_opts` structure 1708to solve the ABI compatibility issue between different SPDK version. 1709 1710A new API `spdk_nvme_ctrlr_cmd_abort_ext` has been added to abort previously submitted 1711commands whose callback argument match. 1712 1713Convenience functions, `spdk_nvme_print_command` and `spdk_nvme-print_completion` were added 1714to the public API. 1715 1716A new function, `spdk_nvmf_cuse_update_namespaces`, updates the cuse representation of an NVMe 1717controller. 1718 1719A new function `qpair_iterate_requests` has been added to the nvme transport interface. ALl 1720implementations of the transport interface will have to implement that function. 1721 1722### nvmf 1723 1724The NVMe-oF target no longer supports connecting scheduling configuration and instead 1725always uses what was previously called "transport" scheduling. 1726 1727`spdk_nvmf_tgt_accept` no longer exists. The accept process now occurs automatically after 1728the creation of an nvmf target and queue pairs are assigned to poll groups by the underlying 1729transport. Further, `spdk_nvmf_transport_ops` has changed such that the accept function 1730pointer no longer takes a function pointer as an argument. Instead, transports should call 1731`spdk_nvmf_tgt_new_qpair` whenever they previously would have called that callback. 1732 1733The NVMe-oF target now supports aborting any submitted NVM or Admin command. Previously, 1734the NVMe-oF target could abort only Asynchronous Event Request commands. 1735 1736### rdma 1737 1738A new `rdma` library has been added. It is an abstraction layer over different RDMA providers. 1739Two providers are available - verbs (used by default when RDMA is enabled or enabled explicitly 1740using --with-rdma=verbs) and mlx5 Direct Verbs aka DV (enabled by --with-rdma=mlx5_dv). 1741Using mlx5_dv requires libmlx5 installed on the system. 1742 1743### rpc 1744 1745Parameter `-p` or `--max-qpairs-per-ctrlr` of `nvmf_create_transport` RPC command accepted by the 1746rpc.py script is deprecated, new parameter `-m` or `--max-io-qpairs-per-ctrlr` was added. 1747 1748Added `sock_impl_get_options` and `sock_impl_set_options` RPC methods. 1749 1750Command line parameters `-r` and `--rpc-socket` will longer accept TCP ports. RPC server 1751must now be started on a Unix domain socket. Exposing RPC on the network, as well as providing 1752proper authentication (if needed) is now a responsibility of the user. 1753 1754The `bdev_set_options` RPC has a new option, `bdev_auto_examine` to control the auto examine function 1755of bdev modules. 1756 1757New RPCs `sock_impl_get_options` and `sock_impl_set_options` been added to expose new socket features. 1758See `sock` section for more details. 1759 1760### sock 1761 1762Added `spdk_sock_impl_get_opts` and `spdk_sock_impl_set_opts` functions to set/get socket layer configuration 1763options. Options can be set independently for each implementation. 1764 1765Added `recv_buf_size` and 'send_buf_size' socket layer options. They are used only in posix implementation. 1766 1767Added `uring` based socket implementation, the code is located in module/sock/uring. This feature is only 1768available in Linux which requires kernel version is greater than 5.4.3. Currently, our CI pool added the uring 1769based socket tests for iSCSI target and also the tests for SPDK NVMe-oF tcp transport. 1770 1771Added `enable_recv_pipe` socket layer option to allow disabling of double buffering on receive. 1772New option is used only in posix implementation. 1773 1774Added `enable_zerocopy_send` socket layer option to allow disabling of zero copy flow on send. 1775New option is used only in posix implementation. 1776 1777### util 1778 1779Some previously exposed CRC32 functions have been removed from the public API - 1780`spdk_crc32_update`, `spdk_crc32_table_init`, and the `spdk_crc32_table` struct. 1781 1782### vhost 1783 1784The function `spdk_vhost_blk_get_dev` has been removed. 1785 1786## v20.04: SPDK Top, IDXD, NVMe qpair groups 1787 1788IDXD engine support for compare has been added. 1789 1790### configuration 1791 1792Legacy INI style configuration for SPDK applications has been deprecated and will be 1793removed in future release. 1794Please switch to JSON-RPC configuration files and/or RPC driven run-time configuration. 1795 1796### copy 1797 1798The copy engine library, modules and public APIs have been renamed. Use of the word `copy` 1799has been replaced with the word `accel` short for accelerator in preparation for adding new 1800capabilities in the future. Additionally, APIs for what was previously called the `memcpy` 1801engine have been renamed to identify the engine as a software accelerator. 1802 1803### crypto 1804 1805Support for AES_XTS was added for the QAT polled mode driver (pmd). The create RPC 1806`bdev_crypto_create` has 2 new optional parameters: cipher and key2. Cipher can be either 1807AES_CBC (default) or AES_XTS. AES_XTS is only valid when using the QAT polled mode driver. 1808The key2 parameter is the second key required for AES_XTS. 1809 1810### event 1811 1812Reactors now accumulate CPU stats and they are retrieved by the RPC `framework_get_reactors`. 1813 1814### idxd 1815 1816IDXD support was added in the form of a low level library that can directly 1817utilize the IDXD offload engine as well as an accel plug-in module allowing for use 1818with the generic accel framework API. IDXD is the first in a family of offload 1819engines that share the same interface, specifically DSA is added here. More info 1820can be found here: https://01.org/blogs/2019/introducing-intel-data-streaming-accelerator 1821 1822Much of the implementation models IOAT, however the low level interfaces are very 1823different. The RPC to enable IDXD requires a configuration number as well. The 1824code includes two pre-defined configurations of IDXD groups/work queues/engines. A future 1825version will provide an interface to allow for setting of individual configuration 1826parameters. 1827 1828IDXD is not yet available so this feature should be considered experimental. It will 1829be built up with additional documentation as an ongoing activity. 1830 1831### iscsi 1832 1833The iSCSI target now creates a lightweight thread per poll group instead of assuming a pool 1834of lightweight threads already exist at start up time. A poll group is a collection of 1835unrelated iSCSI connections. Each poll group is only accessed from the associated 1836lightweight thread. 1837 1838### ftl 1839 1840Several changes have been made to the `spdk_ftl_conf`, `spdk_ftl_dev_init_ops`, and 1841`spdk_ftl_attrs` structs. Please see `include/spdk/ftl.h` for more details. 1842 1843### miscellaneous 1844 1845The `--json-ignore-init-errors` command line parameter has been added to ignore 1846initialization errors on JSON config load. 1847 1848The public header file io_channel.h has been removed. Please use thread.h which has the 1849exact same API. 1850 1851### nvme 1852 1853Exported internal nvme_ctrlr_cmd_security_receive/send() APIs as public APIs with "the spdk_" 1854prefix. 1855 1856Added `priority` field in `spdk_nvme_transport_id`, this field is used to specify the priority 1857of the NVMe-oF connection, and currently it is used for NVMe-oF tcp connection. 1858 1859A new poll group API has been added to allow for pooling of nvme qpairs across a single 1860entity which can be polled for completions. This new API consists of the `spdk_nvme_poll_group` 1861family of functions. As a result of this new API, all NVMe transports are expected to implement 1862several poll group related functions. 1863 1864A new flag, `create_only`, has been added to the `spdk_nvme_io_qpair_opts` structure. This flag 1865allows a user to call `spdk_nvme_ctrlr_get_default_io_qpair` without also connecting the qpair 1866within the context of that call. 1867 1868As a result of the `create_only` flag, two new API functions, `spdk_nvme_ctrlr_connect_io_qpair` 1869and `spdk_nvme_ctrlr_disconnect_io_qpair`, have been added to facilitate connecting newly created 1870qpairs (for example, after they have been added to a poll group) and disconnecting qpairs without 1871destroying them (for example to disconnect a qpair before migrating it to a new poll group and 1872reconnecting it). 1873 1874The functions `spdk_nvme_ctrlr_alloc_cmb_io_buffer` and `spdk_nvme_ctrlr_free_cmb_io_buffer` 1875have been changed to `spdk_nvme_ctrlr_map_cmb` and `spdk_nvme_ctrlr_unmap_cmb` respectively. 1876 1877An additional function, `spdk_nvme_ctrlr_reserve_cmb`, has been added to facilitate reserving 1878the entire size of the controller memory buffer for data transfer. 1879 1880### nvme_cuse 1881 1882`spdk_nvme_cuse_get_ctrlr_name` now takes two additional parameters, `char *name` which 1883stores the pointer to the controller name, and `size_t *size` which stores the length of 1884the name. The return type has also been changed from char * to int. 1885 1886`spdk_nvme_cuse_get_ns_name` now takes two additional parameters, `char *name` which 1887stores the pointer to the namespace name, and `size_t *size` which stores the length of 1888the name. The return type has also been changed from char * to int. 1889 1890### nvme_opal 1891 1892Several public OPAL structure definitions have been changed since the last release. 1893 1894### nvmf 1895 1896`spdk_nvmf_poll_group_destroy()` is now asynchronous and accepts a completion callback. 1897 1898The NVMe-oF target now creates a lightweight thread per poll group instead of assuming a pool 1899of lightweight threads already exist at start up time. A poll group is a collection of 1900unrelated NVMe-oF connections. Each poll group is only accessed from the associated 1901lightweight thread. 1902 1903A new struct, `spdk_nvmf_subsystem_listener`, has been added to encapsulate the subsystem specific 1904nature of a listener object. 1905 1906`spdk_nvmf_tgt_listen` no longer accepts a callback function or argument. It also returns an 1907int to indicate the status of the listen call. 1908 1909The execution of `spdk_nvme_poll_group_destroy` is now asynchronous and the function accepts 1910a cb_fn and cb_arg to call upon completion. 1911 1912The execution of `spdk_nvmf_subsystem_add_listener` is now asynchronous and the function accepts 1913a cb_fn and cb_arg to call upon completion. 1914 1915The `nvmf_transport.h` header has been made public to allow custom NVMe-oF transports to integrate 1916with NVMe-oF libraries without using internal APIs. 1917 1918### ocf 1919 1920Updated the OCF submodule to OCF v20.03 1921 1922New version of OCF comes with API changes and bug fixes 1923 1924### rpc 1925 1926A new RPC `thread_set_cpumask` has been added to set the cpumask of the thread 1927to the specified value. 1928 1929A new RPC `thread_get_pollers` has been added to retrieve pollers from SPDK threads. 1930 1931A new RPC `thread_get_io_channels` has been added to retrieve I/O channels from SPDK threads. 1932 1933A new RPC `bdev_rbd_resize` has been added to resize Ceph RBD bdevs. 1934 1935### sock 1936 1937The `spdk_sock_set_priority` function has been removed since the feature to set the sock priority 1938will be contained in two new functions, i.e., `spdk_sock_listen_ext` and `spdk_sock_connect_ext`. 1939Users may now specify the priority of the socket in the opts that they want to use. 1940 1941### spdk_top 1942 1943A new application, `spdk_top`, has been added which allows users to monitor resource consumption 1944by a running SPDK application. More information on this application can be found in 1945`app/spdk_top/README`. 1946 1947### thread 1948 1949A new function `spdk_thread_lib_init_ext` has been added, and the function 1950`spdk_thread_lib_init` has been deprecated. The user of `spdk_thread_lib_init_ext` is 1951expected to implement both functions `spdk_thread_op_fn` and `spdk_thread_op_supported_fn`. 1952`spdk_thread_op_supported_fn` is called to check whether the SPDK thread operation 1953is supported. `spdk_thread_op_fn` is called to execute the SPDK thread operation. 1954Current SPDK operation types are `SPDK_THREAD_OP_NEW` and `SPDK_THREAD_OP_RESCHED`. 1955The operation `SPDK_THREAD_OP_NEW` is called each time a new thread is created. 1956The operation `SPDK_THREAD_OP_RESCHED` is called when SPDK thread needs to be rescheduled. 1957 1958A unique ID has been added for each created SPDK thread, it is retrieved by a new function 1959`spdk_thread_get_id`, and the SPDK thread which has the specific ID is got by 1960a new function `spdk_thread_get_by_id`. 1961 1962A new function `spdk_thread_cpumask` has been added to set the current thread's cpumask 1963to the specified value. The function requires the operation `SPDK_THREAD_OP_RESCHED` 1964is supported. 1965 1966A new function `spdk_poller_register_named` has been added to set arbitrary name to the 1967created poller. If NULL, the name is set to the pointer of the poller function. 1968 1969The function `spdk_thread_poll` now measures run time per call correctly on multiple SPDK 1970threads configuration, and a new function `spdk_thread_get_last_tsc` has been added to use together. 1971 1972Voluntary termination of SPDK thread has been supported by refining the functions `spdk_thread_exit` 1973and `spdk_thread_poll`. 1974 1975### util 1976 1977New functions `spdk_sn32_lt` and `spdk_sn32_gt` have been added. They compare two sequence 1978numbers based on serial number arithmetic. 1979 1980### vhost 1981 1982Poll groups per session have been replaced by SPDK threads per vhost controller. 1983 1984### vmd 1985 1986A new function, `spdk_vmd_fini`, has been added. It releases all resources acquired by the VMD 1987library through the `spdk_vmd_init` call. 1988 1989## v20.01: Optimized thin provisioning, FTL bdev, VMD hot plug, FUSED support 1990 1991### bdev 1992 1993A new function, `spdk_bdev_set_timeout`, has been added to set per descriptor I/O timeouts. 1994 1995A new class of functions `spdk_bdev_compare*`, have been added to allow native bdev support 1996of block comparisons and compare-and-write. 1997 1998A new class of bdev events, `SPDK_BDEV_EVENT_MEDIA_MANAGEMENT`, has been added to allow bdevs 1999which expose raw media to alert all I/O channels of pending media management events. 2000 2001A new API was added `spdk_bdev_io_get_aux_buf` allowing the caller to request 2002an auxiliary buffer for its own private use. The API is used in the same manner that 2003`spdk_bdev_io_get_buf` is used and the length of the buffer is always the same as the 2004bdev_io primary buffer. 'spdk_bdev_io_put_aux_buf' frees the allocated auxiliary 2005buffer. 2006 2007### blobfs 2008 2009Added boolean return value for function spdk_fs_set_cache_size to indicate its operation result. 2010 2011Added `blobfs_set_cache_size` RPC method to set cache size for blobstore filesystem. 2012 2013### blobstore 2014 2015Added new `use_extent_table` option to `spdk_blob_opts` for creating blobs with Extent Table descriptor. 2016Using this metadata format, dramatically decreases number of writes required to persist each cluster allocation 2017for thin provisioned blobs. Extent Table descriptor is enabled by default. 2018See the [Blobstore Programmer's Guide](https://spdk.io/doc/blob.html#blob_pg_cluster_layout) for more details. 2019 2020### dpdk 2021 2022Updated DPDK submodule to DPDK 19.11. 2023 2024### env_dpdk 2025 2026`spdk_env_dpdk_post_init` now takes a boolean, `legacy_mem`, as an argument. 2027 2028A new function, `spdk_env_dpdk_dump_mem_stats`, prints information about the memory consumed by DPDK to a file specified by 2029the user. A new utility, `scripts/dpdk_mem_info.py`, wraps this function and prints the output in an easy to read way. 2030 2031### event 2032 2033The functions `spdk_reactor_enable_framework_monitor_context_switch()` and 2034`spdk_reactor_framework_monitor_context_switch_enabled()` have been changed to 2035`spdk_framework_enable_context_switch_monitor()` and 2036`spdk_framework_context_switch_monitor_enabled()`, respectively. 2037 2038### ftl 2039 2040All NVMe dependencies were removed from ftl library. 2041From now ftl library is using bdev_zone API. 2042bdev_ftl becomes virtual bdev. 2043 2044`ctrlr` and `trid` fields in `spdk_ftl_dev_init_opts` structure were replaced by 2045`base_bdev_desc`. 2046 2047`bdev_ftl_create` RPC method `trtype` and `traddr` parameters were replaced by `base_bdev` 2048parameter. 2049 2050`spdk_ftl_module_init` and `spdk_ftl_module_fini` functions were removed. 2051 2052`spdk_ftl_punit_range` and `ftl_module_init_opts` structures were removed. 2053 2054### isa-l 2055 2056Updated ISA-L submodule to commit f3993f5c0b6911 which includes implementation and 2057optimization for aarch64. 2058 2059Enabled ISA-L on aarch64 by default in addition to x86. 2060 2061### nvme 2062 2063`delayed_pcie_doorbell` parameter in `spdk_nvme_io_qpair_opts` was renamed to `delay_cmd_submit` 2064to allow reuse in other transports. 2065 2066Added RDMA WR batching to NVMf RDMA initiator. Send and receive WRs are chained together 2067and posted with a single call to ibv_post_send(receive) in the next call to qpair completion 2068processing function. Batching is controlled by 'delay_cmd_submit' qpair option. 2069 2070The NVMe-oF initiator now supports plugging out of tree NVMe-oF transports. In order 2071to facilitate this feature, several small API changes have been made: 2072 2073The `spdk_nvme_transport_id` struct now contains a trstring member used to identify the transport. 2074A new function, `spdk_nvme_transport_available_by_name`, has been added. 2075A function table, `spdk_nvme_transport_ops`, and macro, `SPDK_NVME_TRANSPORT_REGISTER`, have been added which 2076enable registering out of tree transports. 2077 2078A new function, `spdk_nvme_ns_supports_compare`, allows a user to check whether a given namespace supports the compare 2079operation. 2080 2081A new family of functions, `spdk_nvme_ns_compare*`, give the user access to submitting compare commands to NVMe namespaces. 2082 2083A new function, `spdk_nvme_ctrlr_cmd_get_log_page_ext`, gives users more granular control over the command dwords sent in 2084log page requests. 2085 2086### nvmf 2087 2088Add SockPriority option in [Transport] section, this can be used for NVMe-oF target 2089on TCP transport to set sock priority for the incoming TCP connections. 2090 2091The NVMe-oF target now supports plugging out of tree NVMe-oF transports. In order 2092to facilitate this feature, several small API changes have been made: 2093 2094The `spdk_nvme_transport_id` struct now contains a trstring member used to identify the transport. 2095`spdk_nvmf_tgt_get_transport`, `spdk_nvmf_transport_opts_init`, and `spdk_nvmf_transport_create` all have been 2096modified to take a string. 2097A function table, `spdk_nvmf_transport_ops`, and macro, `SPDK_NVMF_TRANSPORT_REGISTER`, have been added which 2098enable registering out of tree transports. 2099 2100Add `spdk_nvmf_tgt_stop_listen()` that can be used to stop listening for 2101incoming connections for specified target and trid. Listener is not stopped 2102implicitly upon destruction of a subsystem any more. 2103 2104A custom NVMe admin command handler has been added which allows the user to use the real drive 2105attributes from one of the target NVMe drives when reporting drive attributes to the initiator. 2106This handler can be enabled via the `nvmf_set_config` RPC. 2107Note: In a future version of SPDK, this handler will be enabled by default. 2108 2109The SPDK target and initiator both now include compare-and-write functionality with one caveat. If using the RDMA transport, 2110the target expects the initiator to send both the compare command and write command either with, or without inline data. The 2111SPDK initiator currently respects this requirement, but this note is included as a flag for other initiators attempting 2112compatibility with this version of SPDK. 2113 2114The `spdk_nvme_ctrlr_opts` struct has been extended with new field `transport_ack_timeout` which allows 2115to configure transport ACK timeout. Applicable for RDMA transport only. 2116 2117### rpc 2118 2119A new RPC, `bdev_zone_block_create`, enables creating an emulated zoned bdev on top of a standard block device. 2120 2121A new RPC, `bdev_ocssd_create`, enables creating an emulated zoned bdev on top of an Open Channel SSD. 2122 2123A new RPC, `blobfs_set_cache_size`, enables managing blobfs cache size. 2124 2125A new RPC, `env_dpdk_get_mem_stats`, has been added to facilitate reading DPDK related memory 2126consumption stats. Please see the env_dpdk section above for more details. 2127 2128A new RPC, `framework_get_reactors`, has been added to retrieve a list of all reactors. 2129 2130`bdev_ftl_create` now takes a `base_bdev` argument in lieu of `trtype`, `traddr`, and `punits`. 2131 2132`bdev_nvme_set_options` now allows users to disable I/O submission batching with the `-d` flag 2133 2134`bdev_nvme_cuse_register` now accepts a `name` parameter. 2135 2136`bdev_uring_create` now takes arguments for `bdev_name` and `block_size` 2137 2138`nvmf_set_config` now takes an argument to enable passthru of identify commands to base NVMe devices. 2139Please see the nvmf section above for more details. 2140 2141### scsi 2142 2143`spdk_scsi_lun_get_dif_ctx` now takes an additional argument of type `spdk_scsi_task`. 2144 2145### sock 2146 2147Added spdk_sock_writev_async for performing asynchronous writes to sockets. This call will 2148never return EAGAIN, instead queueing internally until the data has all been sent. This can 2149simplify many code flows that create pollers to continue attempting to flush writes 2150on sockets. 2151 2152Added `impl_name` parameter in spdk_sock_listen and spdk_sock_connect functions. Users may now 2153specify the sock layer implementation they'd prefer to use. Valid implementations are currently 2154"vpp" and "posix" and NULL, where NULL results in the previous behavior of the functions. 2155 2156### thread 2157 2158`spdk_thread_send_msg` now returns int indicating if the message was successfully 2159sent. 2160 2161A new function `spdk_thread_send_critical_msg`, has been added to support sending a single message from 2162a context that may be interrupted, e.g. a signal handler. 2163 2164Two new functions, `spdk_poller_pause`, and `spdk_poller_resume`, have been added to give greater control 2165of pollers to the application owner. 2166 2167### util 2168 2169`spdk_pipe`, a new utility for buffering data from sockets or files for parsing 2170has been added. The public API is available at `include/spdk/pipe.h`. 2171 2172### bdev 2173 2174Added spdk_bdev_io_get_nvme_fused_status function for translating bdev_io status to NVMe status 2175code for fused compare-and-write operation. 2176 2177Added spdk_bdev_get_acwu function for getting block device atomic compare and write unit size. 2178 2179## v19.10: Zoned bdev API, Opal bdev, NVMe character devices 2180 2181### rpc 2182 2183Many of SPDK's RPCs were renamed to be more consistent and intuitive in this release. 2184The old names will continue to function, but will display a deprecation warning. 2185 2186Added optional parameters '--arbitration-burst' and '--low/medium/high-priority-weight' to 2187'bdev_nvme_set_options' RPC method. 2188 2189Added optional parameter '--md-size' to 'construct_null_bdev' RPC method. 2190 2191Added optional parameters '--dif-type' and '--dif-is-head-of-md' to 'construct_null_bdev' 2192RPC method. 2193 2194Added `blobfs_detect` RPC method to detect whether a blobfs exists on given bdev. 2195 2196Added `blobfs_create` RPC method to build blobfs on given bdev. 2197 2198Added `blobfs_mount` RPC method to mount blobfs on given bdev to a host path by FUSE. 2199Then on the host path, user can directly do some file operations which will be mapped 2200to blobfs. 2201 2202### bdev 2203 2204Added new parameter `cdw0` to `spdk_bdev_io_complete_nvme_status()` and 2205`spdk_bdev_io_get_nvme_status()` that allows setting/getting 2206the NVMe completion queue DW0 entry. This allows vendor specific IO commands 2207to return command specific completion info back to the initiator. 2208 2209Added `spdk_bdev_get_write_unit_size()` function for retrieving required number 2210of logical blocks for write operation. 2211 2212New zone-related fields were added to the result of the `get_bdevs` RPC call: 2213 2214- `zoned`: indicates whether the device is zoned or a regular 2215 block device 2216- `zone_size`: number of blocks in a single zone 2217- `max_open_zones`: maximum number of open zones 2218- `optimal_open_zones`: optimal number of open zones 2219 2220The `zoned` field is a boolean and is always present, while the rest is only available for zoned 2221bdevs. 2222 2223A new `spdk_bdev_open_ext` function has been added and `spdk_bdev_open` function has been deprecated. 2224The new open function introduces requirement to provide callback function that will be called by 2225asynchronous event such as bdev removal. `spdk_bdev_open_ext` function takes bdev name as 2226an argument instead of bdev structure to avoid a race condition that can happen when the bdev 2227is being removed between a call to get its structure based on a name and actually opening it. 2228 2229New 'resize' event has been added to notify about change of block count property of block device. 2230Event is delivered only if block device was opened with `spdk_bdev_open_ext` function. 2231 2232### bdev zone 2233 2234Added new public header for zoned bdev. Zoned bdev is an extension 2235of the bdev interface. 2236 2237`spdk_bdev_get_zone_size()`, `spdk_bdev_get_max_open_zones()`, `spdk_bdev_get_optimal_open_zones()` 2238APIs were added for retrieving zoned device information. 2239`spdk_bdev_get_zone_info()` API was added for retrieving information about zones in zoned 2240device. 2241Added `spdk_bdev_zone_management()` API for changing zone state. 2242`spdk_bdev_zone_append()` and `spdk_bdev_zone_append_with_md()` APIs were added for 2243appending data to a zone. 2244Added `spdk_bdev_io_get_append location()` function for retrieving append location for I/O. 2245Added `spdk_bdev_is_zoned()` function for checking if bdev supports zoned namespace semantics. 2246 2247### bdev opal 2248 2249EXPERIMENTAL: A new opal bdev has been added to support management of 2250NVMe self-encrypting drives through the Opal specification. Users can 2251create opal bdevs from an NVMe namespace bdev, if the controller 2252containing that namespace supports Opal. Currently this is only 2253supported for namespace ID=1. The following RPCs have been added to 2254support Opal: `bdev_nvme_opal_init`, `bdev_nvme_opal_revert`, 2255`bdev_opal_create`, `bdev_opal_delete`, `bdev_opal_get_info`, 2256`bdev_opal_new_user`, `bdev_opal_set_lock_state`. 2257It does not yet support recreating the opal bdevs after application restart. 2258This bdev module should be considered very experimental, and the RPCs may 2259change significantly in future releases. 2260 2261### delay bdev 2262 2263The `bdev_delay_update_latency` has been added to allow users to update 2264a latency value for a given delay bdev. 2265 2266### compress bdev 2267 2268A new RPC `bdev_compress_get_orphans` has been added to list compress bdevs 2269that were not loaded due to a missing pm metadata file. In this state they 2270can only be deleted. 2271 2272### null bdev 2273 2274Metadata support has been added to Null bdev module. 2275 2276Protection information support has been added to Null bdev module. 2277 2278### nvme 2279 2280Added `no_shn_notification` to NVMe controller initialization options, users can enable 2281it for NVMe controllers. When the option is enabled, the controller will not do the 2282shutdown process and just disable the controller, users can start their application 2283later again to initialize the controller to the ready state. 2284 2285A controller flag `SPDK_NVME_CTRLR_WRR_SUPPORTED` was added to indicate the controller 2286can support weighted round robin arbitration feature with submission queue. 2287 2288Added `arbitration_burst` option for arbitration feature, and added three 2289`low/medium/high_priority_weight` options for weighted round robin arbitration. 2290 2291Added `spdk_nvme_ns_cmd_write_uncorrectable`. 2292 2293Added new error handling and reporting functionality. This includes several 2294new API functions to facilitate applications recovering when a qpair or 2295controller fails. 2296 2297`spdk_nvme_ctrlr_reconnect_io_qpair` - Reconnects a failed I/O qpair. 2298`spdk_nvme_ctrlr_set_trid` - Sets the trid of an existing controller. Can be used to 2299change the trid for failover cases. 2300`spdk_nvme_ctrlr_is_failed` - Returns the failed state of a controller. 2301`spdk_nvme_ctrlr_fail` - Forces a controller into a failed state. 2302 2303Modified the return behavior of several API functions to better indicate to 2304applications when a qpair is failed. This list of functions includes: 2305 2306`spdk_nvme_qpair_process_completions` 2307`spdk_nvme_ns_cmd_*` 2308`spdk_nvme_ctrlr_process_admin_completions` 2309`spdk_nvme_ctrlr_cmd_*` 2310 2311These functions now return -ENXIO when the qpair or controller on which they 2312operate is failed. 2313 2314EXPERIMENTAL: Added NVMe character device support to allow to create NVMe device nodes in Linux 2315kernel for controller as well as for namespace and process ioctl requests as usual 2316from linux environment. 2317 2318### nvmf 2319 2320The `spdk_nvmf_tgt_create` function now accepts an object of type `spdk_nvmf_target_opts` 2321as its only parameter. This new structure contains the max_subsystems parameter previously 2322passed into that function. 2323 2324A new public API function `spdk_nvmf_get_tgt` has been added which allows users to 2325retrieve a pointer to an `spdk_nvmf_tgt` object by supplying its name. In the special 2326case where an RPC or application only creates a single target, this function can accept 2327a null name parameter and will return the only available target. 2328 2329The majority of the NVMe-oF RPCs now accept an optional tgt_name parameter. This will 2330allow those RPCs to work with applications that create more than one target. 2331 2332Three new NVMe-oF RPCs have been added `nvmf_create_target`, `nvmf_delete_target`, and 2333`nvmf_get_targets`. These new RPCs provide a basic interface for managing multiple target 2334objects. In SPDK the target object defines a unique discovery service. As of this release, 2335these RPCs are not intended to be used with the in-tree SPDK target applications, spdk_tgt and 2336nvmf_tgt, which use a single, global target structure. As such, they are not included in scripts/rpc.py 2337 2338Three new header functions have also been added to help deal with multiple targets. 2339`spdk_nvmf_tgt_get_name` takes a target pointer as an argument and returns its human readable name. 2340`spdk_nvmf_get_first_target` takes no arguments and returns the first target in the global list. 2341`spdk_nvmf_get_next_tgt` takes a target pointer as an argument and returns the next one in the global list. 2342 2343The `spdk_nvmf_tgt_accept` takes additional argument allowing to pass arbitrary context 2344information to the `new_qpair` callback. This will simplify the code when having multiple 2345nvmf targets or when retrieving the context information from globals is not suitable. 2346 2347### blobstore 2348 2349A new `spdk_bdev_create_bs_dev_from_desc` function has been added and `spdk_bdev_create_bs_dev` 2350function has been deprecated. 2351The new create function can cowork with `spdk_bdev_open_ext` function, which provides callback 2352function that will be called by asynchronous event such as bdev removal. 2353 2354### blobfs_bdev 2355 2356A new blobfs module `bdev` has been added to simplify the operations of blobfs on bdev. 2357 2358Function `spdk_blobfs_bdev_detect` is added to detect whether blobfs exists on the given block device. 2359 2360Function `spdk_blobfs_bdev_create` is added to create a blobfs on the given block device. 2361 2362Function `spdk_blobfs_bdev_mount` is added to mount a blobfs on the given block device to 2363a host path by FUSE. Then, a new thread is created dedicatedly for one mountpoint to handle 2364FUSE request by blobfs API. 2365 2366### build 2367 2368Option to build FUSE components into blobfs_bdev module for mounting a blobfs filesystem. 2369It requires the installation of libfuse3. By default, it is disabled. And it will be 2370enabled if run `./configure` with `--with-fuse` option. 2371 2372### iSCSI 2373 2374Portals may no longer be associated with a cpumask. The scheduling of 2375connections is moving to a more dynamic model. 2376 2377An new RPC `iscsi_portal_group_set_auth` has been added to set CHAP authentication 2378for discovery sessions specific for the existing iSCSI portal group. This RPC overwrites 2379the setting by the global parameters for the iSCSI portal group. 2380 2381### socket 2382 2383Added `spdk_sock_is_connected` to check whether the socket is currently connected. 2384`spdk_sock_group_poll` now returns number of events on success. 2385 2386### env 2387 2388Added `spdk_pci_device_unclaim()` function to cleanup pci claim file. 2389 2390### event 2391 2392`framework_start_init` RPC no longer stops the application on error during 2393initialization. 2394 2395### DPDK 2396 2397Updated DPDK submodule to DPDK 19.08. 2398 2399### ocf 2400 2401Updated OCF submodule to OCF v19.06 2402 2403Along with update, new cache mode 'write only' was added. 2404 2405New cache modes added to use via RPC, wi - write invalidate and wa - write around. 2406 2407New version of OCF provides fully asynchronous management API. 2408 2409## v19.07: NVMe-oF FC Transport, VMD, NVMe-oF Persistent reservations, Bdev I/O with separate metadata 2410 2411### ftl 2412 2413EXPERIMENTAL: Added ability to mirror writes to persistent write buffer cache 2414to allow for recovery from dirty shutdown event. 2415 2416Added handling of Asynchronous Nand Management Events (ANM). 2417 2418### vmd 2419 2420EXPERIMENTAL: Added Intel Volume Management Device (VMD) driver. VMD is an integrated 2421controller inside the CPU PCIe root complex. It enables virtual HBAs for the connected 2422NVMe SSDs. `spdk_vmd_init()` enumerates NVMe SSDs behind VMD device and hook them into 2423SPDK PCI subsystem. `spdk_nvme_probe()` or `spdk_nvme_connect()` can be used to connect 2424NVMe driver to the device located at the given transport ID. 2425 2426To obtain transport ID of NVMe SSD behind VMD `spdk_lspci` can be used. 2427 2428Current implementation does not support hotplug. 2429 2430### blobfs 2431 2432Blobfs file asynchronous operations were added to public APIs. 2433 2434### util 2435 2436A new file API `spdk_posix_file_load` was added to load file content into a data buffer. 2437 2438New APIs `spdk_dif_ctx_set_data_offset`, `spdk_dif_verify_stream`, 2439`spdk_dif_get_range_with_md`, `spdk_dif_get_length_with_md` have been added, 2440and existing APIs `spdk_dif_ctx_init`, `spdk_dif_set_md_interleave_iovs`, and 2441`spdk_dif_generate_stream` have been refined to insert or strip DIF into iovs 2442windows of any alignment. 2443 2444New APIs `spdk_dif_ctx_set_remapped_init_ref_tag`, `spdk_dif_remap_ref_tag`, 2445and `spdk_dix_remap_ref_tag` have been added to remap DIF reference tag. 2446 2447New APIs `spdk_dif_update_crc32c` and `spdk_dif_update_crc32c_stream` have been 2448added to compute CRC-32C checksum for extended LBA payload. 2449 2450Bdevperf and bdevio applications now support starting tests with application specific 2451RPCs. Please see helper Python scripts in their respective directories. 2452This is a move towards simpler RPC-only configuration for all main 2453and auxiliary applications. 2454 2455Legacy INI style configuration for SPDK applications will become deprecated in SPDK 19.10, 2456and removed in SPDK 20.01. Please consider moving to JSON-RPC configuration files and/or 2457RPC driven run-time configuration. 2458 2459### nvmf 2460 2461EXPERIMENTAL: A Fibre Channel transport that supports Broadcom HBAs has been 2462added. This depends on the FC HBA driver at 2463https://github.com/ecdufcdrvr/bcmufctdrvr. See [the documentation](https://spdk.io/doc/nvmf.html#nvmf_fc_transport) 2464for more information. 2465 2466Persistent reservation emulation has been added to the NVMe-oF target. Persistent reservation 2467state is stored in a JSON file on the local filesystem between target restart. To support this, 2468an optional parameter to the RPC method `nvmf_subsystem_add_ns` called `--ptpl-file` was added. 2469This allows the user to specify which file to store the persistent reservation state in. Note 2470that this is done per namespace. 2471 2472The c2h success optimization under which a command capsule response is not sent 2473for reads is turned on by default. A config knob was added to allow disabling 2474the optimization. This will mostly be used for integration testing with 5.0.x kernels 2475while some compatibility fixes make their way down the pipeline for 5.1.x kernels. 2476 2477The sock priority setting of the TCP connection owned by the tcp transport is added. It is 2478used to optimize the TCP connection performance under designated traffic classes. And the 2479priority is used to differentiate the sock priority between SPDK NVMe-oF TCP target application 2480and other TCP based applications. 2481 2482Shared receive queue can now be disabled even for NICs that support it using the 2483`nvmf_create_transport` RPC method parameter `no_srq`. The actual use of a shared 2484receive queue is predicated on hardware support when this flag is not used. 2485 2486spdk_nvmf_get_optimal_poll_group was added, which is used to return the optimal 2487poll group for the qpair. And `ConnectionScheduler` configuration is added into the 2488[Nvmf] section in etc/spdk/nvmf.conf.in to demonstrate how to configure the connection 2489scheduling strategy among different spdk threads. 2490 2491Added infrastructure to retrieve global and per poll group NVMf statistics. 2492 2493DIF strip and insert is now supported for TCP transport. When it is enabled, DIF 2494setting is not exposed to the NVMe-oF initiator, and DIF is attached into data 2495for write I/O and stripped from data for read I/O. 2496 2497Added a field `dif_insert_or_strip` to struct spdk_nvmf_transport_opts, and 2498updated the related rpc function nvmf_create_transport to make this 2499configurable parameter available to users. The `dif_insert_or_strip` is relevant 2500for TCP transport for now and used to configure the DIF strip and insert. 2501 2502Added infrastructure to retrieve NVMf transport statistics. 2503 2504### notify 2505 2506The function `spdk_notify_get_types()` and `spdk_notify_get_events()` were 2507renamed to `spdk_notify_foreach_type()` and `spdk_notify_foreach_event()`, 2508respectively. And update type name of callback accordingly. 2509 2510### bdev 2511 2512The format of the data returned by the get_bdevs_iostat RPC has changed to 2513make it easier to parse. It now returns an object with a "ticks" object 2514and "bdevs" array with the per-bdev statistics. 2515 2516A new bdev module `delay` has been added which simulates a drive latency when placed 2517on top of a Null bdev. This module is intended only for testing and can be created using 2518the new RPC `bdev_delay_create`. That RPC takes the name of the underlying bdev as well 2519as average and p99 latency arguments for both read and write operations. Average latency is 2520defined as a value close to what you would expect a perf tool such as FIO to report back as 2521the mean latency of all I/O submitted to the drive. p99 latency is defined as the value one 2522would expect the drive to see the slowest 1% of I/O report. For underlying drives with already 2523significant latency, the latency values provided to the drive will be additive. This should be 2524taken into account if trying to achieve an artificial latency on top of an nvme drive or aio device. 2525 2526DIF reference tag remapping is now supported for partition type virtual bdev 2527modules. When using partition type virtual bdevs, block address space is 2528remapped during I/O processing and DIF reference tag is remapped accordingly. 2529 2530Added spdk_bdev_*_with_md() functions allowing for IO with metadata being transferred in 2531separate buffer. To check support for separate metadata, use spdk_bdev_is_md_separate(). 2532 2533All bdevs now have a UUID. For devices whose backing hardware does not provide a UUID, 2534one is automatically generated. Across runs of SPDK, bdevs whose UUID is automatically 2535generated may change. 2536 2537A new virtual bdev module `compress` has been added to provide compression services on top of 2538a thinly provisioned logical volume. See documentation for complete details. 2539 2540### nvme 2541 2542Added an optional parameter `--io-queue-requests` to RPC `set_bdev_nvme_options`, which 2543can be used to change the number of requests allocated for one NVMe I/O queue. For 2544very big I/O size, e.g. 128MiB, with this option user will not get an error due to 2545limited requests in NVMe driver layer. 2546 2547Added spdk_nvme_ctrlr_get_transport_id() to get the transport ID from a 2548previously attached controller. 2549 2550Nvme Opal library spdk_opal_cmd deprecated. Adding separate command APIs. 2551NVMe Opal library add support for activating locking SP which will make the transaction 2552from "Manufactured-Inactive" state to "Manufactured" state. Upon successfully invoking 2553of this method, lock and unlock features will be enabled. 2554NVMe Opal library add support for locking/unlocking range and list locking range info. 2555NVMe opal library add support for multiuser. Admin can enable user and add user to specific 2556locking range and the user can lock/unlock his range. 2557 2558Added spdk_nvme_ctrlr_io_cmd_raw_no_payload_build() allowing a caller to pass 2559a completely formed command to an NVMe submission queue (buffer addresses and all). 2560This is supported on the PCIe transport only. 2561 2562Added spdk_nvme_get_ctrlr_registers() to return a pointer to the virtual address 2563of the NVMe controller registers. This is supported on the PCIe transport only. 2564 2565Added additional options to the spdk_nvme_ctrlr_alloc_qpair() option parameter 2566structure to allow caller to override the virtual and optionally physical address 2567of the submission and completion queue pair to be created. This is supported on 2568the PCIe transport only. 2569 2570Added `disable_error_logging` to struct spdk_nvme_ctrlr_opts, which disables 2571logging of failed requests. By default logging is enabled. 2572 2573Added spdk_nvme_qpair_print_command(), spdk_nvme_qpair_print_completion() and 2574spdk_nvme_cpl_get_status_string(). Allowing for easier display of error messages. 2575 2576Added support for NVMe Sanitize command. 2577 2578### env 2579 2580The parameter `free_space` has been added to spdk_ring_enqueue() to wait when 2581the ring is almost full and resume when there is enough space available in 2582the ring. 2583 2584A new API `spdk_mempool_lookup` has been added to lookup the memory pool created 2585by the primary process. 2586 2587Added spdk_pci_get_first_device() and spdk_pci_get_next_device() to allow 2588iterating over PCI devices detected by SPDK. Because of this, all SPDK APIs 2589to attach/detach PCI devices are no longer thread safe. They are now meant to 2590be called from only a single thread only, the same only that called spdk_env_init(). 2591This applies to the newly added APIs as well. 2592 2593### vpp 2594 2595SPDK now supports VPP version 19.04.2, up from VPP 18.01. 2596 2597VPP socket abstraction now uses VPP Session API, instead of VLC (VPP Communications Library). 2598This allows for better control over sessions and queues. 2599Please see VPP documentation for more details: 2600[VPP Host Stack](https://wiki.fd.io/view/VPP/HostStack) 2601 2602### sock 2603 2604Add spdk_sock_get_optimal_sock_group(), which returns the optimal sock group for 2605this socket. When a socket is created, it is often assigned to a sock group using 2606spdk_sock_group_add_sock so that a set of sockets can be polled more efficiently. 2607For some network devices, it is optimal to assign particular sockets to specific 2608sock groups. This API is intended to provide the user with that information. 2609 2610spdk_sock_group_get_ctx() was added to return the context of the spdk_sock_group. 2611spdk_sock_group_create() is updated to allow input the user provided ctx. 2612spdk_sock_set_priority() is added to set the priority of the socket. 2613 2614### rpc 2615 2616Added thread_get_stats RPC method to retrieve existing statistics. 2617 2618Added nvmf_get_stats RPC method to retrieve NVMf subsystem statistics. 2619 2620Response buffers for RPC requests are now always pre-allocated, which implies 2621that all spdk_jsonrpc_begin_result() calls always succeed and return a valid 2622buffer for JSON response. RPC calls no longer need to check if the buffer is 2623non-NULL. 2624 2625Added SPDK_RPC_REGISTER_ALIAS_DEPRECATED to help with deprecation process when 2626renaming existing RPC. First time a deprecated alias is used, it will print 2627a warning message. 2628 2629RPC `get_rpc_methods` was renamed `rpc_get_methods`. The old name is still usable, 2630but is now deprecated. 2631 2632### blobstore 2633 2634A snapshot can now be deleted if there is only a single clone on top of it. 2635 2636### build 2637 2638Preliminary support for cross compilation is now available. Targeting an older 2639CPU on the same architecture using your native compiler can be accomplished by 2640using the `--target-arch` option to `configure` as follows: 2641 2642~~~bash 2643./configure --target-arch=broadwell 2644~~~ 2645 2646Additionally, some support for cross-compiling to other architectures has been 2647added via the `--cross-prefix` argument to `configure`. To cross-compile, set CC 2648and CXX to the cross compilers, then run configure as follows: 2649 2650~~~bash 2651./configure --target-arch=aarm64 --cross-prefix=aarch64-linux-gnu 2652~~~ 2653 2654### vhost 2655 2656A security vulnerability has been identified and fixed in SPDK Vhost-SCSI target. 2657A malicious client (e.g. a virtual machine) could send a carefully prepared, 2658invalid I/O request to crash the entire SPDK process. All users of SPDK Vhost-SCSI 2659target are strongly recommended to update. All SPDK versions < 19.07 are affected. 2660 2661By default, SPDK will now rely on upstream DPDK's rte_vhost instead of its fork 2662located inside SPDK repo. The internal fork is still kept around to support older 2663DPDK versions, but is considered legacy and will be eventually removed. 2664 2665`configure` will now automatically use the upstream rte_vhost if the used DPDK 2666version is >= 19.05. 2667 2668spdk_vhost_init() is now asynchronous and accepts a completion callback. 2669 2670### iscsi target 2671 2672A security vulnerability has been identified and fixed in SPDK iSCSI target. 2673A malicious client (e.g. an iSCSI initiator) could send a carefully prepared, 2674invalid I/O request to crash the entire SPDK process. All users of SPDK iSCSI 2675target are strongly recommended to update. All SPDK versions < 19.07 are affected. 2676 2677### thread 2678 2679Exposed spdk_set_thread() in order for applications to associate 2680with SPDK thread when necessary. 2681 2682Added spdk_thread_destroy() to allow framework polling the thread to 2683release resources associated with that thread. 2684 2685## v19.04: Compression bdev, Notification library, NVMe Opal support 2686 2687### nvme 2688 2689Added asynchronous probe support. New APIs spdk_nvme_probe_async(), 2690spdk_nvme_connect_async() and spdk_nvme_probe_poll_async() were added to enable 2691this feature. spdk_nvme_probe_async() and spdk_nvme_connect_async() return a 2692context associated with the specified controllers. Users then call 2693spdk_nvme_probe_poll_async() until it returns 0, indicating that the operation 2694completed. 2695 2696A new qpair creation option, delay_pcie_doorbell, was added. This can be passed 2697to spdk_nvme_alloc_io_qpair(). This makes the I/O submission functions, such as 2698spdk_nvme_ns_writev(), skip ringing the submission queue doorbell. Instead the 2699doorbell will be rung as necessary inside spdk_nvme_qpair_process_completions(). 2700This can result in significantly fewer MMIO writes to the doorbell register 2701under heavy load, greatly improving performance. 2702 2703spdk_nvme_ctrlr_get_regs_cmbsz() was added to report the size of the controller 2704memory buffer, if available. 2705 2706spdk_nvme_ctrlr_get_flags() was added to return controller feature 2707flags. Two flags are currently tracked: 2708SPDK_NVME_CTRLR_SGL_SUPPORTED 2709SPDK_NVME_CTRLR_SECURITY_SEND_RECV_SUPPORTED 2710 2711The NVMe hotplug poller is now able to detach devices hot-removed from the system 2712via `/sys/bus/pci/devices/<bdf>/remove` and `/sys/bus/pci/devices/<bdf>/driver/unbind`. 2713 2714Opal support was added for scan, take ownership, revert TPer, and dumping device 2715info. The nvme_manage tool can be used to perform these operations. The public 2716API functions are spdk_nvme_ctrlr_security_receive() and 2717spdk_nvme_ctrlr_security_send(). This module should be considered experimental 2718pending additional features and tests. 2719 2720The NVMe-oF initiator is now able to transfer IO whose size is more than 128KiB 2721if the corresponding NVMe-oF target allows. 2722 2723### raid 2724 2725Added new strip_size_kb rpc param on create to replace the more ambiguous 2726strip_size. The strip_size rpc param is deprecated. 2727 2728Changed the raid bdev product_name from "Pooled Device" to "Raid Volume" 2729 2730### thread 2731 2732Added spdk_thread_has_pollers() function to verify if there are any registered 2733pollers to be run on the thread. Added spdk_thread_is_idle() function to check 2734if there are any scheduled operations to be performed on the thread at given 2735time. 2736 2737spdk_thread_create() now takes an optional CPU affinity mask that is passed to 2738the scheduler. 2739 2740spdk_thread_lib_init() now takes an optional context size. For each thread 2741created, an additional region of memory of this size will be allocated. A 2742pointer to this region of memory can be obtained by calling 2743spdk_thread_get_ctx(). The inverse operation is also available via 2744spdk_thread_get_from_ctx(). 2745 2746spdk_thread_poll() now optionally accepts the current time, in ticks. This can 2747be used to avoid some calls to spdk_get_ticks() internally. 2748 2749spdk_app_start() now only accepts a single context argument. 2750 2751### bdev 2752 2753An new API `spdk_bdev_get_data_block_size` has been added to get size of data 2754block except for metadata. 2755 2756spdk_vbdev_register() has been deprecated. spdk_bdev_register() should be used 2757instead. 2758 2759A mechanism for acquiring and releasing data buffers from bdev modules, used 2760to perform zero copy operations, was added. 2761 2762New APIs spdk_bdev_get_md_size(), spdk_bdev_is_md_interleaved(), spdk_bdev_get_dif_type(), 2763spdk_bdev_is_dif_head_of_md(), and spdk_bdev_is_dif_check_enabled() have been 2764added to get metadata and DIF settings. 2765 2766Bdevs claimed by the `examine_config` callback will be now further examined in the 2767`examine_disk` callback. 2768 2769spdk_bdev_io_get_io_channel() was added as a convenient way to get an io_channel 2770from a bdev_io. 2771 2772### ocf 2773 2774Updated OCF submodule to OCF v19.3.2 2775 2776Added support for many-to-one configuration for OCF bdev. 2777Multiple core devices can now be cached on single cache device. 2778 2779Added persistent metadata support, allowing to restore cache state after shutdown. 2780During start of SPDK application, the devices are examined and if OCF metadata 2781is present - appropriate OCF bdevs will be recreated. 2782 2783Added Write-Back mode support. In this mode, data is first written to 2784caching device and periodically synchronized to the core devices. 2785Dirty data is saved as persistent metadata on cache device, 2786allowing for safe restore during application restart. 2787For more details please see OCF documentation: 2788[OpenCAS cache configuration](https://open-cas.github.io/cache_configuration.html) 2789 2790### NVMe-oF Target 2791 2792Support for per-device shared receive queues in the RDMA transport has been added. 2793It is enabled by default for any device that supports it. 2794 2795The size of a shared receive queue is defined by transport configuration file parameter 2796`MaxSRQDepth` and `nvmf_create_transport` RPC method parameter `max_srq_depth`. 2797Default size is 4096. 2798 2799Add model number as parameter to construct_nvmf_subsystem (-d option), 2800rather than using hardcoded define. 2801 2802DIF passthrough feature has been added. DIF setting of the allocated bdevs is 2803exposed to the NVMe-oF initiator and data with DIF from the NVMe-oF initiator is 2804passed through to the allocated bdevs. 2805 2806### env 2807 2808The `phys_addr` parameter in spdk_malloc() and spdk_zmalloc() has been deprecated. 2809For retrieving physical addresses, spdk_vtophys() should be used instead. 2810 2811spdk_realloc() has been added to reallocate DMA/shared memory. 2812 2813spdk_pci_device_is_removed() has been added to let the upper-layer SPDK drivers know 2814that device has a pending external hotremove request. 2815 2816spdk_env_fini() and spdk_env_dpdk_post_fini() were added to release any resources 2817allocated by spdk_env_init() or spdk_env_dpdk_post_init() respectively. It is expected 2818that common usage of those functions is to call them just before terminating the process. 2819 2820Added spdk_iommu_is_enabled() to report if SPDK application is using IOMMU for DMA. 2821 2822### DPDK 2823 2824Dropped support for DPDK 17.07 and earlier, which SPDK won't even compile with right now. 2825 2826Updated DPDK submodule to DPDK 19.02. 2827 2828### rpc 2829 2830New `get_spdk_version` RPC method is introduced to get version info of the running SPDK application. 2831 2832The `start_nbd_disk` RPC method now take nbd_device as an optional parameter. If nbd_device 2833is specified, use that specified nbd device. If it's not specified, pick available one. 2834 2835### iSCSI target 2836 2837DIF strip and insert is now supported. DIF settings are not exposed to the iSCSI initiator. 2838DIF is attached into data for write I/O and stripped from data for read I/O. 2839 2840### vhost 2841 2842Added experimental support for running with the external, upstream rte_vhost library. 2843This can be enabled by configuring SPDK with an `--without-internal-vhost-lib` flag. 2844The minimum supported rte_vhost version (DPDK version) is 19.05-rc1. 2845 2846As a result of fuzz testing, a lot of data races in vhost-scsi LUN hotplug path were identified and 2847fixed. Those data races could have potentially resulted in SPDK crashes, RPC hangs, or memory leaks 2848if Vhost-SCSI LUN hotplug RPCs were executed while connected VMs were in the middle of restarting. 2849 2850The SCSI target id in `add_vhost_scsi_lun` RPC is now optional. If `-1` is passed, the first 2851unoccupied target id will be used. 2852 2853### AIO 2854 2855AIO bdev module can now reap I/O completions directly from userspace, significantly improving 2856the overall performance. 2857 2858### blobfs 2859 2860Synchronous IO operations no longer use spdk_io_channel, but instead use 2861spdk_fs_thread_ctx. The behavior is otherwise identical. 2862 2863### OCF 2864 2865Added support for caching multiple bdevs using a single bdev as a cache. 2866 2867### notify 2868 2869Added the notify library that provides a high performance local event bus 2870between libraries. Example usage was added to bdev module, which reports 2871notifications for added and removed bdevs via RPC. 2872 2873### sock 2874 2875Added new API spdk_sock_readv() to the sock library for performing vectored 2876reads. 2877 2878### event 2879 2880The function spdk_subsystem_init() no longer requires spdk_event as an argument. 2881 2882Changed API of spdk_subsystem_config_json() to no longer be asynchronous. 2883 2884### io_uring 2885 2886A bdev module that issues I/O to kernel block devices using the new io_uring Linux kernel 2887API was added. This module requires liburing. 2888 2889### build 2890 2891Options to easily compile with profile guided optimization have been added to 2892`configure`. To use profile guided optimization with SPDK, run 2893`./configure --with-pgo-capture`, build SPDK, then run a workload of your 2894choosing. Then, simply run `./configure --with-pgo-enable` and recompile to 2895build using the generated profile data. Profile guided optimization can yield 2896very large performance improvements, especially on GCC 8 and clang 7. This may 2897be combined with link time optimization which has been available under the 2898`--enable-lto` configure option for several releases. 2899 2900### compression bdev/reduce library 2901 2902Added "reduce" block compression scheme based on using SSDs for storing 2903compressed blocks of storage and persistent memory for metadata. Please see 2904[compression](https://spdk.io/doc/bdev.html) for more details. 2905 2906## v19.01: NVMe-oF TCP/IP Transport, Open Channel SSD Flash Translation Layer, Caching bdev based on OCF, ISA-L Support, DIF/DIX library 2907 2908### ocf bdev 2909 2910New virtual bdev module based on [Open CAS Framework](https://open-cas.github.io/) has been added. 2911This module allows for the use of one bdev to act as a high performance cache in front of another bdev. 2912Please see [documentation](https://spdk.io/doc/bdev.html#bdev_config_cas) for more details. 2913Only write through mode is currently supported and this feature is considered experimental. 2914 2915### event framework 2916 2917For `spdk_app_parse_args`, add return value to the callback which parses application 2918specific command line parameters to protect SPDK applications from crashing by invalid 2919values from user input. 2920 2921By default, all SPDK applications will now reserve all hugepages at runtime. The pre-reserved 2922memory size can be still set with `-s` or `--mem-size` option, although the default value 2923was reduced down to 0. 2924 2925A custom hugetlbfs directory can now be specified via spdk_app_opts. 2926This can be used to configure hugepages with different sizes, a different size limit, 2927or different access permissions than the system's default hugepage pool. 2928SPDK applications can specify a custom hugetlbfs mount with the `--huge-dir` option. 2929 2930### environment 2931 2932spdk_vtophys() has been refactored to accept length of the translated region as a new 2933parameter. The function will now update that parameter with the largest possible value 2934for which the memory is contiguous in the physical memory address space. 2935 2936The following functions were removed: 2937 2938- spdk_pci_nvme_device_attach() 2939- spdk_pci_nvme_enumerate() 2940- spdk_pci_ioat_device_attach() 2941- spdk_pci_ioat_enumerate() 2942- spdk_pci_virtio_device_attach() 2943- spdk_pci_virtio_enumerate() 2944 2945They were replaced with generic spdk_pci_device_attach() and spdk_pci_enumerate() which 2946require a new spdk_pci_driver object to be provided. It can be one of the following: 2947 2948- spdk_pci_nvme_get_driver() 2949- spdk_pci_ioat_get_driver() 2950- spdk_pci_virtio_get_driver() 2951 2952spdk_pci_hook_device() and spdk_pci_unhook_device() were added. Those allow adding a virtual 2953spdk_pci_device into the SPDK PCI subsystem. A virtual device calls provided callbacks for 2954each BAR mapping request or PCI config access. It's attachable with spdk_pci_device_attach() 2955or spdk_pci_enumerate() like any other device. 2956 2957A new spdk_pause() function was added to pause CPU execution for an implementation specific 2958amount of time. Quoting from DPDK function this is based on: "This call is intended for 2959tight loops which poll a shared resource or wait for an event. A short pause within the loop 2960may reduce the power consumption." 2961 2962A new public header file env_dpdk.h has been introduced, and function spdk_env_dpdk_post_init 2963is added into it. If user is using DPDK, and already called rte_eal_init, then include 2964include/spdk/env_dpdk.h, and call spdk_env_dpdk_post_init() instead of spdk_env_init. 2965 2966ISA-L has been added as an SPDK submodule. ISA-L is enabled by default on x86 architecture 2967to accelerate algorithms such as CRC for iSCSI and NVMe-oF. Users may still disable ISA-L 2968by explicitly passing --without-isal to the configure script. 2969 2970### util 2971 2972A new uuid API `spdk_uuid_copy` was added to make a copy of the source uuid. 2973 2974An new parameter `init_crc` representing the initial CRC value was added to 2975`spdk_crc16_t10dif`. The parameter can be used to calculate a CRC value spanning 2976multiple separate buffers. 2977 2978New DIF APIs were added to generate and verify DIF by byte granularity for both DIF and DIX 2979formats. Among them, DIF with copy APIs will be usable to emulate DIF operations such as DIF 2980insert and strip. 2981 2982Added `spdk_strtol` and `spdk_strtoll` to provide additional error checking around `strtol` 2983and `strtoll`. 2984 2985Added `spdk_sprintf_append_realloc` and `spdk_vsprintf_append_realloc` for appending a string 2986with automatic buffer re-allocation. 2987 2988### nvme 2989 2990Wrapper functions spdk_nvme_ctrlr_security_send() and spdk_nvme_ctrlr_security_receive() are 2991introduced to support further security protocol development. 2992 2993admin_timeout_ms was added to NVMe controller initialization options, users 2994can change the default value when probing a controller. 2995 2996Add two new fields "header_digest" and "data_digest" in struct spdk_nvme_ctrlr_opts, 2997it will be used to enable the digest support for the NVMe/TCP transport. 2998 2999Add a new TCP/IP transport(located in lib/nvme/nvme_tcp.c) in nvme driver. With 3000this new transport, it can be used to connect the NVMe-oF target with the 3001same TCP/IP support. 3002 3003Added API, spdk_nvme_ctrlr_is_discovery(), to indicate whether the ctrlr 3004arg refers to a Discovery Controller or not. 3005 3006Added an API function `spdk_nvme_host_id_parse` and corresponding object `spdk_nvme_host_id` 3007for parsing host address and host service ID arguments on a per connection basis. 3008 3009The RPC `construct_nvme_bdev` now allows a user to specify a source address and service id for the host to 3010use when connecting to the controller backing the NVMe bdev. 3011 3012### NVMe-oF Target 3013 3014The `spdk_nvmf_tgt_opts` struct has been deprecated in favor of `spdk_nvmf_transport_opts`. 3015Users will no longer be able to specify target wide I/O parameters. `spdk_nvmf_tgt_listen` 3016will also no longer implicitly initialize a transport with the default target options (since 3017there are none). Instead, a user must manually instantiate the transport with `spdk_nvmf_transport_create` 3018prior to calling `spdk_nvmf_tgt_listen`. 3019 3020Related to the previous change, the rpc `set_nvmf_target_options` has been renamed to 3021`set_nvmf_target_max_subsystems` to indicate that this is the only target option available for the user to edit. 3022 3023Added fields `num_shared_buffers` and `buf_cache_size` in struct spdk_nvmf_transport_opts, 3024and also updated the related rpc function nvmf_create_transport, to make this 3025configurable parameter available to users. The `num_shared_buffers` is used to 3026configure the shared buffer numbers of the transport used by RDMA or TCP transport. 3027`buf_cache_size` configures number of shared buffers to cache per poll group. 3028 3029### nvmf 3030 3031Add a new TCP/IP transport (located in lib/nvmf/tcp.c). With this transport, 3032the SPDK NVMe-oF target can have a new transport, and can serve the NVMe-oF 3033protocol via TCP/IP from the host. 3034 3035Added optional mechanism to modify the RDMA transport's behavior when creating protection domains and registering memory. 3036By default, the RDMA transport will use the ibverbs library to create protection domains and register memory. 3037Using `spdk_nvme_rdma_init_hooks` will subvert that and use an existing registration. 3038 3039### bdev 3040 3041Added `enable_bdev_histogram` and `get_bdev_histogram` RPC commands to allow gathering latency data for specified bdev. 3042Please see [documentation](https://spdk.io/doc/bdev.html#rpc_bdev_histogram) for more details. 3043 3044Added `required_alignment` field to `spdk_bdev`, that specifies an alignment requirement for data buffers associated with an spdk_bdev_io. 3045Bdev layer will automatically double buffer any spdk_bdev_io that violates this alignment, before the spdk_bdev_io is submitted to the bdev module. 3046 3047On shutdown, bdev unregister now proceeds in top-down fashion, with 3048claimed bdevs skipped (these will be unregistered later, when virtual 3049bdev built on top of the respective base bdev unclaims it). This 3050allows virtual bdevs to be shut down cleanly as opposed to the 3051previous behavior that didn't differentiate between hotremove and 3052planned shutdown. 3053 3054The `delete_bdev` RPC is now deprecated. Users should instead use the specific deletion RPC 3055for the bdev type to be removed (i.e. delete_malloc_bdev). 3056 3057Added support for separate bandwidth rate limits for read and write to QoS in bdev layer. 3058 3059Bdev I/O statistics now track unmap operations. 3060 3061### logical volumes 3062 3063Logical volume bdev can now be marked as read only using `set_read_only_lvol_bdev` RPC. 3064This allows for basing clones on top of lvol_bdev without first creating a snapshot. 3065 3066Added option to change method for data erasure when deleting lvol or resizing down. 3067Default of unmapping clusters can now be changed to writing zeroes or no operation. 3068 3069Added option to change method for erasing data region on lvol store creation. 3070Default of unmapping can now be changed to writing zeroes or no operation. 3071 3072### log 3073 3074"trace flags" are now referred to as "log flags" in the SPDK log API. The 3075set_trace_flag, clear_trace_flag and get_trace_flags RPCs are now deprecated, 3076and set_log_flag, clear_log_flag and get_log_flags RPCs have been added. 3077 3078### trace 3079 3080New `trace_record` application was added. It can be used to poll spdk trace shm file and 3081append any new trace entries into another specified file. This can help retain those entries 3082that would otherwise be overwritten in the shm file. See 3083[Capturing sufficient trace events](https://spdk.io/doc/nvmf_tgt_tracepoints.html#capture_trace_events) 3084for more details. 3085 3086Number of trace entries in circular buffer per lcore can now be assigned by starting SPDK app 3087with argument "--num-trace-entries <NUM>" provided. 3088 3089New `get_tpoint_group_mask` RPC was added to get current tpoint_group_mask, and 3090each tpoint group status. 3091New `enable_tpoint_group` and `disable_tpoint_group` RPC were added to enable or 3092disable a specific tpoint group. 3093 3094### ftl 3095 3096EXPERIMENTAL: Added basic flash translation layer module allowing for using Open Channel SSDs as 3097block devices. The module is split into the library (located in lib/ftl) and bdev_ftl 3098(lib/bdev/ftl). See the [documentation](https://spdk.io/doc/ftl.html) for more details. 3099 3100### vhost 3101 3102A security vulnerability has been identified and fixed in the SPDK vhost target. A malicious 3103vhost client (i.e. virtual machine) could carefully construct a circular descriptor chain which 3104would result in a partial denial of service in the SPDK vhost target. These types of descriptor 3105chains are now properly detected by the vhost target. All SPDK vhost users serving untrusted 3106vhost clients are strongly recommended to upgrade. (Reported by Dima Stepanov and Evgeny 3107Yakovlev.) 3108 3109Vhost SCSI and Vhost Block devices can now accept multiple connections on the same socket file. 3110Each connection (internally called a vhost session) will have access to the same storage, but 3111will use different virtqueues, different features and possibly different memory. 3112 3113### vhost scsi 3114 3115SCSI target hotremove can now be performed even without the VIRTIO_SCSI_F_HOTPLUG feature negotiated. 3116Regardless of VIRTIO_SCSI_F_HOTPLUG support, the hotremoval will be still reported through SCSI sense codes. 3117 3118### DPDK 3119 3120DPDK submodule was updated to DPDK 18.11. Note that SPDK does not fully leverage the new 3121multi-process device hotplug yet and continues to work the same way it always did. 3122 3123Dropped support for DPDK 16.07 and earlier, which SPDK won't even compile with right now. 3124 3125### RPC 3126 3127The following RPC commands deprecated in the previous release are now removed: 3128 3129- construct_virtio_user_scsi_bdev 3130- construct_virtio_pci_scsi_bdev 3131- construct_virtio_user_blk_bdev 3132- construct_virtio_pci_blk_bdev 3133- remove_virtio_scsi_bdev 3134- construct_nvmf_subsystem 3135 3136### Miscellaneous 3137 3138The configure options `--with-raid` and `--without-raid` that were deprecated in the previous 3139release are now removed. 3140 3141### nbd 3142 3143Starting nbd using `spdk_nbd_start` is now performed asynchronously. 3144 3145### net framework 3146 3147Net framework initialization and finish is now done asynchronously. 3148 3149### rpc 3150 3151Added `spdk_rpc_is_method_allowed` function for checking whether method is permitted in a given state. 3152Added `spdk_rpc_get_state` to check current state of RPC server. 3153RPC `wait_subsystem_init` has been added to allow clients to block until all subsystems are initialized. 3154 3155### json rpc 3156 3157JSON RPC client is now running in non-blocking mode. Requests are sent and received during spdk_jsonrpc_client_poll. 3158JSON RPC server can now receive a callback on connection termination or server shutdown using `spdk_jsonrpc_conn_add_close_cb` 3159and `spdk_jsonrpc_conn_del_close_cb`. 3160 3161## v18.10: Dynamic memory allocation, Crypto Virtual Bdev, jsonrpc-client, SPDKCLI iSCSI and NVMe-oF support 3162 3163### nvme 3164 3165spdk_nvme_ctrlr_cmd_security_send() and spdk_nvme_ctrlr_cmd_security_receive() 3166were added to support sending or receiving security protocol data to or from 3167nvme controller. 3168 3169spdk_nvme_ns_get_extended_sector_size() was added. This function includes 3170the metadata size per sector (if any). spdk_nvme_ns_get_sector_size() still 3171returns only the data size per sector, not including metadata. 3172 3173New `send_nvme_cmd` RPC was added to allow sending NVMe commands directly to NVMe controller. 3174See the [send_nvme_cmd](http://spdk.io/doc/jsonrpc.html#rpc_send_nvme_cmd) documentation 3175for more details. 3176 3177### Build System 3178 3179New `configure` options, `--with-shared` and `--without-shared` 3180[default], provide the capability to build, or not, SPDK shared libraries. 3181This includes the single SPDK shared lib encompassing all of the SPDK 3182static libs as well as individual SPDK shared libs corresponding to 3183each of the SPDK static ones. Although the production of the shared 3184libs conforms with conventional version naming practices, such naming 3185does not at this time confer any SPDK ABI compatibility claims. 3186 3187### bdev 3188 3189spdk_bdev_alias_del_all() was added to delete all alias from block device. 3190 3191A new virtual bdev module has been added to perform at rest data encryption using the DPDK CryptoDev 3192Framework. The module initially uses a software AESNI CBC cipher with experimental support for the 3193Intel QAT hardware accelerator also currently implemented with support for CBC cipher. Future work 3194may include additional ciphers as well as consideration for authentication. 3195 3196The RAID virtual bdev module is now always enabled by default. The configure --with-raid and 3197--without-raid options are now ignored and deprecated and will be removed in the next release. 3198 3199Enforcement of bandwidth limits for quality of service (QoS) has been added to the bdev layer. 3200See the new [set_bdev_qos_limit](http://www.spdk.io/doc/jsonrpc.html#rpc_set_bdev_qos_limit) 3201documentation for more details. The previous set_bdev_qos_limit_iops RPC method introduced at 320218.04 release has been deprecated. The new set_bdev_qos_limit RPC method can support both 3203bandwidth and IOPS limits. 3204 3205spdk_bdev_config_json() and corresponding `get_bdevs_config` RPC was removed. 3206 3207### Environment Abstraction Layer and Event Framework 3208 3209The size parameter of spdk_mem_map_translate is now a pointer. This allows the 3210function to report back the actual size of the translation relative to the original 3211request made by the user. 3212 3213A new structure spdk_mem_map_ops has been introduced to hold memory map related 3214callbacks. This structure is now passed as the second argument of spdk_mem_map_alloc 3215in lieu of the notify callback. 3216 3217### DPDK 18.08 3218 3219The DPDK submodule has been updated to the DPDK 18.08 release. SPDK will now automatically 3220utilize DPDK's dynamic memory management with DPDK versions >= 18.05.1. 3221 3222Hugepages can be still reserved with `[-s|--mem-size <size>]` option at application startup, 3223but once we use them all up, instead of failing user allocations with -ENOMEM, we'll try 3224to dynamically reserve even more. This allows starting SPDK with `--mem-size 0` and using 3225only as many hugepages as it is really needed. 3226 3227Due to this change, the memory buffers returned by `spdk_*malloc()` are no longer guaranteed 3228to be physically contiguous. 3229 3230### I/OAT 3231 3232I/OAT driver can now reinitialize I/OAT channels after encountering DMA errors. 3233 3234### iscsi target 3235 3236Parameter names of `set_iscsi_options` and `get_iscsi_global_params` RPC 3237method for CHAP authentication in discovery sessions have been changed to 3238align with `construct_target_node` RPC method. Old names are still usable 3239but will be removed in future release. 3240 3241`set_iscsi_discovery_auth` and `set_iscsi_target_node_auth` RPC methods have 3242been added to set CHAP authentication for discovery sessions and existing 3243target nodes, respectively. 3244 3245The SPDK iSCSI target supports an AuthFile which can be used to load CHAP 3246shared secrets when the iSCSI target starts. SPDK previously provided a 3247default location for this file (`/usr/local/etc/spdk/auth.conf`) if none was 3248specified. This default has been removed. Users must now explicitly specify 3249the location of this file to load CHAP shared secrets from a file, or use 3250the related iSCSI RPC methods to add them at runtime. 3251 3252### iscsi initiator 3253 3254The SPDK iSCSI initiator is no longer considered experimental and becomes 3255a first-class citizen among bdev modules. The basic usage has been briefly 3256described in the bdev user guide: [iSCSI bdev](https://spdk.io/doc/bdev.html#bdev_config_iscsi) 3257 3258### Miscellaneous 3259 3260The SPDK application framework can now parse long name command line parameters. 3261Most single-character parameters have a long name equivalent now. See the 3262[Command Line Parameters](https://spdk.io/doc/app_overview.html) documentation 3263for details or use the `--help` command line parameter to list all available 3264params. 3265 3266bdevperf `-s` param (io size) was renamed to `-o` as `-s` had been already 3267used by existing apps for memory size. 3268 3269bdevio can now accept all SPDK command line parameters. The config now has to 3270be provided with `-c` or `--config` param. 3271 3272The following ioat/perf and nvme/perf parameters were renamed as well: 3273 `-s` (io size) to `-o` 3274 `-d` (mem size) to `-s` 3275 3276The ReactorMask config file parameter has been deprecated. Users should 3277use the -m or --cpumask command line option to specify the CPU core mask 3278for the application. 3279 3280Default config file pathnames have been removed from iscsi_tgt, nvmf_tgt 3281and vhost. Config file pathnames may now only be specified using the 3282-c command line option. 3283 3284Users may no longer set DPDK_DIR in their environment to specify the 3285location of the DPDK installation used to build SPDK. Using DPDK_DIR 3286has not been the documented nor recommended way to specify the DPDK 3287location for several releases, but removing it ensures no unexpected 3288surprises for users who may have DPDK_DIR defined for other reasons. 3289Users should just use the "configure" script to specify the DPDK 3290location before building SPDK. 3291 3292Although we know that many developers still use Python 2 we are officially 3293switching to Python3 with requirement that all new code must be valid also 3294for Python 2 up to the EOL which is year 2020. 3295 3296Invoking interpreter explicitly is forbidden for executable scripts. There 3297is no need to use syntax like "python ./scripts/rpc.py". All executable 3298scripts must contain proper shebang pointing to the right interpreter. 3299Scripts without shebang mustn't be executable. 3300 3301A Python script has been added to enable conversion of old INI config file 3302to new JSON-RPC config file format. This script can be found at 3303scripts/config_converter.py. Example how this script can be used: 3304~~~{.sh} 3305cat old_format.ini | scripts/config_converter.py > new_json_format.json 3306~~~ 3307 3308### Sock 3309 3310Two additional parameters were added to spdk_sock_get_addr() for the server 3311port and client port. These parameters are named "sport" and "cport" 3312respectively. 3313 3314### Virtio 3315 3316The following RPC commands have been deprecated: 3317 3318- construct_virtio_user_scsi_bdev 3319- construct_virtio_pci_scsi_bdev 3320- construct_virtio_user_blk_bdev 3321- construct_virtio_pci_blk_bdev 3322- remove_virtio_scsi_bdev 3323 3324The `construct_virtio_*` ones were replaced with a single `construct_virtio_dev` 3325command that can create any type of Virtio bdev(s). `remove_virtio_scsi_bdev` 3326was replaced with `remove_virtio_bdev` that can delete both Virtio Block and SCSI 3327devices. 3328 3329### Blobfs 3330 3331spdk_file_get_id() returning unique ID for the file was added. 3332 3333### JSON 3334 3335Added jsonrpc-client C library intended for issuing RPC commands from applications. 3336 3337Added API enabling iteration over JSON object: 3338 3339- spdk_json_find() 3340- spdk_json_find_string() 3341- spdk_json_find_array() 3342- spdk_json_object_first() 3343- spdk_json_array_first() 3344- spdk_json_next() 3345 3346### Blobstore 3347 3348Blobstore I/O operations are now based on io_units, instead of blobstore page size. 3349The io_unit size is now the same as the underlying block device's block size. 3350Logical volumes built on a block device with 512B block size can now be used as boot devices 3351in QEMU. 3352 3353### SPDKCLI 3354 3355The SPDKCLI interactive command tool for managing SPDK is no longer considered experimental. 3356Support for the iSCSI and NVMe-oF targets has been added. 3357 3358## v18.07: Raid, Infrastructure Improvements, Bug Fixes 3359 3360### bdev 3361 3362A new public header file bdev_module.h has been introduced to facilitate the 3363development of new bdev modules. This header includes an interface for the 3364spdk_bdev_part and spdk_bdev_part_base objects to enable the creation of 3365multiple virtual bdevs on top of a single base bdev and should act as the 3366primary API for module authors. 3367 3368spdk_bdev_get_opts() and spdk_bdev_set_opts() were added to set bdev-wide 3369options. 3370 3371A mechanism for handling out of memory condition errors (ENOMEM) returned from 3372I/O submission requests at the bdev layer has been added. See 3373spdk_bdev_queue_io_wait(). 3374 3375The spdk_bdev_get_io_stat() function now returns cumulative totals instead of 3376resetting on each call. This allows multiple callers to query I/O statistics 3377without conflicting with each other. Existing users will need to adjust their 3378code to record the previous I/O statistics to calculate the delta between calls. 3379 3380I/O queue depth tracking and samples options have been added. See 3381spdk_bdev_get_qd(), spdk_bdev_get_qd_sampling_period(), and 3382spdk_bdev_set_qd_sampling_period(). 3383 3384### RAID module 3385 3386A new bdev module called "raid" has been added as experimental module which 3387aggregates underlying NVMe bdevs and exposes a single raid bdev. Please note 3388that vhost will not work with this module because it does not yet have support 3389for multi-element io vectors. 3390 3391### Log 3392 3393The debug log component flag available on several SPDK applications has been 3394renamed from `-t` to `-L` to prevent confusion with tracepoints and to allow the 3395option to be added to tools that already use `-t` to mean something else. 3396 3397### Blobstore 3398 3399A new function, spdk_bs_dump(), has been added that dumps all of the contents of 3400a blobstore to a file pointer. This includes the metadata and is very useful for 3401debugging. 3402 3403Two new operations have been added for thin-provisioned blobs. 3404spdk_bs_inflate_blob() will allocate clusters for all thinly provisioned regions 3405of the blob and populate them with the correct data by reading from the backing 3406blob(s). spdk_bs_blob_decouple_parent() works similarly, but will only allocate 3407clusters that correspond to data in the blob's immediate parent. Clusters 3408allocated to grandparents or that aren't allocated at all will remain 3409thin-provisioned. 3410 3411### BlobFS 3412 3413Changed the return type of spdk_file_truncate() from void to int to allow the 3414propagation of `ENOMEM` errors. 3415 3416### NVMe Driver 3417 3418The new API functions spdk_nvme_qpair_add_cmd_error_injection() and 3419spdk_nvme_qpair_remove_cmd_error_injection() have been added for NVMe error 3420emulation. Users can set a specified command to fail with a particular error 3421status. 3422 3423Changed the name `timeout_sec` parameter to `timeout_us` in 3424spdk_nvme_ctrlr_register_timeout_callback(), and also changed the type from 3425uint32_t to uint64_t. This will give users more fine-grained control over the 3426timeout period. 3427 3428Basic support for Open Channel SSDs was added. See nvme_ocssd.h 3429 3430### NVMe Over Fabrics 3431 3432The spdk_nvmf_tgt_destroy() function is now asynchronous and takes a callback 3433as a parameter. 3434 3435spdk_nvmf_qpair_disconnect() was added to allow the user to disconnect qpairs. 3436 3437spdk_nvmf_subsystem_get_max_namespaces() was added to query the maximum allowed 3438number of namespaces for a given subsystem. 3439 3440### Build System 3441 3442The build system now generates a combined shared library (libspdk.so) that may 3443be used in place of the individual static libraries (libspdk_*.a). The combined 3444library includes all components of SPDK and is intended to make linking against 3445SPDK easier. The static libraries are also still provided for users that prefer 3446to link only the minimal set of components required. 3447 3448### git pre-commit and pre-push hooks 3449 3450The pre-commit hook will run `scripts/check_format.sh` and verify there are no 3451formatting errors before allowing `git commit` to run. The pre-push hook runs 3452`make CONFIG_WERROR=y` with and without `CONFIG_DEBUG=y` using both the gcc and 3453clang compiler before allowing `git push` to run. Following each DEBUG build 3454`test/unit/unittest.sh` is run and verified. Results are recorded in the 3455`make.log` file. 3456 3457To enable type: 'git config core.hooksPath .githooks'. To override after 3458configuration use the `git --no-verify` flag. 3459 3460### RPC 3461 3462The `start_nbd_disk` RPC method now returns the path to the kernel NBD device node 3463rather than always returning `true`. 3464 3465### DPDK 18.05 3466 3467The DPDK submodule has been rebased on the DPDK 18.05 release. DPDK 18.05 supports 3468dynamic memory allocation, but due to some issues found after the DPDK 18.05 release, 3469that support is not enabled for SPDK 18.07. Therefore, SPDK 18.07 will continue to use 3470the legacy memory allocation model. The plan is to enable dynamic memory allocation 3471after the DPDK 18.08 release which should fix these issues. 3472 3473### Environment Abstraction Layer and Event Framework 3474 3475The spdk_mem_map_translate() function now takes a size parameter to indicate the size of 3476the memory region. This can be used by environment implementations to validate the 3477requested translation. 3478 3479The I/O Channel implementation has been moved to its own library - lib/thread. The 3480public API that was previously in spdk/io_channel.h is now in spdk/thread.h The 3481file spdk/io_channel.h remains and includes spdk/thread.h. 3482 3483spdk_reactor_get_tsc_stats was added to return interesting statistics for each 3484reactor. 3485 3486### IOAT 3487 3488IOAT for copy engine is disabled by default. It can be enabled by specifying the Enable 3489option with "Yes" in `[Ioat]` section of the configuration file. The Disable option is 3490now deprecated and will be removed in a future release. 3491 3492## v18.04: Logical Volume Snapshot/Clone, iSCSI Initiator, Bdev QoS, VPP Userspace TCP/IP 3493 3494### vhost 3495 3496The SPDK vhost-scsi, vhost-blk and vhost-nvme applications have fixes to address the 3497DPDK rte_vhost vulnerability [CVE-2018-1059](http://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2018-1059). 3498Please see this [security advisory](https://access.redhat.com/security/cve/cve-2018-1059) 3499for additional information on the DPDK vulnerability. 3500 3501Workarounds have been added to ensure vhost compatibility with QEMU 2.12. 3502 3503EXPERIMENTAL: Support for vhost-nvme has been added to the SPDK vhost target. See the 3504[vhost documentation](http://www.spdk.io/doc/vhost.html) for more details. 3505 3506### Unified Target Application 3507 3508A new unified SPDK target application, `spdk_tgt`, has been added. This application combines the 3509functionality of several existing SPDK applications, including the iSCSI target, NVMe-oF target, 3510and vhost target. The new application can be managed through the existing configuration file and 3511[JSON-RPC](http://www.spdk.io/doc/jsonrpc.html) methods. 3512 3513### Env 3514 3515spdk_mempool_get_bulk() has been added to wrap DPDK rte_mempool_get_bulk(). 3516 3517New memory management functions spdk_malloc(), spdk_zmalloc(), and spdk_free() have been added. 3518These new functions have a `flags` parameter that allows the user to specify whether the allocated 3519memory needs to be suitable for DMA and whether it should be shared across processes with the same 3520shm_id. The new functions are intended to replace spdk_dma_malloc() and related functions, which will 3521eventually be deprecated and removed. 3522 3523### Bdev 3524 3525A new optional bdev module interface function, `init_complete`, has been added to notify bdev modules 3526when the bdev subsystem initialization is complete. This may be useful for virtual bdevs that require 3527notification that the set of initialization examine() calls is complete. 3528 3529The bdev layer now allows modules to provide an optional per-bdev UUID, which can be retrieved with 3530the spdk_bdev_get_uuid() function. 3531 3532Enforcement of IOPS limits for quality of service (QoS) has been added to the bdev layer. See the 3533[set_bdev_qos_limit_iops](http://www.spdk.io/doc/jsonrpc.html#rpc_set_bdev_qos_limit_iops) documentation 3534for more details. 3535 3536### RPC 3537 3538The `[Rpc]` configuration file section, which was deprecated in v18.01, has been removed. 3539Users should switch to the `-r` command-line parameter instead. 3540 3541The JSON-RPC server implementation now allows up to 32 megabyte responses, growing as 3542needed; previously, the response was limited to 32 kilobytes. 3543 3544### SPDKCLI 3545 3546EXPERIMENTAL: New SPDKCLI interactive command tool for managing SPDK is available. 3547See the [SPDKCLI](http://www.spdk.io/doc/spdkcli.html) documentation for more details. 3548 3549### NVMe Driver 3550 3551EXPERIMENTAL: Support for WDS and RDS capable CMBs in NVMe controllers has been added. This support is 3552experimental pending a functional allocator to free and reallocate CMB buffers. 3553 3554spdk_nvme_ns_get_uuid() has been added to allow retrieval of per-namespace UUIDs when available. 3555 3556New API functions spdk_nvme_ctrlr_get_first_active_ns() and spdk_nvme_ctrlr_get_next_active_ns() 3557have been added to iterate active namespaces, as well as spdk_nvme_ctrlr_is_active_ns() to check if 3558a namespace ID is active. 3559 3560### NVMe-oF Target 3561 3562Namespaces may now be assigned unique identifiers via new optional `eui64` and `nguid` parameters 3563to the `nvmf_subsystem_add_ns` RPC method. Additionally, the NVMe-oF target automatically exposes 3564the backing bdev's UUID as the namespace UUID when available. 3565 3566spdk_nvmf_subsystem_remove_ns() is now asynchronous and requires a callback to indicate completion. 3567 3568### Blobstore 3569 3570A number of functions have been renamed: 3571 3572- spdk_bs_io_write_blob() => spdk_blob_io_write() 3573- spdk_bs_io_read_blob() => spdk_blob_io_read() 3574- spdk_bs_io_writev_blob() => spdk_blob_io_writev() 3575- spdk_bs_io_readv_blob() => spdk_blob_io_readv() 3576- spdk_bs_io_unmap_blob() => spdk_blob_io_unmap() 3577- spdk_bs_io_write_zeroes_blob() => spdk_blob_io_write_zeroes() 3578 3579The old names still exist but are deprecated. They will be removed in the v18.07 release. 3580 3581spdk_blob_resize() is now an asynchronous operation to enable resizing a blob while I/O 3582are in progress to that blob on other threads. An explicit spdk_blob_sync_md() is still 3583required to sync the updated metadata to disk. 3584 3585### Logical Volumes 3586 3587A new `destroy_lvol_bdev` RPC method to delete logical volumes has been added. 3588 3589Lvols now have their own UUIDs which replace previous LvolStoreUUID_BlobID combination. 3590 3591New Snapshot and Clone functionalities have been added. User may create Snapshots of existing Lvols 3592and Clones of existing Snapshots. 3593See the [lvol snapshots](http://www.spdk.io/doc/logical_volumes.html#lvol_snapshots) documentation 3594for more details. 3595 3596Resizing logical volumes is now supported via the `resize_lvol_bdev` RPC method. 3597 3598### Lib 3599 3600A set of changes were made in the SPDK's lib code altering 3601instances of calls to `exit()` and `abort()` to return a failure instead 3602wherever reasonably possible. 3603 3604spdk_app_start() no longer exit()'s on an internal failure, but 3605instead returns a non-zero error status. 3606 3607spdk_app_parse_args() no longer exit()'s on help, '-h', or an invalid 3608option, but instead returns SPDK_APP_PARSE_ARGS_HELP and 3609SPDK_APP_PARSE_ARGS_FAIL, respectively, and SPDK_APP_PARSE_ARGS_SUCCESS 3610on success. 3611 3612spdk_pci_get_device() has been deprecated and will be removed in SPDK v18.07. 3613 3614### I/O Channels 3615 3616The prototype for spdk_poller_fn() has been modified; it now returns a value indicating 3617whether or not the poller did any work. Existing pollers will need to be updated to 3618return a value. 3619 3620### iSCSI Target 3621 3622The SPDK iSCSI target now supports the fd.io Vector Packet Processing (VPP) framework userspace 3623TCP/IP stack. See the [iSCSI VPP documentation](http://www.spdk.io/doc/iscsi.html#vpp) for more 3624details. 3625 3626### iSCSI initiator 3627 3628An iSCSI initiator bdev module has been added to SPDK. This module should be considered 3629experimental pending additional features and tests. More details can be found in 3630lib/bdev/iscsi/README. 3631 3632### PMDK 3633 3634The persistent memory (PMDK) bdev module is now enabled using `--with-pmdk` instead of 3635`--with-nvml`. This reflects the renaming of the persistent memory library from NVML to 3636PMDK. 3637 3638### Virtio Block driver 3639 3640A userspace driver for Virtio Block devices has been added. It was built on top of the 3641[Virtio](http://www.spdk.io/doc/virtio.html) library and can be managed similarly to 3642the Virtio SCSI driver. See the 3643[Virtio Block](http://www.spdk.io/doc/bdev.html#bdev_config_virtio_blk) reference for 3644more information. 3645 3646### Virtio with 2MB hugepages 3647 3648The previous 1GB hugepage limitation has now been lifted. A new `-g` command-line option 3649enables SPDK Virtio to work with 2MB hugepages. 3650See [2MB hugepages](http://www.spdk.io/doc/virtio.html#virtio_2mb) for details. 3651 3652## v18.01: Blobstore Thin Provisioning 3653 3654### Build System 3655 3656The build system now includes a `make install` rule, including support for the common 3657`DESTDIR` and `prefix` variables as used in other build systems. Additionally, the prefix 3658may be set via the configure `--prefix` option. Example: `make install prefix=/usr`. 3659 3660### RPC 3661 3662A JSON RPC listener is now enabled by default using a UNIX domain socket at /var/tmp/spdk.sock. 3663A -r option command line option has been added to enable an alternative UNIX domain socket location, 3664or a TCP port in the format ip_addr:tcp_port (i.e. 127.0.0.1:5260). The Rpc configuration file 3665section is now deprecated and will be removed in the v18.04 release. 3666 3667### I/O Channels 3668 3669spdk_poller_register() and spdk_poller_unregister() were moved from the event 3670framework (include/spdk/event.h) to the I/O channel library 3671(include/spdk/io_channel.h). This allows code that doesn't depend on the event 3672framework to request registration and unregistration of pollers. 3673 3674spdk_for_each_channel() now allows asynchronous operations during iteration. 3675Instead of immediately continuing the interaction upon returning from the iteration 3676callback, the user must call spdk_for_each_channel_continue() to resume iteration. 3677 3678### Block Device Abstraction Layer (bdev) 3679 3680The poller abstraction was removed from the bdev layer. There is now a general purpose 3681abstraction for pollers available in include/spdk/io_channel.h 3682 3683### Lib 3684 3685A set of changes were made in the SPDK's lib code altering, 3686instances of calls to `exit()` and `abort()` to return a failure instead 3687wherever reasonably possible. This has resulted in return type changes of 3688the API for: 3689 3690- spdk_env_init() from type `void` to `int`. 3691- spdk_mem_map_init() from type `void` to `int`. 3692 3693Applications making use of these APIs should be modified to check for 3694a non-zero return value instead of relying on them to fail without return. 3695 3696### NVMe Driver 3697 3698SPDK now supports hotplug for vfio-attached devices. But there is one thing keep in mind: 3699Only physical removal events are supported; removing devices via the sysfs `remove` file will not work. 3700 3701### NVMe-oF Target 3702 3703Subsystems are no longer tied explicitly to CPU cores. Instead, connections are handed out to the available 3704cores round-robin. The "Core" option in the configuration file has been removed. 3705 3706### Blobstore 3707 3708A number of functions have been renamed: 3709 3710- spdk_bs_md_resize_blob() => spdk_blob_resize() 3711- spdk_bs_md_sync_blob() => spdk_blob_sync_md() 3712- spdk_bs_md_close_blob() => spdk_blob_close() 3713- spdk_bs_md_get_xattr_names() => spdk_blob_get_xattr_names() 3714- spdk_bs_md_get_xattr_value() => spdk_blob_get_xattr_value() 3715- spdk_blob_md_set_xattr() => spdk_blob_set_xattr() 3716- spdk_blob_md_remove_xattr() => spdk_blob_remove_xattr() 3717- spdk_bs_md_create_blob() => spdk_bs_create_blob() 3718- spdk_bs_md_open_blob() => spdk_bs_open_blob() 3719- spdk_bs_md_delete_blob() => spdk_bs_delete_blob() 3720- spdk_bs_md_iter_first() => spdk_bs_iter_first() 3721- spdk_bs_md_iter_next() => spdk_bs_iter_next() 3722 3723The function signature of spdk_blob_close() has changed. It now takes a struct spdk_blob * argument 3724rather than struct spdk_blob **. 3725 3726The function signature of spdk_bs_iter_next() has changed. It now takes a struct spdk_blob * argument 3727rather than struct spdk_blob **. 3728 3729Thin provisioning support has been added to the blobstore. It can be enabled by setting the 3730`thin_provision` flag in struct spdk_blob_opts when calling spdk_bs_create_blob_ext(). 3731 3732### NBD device 3733 3734The NBD application (test/lib/bdev/nbd) has been removed; Same functionality can now be 3735achieved by using the test/app/bdev_svc application and start_nbd_disk RPC method. 3736See the [GPT](http://www.spdk.io/doc/bdev.html#bdev_config_gpt) documentation for more details. 3737 3738### FIO plugin 3739 3740SPDK `fio_plugin` now supports FIO 3.3. The support for previous FIO 2.21 has been dropped, 3741although it still remains to work for now. The new FIO contains huge amount of bugfixes and 3742it's recommended to do an update. 3743 3744### Virtio library 3745 3746Previously a part of the bdev_virtio module, now a separate library. Virtio is now available 3747via `spdk_internal/virtio.h` file. This is an internal interface to be used when implementing 3748new Virtio backends, namely Virtio-BLK. 3749 3750### iSCSI 3751 3752The MinConnectionIdleInterval parameter has been removed, and connections are no longer migrated 3753to an epoll/kqueue descriptor on the master core when idle. 3754 3755## v17.10: Logical Volumes 3756 3757### New dependencies 3758 3759libuuid was added as new dependency for logical volumes. 3760 3761libnuma is now required unconditionally now that the DPDK submodule has been updated to DPDK 17.08. 3762 3763### Block Device Abstraction Layer (bdev) 3764 3765An [fio](http://github.com/axboe/fio) plugin was added that can route 3766I/O to the bdev layer. See the [plugin documentation](https://github.com/spdk/spdk/tree/master/examples/bdev/fio_plugin/) 3767for more information. 3768 3769spdk_bdev_unmap() was modified to take an offset and a length in bytes as 3770arguments instead of requiring the user to provide an array of SCSI 3771unmap descriptors. This limits unmaps to a single contiguous range. 3772 3773spdk_bdev_write_zeroes() was introduced. It ensures that all specified blocks will be zeroed out. 3774If a block device doesn't natively support a write zeroes command, the bdev layer emulates it using 3775write commands. 3776 3777New API functions that accept I/O parameters in units of blocks instead of bytes 3778have been added: 3779 3780- spdk_bdev_read_blocks(), spdk_bdev_readv_blocks() 3781- spdk_bdev_write_blocks(), spdk_bdev_writev_blocks() 3782- spdk_bdev_write_zeroes_blocks() 3783- spdk_bdev_unmap_blocks() 3784 3785The bdev layer now handles temporary out-of-memory I/O failures internally by queueing the I/O to be 3786retried later. 3787 3788### Linux AIO bdev 3789 3790The AIO bdev now allows the user to override the auto-detected block size. 3791 3792### NVMe driver 3793 3794The NVMe driver now recognizes the NVMe 1.3 Namespace Optimal I/O Boundary field. 3795NVMe 1.3 devices may report an optimal I/O boundary, which the driver will take 3796into account when splitting I/O requests. 3797 3798The HotplugEnable option in `[Nvme]` sections of the configuration file is now 3799"No" by default. It was previously "Yes". 3800 3801The NVMe library now includes a spdk_nvme_ns_get_ctrlr() function which returns the 3802NVMe Controller associated with a given namespace. 3803 3804The NVMe library now allows the user to specify a host identifier when attaching 3805to a controller. The host identifier is used as part of the Reservations feature, 3806as well as in the NVMe-oF Connect command. The default host ID is also now a 3807randomly-generated UUID, and the default host NQN uses the host ID to generate 3808a UUID-based NQN. 3809 3810spdk_nvme_connect() was added to allow the user to connect directly to a single 3811NVMe or NVMe-oF controller. 3812 3813### NVMe-oF Target (nvmf_tgt) 3814 3815The NVMe-oF target no longer requires any in-capsule data buffers to run, and 3816the feature is now entirely optional. Previously, at least 4 KiB in-capsule 3817data buffers were required. 3818 3819NVMe-oF subsystems have a new configuration option, AllowAnyHost, to control 3820whether the host NQN whitelist is enforced when accepting new connections. 3821If no Host options have been specified and AllowAnyHost is disabled, the 3822connection will be denied; this is a behavior change from previous releases, 3823which allowed any host NQN to connect if the Host list was empty. 3824AllowAnyHost is disabled by default. 3825 3826NVMe-oF namespaces may now be assigned arbitrary namespace IDs, and the number 3827of namespaces per subsystem is no longer limited. 3828 3829The NVMe-oF target now supports the Write Zeroes command. 3830 3831### Environment Abstraction Layer 3832 3833A new default value, SPDK_MEMPOOL_DEFAULT_CACHE_SIZE, was added to provide 3834additional clarity when constructing spdk_mempools. Previously, -1 could be 3835passed and the library would choose a reasonable default, but this new value 3836makes it explicit that the default is being used. 3837 3838### Blobstore 3839 3840The blobstore super block now contains a bstype field to identify the type of the blobstore. 3841Existing code should be updated to fill out bstype when calling spdk_bs_init() and spdk_bs_load(). 3842 3843spdk_bs_destroy() was added to allow destroying blobstore on device 3844with an initialized blobstore. 3845 3846spdk_bs_io_readv_blob() and spdk_bs_io_writev_blob() were added to enable 3847scattered payloads. 3848 3849A CLI tool for blobstore has been added, allowing basic operations through either command 3850line or shell interface. See the [blobcli](https://github.com/spdk/spdk/tree/master/examples/blob/cli) 3851documentation for more details. 3852 3853### Event Framework 3854 3855The ability to set a thread name, previously only used by the reactor code, is 3856now part of the spdk_thread_allocate() API. Users may specify a thread name 3857which will show up in tools like `gdb`. 3858 3859### Log 3860 3861The spdk_trace_dump() function now takes a new parameter to allow the caller to 3862specify an output file handle (stdout or stderr, for example). 3863 3864### Logical Volumes 3865 3866Logical volumes library built on top of SPDK blobstore has been added. 3867It is possible to create logical volumes on top of other devices using RPC. 3868 3869See the [logical volumes](http://www.spdk.io/doc/logical_volumes.html) documentation for more information. 3870 3871### Persistent Memory 3872 3873A new persistent memory bdev type has been added. 3874The persistent memory block device is built on top of [libpmemblk](http://pmem.io/nvml/libpmemblk/). 3875It is possible to create pmem devices on top of pmem pool files using RPC. 3876 3877See the [Pmem Block Device](http://www.spdk.io/doc/bdev.html#bdev_config_pmem) documentation for more information. 3878 3879### Virtio SCSI driver 3880 3881A userspace driver for Virtio SCSI devices has been added. 3882The driver is capable of creating block devices on top of LUNs exposed by another SPDK vhost-scsi application. 3883 3884See the [Virtio SCSI](http://www.spdk.io/doc/virtio.html) documentation and 3885[Getting Started](http://www.spdk.io/doc/bdev.html#bdev_config_virtio_scsi) guide for more information. 3886 3887### Vhost target 3888 3889The vhost target application now supports live migration between QEMU instances. 3890 3891## v17.07: Build system improvements, userspace vhost-blk target, and GPT bdev 3892 3893### Build System 3894 3895A `configure` script has been added to simplify the build configuration process. 3896The existing CONFIG file and `make CONFIG_...` options are also still supported. 3897Run `./configure --help` for information about available configuration options. 3898 3899A DPDK submodule has been added to make building SPDK easier. If no `--with-dpdk` 3900option is specified to configure, the SPDK build system will automatically build a 3901known-good configuration of DPDK with the minimal options enabled. See the Building 3902section of README.md for more information. 3903 3904A [Vagrant](https://www.vagrantup.com/) setup has been added to make it easier to 3905develop and use SPDK on systems without suitable NVMe hardware. See the Vagrant 3906section of README.md for more information. 3907 3908### Userspace vhost-blk target 3909 3910The vhost library and example app have been updated to support the vhost-blk 3911protocol in addition to the existing vhost-scsi protocol. 3912See the [vhost documentation](http://www.spdk.io/doc/vhost.html) for more details. 3913 3914### Block device abstraction layer (bdev) 3915 3916A GPT virtual block device has been added, which automatically exposes GPT partitions 3917with a special SPDK-specific partition type as bdevs. 3918See the [GPT bdev documentation](http://www.spdk.io/doc/bdev.md#bdev_config_gpt) for 3919more information. 3920 3921### NVMe driver 3922 3923The NVMe driver has been updated to support recent Intel SSDs, including the Intel® 3924Optane™ SSD DC P4800X series. 3925 3926A workaround has been added for devices that failed to recognize register writes 3927during controller reset. 3928 3929The NVMe driver now allocates request tracking objects on a per-queue basis. The 3930number of requests allowed on an I/O queue may be set during `spdk_nvme_probe()` by 3931modifying `io_queue_requests` in the opts structure. 3932 3933The SPDK NVMe `fio_plugin` has been updated to support multiple threads (`numjobs`). 3934 3935spdk_nvme_ctrlr_alloc_io_qpair() has been modified to allow the user to override 3936controller-level options for each individual I/O queue pair. 3937Existing callers with qprio == 0 can be updated to: 3938~~~ 3939... = spdk_nvme_ctrlr_alloc_io_qpair(ctrlr, NULL, 0); 3940~~~ 3941Callers that need to specify a non-default qprio should be updated to: 3942~~~ 3943struct spdk_nvme_io_qpair_opts opts; 3944spdk_nvme_ctrlr_get_default_io_qpair_opts(ctrlr, &opts, sizeof(opts)); 3945opts.qprio = SPDK_NVME_QPRIO_...; 3946... = spdk_nvme_ctrlr_alloc_io_qpair(ctrlr, &opts, sizeof(opts)); 3947~~~ 3948 3949### Environment Abstraction Layer 3950 3951The environment abstraction layer has been updated to include several new functions 3952in order to wrap additional DPDK functionality. See `include/spdk/env.h` for the 3953current set of functions. 3954 3955### SPDK Performance Analysis with Intel® VTune™ Amplifier 3956 3957Support for SPDK performance analysis has been added to Intel® VTune™ Amplifier 2018. 3958 3959This analysis provides: 3960 3961- I/O performance monitoring (calculating standard I/O metrics like IOPS, throughput, etc.) 3962- Tuning insights on the interplay of I/O and compute devices by estimating how many cores 3963 would be reasonable to provide for SPDK to keep up with a current storage workload. 3964 3965See the VTune Amplifier documentation for more information. 3966 3967## v17.03: Blobstore and userspace vhost-scsi target 3968 3969### Blobstore and BlobFS 3970 3971The blobstore is a persistent, power-fail safe block allocator designed to be 3972used as the local storage system backing a higher-level storage service. 3973See the [blobstore documentation](http://www.spdk.io/doc/blob.html) for more details. 3974 3975BlobFS adds basic filesystem functionality like filenames on top of the blobstore. 3976This release also includes a RocksDB Env implementation using BlobFS in place of the 3977kernel filesystem. 3978See the [BlobFS documentation](http://www.spdk.io/doc/blobfs.html) for more details. 3979 3980### Userspace vhost-scsi target 3981 3982A userspace implementation of the QEMU vhost-scsi protocol has been added. 3983The vhost target is capable of exporting SPDK bdevs to QEMU-based VMs as virtio devices. 3984See the [vhost documentation](http://www.spdk.io/doc/vhost.html) for more details. 3985 3986### Event framework 3987 3988The overhead of the main reactor event loop was reduced by optimizing the number of 3989calls to spdk_get_ticks() per iteration. 3990 3991### NVMe library 3992 3993The NVMe library will now automatically split readv/writev requests with scatter-gather 3994lists that do not map to valid PRP lists when the NVMe controller does not natively 3995support SGLs. 3996 3997The `identify` and `perf` NVMe examples were modified to add a consistent format for 3998specifying remote NVMe over Fabrics devices via the `-r` option. 3999This is implemented using the new `spdk_nvme_transport_id_parse()` function. 4000 4001### iSCSI Target 4002 4003The [Nvme] section of the configuration file was modified to remove the `BDF` directive 4004and replace it with a `TransportID` directive. Both local (PCIe) and remote (NVMe-oF) 4005devices can now be specified as the backing block device. A script to generate an 4006entire [Nvme] section based on the local NVMe devices attached was added at 4007`scripts/gen_nvme.sh`. 4008 4009### NVMe-oF Target 4010 4011The [Nvme] section of the configuration file was modified to remove the `BDF` directive 4012and replace it with a `TransportID` directive. Both local (PCIe) and remote (NVMe-oF) 4013devices can now be specified as the backing block device. A script to generate an 4014entire [Nvme] section based on the local NVMe devices attached was added at 4015`scripts/gen_nvme.sh`. 4016 4017## v16.12: NVMe over Fabrics host, hotplug, and multi-process 4018 4019### NVMe library 4020 4021The NVMe library has been changed to create its own request memory pool rather than 4022requiring the user to initialize the global `request_mempool` variable. Apps can be 4023updated by simply removing the initialization of `request_mempool`. Since the NVMe 4024library user no longer needs to know the size of the internal NVMe request 4025structure to create the pool, the `spdk_nvme_request_size()` function was also removed. 4026 4027The `spdk_nvme_ns_cmd_deallocate()` function was renamed and extended to become 4028`spdk_nvme_ns_cmd_dataset_management()`, which allows access to all of the NVMe 4029Dataset Management command's parameters. Existing callers can be updated to use 4030`spdk_nvme_ns_cmd_dataset_management()` with `SPDK_NVME_DSM_ATTR_DEALLOCATE` as the 4031`type` parameter. 4032 4033The NVMe library SGL callback prototype has been changed to return virtual addresses 4034rather than physical addresses. Callers of `spdk_nvme_ns_cmd_readv()` and 4035`spdk_nvme_ns_cmd_writev()` must update their `next_sge_fn` callbacks to match. 4036 4037The NVMe library now supports NVMe over Fabrics devices in addition to the existing 4038support for local PCIe-attached NVMe devices. For an example of how to enable 4039NVMe over Fabrics support in an application, see `examples/nvme/identify` and 4040`examples/nvme/perf`. 4041 4042Hot insert/remove support for NVMe devices has been added. To enable NVMe hotplug 4043support, an application should call the `spdk_nvme_probe()` function on a regular 4044basis to probe for new devices (reported via the existing `probe_cb` callback) and 4045removed devices (reported via a new `remove_cb` callback). Hotplug is currently 4046only supported on Linux with the `uio_pci_generic` driver, and newly-added NVMe 4047devices must be bound to `uio_pci_generic` by an external script or tool. 4048 4049Multiple processes may now coordinate and use a single NVMe device simultaneously 4050using [DPDK Multi-process Support](http://dpdk.org/doc/guides/prog_guide/multi_proc_support.html). 4051 4052### NVMe over Fabrics target (`nvmf_tgt`) 4053 4054The `nvmf_tgt` configuration file format has been updated significantly to enable 4055new features. See the example configuration file `etc/spdk/nvmf.conf.in` for 4056more details on the new and changed options. 4057 4058The NVMe over Fabrics target now supports virtual mode subsystems, which allow the 4059user to export devices from the SPDK block device abstraction layer as NVMe over 4060Fabrics subsystems. Direct mode (raw NVMe device access) is also still supported, 4061and a single `nvmf_tgt` may export both types of subsystems simultaneously. 4062 4063### Block device abstraction layer (bdev) 4064 4065The bdev layer now supports scatter/gather read and write I/O APIs, and the NVMe 4066blockdev driver has been updated to support scatter/gather. Apps can use the 4067new scatter/gather support via the `spdk_bdev_readv()` and `spdk_bdev_writev()` 4068functions. 4069 4070The bdev status returned from each I/O has been extended to pass through NVMe 4071or SCSI status codes directly in cases where the underlying device can provide 4072a more specific status code. 4073 4074A Ceph RBD (RADOS Block Device) blockdev driver has been added. This allows the 4075`iscsi_tgt` and `nvmf_tgt` apps to export Ceph RBD volumes as iSCSI LUNs or 4076NVMe namespaces. 4077 4078### General changes 4079 4080`libpciaccess` has been removed as a dependency and DPDK PCI enumeration is 4081used instead. Prior to DPDK 16.07 enumeration by class code was not supported, 4082so for earlier DPDK versions, only Intel SSD DC P3x00 devices will be discovered 4083by the NVMe library. 4084 4085The `env` environment abstraction library has been introduced, and a default 4086DPDK-based implementation is provided as part of SPDK. The goal of the `env` 4087layer is to enable use of alternate user-mode memory allocation and PCI access 4088libraries. See `doc/porting.md` for more details. 4089 4090The build process has been modified to produce all of the library files in the 4091`build/lib` directory. This is intended to simplify the use of SPDK from external 4092projects, which can now link to SPDK libraries by adding the `build/lib` directory 4093to the library path via `-L` and linking the SPDK libraries by name (for example, 4094`-lspdk_nvme -lspdk_log -lspdk_util`). 4095 4096`nvmf_tgt` and `iscsi_tgt` now have a JSON-RPC interface, which allows the user 4097to query and modify the configuration at runtime. The RPC service is disabled by 4098default, since it currently does not provide any authentication or security 4099mechanisms; it should only be enabled on systems with controlled user access 4100behind a firewall. An example RPC client implemented in Python is provided in 4101`scripts/rpc.py`. 4102 4103## v16.08: iSCSI target, NVMe over Fabrics maturity 4104 4105This release adds a userspace iSCSI target. The iSCSI target is capable of exporting 4106NVMe devices over a network using the iSCSI protocol. The application is located 4107in app/iscsi_tgt and a documented configuration file can be found at etc/spdk/spdk.conf.in. 4108 4109This release also significantly improves the existing NVMe over Fabrics target. 4110 4111- The configuration file format was changed, which will require updates to 4112 any existing nvmf.conf files (see `etc/spdk/nvmf.conf.in`): 4113 - `SubsystemGroup` was renamed to `Subsystem`. 4114 - `AuthFile` was removed (it was unimplemented). 4115 - `nvmf_tgt` was updated to correctly recognize NQN (NVMe Qualified Names) 4116 when naming subsystems. The default node name was changed to reflect this; 4117 it is now "nqn.2016-06.io.spdk". 4118 - `Port` and `Host` sections were merged into the `Subsystem` section 4119 - Global options to control max queue depth, number of queues, max I/O 4120 size, and max in-capsule data size were added. 4121 - The Nvme section was removed. Now a list of devices is specified by 4122 bus/device/function directly in the Subsystem section. 4123 - Subsystems now have a Mode, which can be Direct or Virtual. This is an attempt 4124 to future-proof the interface, so the only mode supported by this release 4125 is "Direct". 4126- Many bug fixes and cleanups were applied to the `nvmf_tgt` app and library. 4127- The target now supports discovery. 4128 4129This release also adds one new feature and provides some better examples and tools 4130for the NVMe driver. 4131 4132- The Weighted Round Robin arbitration method is now supported. This allows 4133 the user to specify different priorities on a per-I/O-queue basis. To 4134 enable WRR, set the `arb_mechanism` field during `spdk_nvme_probe()`. 4135- A simplified "Hello World" example was added to show the proper way to use 4136 the NVMe library API; see `examples/nvme/hello_world/hello_world.c`. 4137- A test for measuring software overhead was added. See `test/lib/nvme/overhead`. 4138 4139## v16.06: NVMf userspace target 4140 4141This release adds a userspace NVMf (NVMe over Fabrics) target, conforming to the 4142newly-released NVMf 1.0/NVMe 1.2.1 specification. The NVMf target exports NVMe 4143devices from a host machine over the network via RDMA. Currently, the target is 4144limited to directly exporting physical NVMe devices, and the discovery subsystem 4145is not supported. 4146 4147This release includes a general API cleanup, including renaming all declarations 4148in public headers to include a `spdk` prefix to prevent namespace clashes with 4149user code. 4150 4151- NVMe 4152 - The `nvme_attach()` API was reworked into a new probe/attach model, which 4153 moves device detection into the NVMe library. The new API also allows 4154 parallel initialization of NVMe controllers, providing a major reduction in 4155 startup time when using multiple controllers. 4156 - I/O queue allocation was changed to be explicit in the API. Each function 4157 that generates I/O requests now takes a queue pair (`spdk_nvme_qpair *`) 4158 argument, and I/O queues may be allocated using 4159 `spdk_nvme_ctrlr_alloc_io_qpair()`. This allows more flexible assignment of 4160 queue pairs than the previous model, which only allowed a single queue 4161 per thread and limited the total number of I/O queues to the lowest number 4162 supported on any attached controller. 4163 - Added support for the Write Zeroes command. 4164 - `examples/nvme/perf` can now report I/O command latency from the 4165 the controller's viewpoint using the Intel vendor-specific read/write latency 4166 log page. 4167 - Added namespace reservation command support, which can be used to coordinate 4168 sharing of a namespace between multiple hosts. 4169 - Added hardware SGL support, which enables use of scattered buffers that 4170 don't conform to the PRP list alignment and length requirements on supported 4171 NVMe controllers. 4172 - Added end-to-end data protection support, including the ability to write and 4173 read metadata in extended LBA (metadata appended to each block of data in the 4174 buffer) and separate metadata buffer modes. 4175 See `spdk_nvme_ns_cmd_write_with_md()` and `spdk_nvme_ns_cmd_read_with_md()` 4176 for details. 4177- IOAT 4178 - The DMA block fill feature is now exposed via the `ioat_submit_fill()` 4179 function. This is functionally similar to `memset()`, except the memory is 4180 filled with an 8-byte repeating pattern instead of a single byte like memset. 4181- PCI 4182 - Added support for using DPDK for PCI device mapping in addition to the 4183 existing libpciaccess option. Using the DPDK PCI support also allows use of 4184 the Linux VFIO driver model, which means that SPDK userspace drivers will work 4185 with the IOMMU enabled. Additionally, SPDK applications may be run as an 4186 unprivileged user with access restricted to a specific set of PCIe devices. 4187 - The PCI library API was made more generic to abstract away differences 4188 between the underlying PCI access implementations. 4189 4190## v1.2.0: IOAT user-space driver 4191 4192This release adds a user-space driver with support for the Intel I/O Acceleration Technology 4193(I/OAT, also known as "Crystal Beach") DMA offload engine. 4194 4195- IOAT 4196 - New user-space driver supporting DMA memory copy offload 4197 - Example programs `ioat/perf` and `ioat/verify` 4198 - Kernel-mode DMA engine test driver `kperf` for performance comparison 4199- NVMe 4200 - Per-I/O flags for Force Unit Access (FUA) and Limited Retry 4201 - Public API for retrieving log pages 4202 - Reservation register/acquire/release/report command support 4203 - Scattered payload support - an alternate API to provide I/O buffers via a sequence of callbacks 4204 - Declarations and `nvme/identify` support for Intel SSD DC P3700 series vendor-specific log pages and features 4205- Updated to support DPDK 2.2.0 4206 4207## v1.0.0: NVMe user-space driver 4208 4209This is the initial open source release of the Storage Performance Development Kit (SPDK). 4210 4211Features: 4212 4213- NVMe user-space driver 4214- NVMe example programs 4215 - `examples/nvme/perf` tests performance (IOPS) using the NVMe user-space driver 4216 - `examples/nvme/identify` displays NVMe controller information in a human-readable format 4217- Linux and FreeBSD support 4218